2024-09-16T09:57:50.958 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T09:57:50.975 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T09:57:51.053 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111 branch: wip-athakkar-testing-2024-09-12-1857 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: athakkar@redhat.com first_in_suite: false job_id: '7907111' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-athakkar-testing-2024-09-12-1857 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: 66e33d6af1496291b0602ee0a859e68145f17dad ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 66e33d6af1496291b0602ee0a859e68145f17dad selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-athakkar-testing-2024-09-12-1857 sha1: 66e33d6af1496291b0602ee0a859e68145f17dad owner: scheduled_avan@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4787 sha1: 66e33d6af1496291b0602ee0a859e68145f17dad sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-athakkar-testing-2024-09-12-1857 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_66e33d6af1496291b0602ee0a859e68145f17dad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 66e33d6af1496291b0602ee0a859e68145f17dad targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/QVqDoxbZ/9evFtcN6RmbGA76jDVY8Fin1nZYq4PDbX74j0PM5coPS87gSjz2T0cAkE4PyW50vHS/gQXFTzRs= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMy6b96LKYVvIUvUwI/mgjcVVLpyXJJm5NOZlfDFbd5yoVxwa0WUja8g2mLl+TUc6dz6tP4Ws7OpCTE+YF8Ocyw= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_08:23:53 tube: smithi user: avan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T09:57:51.053 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_66e33d6af1496291b0602ee0a859e68145f17dad/qa; will attempt to use it 2024-09-16T09:57:51.053 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_66e33d6af1496291b0602ee0a859e68145f17dad/qa/tasks 2024-09-16T09:57:51.054 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T09:57:51.055 INFO:teuthology.task.internal:Checking packages... 2024-09-16T09:57:51.076 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '66e33d6af1496291b0602ee0a859e68145f17dad' 2024-09-16T09:57:51.076 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T09:57:51.076 INFO:teuthology.packaging:ref: None 2024-09-16T09:57:51.076 INFO:teuthology.packaging:tag: None 2024-09-16T09:57:51.076 INFO:teuthology.packaging:branch: wip-athakkar-testing-2024-09-12-1857 2024-09-16T09:57:51.076 INFO:teuthology.packaging:sha1: 66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T09:57:51.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-athakkar-testing-2024-09-12-1857 2024-09-16T09:57:51.214 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4882.g66e33d6a 2024-09-16T09:57:51.215 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T09:57:51.223 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T09:57:51.223 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T09:57:51.252 INFO:teuthology.task.internal:Saving configuration 2024-09-16T09:57:51.264 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T09:57:51.288 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T09:57:51.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 09:52:51.310263', 'locked_by': 'scheduled_avan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/QVqDoxbZ/9evFtcN6RmbGA76jDVY8Fin1nZYq4PDbX74j0PM5coPS87gSjz2T0cAkE4PyW50vHS/gQXFTzRs='} 2024-09-16T09:57:51.329 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 09:52:51.308873', 'locked_by': 'scheduled_avan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMy6b96LKYVvIUvUwI/mgjcVVLpyXJJm5NOZlfDFbd5yoVxwa0WUja8g2mLl+TUc6dz6tP4Ws7OpCTE+YF8Ocyw='} 2024-09-16T09:57:51.329 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T09:57:51.337 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-16T09:57:51.337 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-16T09:57:51.337 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T09:57:51.419 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1dddf13ac0>, signals=[15]) 2024-09-16T09:57:51.419 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T09:57:51.427 INFO:teuthology.task.internal:Opening connections... 2024-09-16T09:57:51.427 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T09:57:51.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T09:57:51.505 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T09:57:51.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T09:57:51.578 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T09:57:51.585 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-09-16T09:57:51.603 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-09-16T09:57:51.604 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-09-16T09:57:51.661 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T09:57:51.662 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T09:57:51.663 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-09-16T09:57:51.687 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-09-16T09:57:51.705 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-09-16T09:57:51.706 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-09-16T09:57:51.763 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-09-16T09:57:51.763 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T09:57:51.764 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T09:57:51.765 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-09-16T09:57:51.787 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T09:57:51.796 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T09:57:51.829 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T09:57:51.829 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-09-16T09:57:51.832 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-09-16T09:57:51.849 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T09:57:51.862 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T09:57:51.862 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-09-16T09:57:51.890 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-09-16T09:57:51.991 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T09:57:52.045 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T09:57:52.060 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T09:57:52.060 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T09:57:52.061 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T09:57:52.061 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T09:57:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T09:57:52.078 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-09-16T09:57:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T09:57:52.079 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-16T09:57:52.136 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-503.el9.x86_64 2024-09-16T09:57:52.136 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.14.0-503.el9.x86_64 2024-09-16T09:57:52.136 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-09-16T09:57:52.137 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-503.el9.x86_64 2024-09-16T09:57:52.137 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-503.el9.x86_64 2024-09-16T09:57:52.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-09-16T09:57:54.686 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-16T09:57:55.499 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-16T09:57:56.428 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 36 MB/s | 20 MB 00:00 2024-09-16T09:58:00.766 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 20 MB 00:04 2024-09-16T09:58:02.287 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-16T09:58:03.904 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 19 kB 00:00 2024-09-16T09:58:04.725 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-16T09:58:07.974 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.5 MB 00:03 2024-09-16T09:58:09.792 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 19 kB 00:00 2024-09-16T09:58:10.356 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T09:58:10.987 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-16T09:58:12.479 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T09:58:12.479 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T09:58:12.586 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T09:58:12.600 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T09:58:12.601 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T09:58:12.602 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-09-16T09:58:12.602 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T09:58:12.602 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 79 M 2024-09-16T09:58:12.602 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 126 M 2024-09-16T09:58:12.603 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T09:58:14.082 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 16 kB/s | 23 kB 00:01 2024-09-16T09:58:16.166 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.4 MB/s | 16 MB 00:03 2024-09-16T09:58:16.460 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T09:58:17.466 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.2 MB/s | 35 MB 00:04 2024-09-16T09:58:18.474 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T09:58:18.474 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T09:58:18.558 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T09:58:18.570 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T09:58:18.571 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T09:58:18.571 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T09:58:18.571 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T09:58:18.571 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T09:58:18.571 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-09-16T09:58:18.571 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T09:58:18.571 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 79 M 2024-09-16T09:58:18.572 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 126 M 2024-09-16T09:58:18.572 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T09:58:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.3 MB/s | 29 MB 00:04 2024-09-16T09:58:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T09:58:18.608 INFO:teuthology.orchestra.run.smithi063.stdout:Total 13 MB/s | 79 MB 00:06 2024-09-16T09:58:18.936 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 208 kB/s | 23 kB 00:00 2024-09-16T09:58:19.329 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T09:58:19.336 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-16T09:58:19.733 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T09:58:19.733 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T09:58:19.836 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 34 MB/s | 35 MB 00:01 2024-09-16T09:58:20.086 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-16T09:58:20.087 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T09:58:20.087 INFO:teuthology.orchestra.run.smithi165.stdout:Total 52 MB/s | 79 MB 00:01 2024-09-16T09:58:20.728 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T09:58:20.729 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T09:58:20.766 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T09:58:21.128 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T09:58:21.128 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T09:58:22.103 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T09:58:22.103 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T09:58:23.316 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T09:58:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T09:58:24.101 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T09:58:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T09:58:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T09:58:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T09:58:26.873 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T09:58:26.995 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T09:58:28.133 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T09:58:28.255 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T09:58:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T09:58:32.171 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:58:33.080 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T09:58:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:58:37.378 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:58:37.985 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:16.207 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T09:59:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T09:59:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T09:59:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T09:59:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T09:59:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T09:59:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T09:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T09:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T09:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.402 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T09:59:17.403 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T09:59:17.603 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T09:59:17.688 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.688 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.688 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.689 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T09:59:17.689 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T09:59:17.689 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T09:59:17.689 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T09:59:17.689 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T09:59:17.747 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.748 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-09-16T09:59:17.784 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.06 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 82.el9 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Group : Unspecified 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 8274670 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T09:59:17.785 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : builder@centos.org 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-09-16T09:59:17.786 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T09:59:17.787 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T09:59:17.788 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T09:59:17.788 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T09:59:17.864 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.864 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.864 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.864 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T09:59:17.864 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T09:59:17.864 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T09:59:17.864 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T09:59:17.865 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T09:59:17.924 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T09:59:17.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo rpm -qi grub2-tools 2024-09-16T09:59:17.956 INFO:teuthology.orchestra.run.smithi165.stdout:Name : grub2-tools 2024-09-16T09:59:17.956 INFO:teuthology.orchestra.run.smithi165.stdout:Epoch : 1 2024-09-16T09:59:17.956 INFO:teuthology.orchestra.run.smithi165.stdout:Version : 2.06 2024-09-16T09:59:17.956 INFO:teuthology.orchestra.run.smithi165.stdout:Release : 82.el9 2024-09-16T09:59:17.956 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: x86_64 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Group : Unspecified 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Size : 8274670 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:License : GPLv3+ 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Packager : builder@centos.org 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Vendor : CentOS 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Summary : Support tools for GRUB. 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout:Description : 2024-09-16T09:59:17.957 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T09:59:17.958 INFO:teuthology.orchestra.run.smithi165.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T09:59:17.958 INFO:teuthology.orchestra.run.smithi165.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T09:59:17.958 INFO:teuthology.orchestra.run.smithi165.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T09:59:17.958 INFO:teuthology.orchestra.run.smithi165.stdout:hardware devices. 2024-09-16T09:59:17.958 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T09:59:17.958 INFO:teuthology.orchestra.run.smithi165.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T09:59:17.959 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T09:59:17.959 INFO:teuthology.task.kernel:Updating grub on smithi165 to boot 5.14.0-505.el9.x86_64 2024-09-16T09:59:17.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T09:59:18.445 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-09-16T09:59:18.655 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-09-16T09:59:19.843 INFO:teuthology.orchestra.run.smithi063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T09:59:19.877 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-09-16T09:59:19.879 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-09-16T09:59:19.894 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.E5S5OCgEAv 2024-09-16T09:59:19.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.E5S5OCgEAv 2024-09-16T09:59:19.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.E5S5OCgEAv 2024-09-16T09:59:20.012 INFO:teuthology.orchestra.run.smithi165.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T09:59:20.037 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-09-16T09:59:20.044 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-16T09:59:20.059 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.pFMUfrBIEB 2024-09-16T09:59:20.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pFMUfrBIEB 2024-09-16T09:59:20.092 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.E5S5OCgEAv is 6KB 2024-09-16T09:59:20.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.pFMUfrBIEB 2024-09-16T09:59:20.140 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.E5S5OCgEAv 2024-09-16T09:59:20.154 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T09:59:20.217 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T09:59:20.217 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T09:59:20.217 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T09:59:20.217 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T09:59:20.219 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T09:59:20.274 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.pFMUfrBIEB is 6KB 2024-09-16T09:59:20.322 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.pFMUfrBIEB 2024-09-16T09:59:20.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T09:59:20.365 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-09-16T09:59:20.406 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T09:59:20.407 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T09:59:20.407 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T09:59:20.407 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T09:59:20.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T09:59:20.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-09-16T09:59:50.397 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T09:59:50.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T09:59:50.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T09:59:50.579 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T09:59:50.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T09:59:50.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:00:24.893 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-16T10:00:25.153 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-16T10:00:33.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T10:00:33.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:00:34.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T10:00:34.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:00:52.285 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-16T10:01:04.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T10:01:04.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:01:04.702 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-09-16T10:01:05.080 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T10:01:05.081 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T10:01:05.081 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-09-16T10:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-505.el9.x86_64 2024-09-16T10:01:05.097 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T10:01:05.097 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:01:05.098 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T10:01:06.098 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T10:01:06.099 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:01:06.104 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-16T10:01:06.214 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-09-16T10:01:06.234 DEBUG:teuthology.parallel:result is None 2024-09-16T10:01:06.463 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T10:01:06.463 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T10:01:06.463 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-16T10:01:06.478 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-505.el9.x86_64 2024-09-16T10:01:06.478 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T10:01:06.478 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:01:06.478 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T10:01:07.479 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T10:01:07.480 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:01:07.561 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-09-16T10:01:07.582 DEBUG:teuthology.parallel:result is None 2024-09-16T10:01:07.582 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T10:01:07.590 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T10:01:07.590 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:01:07.592 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:01:07.607 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T10:01:07.633 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T10:01:07.684 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T10:01:07.684 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:01:07.687 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:01:07.723 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T10:01:07.730 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T10:01:07.731 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:01:07.762 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:01:07.791 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:01:07.801 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:01:07.996 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:01:08.020 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:01:08.022 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T10:01:08.028 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T10:01:08.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:01:08.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:01:08.097 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T10:01:08.107 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T10:01:08.107 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:01:08.110 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:01:08.161 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:01:08.239 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:01:08.313 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:01:08.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:01:08.452 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:01:08.453 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:01:08.520 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:01:08.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:01:08.614 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:01:08.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:01:08.741 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:01:08.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:01:08.809 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-09-16T10:01:08.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-09-16T10:01:08.853 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:01:08.917 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:01:09.116 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T10:01:09.124 INFO:teuthology.task.internal:Starting timer... 2024-09-16T10:01:09.124 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T10:01:09.157 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T10:01:09.202 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-16T10:01:09.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-09-16T10:01:09.345 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-09-16T10:01:09.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-09-16T10:01:09.391 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-09-16T10:01:09.393 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T10:01:09.393 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-09-16T10:01:09.471 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-09-16T10:01:09.471 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-09-16T10:01:09.497 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-09-16T10:01:09.498 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-09-16T10:01:09.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:01:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:01:09.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:01:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:01:09.564 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T10:01:09.564 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-09-16T10:01:09.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-09-16T10:01:09.636 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T10:01:09.646 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T10:01:09.649 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T10:01:09.822 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T10:01:09.834 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T10:01:09.835 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T10:04:56.944 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-09-16T10:04:56.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T10:04:56.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:04:57.025 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-09-16T10:04:57.105 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T10:04:57.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T10:04:57.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:04:57.199 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-16T10:04:57.272 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-16T10:04:57.272 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T10:04:57.286 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T10:04:57.286 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:04:57.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:04:57.289 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:04:57.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:04:57.319 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T10:04:57.335 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T10:04:57.341 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T10:04:57.355 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T10:04:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-09-16T10:04:57.373 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-09-16T10:04:57.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-09-16T10:04:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T10:04:57.391 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-09-16T10:04:57.402 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T10:04:57.405 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T10:04:57.419 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T10:04:57.468 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:04:57.471 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:04:57.471 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-09-16T10:04:57.471 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.471 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.471 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.472 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.479 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:04:57.484 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:04:57.484 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-09-16T10:04:57.484 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.484 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.484 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.484 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:04:57.484 INFO:teuthology.run_tasks:Running task pexec... 2024-09-16T10:04:57.499 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-16T10:04:57.499 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T10:04:57.499 DEBUG:teuthology.orchestra.run.smithi063:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-16T10:04:57.500 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T10:04:57.500 DEBUG:teuthology.orchestra.run.smithi165:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-16T10:04:57.877 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:04:57.886 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:04:57.887 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout:Removing: 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout:Removing unused dependencies: 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:04:57.888 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:04:57.889 INFO:teuthology.orchestra.run.smithi165.stdout:Remove 2 Packages 2024-09-16T10:04:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:04:57.889 INFO:teuthology.orchestra.run.smithi165.stdout:Freed space: 5.5 M 2024-09-16T10:04:57.889 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:04:57.892 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:04:57.892 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout:Removing: 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout:Removing unused dependencies: 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:04:57.900 INFO:teuthology.orchestra.run.smithi063.stdout:Remove 2 Packages 2024-09-16T10:04:57.901 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:04:57.901 INFO:teuthology.orchestra.run.smithi063.stdout:Freed space: 5.5 M 2024-09-16T10:04:57.901 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:04:57.904 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:04:57.905 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:04:57.954 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:04:57.954 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:04:57.965 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:04:57.965 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:04:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:04:58.115 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:04:58.181 INFO:teuthology.orchestra.run.smithi165.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-16T10:04:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-16T10:04:58.237 INFO:teuthology.orchestra.run.smithi165.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T10:04:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T10:04:58.760 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T10:04:58.761 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-16T10:04:58.765 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T10:04:58.766 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-16T10:04:58.982 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-16T10:04:58.982 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:04:58.982 INFO:teuthology.orchestra.run.smithi063.stdout:Removed: 2024-09-16T10:04:58.982 INFO:teuthology.orchestra.run.smithi063.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T10:04:58.982 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:04:58.982 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:04:59.013 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-16T10:04:59.013 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:04:59.014 INFO:teuthology.orchestra.run.smithi165.stdout:Removed: 2024-09-16T10:04:59.014 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T10:04:59.014 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:04:59.014 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:04:59.782 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:46 ago on Mon 16 Sep 2024 10:04:13 AM UTC. 2024-09-16T10:04:59.861 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:49 ago on Mon 16 Sep 2024 10:04:10 AM UTC. 2024-09-16T10:04:59.922 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:04:59.922 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:04:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-16T10:04:59.922 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:04:59.922 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:04:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-09-16T10:04:59.923 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:04:59.926 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.0 M 2024-09-16T10:04:59.926 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 9.2 M 2024-09-16T10:04:59.926 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:04:59.996 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:04:59.996 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:04:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout:Install 6 Packages 2024-09-16T10:04:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:00.005 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 2.0 M 2024-09-16T10:05:00.005 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 9.2 M 2024-09-16T10:05:00.005 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:05:00.160 INFO:teuthology.orchestra.run.smithi165.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 421 kB/s | 44 kB 00:00 2024-09-16T10:05:00.202 INFO:teuthology.orchestra.run.smithi165.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 719 kB/s | 106 kB 00:00 2024-09-16T10:05:00.235 INFO:teuthology.orchestra.run.smithi165.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 953 kB/s | 72 kB 00:00 2024-09-16T10:05:00.269 INFO:teuthology.orchestra.run.smithi165.stdout:(4/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.1 MB/s | 906 kB 00:00 2024-09-16T10:05:00.295 INFO:teuthology.orchestra.run.smithi165.stdout:(5/6): python3-kmod-0.9-32.el9.x86_64.rpm 912 kB/s | 84 kB 00:00 2024-09-16T10:05:00.334 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): libnvme-1.9-3.el9.x86_64.rpm 717 kB/s | 106 kB 00:00 2024-09-16T10:05:00.336 INFO:teuthology.orchestra.run.smithi165.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 8.1 MB/s | 837 kB 00:00 2024-09-16T10:05:00.336 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:05:00.337 INFO:teuthology.orchestra.run.smithi165.stdout:Total 6.0 MB/s | 2.0 MB 00:00 2024-09-16T10:05:00.384 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): python3-configshell-1.1.30-1.el9.noarch. 1.4 MB/s | 72 kB 00:00 2024-09-16T10:05:00.409 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:05:00.423 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:05:00.423 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:05:00.434 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-16T10:05:00.593 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.2 MB/s | 906 kB 00:00 2024-09-16T10:05:00.673 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:05:00.673 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:05:00.726 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 2.8 MB/s | 837 kB 00:00 2024-09-16T10:05:01.129 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:05:01.222 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-16T10:05:01.260 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): nvmetcli-0.7-3.el9.noarch.rpm 41 kB/s | 44 kB 00:01 2024-09-16T10:05:01.260 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:05:01.260 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-09-16T10:05:01.264 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-16T10:05:01.294 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-16T10:05:01.344 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:05:01.360 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:05:01.360 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:05:01.499 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-16T10:05:01.534 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-16T10:05:01.653 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:05:01.653 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:05:01.986 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-16T10:05:02.012 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-16T10:05:02.089 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:05:02.205 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-16T10:05:02.239 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-16T10:05:02.267 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-16T10:05:02.489 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-16T10:05:02.523 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-16T10:05:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-16T10:05:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-16T10:05:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-16T10:05:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-16T10:05:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-16T10:05:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-16T10:05:02.999 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-16T10:05:03.026 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-16T10:05:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-16T10:05:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:03.318 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:05:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T10:05:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-16T10:05:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-16T10:05:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:03.318 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:05:03.675 DEBUG:teuthology.parallel:result is None 2024-09-16T10:05:04.008 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-16T10:05:04.009 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-16T10:05:04.009 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-16T10:05:04.009 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-16T10:05:04.009 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-16T10:05:04.009 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-16T10:05:04.316 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-16T10:05:04.316 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:05:04.316 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:05:04.316 INFO:teuthology.orchestra.run.smithi063.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T10:05:04.316 INFO:teuthology.orchestra.run.smithi063.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-16T10:05:04.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-16T10:05:04.317 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:05:04.317 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:05:04.633 DEBUG:teuthology.parallel:result is None 2024-09-16T10:05:04.633 INFO:teuthology.run_tasks:Running task install... 2024-09-16T10:05:04.645 DEBUG:teuthology.task.install:project ceph 2024-09-16T10:05:04.646 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad'}} 2024-09-16T10:05:04.646 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad'} 2024-09-16T10:05:04.646 INFO:teuthology.task.install:Using flavor: default 2024-09-16T10:05:04.653 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T10:05:04.654 INFO:teuthology.task.install:extra packages: [] 2024-09-16T10:05:04.654 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad', 'tag': None, 'wait_for_package': False} 2024-09-16T10:05:04.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:05:04.657 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad', 'tag': None, 'wait_for_package': False} 2024-09-16T10:05:04.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:05:04.801 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/ 2024-09-16T10:05:04.802 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4882.g66e33d6a 2024-09-16T10:05:04.805 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/ 2024-09-16T10:05:04.805 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4882.g66e33d6a 2024-09-16T10:05:04.914 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T10:05:04.915 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:05:04.915 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T10:05:04.918 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T10:05:04.918 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:05:04.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T10:05:04.944 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T10:05:04.945 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/66e33d6af1496291b0602ee0a859e68145f17dad/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T10:05:04.952 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T10:05:04.953 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/66e33d6af1496291b0602ee0a859e68145f17dad/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T10:05:05.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T10:05:05.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T10:05:05.090 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T10:05:05.099 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T10:05:05.163 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-09-16T10:05:05.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-09-16T10:05:05.165 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-09-16T10:05:05.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-09-16T10:05:05.396 INFO:teuthology.orchestra.run.smithi165.stdout:45 files removed 2024-09-16T10:05:05.403 INFO:teuthology.orchestra.run.smithi063.stdout:45 files removed 2024-09-16T10:05:05.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw 2024-09-16T10:05:05.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-09-16T10:05:05.986 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 303 kB/s | 84 kB 00:00 2024-09-16T10:05:05.989 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 325 kB/s | 84 kB 00:00 2024-09-16T10:05:06.278 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-16T10:05:06.281 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-16T10:05:06.557 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-16T10:05:06.562 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-16T10:05:08.345 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-09-16T10:05:09.915 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.2 MB 00:03 2024-09-16T10:05:14.479 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 4.0 MB/s | 20 MB 00:05 2024-09-16T10:05:18.753 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T10:05:19.949 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 20 MB 00:08 2024-09-16T10:05:20.189 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-16T10:05:21.152 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-16T10:05:24.583 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.5 MB 00:00 2024-09-16T10:05:26.192 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-16T10:05:26.574 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-16T10:05:27.388 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-16T10:05:28.919 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:05:28.921 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:05:28.921 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:05:28.921 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 11 M 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 3.5 M 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 3.2 M 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 5.5 M 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 22 M 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 25 k 2024-09-16T10:05:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 755 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 508 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 5.5 M 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T10:05:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 45 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 143 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 161 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 321 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 306 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 100 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T10:05:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:Installing weak dependencies: 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:Install 31 Packages 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 2 Packages 2024-09-16T10:05:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:28.926 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 62 M 2024-09-16T10:05:28.926 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:05:29.442 INFO:teuthology.orchestra.run.smithi165.stdout:(1/33): ceph-base-19.3.0-4882.g66e33d6a.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-16T10:05:29.509 INFO:teuthology.orchestra.run.smithi165.stdout:(2/33): ceph-selinux-19.3.0-4882.g66e33d6a.el9. 378 kB/s | 25 kB 00:00 2024-09-16T10:05:29.609 INFO:teuthology.orchestra.run.smithi165.stdout:(3/33): libcephfs2-19.3.0-4882.g66e33d6a.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi165.stdout:(4/33): libradosstriper1-19.3.0-4882.g66e33d6a. 6.0 MB/s | 508 kB 00:00 2024-09-16T10:05:30.043 INFO:teuthology.orchestra.run.smithi165.stdout:(5/33): ceph-radosgw-19.3.0-4882.g66e33d6a.el9. 10 MB/s | 11 MB 00:01 2024-09-16T10:05:30.235 INFO:teuthology.orchestra.run.smithi165.stdout:(6/33): ceph-common-19.3.0-4882.g66e33d6a.el9.x 17 MB/s | 22 MB 00:01 2024-09-16T10:05:30.261 INFO:teuthology.orchestra.run.smithi165.stdout:(7/33): python3-ceph-argparse-19.3.0-4882.g66e3 208 kB/s | 45 kB 00:00 2024-09-16T10:05:30.286 INFO:teuthology.orchestra.run.smithi165.stdout:(8/33): python3-ceph-common-19.3.0-4882.g66e33d 2.8 MB/s | 143 kB 00:00 2024-09-16T10:05:30.311 INFO:teuthology.orchestra.run.smithi165.stdout:(9/33): python3-cephfs-19.3.0-4882.g66e33d6a.el 3.1 MB/s | 161 kB 00:00 2024-09-16T10:05:30.337 INFO:teuthology.orchestra.run.smithi165.stdout:(10/33): python3-rados-19.3.0-4882.g66e33d6a.el 6.2 MB/s | 321 kB 00:00 2024-09-16T10:05:30.362 INFO:teuthology.orchestra.run.smithi165.stdout:(11/33): python3-rbd-19.3.0-4882.g66e33d6a.el9. 5.9 MB/s | 306 kB 00:00 2024-09-16T10:05:30.429 INFO:teuthology.orchestra.run.smithi165.stdout:(12/33): librgw2-19.3.0-4882.g66e33d6a.el9.x86_ 7.5 MB/s | 5.5 MB 00:00 2024-09-16T10:05:30.454 INFO:teuthology.orchestra.run.smithi165.stdout:(13/33): python3-rgw-19.3.0-4882.g66e33d6a.el9. 849 kB/s | 100 kB 00:00 2024-09-16T10:05:32.096 INFO:teuthology.orchestra.run.smithi165.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 62 kB/s | 107 kB 00:01 2024-09-16T10:05:32.263 INFO:teuthology.orchestra.run.smithi165.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 89 kB/s | 160 kB 00:01 2024-09-16T10:05:32.288 INFO:teuthology.orchestra.run.smithi165.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 92 kB/s | 171 kB 00:01 2024-09-16T10:05:32.985 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-16T10:05:33.263 INFO:teuthology.orchestra.run.smithi165.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 39 kB/s | 45 kB 00:01 2024-09-16T10:05:33.847 INFO:teuthology.orchestra.run.smithi165.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 188 kB/s | 292 kB 00:01 2024-09-16T10:05:34.022 INFO:teuthology.orchestra.run.smithi165.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 377 kB/s | 662 kB 00:01 2024-09-16T10:05:34.439 INFO:teuthology.orchestra.run.smithi165.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 36 kB/s | 42 kB 00:01 2024-09-16T10:05:35.231 INFO:teuthology.orchestra.run.smithi165.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 30 kB/s | 36 kB 00:01 2024-09-16T10:05:35.281 INFO:teuthology.orchestra.run.smithi165.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-16T10:05:35.390 INFO:teuthology.orchestra.run.smithi165.stdout:(23/33): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-16T10:05:35.415 INFO:teuthology.orchestra.run.smithi165.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-16T10:05:35.424 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:05:35.426 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:05:35.426 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 11 M 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 3.5 M 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 3.2 M 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 5.5 M 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 22 M 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 25 k 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T10:05:35.427 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 755 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 508 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 5.5 M 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 45 k 2024-09-16T10:05:35.428 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 143 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 161 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 321 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 306 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 100 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T10:05:35.429 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:05:35.430 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:05:35.430 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:05:35.430 INFO:teuthology.orchestra.run.smithi063.stdout:Install 31 Packages 2024-09-16T10:05:35.430 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-09-16T10:05:35.430 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:05:35.430 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 62 M 2024-09-16T10:05:35.430 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:05:35.440 INFO:teuthology.orchestra.run.smithi165.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T10:05:35.466 INFO:teuthology.orchestra.run.smithi165.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-16T10:05:35.499 INFO:teuthology.orchestra.run.smithi165.stdout:(27/33): qatlib-24.02.0-1.el9.x86_64.rpm 134 kB/s | 221 kB 00:01 2024-09-16T10:05:35.550 INFO:teuthology.orchestra.run.smithi165.stdout:(28/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-16T10:05:35.592 INFO:teuthology.orchestra.run.smithi165.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 2.0 MB/s | 191 kB 00:00 2024-09-16T10:05:35.630 INFO:teuthology.orchestra.run.smithi165.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 20 MB/s | 1.6 MB 00:00 2024-09-16T10:05:35.680 INFO:teuthology.orchestra.run.smithi165.stdout:(31/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 38 kB/s | 47 kB 00:01 2024-09-16T10:05:35.847 INFO:teuthology.orchestra.run.smithi165.stdout:(32/33): librados2-19.3.0-4882.g66e33d6a.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-16T10:05:35.922 INFO:teuthology.orchestra.run.smithi165.stdout:(33/33): librbd1-19.3.0-4882.g66e33d6a.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-16T10:05:35.927 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:05:35.927 INFO:teuthology.orchestra.run.smithi165.stdout:Total 8.8 MB/s | 62 MB 00:07 2024-09-16T10:05:36.036 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:05:36.074 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:05:36.074 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:05:36.222 INFO:teuthology.orchestra.run.smithi063.stdout:(1/33): ceph-radosgw-19.3.0-4882.g66e33d6a.el9. 18 MB/s | 11 MB 00:00 2024-09-16T10:05:36.253 INFO:teuthology.orchestra.run.smithi063.stdout:(2/33): ceph-selinux-19.3.0-4882.g66e33d6a.el9. 605 kB/s | 25 kB 00:00 2024-09-16T10:05:36.312 INFO:teuthology.orchestra.run.smithi063.stdout:(3/33): libcephfs2-19.3.0-4882.g66e33d6a.el9.x8 13 MB/s | 755 kB 00:00 2024-09-16T10:05:36.362 INFO:teuthology.orchestra.run.smithi063.stdout:(4/33): libradosstriper1-19.3.0-4882.g66e33d6a. 9.9 MB/s | 508 kB 00:00 2024-09-16T10:05:36.596 INFO:teuthology.orchestra.run.smithi063.stdout:(5/33): librgw2-19.3.0-4882.g66e33d6a.el9.x86_6 24 MB/s | 5.5 MB 00:00 2024-09-16T10:05:36.625 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:05:36.628 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:05:36.671 INFO:teuthology.orchestra.run.smithi063.stdout:(6/33): ceph-base-19.3.0-4882.g66e33d6a.el9.x86 5.2 MB/s | 5.5 MB 00:01 2024-09-16T10:05:36.696 INFO:teuthology.orchestra.run.smithi063.stdout:(7/33): python3-ceph-argparse-19.3.0-4882.g66e3 451 kB/s | 45 kB 00:00 2024-09-16T10:05:36.722 INFO:teuthology.orchestra.run.smithi063.stdout:(8/33): python3-ceph-common-19.3.0-4882.g66e33d 2.8 MB/s | 143 kB 00:00 2024-09-16T10:05:36.747 INFO:teuthology.orchestra.run.smithi063.stdout:(9/33): python3-cephfs-19.3.0-4882.g66e33d6a.el 3.1 MB/s | 161 kB 00:00 2024-09-16T10:05:36.772 INFO:teuthology.orchestra.run.smithi063.stdout:(10/33): python3-rados-19.3.0-4882.g66e33d6a.el 6.3 MB/s | 321 kB 00:00 2024-09-16T10:05:36.797 INFO:teuthology.orchestra.run.smithi063.stdout:(11/33): python3-rbd-19.3.0-4882.g66e33d6a.el9. 5.9 MB/s | 306 kB 00:00 2024-09-16T10:05:36.823 INFO:teuthology.orchestra.run.smithi063.stdout:(12/33): python3-rgw-19.3.0-4882.g66e33d6a.el9. 1.9 MB/s | 100 kB 00:00 2024-09-16T10:05:36.990 INFO:teuthology.orchestra.run.smithi063.stdout:(13/33): boost-program-options-1.75.0-8.el9.x86 557 kB/s | 107 kB 00:00 2024-09-16T10:05:37.040 INFO:teuthology.orchestra.run.smithi063.stdout:(14/33): libnbd-1.20.2-2.el9.x86_64.rpm 787 kB/s | 171 kB 00:00 2024-09-16T10:05:37.073 INFO:teuthology.orchestra.run.smithi063.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-09-16T10:05:37.099 INFO:teuthology.orchestra.run.smithi063.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 771 kB/s | 45 kB 00:00 2024-09-16T10:05:37.333 INFO:teuthology.orchestra.run.smithi063.stdout:(17/33): ceph-common-19.3.0-4882.g66e33d6a.el9. 13 MB/s | 22 MB 00:01 2024-09-16T10:05:37.358 INFO:teuthology.orchestra.run.smithi063.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-09-16T10:05:37.383 INFO:teuthology.orchestra.run.smithi063.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-16T10:05:37.416 INFO:teuthology.orchestra.run.smithi063.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-16T10:05:37.442 INFO:teuthology.orchestra.run.smithi063.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 612 kB/s | 36 kB 00:00 2024-09-16T10:05:37.475 INFO:teuthology.orchestra.run.smithi063.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-09-16T10:05:37.500 INFO:teuthology.orchestra.run.smithi063.stdout:(23/33): python3-prettytable-0.7.2-27.el9.noarc 252 kB/s | 42 kB 00:00 2024-09-16T10:05:37.542 INFO:teuthology.orchestra.run.smithi063.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-16T10:05:37.568 INFO:teuthology.orchestra.run.smithi063.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-16T10:05:37.593 INFO:teuthology.orchestra.run.smithi063.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 306 kB/s | 26 kB 00:00 2024-09-16T10:05:37.618 INFO:teuthology.orchestra.run.smithi063.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T10:05:37.643 INFO:teuthology.orchestra.run.smithi063.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-16T10:05:37.702 INFO:teuthology.orchestra.run.smithi063.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-16T10:05:37.744 INFO:teuthology.orchestra.run.smithi063.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T10:05:37.828 INFO:teuthology.orchestra.run.smithi063.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-16T10:05:37.928 INFO:teuthology.orchestra.run.smithi063.stdout:(32/33): librbd1-19.3.0-4882.g66e33d6a.el9.x86_ 18 MB/s | 3.2 MB 00:00 2024-09-16T10:05:38.103 INFO:teuthology.orchestra.run.smithi063.stdout:(33/33): librados2-19.3.0-4882.g66e33d6a.el9.x8 8.7 MB/s | 3.5 MB 00:00 2024-09-16T10:05:38.108 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:05:38.108 INFO:teuthology.orchestra.run.smithi063.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-16T10:05:38.147 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:05:38.229 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:05:38.231 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T10:05:38.261 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T10:05:38.270 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:05:38.270 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:05:38.294 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T10:05:38.325 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T10:05:38.346 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T10:05:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T10:05:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T10:05:38.844 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:05:38.845 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:05:38.940 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T10:05:39.038 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 7/35 2024-09-16T10:05:39.131 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 7/35 2024-09-16T10:05:39.230 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 8/35 2024-09-16T10:05:39.257 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 9/35 2024-09-16T10:05:39.328 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 9/35 2024-09-16T10:05:39.357 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T10:05:39.400 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:19.3.0-4882.g66e33d6a.el9. 11/35 2024-09-16T10:05:39.470 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 12/35 2024-09-16T10:05:39.493 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_6 13/35 2024-09-16T10:05:39.573 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_6 13/35 2024-09-16T10:05:39.608 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T10:05:39.716 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T10:05:39.764 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T10:05:39.808 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T10:05:39.849 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T10:05:39.878 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T10:05:39.915 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T10:05:39.956 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T10:05:40.300 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T10:05:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 23/35 2024-09-16T10:05:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:05:40.441 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 23/35 2024-09-16T10:05:40.456 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T10:05:40.471 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 24/35 2024-09-16T10:05:40.507 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T10:05:40.511 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T10:05:40.542 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T10:05:40.571 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T10:05:40.600 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T10:05:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T10:05:40.845 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T10:05:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T10:05:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 27/35 2024-09-16T10:05:41.102 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 27/35 2024-09-16T10:05:41.177 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 28/35 2024-09-16T10:05:41.203 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T10:05:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:19.3.0-4882.g66e33d6a.el9.x8 29/35 2024-09-16T10:05:41.317 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 7/35 2024-09-16T10:05:41.420 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 7/35 2024-09-16T10:05:41.508 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 8/35 2024-09-16T10:05:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 9/35 2024-09-16T10:05:41.598 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 9/35 2024-09-16T10:05:41.627 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T10:05:41.670 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.3.0-4882.g66e33d6a.el9. 11/35 2024-09-16T10:05:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 12/35 2024-09-16T10:05:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_6 13/35 2024-09-16T10:05:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_6 13/35 2024-09-16T10:05:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T10:05:41.948 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T10:05:41.987 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T10:05:42.034 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T10:05:42.069 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T10:05:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T10:05:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T10:05:42.168 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T10:05:42.512 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T10:05:42.552 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 23/35 2024-09-16T10:05:42.644 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 23/35 2024-09-16T10:05:42.683 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 24/35 2024-09-16T10:05:42.718 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T10:05:43.240 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T10:05:43.271 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 27/35 2024-09-16T10:05:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 27/35 2024-09-16T10:05:43.423 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 28/35 2024-09-16T10:05:43.457 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.3.0-4882.g66e33d6a.el9.x8 29/35 2024-09-16T10:05:43.549 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 30/35 2024-09-16T10:05:43.601 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 30/35 2024-09-16T10:05:44.188 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 30/35 2024-09-16T10:05:44.230 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 31/35 2024-09-16T10:05:44.323 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 31/35 2024-09-16T10:05:44.323 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T10:05:44.323 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T10:05:44.323 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:44.365 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 32/35 2024-09-16T10:05:45.871 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 30/35 2024-09-16T10:05:45.911 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 30/35 2024-09-16T10:05:46.524 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 30/35 2024-09-16T10:05:46.556 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 31/35 2024-09-16T10:05:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 31/35 2024-09-16T10:05:46.649 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T10:05:46.650 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T10:05:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:05:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 32/35 2024-09-16T10:05:54.374 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 32/35 2024-09-16T10:05:54.374 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-09-16T10:05:54.374 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-09-16T10:05:54.374 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-09-16T10:05:54.374 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-09-16T10:05:54.374 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-09-16T10:05:54.374 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-09-16T10:05:54.375 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-09-16T10:05:54.375 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:55.392 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/35 2024-09-16T10:05:55.417 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/35 2024-09-16T10:05:55.417 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:05:55.417 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T10:05:55.418 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:05:55.418 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:05:55.418 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:55.420 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T10:05:55.507 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T10:05:55.507 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 5/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_6 6/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 7/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4882.g66e33d6a.el9. 8/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:19.3.0-4882.g66e33d6a.el9.x8 9/35 2024-09-16T10:05:56.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 10/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 11/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 12/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 13/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T10:05:56.634 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 32/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T10:05:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 34/35 2024-09-16T10:05:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 32/35 2024-09-16T10:05:57.305 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-09-16T10:05:57.305 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-09-16T10:05:57.305 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-09-16T10:05:57.305 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-09-16T10:05:57.306 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-09-16T10:05:57.306 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-09-16T10:05:57.306 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-09-16T10:05:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:05:57.747 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T10:05:57.747 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:57.747 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-09-16T10:05:57.747 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T10:05:57.748 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:05:57.749 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T10:05:57.750 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T10:05:57.750 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T10:05:57.750 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T10:05:57.750 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T10:05:57.750 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:57.750 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:05:58.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-test 2024-09-16T10:05:58.340 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/35 2024-09-16T10:05:58.367 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/35 2024-09-16T10:05:58.367 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:05:58.367 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T10:05:58.367 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:05:58.367 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:05:58.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:05:58.369 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T10:05:58.459 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T10:05:58.459 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T10:05:58.707 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:32 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:05:58.842 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 50 M 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 161 k 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T10:05:58.843 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T10:05:58.844 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:58.844 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:05:58.844 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:05:58.844 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-09-16T10:05:58.844 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:05:58.844 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 51 M 2024-09-16T10:05:58.845 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 217 M 2024-09-16T10:05:58.845 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:05:59.337 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): libcephsqlite-19.3.0-4882.g66e33d6a.el9. 762 kB/s | 161 kB 00:00 2024-09-16T10:05:59.362 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-16T10:05:59.479 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 449 kB/s | 64 kB 00:00 2024-09-16T10:05:59.607 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 5/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_6 6/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 7/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4882.g66e33d6a.el9. 8/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.3.0-4882.g66e33d6a.el9.x8 9/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 10/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 11/35 2024-09-16T10:05:59.608 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 12/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 13/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T10:05:59.609 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 32/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T10:05:59.610 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 34/35 2024-09-16T10:06:00.695 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): ceph-test-19.3.0-4882.g66e33d6a.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-16T10:06:00.698 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:00.698 INFO:teuthology.orchestra.run.smithi165.stdout:Total 27 MB/s | 51 MB 00:01 2024-09-16T10:06:00.726 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:06:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T10:06:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:00.732 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-09-16T10:06:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.732 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.732 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T10:06:00.733 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.734 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:00.735 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T10:06:00.735 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T10:06:00.735 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T10:06:00.735 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T10:06:00.735 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T10:06:00.735 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:00.735 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:00.746 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:06:00.746 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:06:01.099 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-09-16T10:06:01.130 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:06:01.130 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:06:01.666 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:06:01.686 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:01.781 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T10:06:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T10:06:01.825 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:01.826 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:06:01.826 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:01.826 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:06:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 50 M 2024-09-16T10:06:01.826 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 161 k 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-09-16T10:06:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:01.828 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 51 M 2024-09-16T10:06:01.828 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 217 M 2024-09-16T10:06:01.828 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:06:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/4 2024-09-16T10:06:02.073 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-16T10:06:02.098 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): libcephsqlite-19.3.0-4882.g66e33d6a.el9. 726 kB/s | 161 kB 00:00 2024-09-16T10:06:02.115 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T10:06:03.433 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): ceph-test-19.3.0-4882.g66e33d6a.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-16T10:06:03.434 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:03.434 INFO:teuthology.orchestra.run.smithi063.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-16T10:06:03.462 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:06:03.485 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:06:03.485 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:06:03.886 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:06:03.887 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:06:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:06:04.539 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T10:06:04.590 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T10:06:04.617 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/4 2024-09-16T10:06:06.624 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/4 2024-09-16T10:06:06.710 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/4 2024-09-16T10:06:09.188 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/4 2024-09-16T10:06:09.188 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/4 2024-09-16T10:06:09.188 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/4 2024-09-16T10:06:09.188 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:09.633 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:09.647 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/4 2024-09-16T10:06:09.726 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/4 2024-09-16T10:06:09.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph 2024-09-16T10:06:10.532 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:44 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:10.673 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 6.6 k 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 2.4 M 2024-09-16T10:06:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 1.5 M 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 291 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 4.8 M 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 17 M 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T10:06:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T10:06:10.678 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout:Install 37 Packages 2024-09-16T10:06:10.679 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:10.680 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 31 M 2024-09-16T10:06:10.680 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 109 M 2024-09-16T10:06:10.680 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:06:11.413 INFO:teuthology.orchestra.run.smithi165.stdout:(1/37): ceph-19.3.0-4882.g66e33d6a.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-16T10:06:11.564 INFO:teuthology.orchestra.run.smithi165.stdout:(2/37): ceph-mgr-19.3.0-4882.g66e33d6a.el9.x86_ 5.3 MB/s | 1.5 MB 00:00 2024-09-16T10:06:11.633 INFO:teuthology.orchestra.run.smithi165.stdout:(3/37): ceph-mds-19.3.0-4882.g66e33d6a.el9.x86_ 6.8 MB/s | 2.4 MB 00:00 2024-09-16T10:06:11.731 INFO:teuthology.orchestra.run.smithi165.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4882.g66e3 2.8 MB/s | 291 kB 00:00 2024-09-16T10:06:11.806 INFO:teuthology.orchestra.run.smithi165.stdout:(5/37): ceph-mon-19.3.0-4882.g66e33d6a.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-16T10:06:11.981 INFO:teuthology.orchestra.run.smithi165.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-16T10:06:12.148 INFO:teuthology.orchestra.run.smithi165.stdout:(7/37): ceph-osd-19.3.0-4882.g66e33d6a.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-16T10:06:12.173 INFO:teuthology.orchestra.run.smithi165.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-09-16T10:06:12.184 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/4 2024-09-16T10:06:12.185 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/4 2024-09-16T10:06:12.185 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/4 2024-09-16T10:06:12.185 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T10:06:12.224 INFO:teuthology.orchestra.run.smithi165.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-16T10:06:12.258 INFO:teuthology.orchestra.run.smithi165.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-09-16T10:06:12.283 INFO:teuthology.orchestra.run.smithi165.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T10:06:12.316 INFO:teuthology.orchestra.run.smithi165.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 806 kB/s | 135 kB 00:00 2024-09-16T10:06:12.392 INFO:teuthology.orchestra.run.smithi165.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-16T10:06:12.409 INFO:teuthology.orchestra.run.smithi165.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 378 kB/s | 35 kB 00:00 2024-09-16T10:06:12.434 INFO:teuthology.orchestra.run.smithi165.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-16T10:06:12.459 INFO:teuthology.orchestra.run.smithi165.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 978 kB/s | 172 kB 00:00 2024-09-16T10:06:12.476 INFO:teuthology.orchestra.run.smithi165.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 756 kB/s | 51 kB 00:00 2024-09-16T10:06:12.493 INFO:teuthology.orchestra.run.smithi165.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 705 kB/s | 42 kB 00:00 2024-09-16T10:06:12.535 INFO:teuthology.orchestra.run.smithi165.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-16T10:06:12.552 INFO:teuthology.orchestra.run.smithi165.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 241 kB/s | 22 kB 00:00 2024-09-16T10:06:12.577 INFO:teuthology.orchestra.run.smithi165.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 519 kB/s | 43 kB 00:00 2024-09-16T10:06:12.596 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T10:06:12.596 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:12.596 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:06:12.596 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:12.596 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:12.596 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T10:06:12.596 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T10:06:12.597 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:12.597 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:12.628 INFO:teuthology.orchestra.run.smithi165.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-16T10:06:12.645 INFO:teuthology.orchestra.run.smithi165.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 163 kB/s | 11 kB 00:00 2024-09-16T10:06:12.662 INFO:teuthology.orchestra.run.smithi165.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 529 kB/s | 18 kB 00:00 2024-09-16T10:06:12.679 INFO:teuthology.orchestra.run.smithi165.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 690 kB/s | 23 kB 00:00 2024-09-16T10:06:12.696 INFO:teuthology.orchestra.run.smithi165.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2024-09-16T10:06:12.713 INFO:teuthology.orchestra.run.smithi165.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 584 kB/s | 20 kB 00:00 2024-09-16T10:06:12.730 INFO:teuthology.orchestra.run.smithi165.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-16T10:06:12.755 INFO:teuthology.orchestra.run.smithi165.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-16T10:06:12.789 INFO:teuthology.orchestra.run.smithi165.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-16T10:06:12.805 INFO:teuthology.orchestra.run.smithi165.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T10:06:12.831 INFO:teuthology.orchestra.run.smithi165.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T10:06:12.856 INFO:teuthology.orchestra.run.smithi165.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-16T10:06:12.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-09-16T10:06:12.890 INFO:teuthology.orchestra.run.smithi165.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-16T10:06:12.915 INFO:teuthology.orchestra.run.smithi165.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-16T10:06:12.940 INFO:teuthology.orchestra.run.smithi165.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-16T10:06:12.966 INFO:teuthology.orchestra.run.smithi165.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T10:06:12.969 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:12.969 INFO:teuthology.orchestra.run.smithi165.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-16T10:06:13.115 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:06:13.156 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:06:13.156 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:06:13.442 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:41 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:13.582 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:13.584 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:13.584 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:06:13.584 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 6.6 k 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 2.4 M 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 1.5 M 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 291 k 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 4.8 M 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 17 M 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T10:06:13.585 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T10:06:13.586 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T10:06:13.587 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:13.588 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:06:13.588 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:13.588 INFO:teuthology.orchestra.run.smithi063.stdout:Install 37 Packages 2024-09-16T10:06:13.588 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:13.588 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 31 M 2024-09-16T10:06:13.588 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 109 M 2024-09-16T10:06:13.589 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:06:13.625 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:06:13.626 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:06:13.950 INFO:teuthology.orchestra.run.smithi063.stdout:(1/37): ceph-19.3.0-4882.g66e33d6a.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-16T10:06:14.133 INFO:teuthology.orchestra.run.smithi063.stdout:(2/37): ceph-mgr-19.3.0-4882.g66e33d6a.el9.x86_ 4.8 MB/s | 1.5 MB 00:00 2024-09-16T10:06:14.192 INFO:teuthology.orchestra.run.smithi063.stdout:(3/37): ceph-mds-19.3.0-4882.g66e33d6a.el9.x86_ 6.3 MB/s | 2.4 MB 00:00 2024-09-16T10:06:14.284 INFO:teuthology.orchestra.run.smithi063.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4882.g66e3 3.1 MB/s | 291 kB 00:00 2024-09-16T10:06:14.384 INFO:teuthology.orchestra.run.smithi063.stdout:(5/37): ceph-mon-19.3.0-4882.g66e33d6a.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-16T10:06:14.543 INFO:teuthology.orchestra.run.smithi063.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 980 kB/s | 253 kB 00:00 2024-09-16T10:06:14.710 INFO:teuthology.orchestra.run.smithi063.stdout:(7/37): ceph-osd-19.3.0-4882.g66e33d6a.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-16T10:06:14.735 INFO:teuthology.orchestra.run.smithi063.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-09-16T10:06:14.826 INFO:teuthology.orchestra.run.smithi063.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.3 MB/s | 125 kB 00:00 2024-09-16T10:06:14.899 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:06:14.905 INFO:teuthology.orchestra.run.smithi063.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 705 kB/s | 135 kB 00:00 2024-09-16T10:06:14.952 INFO:teuthology.orchestra.run.smithi063.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 2.2 MB/s | 1.2 MB 00:00 2024-09-16T10:06:14.960 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T10:06:14.977 INFO:teuthology.orchestra.run.smithi063.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.4 MB/s | 215 kB 00:00 2024-09-16T10:06:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T10:06:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T10:06:15.078 INFO:teuthology.orchestra.run.smithi063.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-09-16T10:06:15.110 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T10:06:15.111 INFO:teuthology.orchestra.run.smithi063.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T10:06:15.137 INFO:teuthology.orchestra.run.smithi063.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 804 kB/s | 188 kB 00:00 2024-09-16T10:06:15.162 INFO:teuthology.orchestra.run.smithi063.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T10:06:15.169 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T10:06:15.187 INFO:teuthology.orchestra.run.smithi063.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 708 kB/s | 77 kB 00:00 2024-09-16T10:06:15.203 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T10:06:15.212 INFO:teuthology.orchestra.run.smithi063.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-16T10:06:15.263 INFO:teuthology.orchestra.run.smithi063.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-16T10:06:15.288 INFO:teuthology.orchestra.run.smithi063.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.3 MB/s | 172 kB 00:00 2024-09-16T10:06:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T10:06:15.313 INFO:teuthology.orchestra.run.smithi063.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-16T10:06:15.338 INFO:teuthology.orchestra.run.smithi063.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-09-16T10:06:15.363 INFO:teuthology.orchestra.run.smithi063.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-16T10:06:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T10:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-16T10:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-16T10:06:15.415 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T10:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-16T10:06:15.449 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T10:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T10:06:15.478 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T10:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout:(28/37): lua-devel-5.4.4-4.el9.x86_64.rpm 68 kB/s | 22 kB 00:00 2024-09-16T10:06:15.513 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T10:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-09-16T10:06:15.547 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T10:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2024-09-16T10:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 339 kB/s | 16 kB 00:00 2024-09-16T10:06:15.587 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T10:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T10:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-16T10:06:15.638 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T10:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-16T10:06:15.673 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T10:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-16T10:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T10:06:15.722 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T10:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-16T10:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-16T10:06:15.774 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T10:06:15.870 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T10:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:06:16.018 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T10:06:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T10:06:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T10:06:16.226 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T10:06:16.282 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T10:06:16.345 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T10:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:06:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T10:06:16.592 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T10:06:16.671 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T10:06:16.762 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T10:06:16.809 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T10:06:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T10:06:17.113 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4882.g66e33d6a.el9. 32/37 2024-09-16T10:06:17.152 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/37 2024-09-16T10:06:17.178 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/37 2024-09-16T10:06:17.178 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:17.178 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T10:06:17.178 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:06:17.178 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:06:17.178 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:17.730 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:06:17.820 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T10:06:17.853 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T10:06:17.881 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T10:06:17.956 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T10:06:18.005 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T10:06:18.038 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T10:06:18.133 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T10:06:18.219 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T10:06:18.265 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T10:06:18.317 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T10:06:18.346 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T10:06:18.389 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T10:06:18.423 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T10:06:18.464 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T10:06:18.507 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T10:06:18.541 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T10:06:18.593 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T10:06:18.630 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T10:06:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 34/37 2024-09-16T10:06:18.663 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 34/37 2024-09-16T10:06:18.663 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:18.663 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T10:06:18.663 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:06:18.663 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:06:18.663 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:18.706 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T10:06:18.859 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T10:06:18.941 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T10:06:18.992 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T10:06:19.030 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T10:06:19.088 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T10:06:19.120 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 35/37 2024-09-16T10:06:19.144 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 35/37 2024-09-16T10:06:19.144 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:19.144 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T10:06:19.144 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:06:19.144 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:06:19.145 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:19.152 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T10:06:19.350 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T10:06:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 36/37 2024-09-16T10:06:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 36/37 2024-09-16T10:06:19.394 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:19.395 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T10:06:19.395 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:06:19.395 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:06:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:19.400 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T10:06:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 37/37 2024-09-16T10:06:19.481 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T10:06:19.571 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T10:06:19.626 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T10:06:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T10:06:19.937 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4882.g66e33d6a.el9. 32/37 2024-09-16T10:06:19.967 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/37 2024-09-16T10:06:19.995 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 33/37 2024-09-16T10:06:19.995 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:19.995 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T10:06:19.995 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:06:19.995 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:06:19.995 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 37/37 2024-09-16T10:06:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/37 2024-09-16T10:06:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/37 2024-09-16T10:06:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/37 2024-09-16T10:06:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/37 2024-09-16T10:06:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 5/37 2024-09-16T10:06:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4882.g66e33d6a.el9. 6/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T10:06:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T10:06:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T10:06:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T10:06:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T10:06:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T10:06:21.559 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 34/37 2024-09-16T10:06:21.588 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 34/37 2024-09-16T10:06:21.588 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:21.589 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T10:06:21.589 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:06:21.589 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:06:21.589 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 35/37 2024-09-16T10:06:22.094 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 35/37 2024-09-16T10:06:22.094 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:22.094 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T10:06:22.094 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:06:22.094 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:06:22.094 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T10:06:22.192 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T10:06:22.193 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T10:06:22.194 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T10:06:22.195 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T10:06:22.195 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T10:06:22.195 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T10:06:22.195 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:22.195 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:22.336 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 36/37 2024-09-16T10:06:22.361 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 36/37 2024-09-16T10:06:22.361 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:22.361 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T10:06:22.361 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:06:22.361 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:06:22.362 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:22.427 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 37/37 2024-09-16T10:06:22.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-base 2024-09-16T10:06:23.198 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:23.289 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:06:23.332 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:23.333 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:06:23.333 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:23.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install cephadm 2024-09-16T10:06:23.971 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:24.102 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 784 k 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:06:24.103 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:24.104 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:06:24.104 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:24.104 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 784 k 2024-09-16T10:06:24.104 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 791 k 2024-09-16T10:06:24.104 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:06:24.173 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 37/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 3/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 4/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 5/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4882.g66e33d6a.el9. 6/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T10:06:24.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T10:06:24.175 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T10:06:24.176 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T10:06:24.176 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T10:06:24.178 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T10:06:24.397 INFO:teuthology.orchestra.run.smithi165.stdout:cephadm-19.3.0-4882.g66e33d6a.el9.noarch.rpm 2.6 MB/s | 784 kB 00:00 2024-09-16T10:06:24.397 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:24.398 INFO:teuthology.orchestra.run.smithi165.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-16T10:06:24.398 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:06:24.403 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:06:24.403 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:06:24.409 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:06:24.409 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:06:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:06:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:25.315 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T10:06:25.315 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:25.315 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:06:25.315 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:25.315 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:25.315 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:25.315 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T10:06:25.316 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T10:06:25.317 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:25.318 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:25.461 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:25.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-09-16T10:06:25.737 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:25.737 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:25.737 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:06:25.737 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:25.737 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:25.737 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:25.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T10:06:26.232 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:26.326 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:06:26.370 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:26.371 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:06:26.371 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:26.441 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-09-16T10:06:26.535 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:26.673 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:26.673 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 145 k 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 145 k 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 432 k 2024-09-16T10:06:26.674 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:06:26.888 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-immutable-object-cache-19.3.0-4882.g66e33d 680 kB/s | 145 kB 00:00 2024-09-16T10:06:26.888 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:26.889 INFO:teuthology.orchestra.run.smithi165.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-16T10:06:26.889 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:06:26.896 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:06:26.896 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:06:26.955 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:06:26.955 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:06:27.020 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:27.111 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:06:27.158 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 784 k 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:27.159 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 784 k 2024-09-16T10:06:27.160 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 791 k 2024-09-16T10:06:27.160 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:06:27.189 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a. 1/1 2024-09-16T10:06:27.213 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a. 1/1 2024-09-16T10:06:27.213 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:27.213 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T10:06:27.213 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:27.416 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-19.3.0-4882.g66e33d6a.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-16T10:06:27.417 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:27.417 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-16T10:06:27.417 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:06:27.423 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:06:27.423 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:06:27.429 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:06:27.430 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:06:27.502 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:06:27.989 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:28.064 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:28.121 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a. 1/1 2024-09-16T10:06:28.121 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:28.121 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:06:28.121 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:28.121 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:28.121 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:28.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr 2024-09-16T10:06:28.432 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:28.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:06:28.672 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:28.672 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:06:28.672 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:28.672 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:28.672 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:28.887 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T10:06:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:02 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:29.018 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:06:29.061 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:29.062 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:06:29.062 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:29.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T10:06:29.483 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:29.622 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:29.622 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:29.622 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-16T10:06:29.622 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 145 k 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 145 k 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 432 k 2024-09-16T10:06:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:06:29.697 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:03 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:29.804 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-19.3.0-4882.g66e33d 801 kB/s | 145 kB 00:00 2024-09-16T10:06:29.805 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:29.805 INFO:teuthology.orchestra.run.smithi063.stdout:Total 794 kB/s | 145 kB 00:00 2024-09-16T10:06:29.805 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:06:29.813 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:06:29.813 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:06:29.830 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:29.831 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:29.831 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:06:29.831 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:29.831 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 14 M 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 34 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 16 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T10:06:29.832 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T10:06:29.833 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:29.833 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:06:29.833 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:29.833 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-09-16T10:06:29.833 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:29.833 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 19 M 2024-09-16T10:06:29.834 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 159 M 2024-09-16T10:06:29.834 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:06:29.874 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:06:29.876 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:06:30.025 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:06:30.089 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a. 1/1 2024-09-16T10:06:30.115 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a. 1/1 2024-09-16T10:06:30.115 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:06:30.115 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T10:06:30.116 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:30.342 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4882.g66 218 kB/s | 34 kB 00:00 2024-09-16T10:06:30.368 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4882.g66e 90 kB/s | 16 kB 00:00 2024-09-16T10:06:30.560 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-16T10:06:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): ceph-mgr-dashboard-19.3.0-4882.g66e33d6 21 MB/s | 14 MB 00:00 2024-09-16T10:06:30.902 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-16T10:06:30.927 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-09-16T10:06:30.978 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 258 kB/s | 19 kB 00:00 2024-09-16T10:06:31.020 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.5 MB/s | 548 kB 00:00 2024-09-16T10:06:31.045 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.2 MB/s | 144 kB 00:00 2024-09-16T10:06:31.070 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 1.2 MB/s | 31 kB 00:00 2024-09-16T10:06:31.072 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a. 1/1 2024-09-16T10:06:31.072 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:31.072 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:06:31.072 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:06:31.072 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:31.072 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:31.104 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-16T10:06:31.179 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.6 MB/s | 2.0 MB 00:00 2024-09-16T10:06:31.183 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:31.183 INFO:teuthology.orchestra.run.smithi165.stdout:Total 14 MB/s | 19 MB 00:01 2024-09-16T10:06:31.231 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:06:31.249 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:06:31.249 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:06:31.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-09-16T10:06:31.523 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:06:31.523 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:06:31.892 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:31.987 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:06:32.032 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:32.033 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:06:32.033 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:32.105 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T10:06:32.129 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:06:32.213 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T10:06:32.257 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T10:06:32.304 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T10:06:32.342 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T10:06:32.432 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T10:06:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T10:06:32.671 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T10:06:32.695 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:32.723 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T10:06:32.751 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T10:06:32.784 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4882.g66e33d6a.el9 10/12 2024-09-16T10:06:32.834 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:32.835 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:32.835 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:06:32.835 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:32.835 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:06:32.835 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 14 M 2024-09-16T10:06:32.835 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:06:32.836 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T10:06:32.836 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 34 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 16 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:06:32.837 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:32.838 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-09-16T10:06:32.838 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:32.838 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 19 M 2024-09-16T10:06:32.838 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 159 M 2024-09-16T10:06:32.838 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:06:33.624 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4882.g66e 105 kB/s | 16 kB 00:00 2024-09-16T10:06:33.650 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4882.g66 189 kB/s | 34 kB 00:00 2024-09-16T10:06:33.850 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-16T10:06:33.952 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-09-16T10:06:34.117 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4882.g66e33d6 22 MB/s | 14 MB 00:00 2024-09-16T10:06:34.167 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 410 kB/s | 19 kB 00:00 2024-09-16T10:06:34.201 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-16T10:06:34.226 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-09-16T10:06:34.260 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T10:06:34.293 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.9 MB/s | 188 kB 00:00 2024-09-16T10:06:34.310 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.3 MB/s | 144 kB 00:00 2024-09-16T10:06:34.361 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-16T10:06:34.365 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:34.365 INFO:teuthology.orchestra.run.smithi063.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-16T10:06:34.417 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:06:34.438 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:06:34.438 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:06:34.723 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:06:34.723 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:06:35.337 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:06:35.412 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T10:06:35.447 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T10:06:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T10:06:35.531 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T10:06:35.625 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T10:06:35.754 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T10:06:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T10:06:35.895 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T10:06:35.923 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T10:06:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4882.g66e33d6a.el9 10/12 2024-09-16T10:06:35.961 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4882.g66e33d6a.el 11/12 2024-09-16T10:06:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noa 12/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noa 12/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4882.g66e33d6a.el 1/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noa 2/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4882.g66e33d6a.el9 3/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T10:06:37.999 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T10:06:38.000 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T10:06:38.000 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T10:06:38.000 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T10:06:38.318 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T10:06:38.318 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T10:06:38.319 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:38.320 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:38.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T10:06:39.156 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:13 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:39.291 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:39.292 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 7.4 M 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T10:06:39.293 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-09-16T10:06:39.294 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:39.295 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 38 M 2024-09-16T10:06:39.295 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 212 M 2024-09-16T10:06:39.295 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:06:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4882.g66e33d6a.el 11/12 2024-09-16T10:06:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noa 12/12 2024-09-16T10:06:39.712 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 975 kB/s | 191 kB 00:00 2024-09-16T10:06:39.863 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-16T10:06:39.971 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-16T10:06:39.997 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 115 kB/s | 33 kB 00:00 2024-09-16T10:06:40.205 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 185 kB/s | 38 kB 00:00 2024-09-16T10:06:40.255 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-16T10:06:40.722 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 524 kB/s | 245 kB 00:00 2024-09-16T10:06:41.006 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-16T10:06:41.106 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 5.5 MB/s | 4.9 MB 00:00 2024-09-16T10:06:41.131 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-09-16T10:06:41.248 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noa 12/12 2024-09-16T10:06:41.248 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4882.g66e33d6a.el 1/12 2024-09-16T10:06:41.248 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noa 2/12 2024-09-16T10:06:41.248 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4882.g66e33d6a.el9 3/12 2024-09-16T10:06:41.248 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T10:06:41.248 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T10:06:41.249 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T10:06:41.249 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T10:06:41.249 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T10:06:41.249 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T10:06:41.249 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T10:06:41.249 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T10:06:41.563 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T10:06:41.564 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T10:06:41.565 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T10:06:41.565 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T10:06:41.565 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T10:06:41.565 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:41.565 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:41.573 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.0 MB/s | 5.1 MB 00:00 2024-09-16T10:06:41.821 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T10:06:42.420 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:42.565 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:42.565 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-09-16T10:06:42.565 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:06:42.565 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 7.4 M 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T10:06:42.566 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T10:06:42.567 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T10:06:42.567 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T10:06:42.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:42.567 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:06:42.567 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-09-16T10:06:42.567 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-09-16T10:06:42.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:42.568 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 38 M 2024-09-16T10:06:42.568 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 212 M 2024-09-16T10:06:42.568 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:06:42.850 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-16T10:06:42.908 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.2 MB/s | 801 kB 00:00 2024-09-16T10:06:42.942 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-09-16T10:06:43.025 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-09-16T10:06:43.109 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-16T10:06:43.134 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 355 kB/s | 38 kB 00:00 2024-09-16T10:06:43.218 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-16T10:06:43.360 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 20 MB/s | 4.9 MB 00:00 2024-09-16T10:06:43.443 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-16T10:06:43.469 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-09-16T10:06:43.527 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 4.8 MB/s | 3.0 MB 00:00 2024-09-16T10:06:43.778 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-16T10:06:43.779 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:43.779 INFO:teuthology.orchestra.run.smithi063.stdout:Total 32 MB/s | 38 MB 00:01 2024-09-16T10:06:44.007 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.6 MB/s | 16 MB 00:02 2024-09-16T10:06:44.008 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:44.008 INFO:teuthology.orchestra.run.smithi165.stdout:Total 8.1 MB/s | 38 MB 00:04 2024-09-16T10:06:44.144 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:06:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:06:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:06:44.362 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:06:44.388 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:06:44.388 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:06:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:06:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:06:44.814 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:06:44.814 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:06:45.407 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:06:45.471 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T10:06:45.520 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:06:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:06:45.625 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:06:45.669 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T10:06:45.680 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T10:06:45.735 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:06:45.840 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:06:45.869 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T10:06:45.882 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T10:06:45.909 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T10:06:45.988 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T10:06:46.066 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T10:06:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T10:06:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T10:06:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T10:06:46.840 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T10:06:46.872 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T10:06:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T10:06:48.173 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:06:48.260 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:06:49.258 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T10:06:49.299 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33 12/12 2024-09-16T10:06:49.308 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T10:06:49.340 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33 12/12 2024-09-16T10:06:51.783 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33 12/12 2024-09-16T10:06:51.783 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33 1/12 2024-09-16T10:06:51.783 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:06:51.784 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33 12/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33 1/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T10:06:51.907 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T10:06:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T10:06:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T10:06:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:06:51.908 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T10:06:52.169 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T10:06:52.169 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:52.169 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T10:06:52.170 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T10:06:52.171 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T10:06:52.171 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T10:06:52.171 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:52.171 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:06:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T10:06:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T10:06:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T10:06:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T10:06:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:52.330 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:52.547 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-09-16T10:06:52.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-rook 2024-09-16T10:06:53.142 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:21 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:06:53.285 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:06:53.286 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:53.286 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:06:53.286 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:53.286 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:06:53.286 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 49 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T10:06:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T10:06:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T10:06:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:53.288 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:06:53.288 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:06:53.288 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-09-16T10:06:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:06:53.288 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.2 M 2024-09-16T10:06:53.289 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-09-16T10:06:53.289 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:06:53.344 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:27 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:53.480 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:53.481 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:53.481 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:06:53.481 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:53.482 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:06:53.482 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 49 k 2024-09-16T10:06:53.482 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:06:53.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T10:06:53.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T10:06:53.484 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T10:06:53.484 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:53.484 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:06:53.484 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:53.484 INFO:teuthology.orchestra.run.smithi165.stdout:Install 13 Packages 2024-09-16T10:06:53.484 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:53.484 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 2.2 M 2024-09-16T10:06:53.485 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 26 M 2024-09-16T10:06:53.485 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:06:53.865 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): ceph-mgr-rook-19.3.0-4882.g66e33d6a.el9 303 kB/s | 49 kB 00:00 2024-09-16T10:06:54.226 INFO:teuthology.orchestra.run.smithi165.stdout:(1/13): ceph-mgr-rook-19.3.0-4882.g66e33d6a.el9 282 kB/s | 49 kB 00:00 2024-09-16T10:06:54.343 INFO:teuthology.orchestra.run.smithi165.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 66 kB/s | 19 kB 00:00 2024-09-16T10:06:54.369 INFO:teuthology.orchestra.run.smithi165.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 81 kB/s | 26 kB 00:00 2024-09-16T10:06:54.719 INFO:teuthology.orchestra.run.smithi165.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-16T10:06:54.790 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 18 kB/s | 19 kB 00:01 2024-09-16T10:06:54.828 INFO:teuthology.orchestra.run.smithi165.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 608 kB/s | 279 kB 00:00 2024-09-16T10:06:54.853 INFO:teuthology.orchestra.run.smithi165.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 402 kB/s | 54 kB 00:00 2024-09-16T10:06:54.895 INFO:teuthology.orchestra.run.smithi165.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 332 kB/s | 222 kB 00:00 2024-09-16T10:06:54.907 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 21 kB/s | 26 kB 00:01 2024-09-16T10:06:54.911 INFO:teuthology.orchestra.run.smithi165.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 383 kB/s | 32 kB 00:00 2024-09-16T10:06:54.937 INFO:teuthology.orchestra.run.smithi165.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 168 kB/s | 14 kB 00:00 2024-09-16T10:06:54.962 INFO:teuthology.orchestra.run.smithi165.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-16T10:06:54.996 INFO:teuthology.orchestra.run.smithi165.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-16T10:06:55.021 INFO:teuthology.orchestra.run.smithi165.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.7 MB/s | 220 kB 00:00 2024-09-16T10:06:55.057 INFO:teuthology.orchestra.run.smithi165.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-16T10:06:55.058 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:55.058 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-09-16T10:06:55.125 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:06:55.140 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:06:55.140 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:06:55.232 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:06:55.233 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:06:55.591 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 129 kB/s | 222 kB 00:01 2024-09-16T10:06:55.736 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:06:55.913 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T10:06:55.972 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T10:06:56.028 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T10:06:56.074 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T10:06:56.115 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T10:06:56.218 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T10:06:56.337 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T10:06:56.384 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T10:06:56.441 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 96 kB/s | 159 kB 00:01 2024-09-16T10:06:56.491 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 651 kB/s | 32 kB 00:00 2024-09-16T10:06:56.517 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 564 kB/s | 14 kB 00:00 2024-09-16T10:06:56.550 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-16T10:06:56.609 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 18 MB/s | 1.0 MB 00:00 2024-09-16T10:06:56.634 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T10:06:56.659 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-16T10:06:56.684 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): python3-pyasn1-modules-0.4.8-6.el9.noa 157 kB/s | 279 kB 00:01 2024-09-16T10:06:56.851 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 43 kB/s | 54 kB 00:01 2024-09-16T10:06:56.853 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:06:56.854 INFO:teuthology.orchestra.run.smithi063.stdout:Total 636 kB/s | 2.2 MB 00:03 2024-09-16T10:06:56.928 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:06:56.946 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:06:56.946 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:06:57.045 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:06:57.045 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:06:57.127 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T10:06:57.190 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T10:06:57.244 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T10:06:57.289 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T10:06:57.319 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 13/13 2024-09-16T10:06:57.576 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:06:57.756 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T10:06:57.821 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T10:06:57.884 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T10:06:57.920 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T10:06:57.962 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T10:06:58.068 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 13/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 1/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T10:06:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T10:06:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T10:06:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T10:06:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T10:06:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T10:06:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T10:06:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T10:06:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T10:06:58.439 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T10:06:58.440 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T10:06:58.441 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T10:06:58.441 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:58.441 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:06:58.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T10:06:59.009 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T10:06:59.069 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T10:06:59.102 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T10:06:59.151 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T10:06:59.180 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 13/13 2024-09-16T10:06:59.287 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:06:59.423 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:06:59.423 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:06:59.424 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:59.424 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:06:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 154 k 2024-09-16T10:06:59.424 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:06:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T10:06:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T10:06:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T10:06:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T10:06:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T10:06:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:59.425 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:06:59.425 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:06:59.426 INFO:teuthology.orchestra.run.smithi165.stdout:Install 6 Packages 2024-09-16T10:06:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:06:59.427 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 7.0 M 2024-09-16T10:06:59.427 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 33 M 2024-09-16T10:06:59.427 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:06:59.933 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 13/13 2024-09-16T10:06:59.933 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 1/13 2024-09-16T10:06:59.933 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T10:06:59.934 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T10:07:00.100 INFO:teuthology.orchestra.run.smithi165.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4882.g66e33d6a.e 705 kB/s | 154 kB 00:00 2024-09-16T10:07:00.167 INFO:teuthology.orchestra.run.smithi165.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-16T10:07:00.192 INFO:teuthology.orchestra.run.smithi165.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T10:07:00.226 INFO:teuthology.orchestra.run.smithi165.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T10:07:00.269 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:00.270 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:00.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T10:07:01.121 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:01.262 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 154 k 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T10:07:01.263 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T10:07:01.264 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T10:07:01.264 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T10:07:01.264 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:01.264 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:01.264 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:01.264 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-09-16T10:07:01.264 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:01.265 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 7.0 M 2024-09-16T10:07:01.265 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 33 M 2024-09-16T10:07:01.265 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:01.985 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4882.g66e33d6a.e 744 kB/s | 154 kB 00:00 2024-09-16T10:07:02.052 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-16T10:07:02.069 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-16T10:07:02.094 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T10:07:02.101 INFO:teuthology.orchestra.run.smithi165.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 112 kB/s | 248 kB 00:02 2024-09-16T10:07:02.461 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 363 kB/s | 248 kB 00:00 2024-09-16T10:07:03.153 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.3 MB/s | 6.0 MB 00:01 2024-09-16T10:07:03.155 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:03.155 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.7 MB/s | 7.0 MB 00:01 2024-09-16T10:07:03.159 INFO:teuthology.orchestra.run.smithi165.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.8 MB/s | 6.0 MB 00:03 2024-09-16T10:07:03.161 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:03.162 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.9 MB/s | 7.0 MB 00:03 2024-09-16T10:07:03.232 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:03.235 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:03.290 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:03.290 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:03.290 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:03.290 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:03.349 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:03.349 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:03.352 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:03.352 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:03.671 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:03.841 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T10:07:03.843 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T10:07:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T10:07:03.956 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T10:07:04.439 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T10:07:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T10:07:04.530 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T10:07:04.599 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T10:07:04.611 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T10:07:04.624 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 6/6 2024-09-16T10:07:04.687 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T10:07:04.717 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 6/6 2024-09-16T10:07:05.445 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 6/6 2024-09-16T10:07:05.445 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/6 2024-09-16T10:07:05.445 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T10:07:05.445 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T10:07:05.445 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T10:07:05.445 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T10:07:05.678 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 6/6 2024-09-16T10:07:05.679 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 1/6 2024-09-16T10:07:05.679 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T10:07:05.679 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T10:07:05.679 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T10:07:05.679 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T10:07:05.719 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T10:07:05.720 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:05.720 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T10:07:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:05.964 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:07:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T10:07:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T10:07:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T10:07:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T10:07:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T10:07:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:05.965 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:05.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-09-16T10:07:06.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-fuse 2024-09-16T10:07:06.562 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:06.702 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:06.702 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 900 k 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-09-16T10:07:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:06.704 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 980 k 2024-09-16T10:07:06.704 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.9 M 2024-09-16T10:07:06.704 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:06.822 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:06.896 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 566 kB/s | 80 kB 00:00 2024-09-16T10:07:06.957 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 900 k 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T10:07:06.958 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:06.959 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:07:06.959 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:06.959 INFO:teuthology.orchestra.run.smithi165.stdout:Install 2 Packages 2024-09-16T10:07:06.959 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:06.959 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 980 k 2024-09-16T10:07:06.959 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 2.9 M 2024-09-16T10:07:06.960 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:07:07.029 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-19.3.0-4882.g66e33d6a.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-16T10:07:07.030 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:07.030 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.9 MB/s | 980 kB 00:00 2024-09-16T10:07:07.042 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:07.065 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:07.065 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:07.150 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:07.150 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:07.271 INFO:teuthology.orchestra.run.smithi165.stdout:(1/2): ceph-fuse-19.3.0-4882.g66e33d6a.el9.x86_ 3.3 MB/s | 900 kB 00:00 2024-09-16T10:07:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:07.506 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T10:07:07.582 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/2 2024-09-16T10:07:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/2 2024-09-16T10:07:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/2 2024-09-16T10:07:08.544 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T10:07:08.544 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:08.544 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:08.544 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T10:07:08.544 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:08.544 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:08.755 INFO:teuthology.orchestra.run.smithi165.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 46 kB/s | 80 kB 00:01 2024-09-16T10:07:08.756 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:08.756 INFO:teuthology.orchestra.run.smithi165.stdout:Total 545 kB/s | 980 kB 00:01 2024-09-16T10:07:08.770 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:08.796 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:08.796 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:08.800 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-volume 2024-09-16T10:07:08.884 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:08.884 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:09.084 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:09.236 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T10:07:09.313 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/2 2024-09-16T10:07:09.403 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:09.543 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:09.544 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-16T10:07:09.544 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:09.544 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 285 k 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 285 k 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.3 M 2024-09-16T10:07:09.545 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:09.750 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-volume-19.3.0-4882.g66e33d6a.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-16T10:07:09.751 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:09.751 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-16T10:07:09.751 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:09.760 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:09.761 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:09.793 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:09.987 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2/2 2024-09-16T10:07:09.987 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/2 2024-09-16T10:07:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:10.094 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:07:10.111 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:07:10.111 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:07:10.111 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T10:07:10.111 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T10:07:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:10.272 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T10:07:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:10.272 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:10.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-volume 2024-09-16T10:07:11.141 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:11.156 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:07:11.156 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:11.156 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:11.156 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:07:11.156 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:11.156 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:11.278 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:19.3.0-4882.g66e33d6a.el9 ceph-noarch 285 k 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:11.279 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:07:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:11.280 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 285 k 2024-09-16T10:07:11.280 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 1.3 M 2024-09-16T10:07:11.280 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:07:11.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-09-16T10:07:11.510 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-volume-19.3.0-4882.g66e33d6a.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T10:07:11.510 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:11.511 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T10:07:11.511 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:11.519 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:11.519 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:11.545 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:11.546 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:11.845 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:07:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:07:11.862 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:07:11.862 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T10:07:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:12.073 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:12.212 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 127 k 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 127 k 2024-09-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 456 k 2024-09-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:12.405 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-19.3.0-4882.g66e33d6a.el9.x86_64 663 kB/s | 127 kB 00:00 2024-09-16T10:07:12.405 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:12.405 INFO:teuthology.orchestra.run.smithi063.stdout:Total 658 kB/s | 127 kB 00:00 2024-09-16T10:07:12.405 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:12.411 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:12.411 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:12.480 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:12.480 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:12.647 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:12.748 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:12.897 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 1/1 2024-09-16T10:07:12.897 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:12.897 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:12.897 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:19.3.0-4882.g66e33d6a.el9.noarch 2024-09-16T10:07:12.897 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:12.897 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:13.180 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:13.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados-devel 2024-09-16T10:07:13.436 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:13.436 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:13.436 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:13.436 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:13.436 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:13.436 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:13.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-09-16T10:07:13.796 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:13.936 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 127 k 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:07:13.937 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:13.938 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 127 k 2024-09-16T10:07:13.938 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 456 k 2024-09-16T10:07:13.938 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:07:14.132 INFO:teuthology.orchestra.run.smithi165.stdout:librados-devel-19.3.0-4882.g66e33d6a.el9.x86_64 650 kB/s | 127 kB 00:00 2024-09-16T10:07:14.133 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:14.133 INFO:teuthology.orchestra.run.smithi165.stdout:Total 645 kB/s | 127 kB 00:00 2024-09-16T10:07:14.133 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:14.139 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:14.139 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:14.203 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:14.203 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:14.362 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:14.457 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:14.503 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:14.504 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:07:14.504 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:14.577 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-09-16T10:07:14.933 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:15.172 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:15.191 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:15.191 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:15.191 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:15.191 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:15.191 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:15.191 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:15.313 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:15.314 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-16T10:07:15.314 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:15.314 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 32 k 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 32 k 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 142 k 2024-09-16T10:07:15.315 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:15.459 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-19.3.0-4882.g66e33d6a.el9.x86_6 222 kB/s | 32 kB 00:00 2024-09-16T10:07:15.459 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:15.460 INFO:teuthology.orchestra.run.smithi063.stdout:Total 219 kB/s | 32 kB 00:00 2024-09-16T10:07:15.460 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:15.463 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:15.463 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:15.479 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:15.479 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:15.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs2 2024-09-16T10:07:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:16.087 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:16.178 INFO:teuthology.orchestra.run.smithi165.stdout:Package libcephfs2-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:16.223 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:16.224 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:07:16.224 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:16.291 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:16.291 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:16.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs-devel 2024-09-16T10:07:16.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-09-16T10:07:16.904 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:17.050 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:17.050 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 32 k 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 32 k 2024-09-16T10:07:17.051 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 142 k 2024-09-16T10:07:17.052 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:07:17.090 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:17.183 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:17.197 INFO:teuthology.orchestra.run.smithi165.stdout:libcephfs-devel-19.3.0-4882.g66e33d6a.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-16T10:07:17.198 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:17.198 INFO:teuthology.orchestra.run.smithi165.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-16T10:07:17.199 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:17.202 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:17.202 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:17.221 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:17.222 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:17.228 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:17.229 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:07:17.229 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:17.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-09-16T10:07:17.327 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:17.463 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:17.867 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:17.887 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:17.981 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:18.027 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:18.027 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:07:18.027 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:18.096 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-09-16T10:07:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:18.111 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:18.111 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:18.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados2 2024-09-16T10:07:18.687 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:18.781 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:18.829 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:18.830 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:07:18.830 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:18.906 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:18.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-09-16T10:07:18.998 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:19.042 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:19.043 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:07:19.043 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:19.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librbd1 2024-09-16T10:07:19.505 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:19.598 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:19.643 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:19.645 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:07:19.645 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:19.707 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:19.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-09-16T10:07:19.797 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:19.841 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:19.842 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:07:19.842 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:19.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rados 2024-09-16T10:07:20.319 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:20.413 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:20.459 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:20.461 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:07:20.461 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:20.494 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:54 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:20.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-09-16T10:07:20.584 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rados-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:20.628 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:20.629 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:07:20.630 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:20.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rgw 2024-09-16T10:07:21.137 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:21.230 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:21.276 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:21.277 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-16T10:07:21.277 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:21.281 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:55 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:21.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-09-16T10:07:21.372 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rgw-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:21.416 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:21.417 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:07:21.417 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:21.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-cephfs 2024-09-16T10:07:21.942 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:22.073 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:22.081 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:22.082 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-16T10:07:22.082 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:22.082 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 85 k 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 85 k 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 231 k 2024-09-16T10:07:22.083 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:22.163 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-cephfs-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:22.207 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:22.208 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:07:22.209 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:22.278 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-19.3.0-4882.g66e33d6a.el9.x86_64.rpm 436 kB/s | 85 kB 00:00 2024-09-16T10:07:22.279 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:22.279 INFO:teuthology.orchestra.run.smithi063.stdout:Total 433 kB/s | 85 kB 00:00 2024-09-16T10:07:22.279 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:22.287 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:22.287 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:22.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rbd 2024-09-16T10:07:22.355 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:22.355 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:22.507 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:22.866 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:22.957 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 is already installed. 2024-09-16T10:07:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:23.001 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:23.002 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-16T10:07:23.002 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:23.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-fuse 2024-09-16T10:07:23.239 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:23.239 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:23.239 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:23.239 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:23.240 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:23.240 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:23.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-09-16T10:07:23.655 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:57 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:23.789 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:23.789 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 85 k 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 85 k 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 231 k 2024-09-16T10:07:23.790 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:07:24.000 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-fuse-19.3.0-4882.g66e33d6a.el9.x86_64.rpm 406 kB/s | 85 kB 00:00 2024-09-16T10:07:24.001 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:24.001 INFO:teuthology.orchestra.run.smithi165.stdout:Total 403 kB/s | 85 kB 00:00 2024-09-16T10:07:24.001 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:24.008 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:24.008 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:24.072 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:24.073 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:24.083 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:24.223 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:24.237 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:24.238 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-16T10:07:24.238 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:24.238 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 3.2 M 2024-09-16T10:07:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:07:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.2 M 2024-09-16T10:07:24.239 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 14 M 2024-09-16T10:07:24.240 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:24.314 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:24.573 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-19.3.0-4882.g66e33d6a.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-16T10:07:24.573 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:24.574 INFO:teuthology.orchestra.run.smithi063.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-16T10:07:24.574 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:24.584 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:24.584 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:24.672 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:24.673 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:24.708 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:24.969 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:24.969 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:24.969 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:24.970 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:24.970 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:24.970 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:25.187 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:25.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-mirror 2024-09-16T10:07:25.249 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:25.276 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:07:25.276 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T10:07:25.276 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:07:25.276 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:07:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:25.795 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:59 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:25.933 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:25.933 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 3.2 M 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:07:25.934 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:25.935 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.2 M 2024-09-16T10:07:25.935 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 14 M 2024-09-16T10:07:25.935 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:07:26.278 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-mirror-19.3.0-4882.g66e33d6a.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-16T10:07:26.279 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:26.279 INFO:teuthology.orchestra.run.smithi165.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-16T10:07:26.279 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:26.289 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:26.289 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:26.367 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:26.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:26.367 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:26.367 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:26.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:26.367 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:26.372 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:26.373 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:26.567 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-09-16T10:07:26.892 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:26.954 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:26.980 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:07:26.980 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T10:07:26.980 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:07:26.980 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:07:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:27.168 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:55 ago on Mon 16 Sep 2024 10:05:32 AM UTC. 2024-09-16T10:07:27.309 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 171 k 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-16T10:07:27.310 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-16T10:07:27.311 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-16T10:07:27.311 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:27.311 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 171 k 2024-09-16T10:07:27.311 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 490 k 2024-09-16T10:07:27.311 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-16T10:07:27.545 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-19.3.0-4882.g66e33d6a.el9.x86_64.rpm 730 kB/s | 171 kB 00:00 2024-09-16T10:07:27.546 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:27.546 INFO:teuthology.orchestra.run.smithi063.stdout:Total 725 kB/s | 171 kB 00:00 2024-09-16T10:07:27.546 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-16T10:07:27.554 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-16T10:07:27.555 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-16T10:07:27.623 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-16T10:07:27.624 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-16T10:07:27.796 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-16T10:07:27.909 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:28.110 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:28.110 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:28.110 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:28.110 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:28.110 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:28.110 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:28.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-nbd 2024-09-16T10:07:28.361 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:28.615 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-16T10:07:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:07:28.616 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-16T10:07:28.830 DEBUG:teuthology.parallel:result is None 2024-09-16T10:07:28.934 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:02:02 ago on Mon 16 Sep 2024 10:05:26 AM UTC. 2024-09-16T10:07:29.072 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-16T10:07:29.072 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:29.072 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-16T10:07:29.072 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:29.072 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-16T10:07:29.073 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:19.3.0-4882.g66e33d6a.el9 ceph 171 k 2024-09-16T10:07:29.073 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:29.073 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-16T10:07:29.073 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-16T10:07:29.073 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-16T10:07:29.073 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:29.073 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 171 k 2024-09-16T10:07:29.074 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 490 k 2024-09-16T10:07:29.074 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-16T10:07:29.281 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-nbd-19.3.0-4882.g66e33d6a.el9.x86_64.rpm 822 kB/s | 171 kB 00:00 2024-09-16T10:07:29.282 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:07:29.282 INFO:teuthology.orchestra.run.smithi165.stdout:Total 816 kB/s | 171 kB 00:00 2024-09-16T10:07:29.282 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-16T10:07:29.290 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-16T10:07:29.290 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-16T10:07:29.354 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-16T10:07:29.354 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-16T10:07:29.517 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-16T10:07:29.621 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:30.344 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 1/1 2024-09-16T10:07:30.344 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:30.344 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-16T10:07:30.344 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:19.3.0-4882.g66e33d6a.el9.x86_64 2024-09-16T10:07:30.344 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:07:30.344 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-16T10:07:30.621 DEBUG:teuthology.parallel:result is None 2024-09-16T10:07:30.621 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:07:30.768 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T10:07:30.795 INFO:teuthology.orchestra.run.smithi063.stdout:19.3.0-4882.g66e33d6a.el9 2024-09-16T10:07:30.795 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4882.g66e33d6a.el9 2024-09-16T10:07:30.796 INFO:teuthology.task.install:The correct ceph version 19.3.0-4882.g66e33d6a is installed. 2024-09-16T10:07:30.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:07:30.940 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T10:07:30.962 INFO:teuthology.orchestra.run.smithi165.stdout:19.3.0-4882.g66e33d6a.el9 2024-09-16T10:07:30.962 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4882.g66e33d6a.el9 2024-09-16T10:07:30.962 INFO:teuthology.task.install:The correct ceph version 19.3.0-4882.g66e33d6a is installed. 2024-09-16T10:07:30.965 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T10:07:30.965 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:07:30.965 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:07:30.996 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:07:30.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:07:31.028 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T10:07:31.029 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:07:31.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:07:31.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:07:31.138 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:07:31.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:07:31.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:07:31.229 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T10:07:31.230 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:07:31.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:07:31.259 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:07:31.327 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:07:31.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:07:31.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:07:31.431 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T10:07:31.432 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:07:31.432 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:07:31.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:07:31.527 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:07:31.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:07:31.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:07:31.616 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-16T10:07:31.721 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad', 'cephadm_mode': 'root'} 2024-09-16T10:07:31.721 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:07:31.722 INFO:tasks.cephadm:Cluster fsid is 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:07:31.722 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-16T10:07:31.722 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.63:6789]', 'mon.c': '[v1:172.21.15.63:6790]', 'mon.b': '[v1:172.21.15.165:6789]'} 2024-09-16T10:07:31.722 INFO:tasks.cephadm:First mon is mon.a on smithi063 2024-09-16T10:07:31.722 INFO:tasks.cephadm:First mgr is y 2024-09-16T10:07:31.722 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-16T10:07:31.722 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-09-16T10:07:31.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-09-16T10:07:31.775 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-16T10:07:31.775 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:07:31.927 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83025/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4882-g66e33d6a', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4882.g66e33d6a'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-12 19:44:45.618408', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-athakkar-testing-2024-09-12-1857', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-16T10:07:32.078 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-athakkar-testing-2024-09-12-1857, sha1 66e33d6af1496291b0602ee0a859e68145f17dad from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:07:32.080 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/x86_64/flavors/default/cephadm 2024-09-16T10:07:32.080 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/x86_64/flavors/default/cephadm 2024-09-16T10:07:32.080 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T10:07:32.455 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-r--r--. 1 ubuntu ubuntu 804193 Sep 16 10:07 /home/ubuntu/cephtest/cephadm 2024-09-16T10:07:32.456 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-athakkar-testing-2024-09-12-1857/66e33d6af1496291b0602ee0a859e68145f17dad/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T10:07:32.840 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 804193 Sep 16 10:07 /home/ubuntu/cephtest/cephadm 2024-09-16T10:07:32.841 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T10:07:32.862 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T10:07:32.892 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad on all hosts... 2024-09-16T10:07:32.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad pull 2024-09-16T10:07:32.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad pull 2024-09-16T10:07:34.485 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad... 2024-09-16T10:07:34.486 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad... 2024-09-16T10:08:12.218 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:08:12.218 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 19.3.0-4882-g66e33d6a (66e33d6af1496291b0602ee0a859e68145f17dad) squid (dev)", 2024-09-16T10:08:12.218 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e", 2024-09-16T10:08:12.218 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-09-16T10:08:12.218 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f" 2024-09-16T10:08:12.219 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:08:12.219 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:08:12.473 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:08:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 19.3.0-4882-g66e33d6a (66e33d6af1496291b0602ee0a859e68145f17dad) squid (dev)", 2024-09-16T10:08:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e", 2024-09-16T10:08:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-09-16T10:08:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f" 2024-09-16T10:08:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:08:12.474 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:08:12.503 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-09-16T10:08:12.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-09-16T10:08:12.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-09-16T10:08:12.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-09-16T10:08:12.631 INFO:tasks.cephadm:Writing seed config... 2024-09-16T10:08:12.633 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-16T10:08:12.633 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-16T10:08:12.633 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-16T10:08:12.633 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-16T10:08:12.633 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-16T10:08:12.633 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-16T10:08:12.634 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-16T10:08:12.635 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-16T10:08:12.636 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-16T10:08:12.637 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:08:12.637 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-16T10:08:12.658 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7cb51a9a-7413-11ef-bceb-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-16T10:08:12.658 DEBUG:teuthology.orchestra.run.smithi063:mon.a> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a.service 2024-09-16T10:08:12.701 DEBUG:teuthology.orchestra.run.smithi063:mgr.y> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.y.service 2024-09-16T10:08:12.744 INFO:tasks.cephadm:Bootstrapping... 2024-09-16T10:08:12.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad -v bootstrap --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.63:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-16T10:08:12.956 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:08:12.956 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad', '-v', 'bootstrap', '--fsid', '7cb51a9a-7413-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.63:6789]', '--skip-admin-label'] 2024-09-16T10:08:12.957 INFO:teuthology.orchestra.run.smithi063.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-16T10:08:12.957 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying podman|docker is present... 2024-09-16T10:08:12.980 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 5.2.2 2024-09-16T10:08:12.980 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying lvm2 is present... 2024-09-16T10:08:12.980 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying time synchronization is in place... 2024-09-16T10:08:12.988 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T10:08:12.988 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T10:08:12.994 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T10:08:12.994 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-09-16T10:08:13.002 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-09-16T10:08:13.008 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-09-16T10:08:13.008 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-09-16T10:08:13.009 INFO:teuthology.orchestra.run.smithi063.stdout:Repeating the final host check... 2024-09-16T10:08:13.031 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 5.2.2 2024-09-16T10:08:13.031 INFO:teuthology.orchestra.run.smithi063.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-16T10:08:13.031 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl is present 2024-09-16T10:08:13.031 INFO:teuthology.orchestra.run.smithi063.stdout:lvcreate is present 2024-09-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T10:08:13.044 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T10:08:13.045 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-09-16T10:08:13.051 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:Host looks OK 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:Cluster fsid: 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:Acquiring lock 139666644250240 on /run/cephadm/7cb51a9a-7413-11ef-bceb-c7b262605968.lock 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:Lock 139666644250240 acquired on /run/cephadm/7cb51a9a-7413-11ef-bceb-c7b262605968.lock 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying IP 172.21.15.63 port 6789 ... 2024-09-16T10:08:13.059 INFO:teuthology.orchestra.run.smithi063.stdout:Base mon IP(s) is [172.21.15.63:6789], mon addrv is [v1:172.21.15.63:6789] 2024-09-16T10:08:13.062 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.63 metric 100 2024-09-16T10:08:13.062 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.63 metric 100 2024-09-16T10:08:13.065 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-16T10:08:13.065 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-16T10:08:13.065 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-16T10:08:13.067 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-16T10:08:13.068 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-16T10:08:13.068 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T10:08:13.068 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-16T10:08:13.068 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8d7/64 scope link noprefixroute 2024-09-16T10:08:13.068 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T10:08:13.068 INFO:teuthology.orchestra.run.smithi063.stdout:Mon IP `172.21.15.63` is in CIDR network `172.21.0.0/20` 2024-09-16T10:08:13.069 INFO:teuthology.orchestra.run.smithi063.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-09-16T10:08:13.069 INFO:teuthology.orchestra.run.smithi063.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-16T10:08:13.069 INFO:teuthology.orchestra.run.smithi063.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad... 2024-09-16T10:08:13.556 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e 2024-09-16T10:08:13.556 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad... 2024-09-16T10:08:13.556 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Getting image source signatures 2024-09-16T10:08:13.556 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:b030955614448665a7d6afdb71ea500fa5c71d8f06361a0bc26d4302d02cd82d 2024-09-16T10:08:13.557 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-16T10:08:13.557 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying config sha256:261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e 2024-09-16T10:08:13.557 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-16T10:08:14.452 INFO:teuthology.orchestra.run.smithi063.stdout:ceph: stdout ceph version 19.3.0-4882-g66e33d6a (66e33d6af1496291b0602ee0a859e68145f17dad) squid (dev) 2024-09-16T10:08:14.452 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph version: ceph version 19.3.0-4882-g66e33d6a (66e33d6af1496291b0602ee0a859e68145f17dad) squid (dev) 2024-09-16T10:08:14.453 INFO:teuthology.orchestra.run.smithi063.stdout:Extracting ceph user uid/gid from container image... 2024-09-16T10:08:15.283 INFO:teuthology.orchestra.run.smithi063.stdout:stat: stdout 167 167 2024-09-16T10:08:15.283 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial keys... 2024-09-16T10:08:16.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCPA+hm2fIoIxAAcuShTSUiT3LCTtHpcETeSg== 2024-09-16T10:08:17.054 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCQA+hmzl6kHBAAdx4KMrQhGuwvPErM/4DYgg== 2024-09-16T10:08:17.935 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCRA+hmh4IlFBAA5llQV0R7slM/P6/QArlsPA== 2024-09-16T10:08:17.935 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial monmap... 2024-09-16T10:08:18.766 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T10:08:18.766 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool for a [v1:172.21.15.63:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = quincy 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: set fsid to 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:08:18.767 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mon... 2024-09-16T10:08:19.697 INFO:teuthology.orchestra.run.smithi063.stdout:create mon.a on 2024-09-16T10:08:19.904 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-16T10:08:20.050 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-16T10:08:20.222 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968.target. 2024-09-16T10:08:20.222 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968.target. 2024-09-16T10:08:20.450 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a 2024-09-16T10:08:20.450 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-16T10:08:20.621 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968.target.wants/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@.service. 2024-09-16T10:08:21.108 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:21 smithi063 ceph-mon[29626]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T10:08:21.108 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:21 smithi063 bash[29607]: 0e293b98998b68a2e8c8b1bd5ce4c1b806b24797d603c90dbe27b5343385cad3 2024-09-16T10:08:21.123 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T10:08:21.123 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-09-16T10:08:21.130 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T10:08:21.130 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-09-16T10:08:21.130 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-09-16T10:08:21.130 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T10:08:21.130 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon to start... 2024-09-16T10:08:21.130 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon... 2024-09-16T10:08:21.428 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:21 smithi063 systemd[1]: Started Ceph mon.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:08:22.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout cluster: 2024-09-16T10:08:22.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout id: 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:22.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout services: 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.495031s) 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout data: 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pgs: 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:mon is available 2024-09-16T10:08:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:Assimilating anything we can from ceph.conf... 2024-09-16T10:08:22.376 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: monmap epoch 1 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: last_changed 2024-09-16T10:08:18.256111+0000 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: created 2024-09-16T10:08:18.256111+0000 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: min_mon_release 19 (squid) 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: election_strategy: 1 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: 0: v1:172.21.15.63:6789/0 mon.a 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: fsmap 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: mgrmap e1: no daemons active 2024-09-16T10:08:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:22 smithi063 ceph-mon[29626]: from='client.? v1:172.21.15.63:0/1293982177' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-16T10:08:23.051 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:23.051 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T10:08:23.051 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:23.051 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-16T10:08:23.051 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.63:6789] 2024-09-16T10:08:23.051 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T10:08:23.052 INFO:teuthology.orchestra.run.smithi063.stdout:Generating new minimal ceph.conf... 2024-09-16T10:08:23.305 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:23 smithi063 ceph-mon[29626]: from='client.? v1:172.21.15.63:0/2302618533' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-16T10:08:23.305 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:23 smithi063 ceph-mon[29626]: from='client.? v1:172.21.15.63:0/2302618533' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-16T10:08:24.032 INFO:teuthology.orchestra.run.smithi063.stdout:Restarting the monitor... 2024-09-16T10:08:24.316 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:24 smithi063 systemd[1]: Stopping Ceph mon.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:08:24.316 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:24 smithi063 ceph-mon[29626]: from='client.? v1:172.21.15.63:0/1278002855' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:08:24.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:24 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29622]: 2024-09-16T10:08:24.313+0000 7fce96d4e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:08:24.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:24 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29622]: 2024-09-16T10:08:24.313+0000 7fce96d4e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-16T10:08:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:24 smithi063 podman[29870]: 2024-09-16 10:08:24.518695512 +0000 UTC m=+0.314019464 container died 0e293b98998b68a2e8c8b1bd5ce4c1b806b24797d603c90dbe27b5343385cad3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-09-16T10:08:25.024 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:24 smithi063 podman[29870]: 2024-09-16 10:08:24.78034579 +0000 UTC m=+0.575669795 container cleanup 0e293b98998b68a2e8c8b1bd5ce4c1b806b24797d603c90dbe27b5343385cad3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T10:08:25.024 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:24 smithi063 bash[29870]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a 2024-09-16T10:08:25.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 podman[29884]: 2024-09-16 10:08:25.022864522 +0000 UTC m=+0.502509308 container remove 0e293b98998b68a2e8c8b1bd5ce4c1b806b24797d603c90dbe27b5343385cad3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T10:08:25.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-16T10:08:25.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 systemd[1]: Stopped Ceph mon.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:08:25.359 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 systemd[1]: Starting Ceph mon.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:08:25.608 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 podman[29975]: 2024-09-16 10:08:25.35687561 +0000 UTC m=+0.103796051 container create c4c048569e6bdc80f3a9f58f75d86c89dc5d61095591bacc7a4ae2b89916c382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6) 2024-09-16T10:08:25.608 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 podman[29975]: 2024-09-16 10:08:25.26981219 +0000 UTC m=+0.016732641 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 podman[29975]: 2024-09-16 10:08:25.576274221 +0000 UTC m=+0.323194663 container init c4c048569e6bdc80f3a9f58f75d86c89dc5d61095591bacc7a4ae2b89916c382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6) 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 podman[29975]: 2024-09-16 10:08:25.579408395 +0000 UTC m=+0.326328840 container start c4c048569e6bdc80f3a9f58f75d86c89dc5d61095591bacc7a4ae2b89916c382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: ceph version 19.3.0-4882-g66e33d6a (66e33d6af1496291b0602ee0a859e68145f17dad) squid (dev), process ceph-mon, pid 2 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: pidfile_write: ignore empty --pid-file 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: load: jerasure load: lrc 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: RocksDB version: 7.9.2 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Git sha 0 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Compile date 2024-09-12 18:19:39 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: DB SUMMARY 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: DB Session ID: BLEFRIGXGKWX8H8D3LMO 2024-09-16T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: CURRENT file: CURRENT 2024-09-16T10:08:25.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T10:08:25.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-16T10:08:25.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-16T10:08:25.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92894 ; 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.error_if_exists: 0 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.create_if_missing: 0 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.paranoid_checks: 1 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.env: 0x5622d7df8120 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.info_log: 0x5622da0956a0 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.statistics: (nil) 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.use_fsync: 0 2024-09-16T10:08:25.611 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_log_file_size: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.allow_fallocate: 1 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.use_direct_reads: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T10:08:25.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.db_log_dir: 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.wal_dir: 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T10:08:25.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.write_buffer_manager: 0x5622da099b80 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.unordered_write: 0 2024-09-16T10:08:25.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.row_cache: None 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.wal_filter: None 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.two_write_queues: 0 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.wal_compression: 0 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.atomic_flush: 0 2024-09-16T10:08:25.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.log_readahead_size: 0 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T10:08:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_background_jobs: 2 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_background_compactions: -1 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_subcompactions: 1 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_open_files: -1 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T10:08:25.617 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_background_flushes: -1 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Compression algorithms supported: 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kZSTD supported: 0 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kXpressCompression supported: 0 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kBZip2Compression supported: 0 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kLZ4Compression supported: 1 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kZlibCompression supported: 1 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: kSnappyCompression supported: 1 2024-09-16T10:08:25.618 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.merge_operator: 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_filter: None 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5622da0952c0) 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout: cache_index_and_filter_blocks: 1 2024-09-16T10:08:25.619 INFO:journalctl@ceph.mon.a.smithi063.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: pin_top_level_index_and_filter: 1 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: index_type: 0 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: data_block_index_type: 0 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: index_shortening: 1 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: checksum: 4 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: no_block_cache: 0 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache: 0x5622da089610 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache_name: BinnedLRUCache 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache_options: 2024-09-16T10:08:25.620 INFO:journalctl@ceph.mon.a.smithi063.stdout: capacity : 536870912 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: num_shard_bits : 4 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: strict_capacity_limit : 0 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: high_pri_pool_ratio: 0.000 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache_compressed: (nil) 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: persistent_cache: (nil) 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_size: 4096 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_size_deviation: 10 2024-09-16T10:08:25.621 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_restart_interval: 16 2024-09-16T10:08:25.622 INFO:journalctl@ceph.mon.a.smithi063.stdout: index_block_restart_interval: 1 2024-09-16T10:08:25.622 INFO:journalctl@ceph.mon.a.smithi063.stdout: metadata_block_size: 4096 2024-09-16T10:08:25.622 INFO:journalctl@ceph.mon.a.smithi063.stdout: partition_filters: 0 2024-09-16T10:08:25.622 INFO:journalctl@ceph.mon.a.smithi063.stdout: use_delta_encoding: 1 2024-09-16T10:08:25.622 INFO:journalctl@ceph.mon.a.smithi063.stdout: filter_policy: bloomfilter 2024-09-16T10:08:25.622 INFO:journalctl@ceph.mon.a.smithi063.stdout: whole_key_filtering: 1 2024-09-16T10:08:25.622 INFO:journalctl@ceph.mon.a.smithi063.stdout: verify_compression: 0 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: read_amp_bytes_per_bit: 0 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: format_version: 5 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: enable_index_compression: 1 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_align: 0 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: max_auto_readahead_size: 262144 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: prepopulate_block_cache: 0 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: initial_auto_readahead_size: 8192 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression: NoCompression 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T10:08:25.623 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.num_levels: 7 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T10:08:25.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T10:08:25.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T10:08:25.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T10:08:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.inplace_update_support: 0 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T10:08:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.bloom_locality: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.max_successive_merges: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.ttl: 2592000 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.enable_blob_files: false 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.min_blob_size: 0 2024-09-16T10:08:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5e667d6b-1b91-40e6-afbe-54732faa781e 2024-09-16T10:08:25.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726481305606360, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-16T10:08:25.631 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-16T10:08:25.631 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726481305607392, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 287, "table_properties": {"data_size": 86936, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11809, "raw_average_key_size": 48, "raw_value_size": 80408, "raw_average_value_size": 326, "num_data_blocks": 11, "num_entries": 246, "num_filter_entries": 246, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726481305, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5e667d6b-1b91-40e6-afbe-54732faa781e", "db_session_id": "BLEFRIGXGKWX8H8D3LMO", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-16T10:08:25.631 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726481305607532, "job": 1, "event": "recovery_finished"} 2024-09-16T10:08:25.631 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-16T10:08:25.631 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T10:08:25.631 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5622da0b8e00 2024-09-16T10:08:25.631 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: DB pointer 0x5622da1ba000 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** DB Stats ** 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** Compaction Stats [default] ** 2024-09-16T10:08:25.632 INFO:journalctl@ceph.mon.a.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: L0 2/0 88.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: Sum 2/0 88.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** Compaction Stats [default] ** 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-09-16T10:08:25.633 INFO:journalctl@ceph.mon.a.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative compaction: 0.00 GB write, 22.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval compaction: 0.00 GB write, 22.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: Block cache BinnedLRUCache@0x5622da089610#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-09-16T10:08:25.634 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T10:08:25.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: starting mon.a rank 0 at public addrs v1:172.21.15.63:6789/0 at bind addrs v1:172.21.15.63:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:25.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???) e1 preinit fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:25.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???).mds e1 new map 2024-09-16T10:08:25.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???).mds e1 print_map 2024-09-16T10:08:25.635 INFO:journalctl@ceph.mon.a.smithi063.stdout: e1 2024-09-16T10:08:25.635 INFO:journalctl@ceph.mon.a.smithi063.stdout: btime 2024-09-16T10:08:21:081733+0000 2024-09-16T10:08:25.635 INFO:journalctl@ceph.mon.a.smithi063.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout: legacy client fscid: -1 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout: No filesystems configured 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T10:08:25.636 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-16T10:08:25.649 INFO:teuthology.orchestra.run.smithi063.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-16T10:08:25.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 bash[29975]: c4c048569e6bdc80f3a9f58f75d86c89dc5d61095591bacc7a4ae2b89916c382 2024-09-16T10:08:25.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 systemd[1]: Started Ceph mon.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:08:25.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T10:08:25.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: monmap epoch 1 2024-09-16T10:08:25.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:25.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: last_changed 2024-09-16T10:08:18.256111+0000 2024-09-16T10:08:25.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: created 2024-09-16T10:08:18.256111+0000 2024-09-16T10:08:25.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: min_mon_release 19 (squid) 2024-09-16T10:08:25.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: election_strategy: 1 2024-09-16T10:08:25.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: 0: v1:172.21.15.63:6789/0 mon.a 2024-09-16T10:08:25.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: fsmap 2024-09-16T10:08:25.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T10:08:25.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:25 smithi063 ceph-mon[29993]: mgrmap e1: no daemons active 2024-09-16T10:08:26.637 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-16T10:08:26.661 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-16T10:08:26.661 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mgr... 2024-09-16T10:08:26.661 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-16T10:08:26.661 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-16T10:08:26.912 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.y 2024-09-16T10:08:26.912 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-16T10:08:27.070 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968.target.wants/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@.service. 2024-09-16T10:08:27.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3437405009' entity='client.admin' 2024-09-16T10:08:27.590 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:27 smithi063 podman[30203]: 2024-09-16 10:08:27.516847385 +0000 UTC m=+0.326854586 container start 7e145543eced72c9a35c52fb052a4700300b514fb6991e436df82e66e7c9a5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-16T10:08:27.591 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:27 smithi063 bash[30203]: 7e145543eced72c9a35c52fb052a4700300b514fb6991e436df82e66e7c9a5fa 2024-09-16T10:08:27.605 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T10:08:27.605 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-09-16T10:08:27.613 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T10:08:27.613 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-09-16T10:08:27.613 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-09-16T10:08:27.613 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T10:08:27.622 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T10:08:27.622 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-09-16T10:08:27.632 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T10:08:27.633 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-09-16T10:08:27.633 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-09-16T10:08:27.633 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-16T10:08:27.633 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr to start... 2024-09-16T10:08:27.633 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr... 2024-09-16T10:08:27.843 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:27 smithi063 systemd[1]: Started Ceph mgr.y for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:08:28.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1085360793' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-16T10:08:28.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T10:08:28.880 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T10:08:28.881 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T10:08:28.882 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T10:08:28.882 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T10:08:28.882 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T10:08:28.882 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T10:08:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T10:08:21:081733+0000", 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T10:08:28.886 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T10:08:21.082417+0000", 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-09-16T10:08:28.887 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (1/15)... 2024-09-16T10:08:31.692 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/473524723' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-16T10:08:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-09-16T10:08:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T10:08:32.017 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:32.018 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T10:08:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T10:08:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T10:08:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T10:08:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T10:08:32.021 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T10:08:21:081733+0000", 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:32.022 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T10:08:21.082417+0000", 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-09-16T10:08:32.024 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (2/15)... 2024-09-16T10:08:33.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: Activating manager daemon y 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: mgrmap e2: y(active, starting, since 0.00322514s) 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: Manager daemon y is now available 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' 2024-09-16T10:08:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:32 smithi063 ceph-mon[29993]: from='mgr.14100 v1:172.21.15.63:0/3792120892' entity='mgr.y' 2024-09-16T10:08:34.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:34 smithi063 ceph-mon[29993]: mgrmap e3: y(active, since 1.00749s) 2024-09-16T10:08:34.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3526157685' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-16T10:08:36.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:35 smithi063 ceph-mon[29993]: mgrmap e4: y(active, since 2s) 2024-09-16T10:08:39.813 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:39.813 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-09-16T10:08:39.813 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:08:39.813 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T10:08:39.813 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-16T10:08:39.814 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T10:08:39.815 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T10:08:39.816 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T10:08:39.819 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:39.819 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T10:08:39.819 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T10:08:39.819 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T10:08:21:081733+0000", 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T10:08:39.820 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T10:08:39.821 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T10:08:21.082417+0000", 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-09-16T10:08:39.822 INFO:teuthology.orchestra.run.smithi063.stdout:mgr is available 2024-09-16T10:08:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1114984109' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-16T10:08:45.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:45.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T10:08:45.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:45.052 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.63:6789] 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T10:08:45.053 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling cephadm module... 2024-09-16T10:08:46.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:45 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/446681093' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-16T10:08:46.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:45 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ignoring --setuser ceph since I am not root 2024-09-16T10:08:46.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:45 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ignoring --setgroup ceph since I am not root 2024-09-16T10:08:47.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/446681093' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-16T10:08:47.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:46 smithi063 ceph-mon[29993]: mgrmap e5: y(active, since 12s) 2024-09-16T10:08:47.447 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-09-16T10:08:47.447 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-16T10:08:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T10:08:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-16T10:08:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T10:08:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-09-16T10:08:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-09-16T10:08:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 5... 2024-09-16T10:08:47.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1129253765' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: Active manager daemon y restarted 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: Activating manager daemon y 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: osdmap e2: 0 total, 0 up, 0 in 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: mgrmap e6: y(active, starting, since 0.00344283s) 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T10:08:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T10:08:51.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T10:08:51.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:51 smithi063 ceph-mon[29993]: Manager daemon y is now available 2024-09-16T10:08:52.610 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-09-16T10:08:52.611 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-16T10:08:52.611 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T10:08:52.611 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-09-16T10:08:52.611 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 5 is available 2024-09-16T10:08:52.611 INFO:teuthology.orchestra.run.smithi063.stdout:Setting orchestrator backend to cephadm... 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: Found migration_current of "None". Setting to last migration. 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:08:52.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:08:52.912 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:08:52.912 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:52 smithi063 ceph-mon[29993]: mgrmap e7: y(active, since 1.00618s) 2024-09-16T10:08:53.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:08:53.071+0000 7f1b5f80e640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:10:08:53] ENGINE Error in HTTPServer.serve 2024-09-16T10:08:53.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: Traceback (most recent call last): 2024-09-16T10:08:53.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:08:53.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._connections.run(self.expiration_interval) 2024-09-16T10:08:53.203 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:08:53.203 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._run(expiration_interval) 2024-09-16T10:08:53.203 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:08:53.203 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:08:53.203 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:08:53.204 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:08:53.204 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:08:53.204 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: s = self.context.wrap_socket( 2024-09-16T10:08:53.204 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:08:53.204 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: return self.sslsocket_class._create( 2024-09-16T10:08:53.204 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:08:53.205 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self.do_handshake() 2024-09-16T10:08:53.205 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:08:53.205 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._sslobj.do_handshake() 2024-09-16T10:08:53.205 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:08:53.205 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:08:53.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-mon[29993]: from='client.14122 v1:172.21.15.63:0/316728917' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-16T10:08:53.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-mon[29993]: from='client.14122 v1:172.21.15.63:0/316728917' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-16T10:08:53.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:08:53.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:08:53.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:53 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:54 smithi063 ceph-mon[29993]: [16/Sep/2024:10:08:52] ENGINE Bus STARTING 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:54 smithi063 ceph-mon[29993]: [16/Sep/2024:10:08:52] ENGINE Serving on http://172.21.15.63:8765 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:54 smithi063 ceph-mon[29993]: [16/Sep/2024:10:08:53] ENGINE Serving on https://172.21.15.63:7150 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:54 smithi063 ceph-mon[29993]: [16/Sep/2024:10:08:53] ENGINE Bus STARTED 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:54 smithi063 ceph-mon[29993]: [16/Sep/2024:10:08:53] ENGINE Error in HTTPServer.serve 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout: Traceback (most recent call last): 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._connections.run(self.expiration_interval) 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:08:54.846 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._run(expiration_interval) 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: s = self.context.wrap_socket( 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: return self.sslsocket_class._create( 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: self.do_handshake() 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._sslobj.do_handshake() 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:54 smithi063 ceph-mon[29993]: from='client.14130 v1:172.21.15.63:0/811712358' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:08:54.847 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:54 smithi063 ceph-mon[29993]: mgrmap e8: y(active, since 2s) 2024-09-16T10:08:54.848 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-16T10:08:54.848 INFO:teuthology.orchestra.run.smithi063.stdout:Generating ssh key... 2024-09-16T10:08:55.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-mon[29993]: from='client.14132 v1:172.21.15.63:0/2310589112' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:08:55.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:08:55.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:08:55.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: Generating public/private rsa key pair. 2024-09-16T10:08:55.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: Your identification has been saved in /tmp/tmp40dyalru/key 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: Your public key has been saved in /tmp/tmp40dyalru/key.pub 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: The key fingerprint is: 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: SHA256:IvFGP6zqGxEde247CbJZxGX0oMwhUWdyz+w1K0pekBo ceph-7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: The key's randomart image is: 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: +---[RSA 3072]----+ 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | oo=.O | 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | * % B | 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | o E + = o | 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | * B o . o | 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | + B S + . | 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | O B * . | 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | + . * | 2024-09-16T10:08:55.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | o . | 2024-09-16T10:08:55.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: | .+. | 2024-09-16T10:08:55.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:08:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: +----[SHA256]-----+ 2024-09-16T10:08:56.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:56 smithi063 ceph-mon[29993]: from='client.14134 v1:172.21.15.63:0/2141862741' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:08:56.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:56 smithi063 ceph-mon[29993]: Generating ssh key... 2024-09-16T10:08:57.385 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC05iAEvuyd+CXNAnBV3ZHPpQF4TbJLtdOHQvfzT32qyrYp427/ubBGRnwsV8GLxun+AcJ6lMDhVotjDO2wEhQjP+E7vXj2lJe4Wr9f1kNO+h3MHqCTXUpd/9tifiw4jMM56YaGJs7wc5KKDWKLd95g3jIdIGQK14r10KuXSx1iUfVp8P6XnHhB0cORet22B0OEjIAkwRDqeL17gWXWXt/hZSFp0K60TF32OH+vO1lBmL1o1dmbOaPqUCNRCVOGwiGkcXw5gpMo27/1jsjxkx0IQUGcjMrcGXst4AbJSlRJJMlq74ihf4TI8VVbZ2nhdr7zJ9F1r0CzGLpBGyYIwbQEmqmB0vo+7ORupC5rp4OrJ+hI/m9A06xEytUS5TICmIzl1+MKvHN0CxvdtyQvZA/RzTn67cF1MmrALj5OBRkL25iiETbrkKkHs3XuRQukAgj/yCK8VWVSDsDs35pzdy/RwZGLDiombSSIfUW3aYRGcEBie33VtlWnCzOdSsUwfRs= ceph-7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:08:57.385 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-16T10:08:57.385 INFO:teuthology.orchestra.run.smithi063.stdout:Adding key to root@localhost authorized_keys... 2024-09-16T10:08:57.397 INFO:teuthology.orchestra.run.smithi063.stdout:Adding host smithi063... 2024-09-16T10:08:57.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:57 smithi063 ceph-mon[29993]: from='client.14136 v1:172.21.15.63:0/1253256400' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:08:58.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:58 smithi063 ceph-mon[29993]: from='client.14138 v1:172.21.15.63:0/120840960' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "addr": "172.21.15.63", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:00.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:08:59 smithi063 ceph-mon[29993]: Deploying cephadm binary to smithi063 2024-09-16T10:09:02.284 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Added host 'smithi063' with addr '172.21.15.63' 2024-09-16T10:09:02.284 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying unmanaged mon service... 2024-09-16T10:09:02.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:02 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:09:02.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:02 smithi063 ceph-mon[29993]: Added host smithi063 2024-09-16T10:09:02.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:02 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:03.501 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-16T10:09:03.501 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying unmanaged mgr service... 2024-09-16T10:09:04.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:03 smithi063 ceph-mon[29993]: from='client.14140 v1:172.21.15.63:0/1216404631' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:04.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:03 smithi063 ceph-mon[29993]: Saving service mon spec with placement count:5 2024-09-16T10:09:04.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:03 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:09:05.612 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-16T10:09:05.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:05 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:09:05.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:05 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:09:05.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:05 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:09:06.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:06 smithi063 ceph-mon[29993]: from='client.14142 v1:172.21.15.63:0/3673390456' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:06.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:06 smithi063 ceph-mon[29993]: Saving service mgr spec with placement count:2 2024-09-16T10:09:06.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2192309927' entity='client.admin' 2024-09-16T10:09:08.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:08 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:09:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:08 smithi063 ceph-mon[29993]: from='mgr.14118 v1:172.21.15.63:0/3965811624' entity='mgr.y' 2024-09-16T10:09:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4174859467' entity='client.admin' 2024-09-16T10:09:08.670 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling the dashboard module... 2024-09-16T10:09:09.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2220170475' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-16T10:09:09.701 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:09 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ignoring --setuser ceph since I am not root 2024-09-16T10:09:09.702 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:09 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ignoring --setgroup ceph since I am not root 2024-09-16T10:09:10.396 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2220170475' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-16T10:09:10.396 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:10 smithi063 ceph-mon[29993]: mgrmap e9: y(active, since 18s) 2024-09-16T10:09:11.129 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-09-16T10:09:11.129 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-16T10:09:11.130 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T10:09:11.130 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-16T10:09:11.130 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T10:09:11.130 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-09-16T10:09:11.130 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-09-16T10:09:11.130 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 9... 2024-09-16T10:09:11.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/515693668' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-16T10:09:14.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: Active manager daemon y restarted 2024-09-16T10:09:14.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: Activating manager daemon y 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: osdmap e3: 0 total, 0 up, 0 in 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: mgrmap e10: y(active, starting, since 0.00386457s) 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T10:09:14.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:14 smithi063 ceph-mon[29993]: Manager daemon y is now available 2024-09-16T10:09:15.900 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:15 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:15.900 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:15 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:09:15.900 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:15 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:09:15.900 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:15 smithi063 ceph-mon[29993]: mgrmap e11: y(active, since 1.0059s) 2024-09-16T10:09:16.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-09-16T10:09:16.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-16T10:09:16.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T10:09:16.145 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-09-16T10:09:16.145 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 9 is available 2024-09-16T10:09:16.145 INFO:teuthology.orchestra.run.smithi063.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-16T10:09:16.814 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-mon[29993]: [16/Sep/2024:10:09:16] ENGINE Bus STARTING 2024-09-16T10:09:16.814 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:09:16.547+0000 7fd3fcb3f640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:10:09:16] ENGINE Error in HTTPServer.serve 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: Traceback (most recent call last): 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._connections.run(self.expiration_interval) 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._run(expiration_interval) 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: s = self.context.wrap_socket( 2024-09-16T10:09:16.815 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:09:16.816 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: return self.sslsocket_class._create( 2024-09-16T10:09:16.816 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:09:16.816 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self.do_handshake() 2024-09-16T10:09:16.816 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:09:16.816 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._sslobj.do_handshake() 2024-09-16T10:09:16.816 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:09:16.816 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:16 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:09:17.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:17 smithi063 ceph-mon[29993]: [16/Sep/2024:10:09:16] ENGINE Serving on http://172.21.15.63:8765 2024-09-16T10:09:17.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:17 smithi063 ceph-mon[29993]: [16/Sep/2024:10:09:16] ENGINE Serving on https://172.21.15.63:7150 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:17 smithi063 ceph-mon[29993]: [16/Sep/2024:10:09:16] ENGINE Bus STARTED 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:17 smithi063 ceph-mon[29993]: [16/Sep/2024:10:09:16] ENGINE Error in HTTPServer.serve 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: Traceback (most recent call last): 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._connections.run(self.expiration_interval) 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._run(expiration_interval) 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:09:17.952 INFO:journalctl@ceph.mon.a.smithi063.stdout: s = self.context.wrap_socket( 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout: return self.sslsocket_class._create( 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout: self.do_handshake() 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._sslobj.do_handshake() 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:17 smithi063 ceph-mon[29993]: mgrmap e12: y(active, since 2s) 2024-09-16T10:09:17.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:17 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:18.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:18 smithi063 ceph-mon[29993]: from='client.14162 v1:172.21.15.63:0/360375063' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:18.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:18 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:19.024 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFEDCCAvigAwIBAgIUVeNIbP7Vb8L8ODhFhftAkw4WHN0wDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkxNjEwMDkxOFoXDTM0\nMDkxNzEwMDkxOFowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjYzMIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAx3ULZmnGv7A8rnSrMTHVOSDj0oT/rD+7R90O\npv1BpXQZLDWmxtk+1QEqIXvwnf+FCYq1ncZpgSUhhTcCCwzcZ7TgpQ88hljafo8m\nNG3RG/iU7Xw54iy2R/m32FY3W1OkHGononGZHlZo1n9o6Sbs8z5Wul7pgWWPp+qa\nszrIp5EjOb0FLFhuDdz9a9M1525Gk5O9Z8/e6ZW/Bi5GI8yvo0XHzU58uQGuXyf6\nyxPh2H2cY9wHgMIscw0Pin1BFVgjO9NItKLyKu/vfe16acNVjXsyYnCNZIuWUkJd\n+ArlcokwrLF21sfHP0z8ZoYxgooiTHWRi7AVbzzfxNVjHSoa3Mfn57xtgjiKNoZ9\n7d67bw+oZup/2tF4voqH4LbbQUya4SG4ReIniWQG2/TaA8Cxy7eUka3hLYLRwc9/\nIuQmudFLi518ZGz5UIRQBdWb4SylbgcHCw/4fLOzVOUBo/aeCtUOT1huAAvVg62r\njamhFqNFkHZh1hnpMC54ROrnGZ+lfAgFyuw1pjmvAJm1aZ8aJjAS4XC/pmXfxxsy\noMz6eoMIRYM0Q4iW/RZxfN79wCqJKkc/v0BCy4zdFi7I4hXQ0s+UXikROU7PCvgA\njca4oL84atM+LsOghvEr/jli8OUbXYgzcOlSlxmFzhhhaKS1iq9Dc7sBjwAfvrz9\nCg2ubS0CAwEAAaNUMFIwQgYDVR0RBDswOYIec21pdGhpMDYzLmZyb250LnNlcGlh\nLmNlcGguY29tghFkYXNoYm9hcmRfc2VydmVyc4cErBUPPzAMBgNVHRMBAf8EAjAA\nMA0GCSqGSIb3DQEBCwUAA4ICAQAImwcgFibZxLVOG/hS5eBdxSLPRM8aHARhC60A\ntDtdF59CEzHko+aICWjBu4OfTMyo3Wypx8XIjlRNFnNJa0BSL7+0a4Se78w9UdO1\n/RuW0YS1xkgJKG6nPzv3mpZyKM1P6ai8N5pdc9OVACyVFmKpNBrUVYhbEp+G8QK/\nEY+E5OtaKEZahfUF5u/WY2WZ27/IP/vXFh8FxBI4ovMmN6sPti/w9Jun79uO20iZ\nUaSvXViRVN8mXePxZteKCJ/qDmE+L5VVZDal6YW0m8Bis3HELokbqfHT3Bs16OaV\ngEnn/nuEjP9l4jgF0LIOMUYAcUm4kbxqBIfu7OfoQD2EFxmcyX9/fG3XLSJRP0gl\nVkF4F5jJoJsAJ1ZpPXnJ2I+NvRtMl36MpA4zSROBzVAZxNPozHB/MQcHmnTV1Uhd\njs9lDsc6oqBU9aW+ZdKN5DVklh2pPyLo20oam4DjqRPJC2yNhPjx7EezgzdjmBUa\nrlpGtS5+r/Hf/fYBEzvlCj2NRiR1ina5IYCbNp5YnSOgSnVUL1V/wKfGTvaYgKqR\nAjpSlnG5Fgcgn415E2PfC57XV1E2Y+qgRZz/9cRRdTvxmkvctCM6mJs7K/fRXKwP\nkKQihdQH1qzYTKa2nWE497+zNP5yhB4AOn6N66Yzj4DLqS+15O6wbeh9yHIokDcw\neXRcWg==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEAx3ULZmnGv7A8rnSrMTHVOSDj0oT/rD+7R90Opv1BpXQZLDWm\nxtk+1QEqIXvwnf+FCYq1ncZpgSUhhTcCCwzcZ7TgpQ88hljafo8mNG3RG/iU7Xw5\n4iy2R/m32FY3W1OkHGononGZHlZo1n9o6Sbs8z5Wul7pgWWPp+qaszrIp5EjOb0F\nLFhuDdz9a9M1525Gk5O9Z8/e6ZW/Bi5GI8yvo0XHzU58uQGuXyf6yxPh2H2cY9wH\ngMIscw0Pin1BFVgjO9NItKLyKu/vfe16acNVjXsyYnCNZIuWUkJd+ArlcokwrLF2\n1sfHP0z8ZoYxgooiTHWRi7AVbzzfxNVjHSoa3Mfn57xtgjiKNoZ97d67bw+oZup/\n2tF4voqH4LbbQUya4SG4ReIniWQG2/TaA8Cxy7eUka3hLYLRwc9/IuQmudFLi518\nZGz5UIRQBdWb4SylbgcHCw/4fLOzVOUBo/aeCtUOT1huAAvVg62rjamhFqNFkHZh\n1hnpMC54ROrnGZ+lfAgFyuw1pjmvAJm1aZ8aJjAS4XC/pmXfxxsyoMz6eoMIRYM0\nQ4iW/RZxfN79wCqJKkc/v0BCy4zdFi7I4hXQ0s+UXikROU7PCvgAjca4oL84atM+\nLsOghvEr/jli8OUbXYgzcOlSlxmFzhhhaKS1iq9Dc7sBjwAfvrz9Cg2ubS0CAwEA\nAQKCAgA6oCWEqayYZTXA1VwILqyMiSVNmHgJ6gWjHs+St7fW6OQJQY/OvMYiF6CJ\njQNxfUxePJ4bTNlv8zkCDeQU2WfjN6AcA/l3R6CZOHwzRuACx7MGhtAAscG2MCkg\ndl75AgGntHAH6DnoV9+ixHpSpXVaHcWiLAmYVgC+i0+DRND7p8ZhzzQ5D5jxjDYU\nvlUGvDwUXoV6gPf1UmV+bJWDKkOpyvXqW5p/SoY1Q+wMSsDEQE96bo7qAqjL/FfI\nDf7WRjy89eJV7dseUb3kFqw3cAmydzadBpYYqoq5jPZP1OwSGU7sQ5E5mijvazJ4\n9BKTWwSv0pLAKzZxpsPi+BGU3e5c1x+Jj8YDQs2d9bpMQ/QLnGcTPGAs8m1Vu65o\n9nOOiYAs/N/2z6vazQCBqqPQzxoitdRaWJ9wb5k4gdEllrvJbJRrkK6ClxZEHGAJ\no19H9D98FOj2VMkI1635+8jqgI6r1bF6uFzQ1V4LICfUVWMTslUZG79dh56HEZB9\nF66/uK7lhOeSaKtLmdR3y5yguMYcf+AtDSMCRsa8QsmXLCX5TLfFLUn6SP6TZ+RM\n4P1Y19e4XTVFXQ3DJKOX0K4zVIk1VMdO+lJmJ53F8/DOEMhD4xsJxBf7DFCTv86k\nP8+aItOJbTLq+k6egADWN9HzQZeY22rP387BPMymngXKlzhehQKCAQEA+AQ5pU1y\neLbVh2KFw4HZJu+G3jG2A8g706dtNrqk35R2eS4mu5LnJLy4weXsmHKkaoGdR3yu\nHTYO3cu+OHyOWvz0ElYobjVkLz6DaYridkNO3y+qn22tTJm70LAnyvvaSQ+lU5Lm\nUsfnL+ASPd/9hapvJDIp8tGwP+xTUT0w5pwixORID5t1gU3N5o0azVjD9rwfgwXY\nQhTfa9STaTzJWv4OO3CJE6+n5LYIAq8a3837HkvccBZMxDrKHNACNQrCuClza0yX\ntX/4QvVM3PdwN6yY1qN5FjkXH8mB2X67CPHQLaM3hQLVUL9oEHkC7gJq9r8Nwd+T\n+DAa8jvZX3ccJwKCAQEAzeCq3r4aPMOYBlHobbkzBw2x7T3fcx3DaJXH6y65kcnL\nswMx/LIhdVkekoANSLbX4dM6ZbRQoT5nWgea11EFK2h94TZuUMBmFZFvqjXRA+xK\nZSjW678VZ00/DtS7BkmAiGppaAsS47b+32HE5Z2ZoplX7jpXJt13t7FK3XTq5YW2\nFI//b7k33lIot9s8gDLVnmzEK6DoeDjZtehs621Zg8coSepanIx6mT/1Rn03EHz1\nfy5oBmu5o3q6Q9XDTwE7ARGxZqmG8+fpIKCkrz+2T+3+QMi0GvFlhswto+Weq56J\ny65StK4M1h0IBh/vH0qzV8W6DI3QFaOoKy3x8+88iwKCAQEAsBBvy8VbCg7hLqbm\nRpAbU+SeXYX0rGKs9efZRAzjlkSShhwZKvdyV0YR8rQCbYaU6Xbx7Bm6kMoba6CD\nfR4PANCpg4reUL5eTJuYNoh9k27cwtvXyC3X5JUOG2uyFAsOPud4egfCselvWTE8\nQeiM1SLUm2FxHgt1jyt5SiaiXFNi61sTPOEjXZOaI5zv9e6ZzsJlrGGtz9t5b9cM\nfpzEZHAOMmwFI5CDI32F2kHsVgynxC4eYwOxo2wIpsOTjYwCBzKF2RoaaKJb30y8\nL/EIHL3OEjEieIA2xj4KjBfHNuZS3tMLSsVkIVp/yZHeXpAg1uFyJrtNE9cyiduq\nyHaJ6QKCAQB8bj+X5JoDA8yMdVDfkEGxD6vsUasN7n0+Ztiv4kuyO57sERBTRG5I\neJarwM1Gl4wfJwVylursqcTZeyMw4/0V4U/DoJxIHiEVBBPTcjdkAZptTaOoVJNh\nHiTwl8553E7dJEOU6KyACQg+ht70UK6cwcWNbKFkEhLbaUJwWsvl8Uy9XNLr+aw0\nqL+Hb/A/I3pNCDwAin42u1WKRnIeULb7mJFXK1ecqqMmoKtj8SYoxv2x4qBo0Q8g\nKXsOqNIkm//Hg6QPAoNYQ2B5od4nNYDBCnzTNvr0FdZhBGLJXmzqGSX7ft7p6RBF\nEMLawM5LwWBnuk/aapPLoGBxyJEl+ABFAoIBAFtp2J5u33EPt50PQjbKPxq+A4IS\nMIddQauEMl3QoRsNJC+Up1keLDnc42WMdKCi5Cme3SupdaGLztUK6Aygrywf2lXD\na1rIXQj09XqE9270gQedYhjEJCzrZ4Q3H2NJAJ16oyVvn68ISRPHh2jRZSRloU8r\nJ2ZgU3Rq2CgxA2slLhttSndtjdhG2JZpBpWJOWGCAsNLANWuEFuwItLzGn9UacwO\no/pVDkkvu+cDMYsK6mOSD/X36yLbadW5e430v+MW9hzx/pcB1J4NirOfFqzeOOU9\n4GkEBRLYAXed/jttF05806IpatxMWq36j1dTIrAk9lRO8euaWUquWgDuhSo=\n-----END RSA PRIVATE KEY-----\n"} 2024-09-16T10:09:20.200 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-16T10:09:20.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:20 smithi063 ceph-mon[29993]: from='client.14164 v1:172.21.15.63:0/3649247555' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:20.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:21.306 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-16T10:09:21.306 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial admin user... 2024-09-16T10:09:21.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:21 smithi063 ceph-mon[29993]: from='client.14166 v1:172.21.15.63:0/1555095795' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:21.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:21 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:22.739 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$UP0iSllN9E2hpdsTWSVfgOdz3rgCWRqOwbn82OgiJyML1gPNQOzzC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726481362, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-16T10:09:22.739 INFO:teuthology.orchestra.run.smithi063.stdout:Fetching dashboard port number... 2024-09-16T10:09:23.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:23 smithi063 ceph-mon[29993]: from='client.14168 v1:172.21.15.63:0/1534194261' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:23.927 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 8443 2024-09-16T10:09:23.937 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T10:09:23.937 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-09-16T10:09:23.945 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T10:09:23.945 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-09-16T10:09:23.945 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-09-16T10:09:23.945 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-16T10:09:23.947 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph Dashboard is now available at: 2024-09-16T10:09:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: URL: https://smithi063.front.sepia.ceph.com:8443/ 2024-09-16T10:09:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: User: admin 2024-09-16T10:09:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: Password: 00tv55inmq 2024-09-16T10:09:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:23.948 INFO:teuthology.orchestra.run.smithi063.stdout:Saving cluster configuration to /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config directory 2024-09-16T10:09:24.319 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2392140480' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-16T10:09:25.150 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-16T10:09:25.150 INFO:teuthology.orchestra.run.smithi063.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-16T10:09:25.150 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.150 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-16T10:09:25.150 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout:Or, if you are only running a single cluster on this host: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: ceph telemetry on 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout:For more information see: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:25.151 INFO:teuthology.orchestra.run.smithi063.stdout:Bootstrap complete. 2024-09-16T10:09:25.190 INFO:tasks.cephadm:Fetching config... 2024-09-16T10:09:25.190 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:09:25.191 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-16T10:09:25.207 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-16T10:09:25.208 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:09:25.208 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-16T10:09:25.265 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-16T10:09:25.265 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:09:25.265 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-16T10:09:25.331 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-16T10:09:25.331 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:09:25.331 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-16T10:09:25.387 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-16T10:09:25.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC05iAEvuyd+CXNAnBV3ZHPpQF4TbJLtdOHQvfzT32qyrYp427/ubBGRnwsV8GLxun+AcJ6lMDhVotjDO2wEhQjP+E7vXj2lJe4Wr9f1kNO+h3MHqCTXUpd/9tifiw4jMM56YaGJs7wc5KKDWKLd95g3jIdIGQK14r10KuXSx1iUfVp8P6XnHhB0cORet22B0OEjIAkwRDqeL17gWXWXt/hZSFp0K60TF32OH+vO1lBmL1o1dmbOaPqUCNRCVOGwiGkcXw5gpMo27/1jsjxkx0IQUGcjMrcGXst4AbJSlRJJMlq74ihf4TI8VVbZ2nhdr7zJ9F1r0CzGLpBGyYIwbQEmqmB0vo+7ORupC5rp4OrJ+hI/m9A06xEytUS5TICmIzl1+MKvHN0CxvdtyQvZA/RzTn67cF1MmrALj5OBRkL25iiETbrkKkHs3XuRQukAgj/yCK8VWVSDsDs35pzdy/RwZGLDiombSSIfUW3aYRGcEBie33VtlWnCzOdSsUwfRs= ceph-7cb51a9a-7413-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T10:09:25.471 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC05iAEvuyd+CXNAnBV3ZHPpQF4TbJLtdOHQvfzT32qyrYp427/ubBGRnwsV8GLxun+AcJ6lMDhVotjDO2wEhQjP+E7vXj2lJe4Wr9f1kNO+h3MHqCTXUpd/9tifiw4jMM56YaGJs7wc5KKDWKLd95g3jIdIGQK14r10KuXSx1iUfVp8P6XnHhB0cORet22B0OEjIAkwRDqeL17gWXWXt/hZSFp0K60TF32OH+vO1lBmL1o1dmbOaPqUCNRCVOGwiGkcXw5gpMo27/1jsjxkx0IQUGcjMrcGXst4AbJSlRJJMlq74ihf4TI8VVbZ2nhdr7zJ9F1r0CzGLpBGyYIwbQEmqmB0vo+7ORupC5rp4OrJ+hI/m9A06xEytUS5TICmIzl1+MKvHN0CxvdtyQvZA/RzTn67cF1MmrALj5OBRkL25iiETbrkKkHs3XuRQukAgj/yCK8VWVSDsDs35pzdy/RwZGLDiombSSIfUW3aYRGcEBie33VtlWnCzOdSsUwfRs= ceph-7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:25.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC05iAEvuyd+CXNAnBV3ZHPpQF4TbJLtdOHQvfzT32qyrYp427/ubBGRnwsV8GLxun+AcJ6lMDhVotjDO2wEhQjP+E7vXj2lJe4Wr9f1kNO+h3MHqCTXUpd/9tifiw4jMM56YaGJs7wc5KKDWKLd95g3jIdIGQK14r10KuXSx1iUfVp8P6XnHhB0cORet22B0OEjIAkwRDqeL17gWXWXt/hZSFp0K60TF32OH+vO1lBmL1o1dmbOaPqUCNRCVOGwiGkcXw5gpMo27/1jsjxkx0IQUGcjMrcGXst4AbJSlRJJMlq74ihf4TI8VVbZ2nhdr7zJ9F1r0CzGLpBGyYIwbQEmqmB0vo+7ORupC5rp4OrJ+hI/m9A06xEytUS5TICmIzl1+MKvHN0CxvdtyQvZA/RzTn67cF1MmrALj5OBRkL25iiETbrkKkHs3XuRQukAgj/yCK8VWVSDsDs35pzdy/RwZGLDiombSSIfUW3aYRGcEBie33VtlWnCzOdSsUwfRs= ceph-7cb51a9a-7413-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T10:09:25.619 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC05iAEvuyd+CXNAnBV3ZHPpQF4TbJLtdOHQvfzT32qyrYp427/ubBGRnwsV8GLxun+AcJ6lMDhVotjDO2wEhQjP+E7vXj2lJe4Wr9f1kNO+h3MHqCTXUpd/9tifiw4jMM56YaGJs7wc5KKDWKLd95g3jIdIGQK14r10KuXSx1iUfVp8P6XnHhB0cORet22B0OEjIAkwRDqeL17gWXWXt/hZSFp0K60TF32OH+vO1lBmL1o1dmbOaPqUCNRCVOGwiGkcXw5gpMo27/1jsjxkx0IQUGcjMrcGXst4AbJSlRJJMlq74ihf4TI8VVbZ2nhdr7zJ9F1r0CzGLpBGyYIwbQEmqmB0vo+7ORupC5rp4OrJ+hI/m9A06xEytUS5TICmIzl1+MKvHN0CxvdtyQvZA/RzTn67cF1MmrALj5OBRkL25iiETbrkKkHs3XuRQukAgj/yCK8VWVSDsDs35pzdy/RwZGLDiombSSIfUW3aYRGcEBie33VtlWnCzOdSsUwfRs= ceph-7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:25.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-16T10:09:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:09:25.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1433222345' entity='client.admin' 2024-09-16T10:09:27.923 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2531509360' entity='client.admin' 2024-09-16T10:09:27.951 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-16T10:09:27.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-16T10:09:28.669 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:09:28.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:28 smithi063 ceph-mon[29993]: mgrmap e13: y(active, since 12s) 2024-09-16T10:09:29.945 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"} : dispatch 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='client.14176 v1:172.21.15.63:0/3052646860' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:09:29.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:29 smithi063 ceph-mon[29993]: Updating smithi063:/etc/ceph/ceph.conf 2024-09-16T10:09:29.977 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-09-16T10:09:29.977 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:09:29.977 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-09-16T10:09:29.997 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:09:29.997 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:09:30.056 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-09-16T10:09:30.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch host add smithi165 2024-09-16T10:09:30.307 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:09:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:30 smithi063 ceph-mon[29993]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:30 smithi063 ceph-mon[29993]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:09:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:30 smithi063 ceph-mon[29993]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:09:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:32.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:31 smithi063 ceph-mon[29993]: from='client.14178 v1:172.21.15.63:0/2661863073' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:33.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:32 smithi063 ceph-mon[29993]: Deploying cephadm binary to smithi165 2024-09-16T10:09:34.771 INFO:teuthology.orchestra.run.smithi063.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-09-16T10:09:35.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T10:09:35.572 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:09:35.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:35 smithi063 ceph-mon[29993]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:35 smithi063 ceph-mon[29993]: Added host smithi165 2024-09-16T10:09:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:36.216 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:09:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:[{"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-09-16T10:09:36.821 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-16T10:09:36.821 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-16T10:09:37.073 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:09:37.385 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:37 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:37.386 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:37 smithi063 ceph-mon[29993]: from='client.14180 v1:172.21.15.63:0/3451467039' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:09:38.104 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-09-16T10:09:38.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:38 smithi063 ceph-mon[29993]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:38.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3640956994' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-16T10:09:38.689 INFO:tasks.cephadm:Adding mon.a on smithi063 2024-09-16T10:09:38.690 INFO:tasks.cephadm:Adding mon.c on smithi063 2024-09-16T10:09:38.690 INFO:tasks.cephadm:Adding mon.b on smithi165 2024-09-16T10:09:38.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi063:[v1:172.21.15.63:6789]=a;smithi063:[v1:172.21.15.63:6790]=c;smithi165:[v1:172.21.15.165:6789]=b' 2024-09-16T10:09:38.944 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T10:09:38.991 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3640956994' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"} : dispatch 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:09:39.684 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mon update... 2024-09-16T10:09:40.256 DEBUG:teuthology.orchestra.run.smithi063:mon.c> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.c.service 2024-09-16T10:09:40.259 DEBUG:teuthology.orchestra.run.smithi165:mon.b> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.b.service 2024-09-16T10:09:40.261 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T10:09:40.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:40.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:40.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:09:40.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:40.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:09:40.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:41.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:41 smithi063 ceph-mon[29993]: from='client.14184 v1:172.21.15.165:0/1562738500' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi063:[v1:172.21.15.63:6789]=a;smithi063:[v1:172.21.15.63:6790]=c;smithi165:[v1:172.21.15.165:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:09:41.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:41 smithi063 ceph-mon[29993]: Saving service mon spec with placement smithi063:[v1:172.21.15.63:6789]=a;smithi063:[v1:172.21.15.63:6790]=c;smithi165:[v1:172.21.15.165:6789]=b;count:3 2024-09-16T10:09:41.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:41 smithi063 ceph-mon[29993]: Deploying daemon mon.b on smithi165 2024-09-16T10:09:41.814 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:42.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:42 smithi063 ceph-mon[29993]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:43.206 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:09:43.206 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":1,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","modified":"2024-09-16T10:08:18.256111Z","created":"2024-09-16T10:08:18.256111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-16T10:09:43.206 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 1 2024-09-16T10:09:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:44 smithi165 ceph-mon[32696]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-16T10:09:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:44 smithi063 ceph-mon[29993]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:44 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/3143328716' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:44.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:44 smithi165 bash[32677]: 7c7bce6d8fc5e6fab9e6c6cd91112185f3b1b76a5b9e03e10d63f11334113818 2024-09-16T10:09:44.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:44 smithi165 systemd[1]: Started Ceph mon.b for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:09:44.902 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T10:09:44.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T10:09:45.172 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:09:49.675 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:09:49.675 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":2,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","modified":"2024-09-16T10:09:44.395108Z","created":"2024-09-16T10:08:18.256111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T10:09:49.676 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 2 2024-09-16T10:09:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:09:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: mon.a calling monitor election 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: mon.b calling monitor election 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: monmap epoch 2 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: last_changed 2024-09-16T10:09:44.395108+0000 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: created 2024-09-16T10:08:18.256111+0000 2024-09-16T10:09:49.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: min_mon_release 19 (squid) 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: election_strategy: 1 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: 0: v1:172.21.15.63:6789/0 mon.a 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: fsmap 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: mgrmap e13: y(active, since 34s) 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: overall HEALTH_OK 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:49.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:09:49.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:49.806 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:09:49.806 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.806 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: mon.a calling monitor election 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: mon.b calling monitor election 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: monmap epoch 2 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:49.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: last_changed 2024-09-16T10:09:44.395108+0000 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: created 2024-09-16T10:08:18.256111+0000 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: min_mon_release 19 (squid) 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: election_strategy: 1 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: 0: v1:172.21.15.63:6789/0 mon.a 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: fsmap 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: mgrmap e13: y(active, since 34s) 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: overall HEALTH_OK 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:49.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:09:49.809 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:50.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:50 smithi165 ceph-mon[32696]: Deploying daemon mon.c on smithi063 2024-09-16T10:09:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:50 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/369953334' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:50 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:50.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:50 smithi063 ceph-mon[29993]: Deploying daemon mon.c on smithi063 2024-09-16T10:09:50.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:50 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/369953334' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:50.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:50 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:51.246 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T10:09:51.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T10:09:51.501 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:09:51.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:51 smithi165 ceph-mon[32696]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:51.734 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:51 smithi063 ceph-mon[29993]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:09:52.166 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":2,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","modified":"2024-09-16T10:09:44.395108Z","created":"2024-09-16T10:08:18.256111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T10:09:52.167 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 2 2024-09-16T10:09:52.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:52 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/842651755' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:52 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/842651755' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:52 smithi063 systemd[1]: Started Ceph mon.c for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:09:53.760 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T10:09:53.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T10:09:54.019 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:09:57.824 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:09:57.824 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":3,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","modified":"2024-09-16T10:09:52.531650Z","created":"2024-09-16T10:08:18.256111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-16T10:09:57.825 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 3 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: mon.a calling monitor election 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: mon.b calling monitor election 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: mon.c calling monitor election 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: monmap epoch 3 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: last_changed 2024-09-16T10:09:52.531650+0000 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: created 2024-09-16T10:08:18.256111+0000 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: min_mon_release 19 (squid) 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: election_strategy: 1 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: 0: v1:172.21.15.63:6789/0 mon.a 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-16T10:09:57.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: 2: v1:172.21.15.63:6790/0 mon.c 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: fsmap 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: mgrmap e13: y(active, since 42s) 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: overall HEALTH_OK 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: mon.a calling monitor election 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: mon.b calling monitor election 2024-09-16T10:09:57.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: mon.c calling monitor election 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: monmap epoch 3 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:57.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: last_changed 2024-09-16T10:09:52.531650+0000 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: created 2024-09-16T10:08:18.256111+0000 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: min_mon_release 19 (squid) 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: election_strategy: 1 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: 0: v1:172.21.15.63:6789/0 mon.a 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: 2: v1:172.21.15.63:6790/0 mon.c 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: fsmap 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: mgrmap e13: y(active, since 42s) 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: overall HEALTH_OK 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:57.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:57.957 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:57.957 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:09:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: mon.a calling monitor election 2024-09-16T10:09:58.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: mon.b calling monitor election 2024-09-16T10:09:58.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:09:58.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:58.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: mon.c calling monitor election 2024-09-16T10:09:58.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:58.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:58.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.055 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-16T10:09:58.055 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: monmap epoch 3 2024-09-16T10:09:58.055 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:58.055 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: last_changed 2024-09-16T10:09:52.531650+0000 2024-09-16T10:09:58.055 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: created 2024-09-16T10:08:18.256111+0000 2024-09-16T10:09:58.055 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: min_mon_release 19 (squid) 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: election_strategy: 1 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: 0: v1:172.21.15.63:6789/0 mon.a 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: 2: v1:172.21.15.63:6790/0 mon.c 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: fsmap 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: mgrmap e13: y(active, since 42s) 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: overall HEALTH_OK 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:58.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:58.461 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-16T10:09:58.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-16T10:09:58.725 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:09:58.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:58 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:58.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:58 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/736718996' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:58.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:58 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.834 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:58 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:58.834 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:58 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/736718996' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:58.834 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:58 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:58.835 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:58 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:09:58.835 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:58 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/736718996' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:09:58.835 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:58 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:09:59.340 INFO:teuthology.orchestra.run.smithi063.stdout:# minimal ceph.conf for 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:59.340 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-09-16T10:09:59.341 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:09:59.341 INFO:teuthology.orchestra.run.smithi063.stdout: mon_host = 172.21.15.63:6789/0 172.21.15.165:6789/0 v1:172.21.15.63:6790/0 2024-09-16T10:09:59.788 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:09:59.531+0000 7fd435d7f640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T10:09:59.788 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: Updating smithi063:/etc/ceph/ceph.conf 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/23584344' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.790 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: Updating smithi063:/etc/ceph/ceph.conf 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/23584344' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:09:59.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:09:59.792 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:09:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:09:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: Updating smithi063:/etc/ceph/ceph.conf 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/23584344' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:09:59.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:09:59.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:09:59.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:09:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:00.028 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-16T10:10:00.028 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:10:00.028 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T10:10:00.056 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:10:00.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:10:00.125 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:10:00.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T10:10:00.153 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:10:00.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:10:00.220 INFO:tasks.cephadm:Adding mgr.y on smithi063 2024-09-16T10:10:00.220 INFO:tasks.cephadm:Adding mgr.x on smithi165 2024-09-16T10:10:00.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi063=y;smithi165=x' 2024-09-16T10:10:00.507 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:10:00.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:00 smithi165 ceph-mon[32696]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T10:10:00.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:00 smithi165 ceph-mon[32696]: Reconfiguring daemon mon.a on smithi063 2024-09-16T10:10:00.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:00 smithi165 ceph-mon[32696]: overall HEALTH_OK 2024-09-16T10:10:00.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:00 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:00.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[29993]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T10:10:00.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[29993]: Reconfiguring daemon mon.a on smithi063 2024-09-16T10:10:00.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[29993]: overall HEALTH_OK 2024-09-16T10:10:00.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[39470]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T10:10:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[39470]: Reconfiguring daemon mon.a on smithi063 2024-09-16T10:10:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[39470]: overall HEALTH_OK 2024-09-16T10:10:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:00 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:01.142 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mgr update... 2024-09-16T10:10:01.773 DEBUG:teuthology.orchestra.run.smithi165:mgr.x> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.x.service 2024-09-16T10:10:01.775 DEBUG:tasks.cephadm:set 0 configs 2024-09-16T10:10:01.775 INFO:tasks.cephadm:Deploying OSDs... 2024-09-16T10:10:01.775 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:10:01.775 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T10:10:01.792 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T10:10:01.792 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 10:09:52.024487968 +0000 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 10:05:03.118359529 +0000 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 10:05:03.118359529 +0000 2024-09-16T10:10:01.849 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-16 10:05:03.118359529 +0000 2024-09-16T10:10:01.850 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T10:10:01.915 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T10:10:01.915 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T10:10:01.916 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000266993 s, 1.9 MB/s 2024-09-16T10:10:01.917 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T10:10:01.976 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 10:09:52.024487968 +0000 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 10:05:03.117359273 +0000 2024-09-16T10:10:02.033 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 10:05:03.117359273 +0000 2024-09-16T10:10:02.034 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-16 10:05:03.117359273 +0000 2024-09-16T10:10:02.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T10:10:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T10:10:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T10:10:02.097 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000244638 s, 2.1 MB/s 2024-09-16T10:10:02.098 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T10:10:02.155 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-09-16T10:10:02.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:02.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: from='client.24109 v1:172.21.15.165:0/2769506704' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi063=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:02.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: Saving service mgr spec with placement smithi063=y;smithi165=x;count:2 2024-09-16T10:10:02.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[39470]: Reconfiguring daemon mgr.y on smithi063 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: from='client.24109 v1:172.21.15.165:0/2769506704' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi063=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: Saving service mgr spec with placement smithi063=y;smithi165=x;count:2 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T10:10:02.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T10:10:02.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T10:10:02.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:02.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:02 smithi063 ceph-mon[29993]: Reconfiguring daemon mgr.y on smithi063 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 10:09:52.025487958 +0000 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 10:05:03.119359786 +0000 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 10:05:03.119359786 +0000 2024-09-16T10:10:02.212 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-16 10:05:03.119359786 +0000 2024-09-16T10:10:02.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T10:10:02.277 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T10:10:02.277 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T10:10:02.277 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000216169 s, 2.4 MB/s 2024-09-16T10:10:02.278 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T10:10:02.335 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-09-16T10:10:02.391 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T10:10:02.391 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:02.391 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-16T10:10:02.391 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:02.392 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:02.392 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 10:09:52.025487958 +0000 2024-09-16T10:10:02.392 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 10:05:03.120360171 +0000 2024-09-16T10:10:02.392 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 10:05:03.120360171 +0000 2024-09-16T10:10:02.392 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-16 10:05:03.120360171 +0000 2024-09-16T10:10:02.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T10:10:02.456 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T10:10:02.456 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T10:10:02.456 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000203653 s, 2.5 MB/s 2024-09-16T10:10:02.457 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T10:10:02.515 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:10:02.515 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T10:10:02.533 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T10:10:02.533 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: from='client.24109 v1:172.21.15.165:0/2769506704' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi063=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: Saving service mgr spec with placement smithi063=y;smithi165=x;count:2 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:02 smithi165 ceph-mon[32696]: Reconfiguring daemon mgr.y on smithi063 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 10:09:43.415812845 +0000 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:02.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T10:10:02.633 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T10:10:02.633 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T10:10:02.634 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000202837 s, 2.5 MB/s 2024-09-16T10:10:02.635 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T10:10:02.692 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 10:09:43.416812830 +0000 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 10:05:02.084901006 +0000 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 10:05:02.084901006 +0000 2024-09-16T10:10:02.748 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-16 10:05:02.084901006 +0000 2024-09-16T10:10:02.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T10:10:02.812 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T10:10:02.812 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T10:10:02.812 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000322776 s, 1.6 MB/s 2024-09-16T10:10:02.813 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T10:10:02.870 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-09-16T10:10:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 10:09:43.416812830 +0000 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:02.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T10:10:02.990 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T10:10:02.991 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T10:10:02.991 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000177464 s, 2.9 MB/s 2024-09-16T10:10:02.992 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T10:10:03.049 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-09-16T10:10:03.105 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T10:10:03.105 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:10:03.106 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-16T10:10:03.106 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:10:03.106 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:10:03.106 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-16 10:09:43.417812815 +0000 2024-09-16T10:10:03.106 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:03.106 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:03.107 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-16 10:05:02.085900990 +0000 2024-09-16T10:10:03.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T10:10:03.169 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-16T10:10:03.169 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-16T10:10:03.169 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000199637 s, 2.6 MB/s 2024-09-16T10:10:03.170 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T10:10:03.226 INFO:tasks.cephadm:Deploying osd.0 on smithi063 with /dev/vg_nvme/lv_4... 2024-09-16T10:10:03.227 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T10:10:03.777 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:03.777 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:03.777 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:03.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: Reconfiguring mon.c (monmap changed)... 2024-09-16T10:10:03.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:10:03.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:10:03.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:03.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[39470]: Reconfiguring daemon mon.c on smithi063 2024-09-16T10:10:03.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:03.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:03.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:03.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: Reconfiguring mon.c (monmap changed)... 2024-09-16T10:10:03.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:10:03.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:10:03.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:03.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:03 smithi063 ceph-mon[29993]: Reconfiguring daemon mon.c on smithi063 2024-09-16T10:10:03.905 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: Reconfiguring mon.c (monmap changed)... 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:03 smithi165 ceph-mon[32696]: Reconfiguring daemon mon.c on smithi063 2024-09-16T10:10:05.155 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:05.156 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:05.156 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:04 smithi165 ceph-mon[32696]: Reconfiguring mon.b (monmap changed)... 2024-09-16T10:10:05.156 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:10:05.156 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:10:05.156 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:05.156 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:04 smithi165 ceph-mon[32696]: Reconfiguring daemon mon.b on smithi165 2024-09-16T10:10:05.169 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:05.169 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:05.169 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[39470]: Reconfiguring mon.b (monmap changed)... 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[39470]: Reconfiguring daemon mon.b on smithi165 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[29993]: Reconfiguring mon.b (monmap changed)... 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:05.170 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:04 smithi063 ceph-mon[29993]: Reconfiguring daemon mon.b on smithi165 2024-09-16T10:10:06.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:10:06.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_4 2024-09-16T10:10:06.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[29993]: Deploying daemon mgr.x on smithi165 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:06 smithi063 ceph-mon[39470]: Deploying daemon mgr.x on smithi165 2024-09-16T10:10:06.298 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:10:06.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:06.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:06.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:06.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:06.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:06.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T10:10:06.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T10:10:06.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T10:10:06.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:06.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:06 smithi165 ceph-mon[32696]: Deploying daemon mgr.x on smithi165 2024-09-16T10:10:07.409 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:07 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:07.410 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:07 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:07.410 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:07 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:07.410 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:07 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:07.410 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:07 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:07.410 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:07 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:07.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:07 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:07 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:07 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:07.552 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:10:07 smithi165 systemd[1]: Started Ceph mgr.x for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:10:08.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[29993]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:08.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[29993]: from='client.14217 v1:172.21.15.63:0/3756842168' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:08.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[39470]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[39470]: from='client.14217 v1:172.21.15.63:0/3756842168' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.571 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.571 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:08 smithi165 ceph-mon[32696]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:08 smithi165 ceph-mon[32696]: from='client.14217 v1:172.21.15.63:0/3756842168' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:09.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1483683705' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c4234e56-a2f2-467f-9616-190a52181d42"} : dispatch 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1483683705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4234e56-a2f2-467f-9616-190a52181d42"}]': finished 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T10:10:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1483683705' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c4234e56-a2f2-467f-9616-190a52181d42"} : dispatch 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1483683705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4234e56-a2f2-467f-9616-190a52181d42"}]': finished 2024-09-16T10:10:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T10:10:09.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1483683705' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c4234e56-a2f2-467f-9616-190a52181d42"} : dispatch 2024-09-16T10:10:09.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1483683705' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4234e56-a2f2-467f-9616-190a52181d42"}]': finished 2024-09-16T10:10:09.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T10:10:09.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:10.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:10 smithi063 ceph-mon[29993]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:10.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2374845860' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:10:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:10.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:10 smithi063 ceph-mon[39470]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:10.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2374845860' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:10:10.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:10 smithi165 ceph-mon[32696]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2374845860' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:10:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:12.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[29993]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:12.706 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[29993]: Standby manager daemon x started 2024-09-16T10:10:12.706 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T10:10:12.706 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[39470]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[39470]: Standby manager daemon x started 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T10:10:12.707 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:12 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T10:10:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:12 smithi165 ceph-mon[32696]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:12 smithi165 ceph-mon[32696]: Standby manager daemon x started 2024-09-16T10:10:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:12 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T10:10:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:12 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T10:10:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:12 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T10:10:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:12 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/362075251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T10:10:13.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:13 smithi063 ceph-mon[29993]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-16T10:10:13.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:13 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T10:10:13.650 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:13 smithi063 ceph-mon[39470]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-16T10:10:13.650 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:13 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T10:10:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:13 smithi165 ceph-mon[32696]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-16T10:10:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:13 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T10:10:14.602 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:14 smithi063 ceph-mon[29993]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:14.603 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:14 smithi063 ceph-mon[39470]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:14.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:14 smithi165 ceph-mon[32696]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:15.633 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:15 smithi063 ceph-mon[29993]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:15.633 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:15 smithi063 ceph-mon[39470]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:15 smithi165 ceph-mon[32696]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:16.948 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:16 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-16T10:10:16.948 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:16 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:16.948 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:16 smithi063 ceph-mon[39470]: Deploying daemon osd.0 on smithi063 2024-09-16T10:10:16.948 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:16 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-16T10:10:16.948 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:16 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:16.948 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:16 smithi063 ceph-mon[29993]: Deploying daemon osd.0 on smithi063 2024-09-16T10:10:17.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:16 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-16T10:10:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:16 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:16 smithi165 ceph-mon[32696]: Deploying daemon osd.0 on smithi063 2024-09-16T10:10:17.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:17 smithi063 ceph-mon[29993]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:17.775 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:17 smithi063 ceph-mon[39470]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:18.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:17 smithi165 ceph-mon[32696]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:19 smithi063 ceph-mon[29993]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:19.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:19 smithi063 ceph-mon[39470]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:20.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:19 smithi165 ceph-mon[32696]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:21.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:21 smithi063 ceph-mon[29993]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:21.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:21 smithi063 ceph-mon[39470]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:21 smithi165 ceph-mon[32696]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:22 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:22 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:22 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:22 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:22 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:22 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:22 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:22 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:22 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:23.949 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:23 smithi063 ceph-mon[29993]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:23.950 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:23 smithi063 ceph-mon[39470]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:23 smithi165 ceph-mon[32696]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:26.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[29993]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:26.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[39470]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:25 smithi165 ceph-mon[32696]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.695 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 0 on host 'smithi063' 2024-09-16T10:10:26.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:26 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:26 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:26 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:26 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:26 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:26 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:27.258 DEBUG:teuthology.orchestra.run.smithi063:osd.0> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.0.service 2024-09-16T10:10:27.261 INFO:tasks.cephadm:Deploying osd.1 on smithi063 with /dev/vg_nvme/lv_3... 2024-09-16T10:10:27.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T10:10:27.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:26 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:26 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:26 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:27.538 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:10:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0[42760]: 2024-09-16T10:10:27.338+0000 7fdb40f99640 -1 osd.0 0 log_to_monitors true 2024-09-16T10:10:28.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:27 smithi063 ceph-mon[29993]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:28.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:27 smithi063 ceph-mon[29993]: from='osd.0 v1:172.21.15.63:6801/2031112888' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T10:10:28.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:27 smithi063 ceph-mon[29993]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T10:10:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:27 smithi063 ceph-mon[39470]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:27 smithi063 ceph-mon[39470]: from='osd.0 v1:172.21.15.63:6801/2031112888' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T10:10:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:27 smithi063 ceph-mon[39470]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T10:10:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:27 smithi165 ceph-mon[32696]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:27 smithi165 ceph-mon[32696]: from='osd.0 v1:172.21.15.63:6801/2031112888' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T10:10:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:27 smithi165 ceph-mon[32696]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T10:10:28.735 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:10:29.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[29993]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[29993]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[29993]: from='osd.0 v1:172.21.15.63:6801/2031112888' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[29993]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[39470]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[39470]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:29.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[39470]: from='osd.0 v1:172.21.15.63:6801/2031112888' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:29.160 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-mon[39470]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:29.160 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0[42760]: 2024-09-16T10:10:28.900+0000 7fdb3cf1c640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T10:10:29.160 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:10:28 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0[42760]: 2024-09-16T10:10:28.902+0000 7fdb38d43640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:10:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:28 smithi165 ceph-mon[32696]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T10:10:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:28 smithi165 ceph-mon[32696]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T10:10:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:28 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:28 smithi165 ceph-mon[32696]: from='osd.0 v1:172.21.15.63:6801/2031112888' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:28 smithi165 ceph-mon[32696]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:29.971 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[39470]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:29.971 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[39470]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:10:29.971 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[39470]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T10:10:29.972 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:29.972 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:29.972 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[29993]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:29.972 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[29993]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:10:29.972 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[29993]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T10:10:29.972 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:29.972 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:29 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:29 smithi165 ceph-mon[32696]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T10:10:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:29 smithi165 ceph-mon[32696]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:10:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:29 smithi165 ceph-mon[32696]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T10:10:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:29 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:29 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:31.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:10:31.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:10:31.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: osd.0 v1:172.21.15.63:6801/2031112888 boot 2024-09-16T10:10:31.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T10:10:31.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:31.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: osd.0 v1:172.21.15.63:6801/2031112888 boot 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.086 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.087 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:31.087 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.087 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:31.087 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:31.087 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:30 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:10:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:10:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: osd.0 v1:172.21.15.63:6801/2031112888 boot 2024-09-16T10:10:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T10:10:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:10:31.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:31.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:31.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:31.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:30 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:31.809 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:10:31.837 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_3 2024-09-16T10:10:32.095 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:10:32.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi063 2024-09-16T10:10:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi063 to 16064M 2024-09-16T10:10:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[29993]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[29993]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T10:10:32.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi063 2024-09-16T10:10:32.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi063 to 16064M 2024-09-16T10:10:32.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[39470]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:32.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:31 smithi063 ceph-mon[39470]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T10:10:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:31 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi063 2024-09-16T10:10:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:31 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi063 to 16064M 2024-09-16T10:10:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:31 smithi165 ceph-mon[32696]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:31 smithi165 ceph-mon[32696]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T10:10:33.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:33.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:33.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:33.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:33.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:34.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:33 smithi063 ceph-mon[29993]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:34.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:33 smithi063 ceph-mon[29993]: from='client.44141 v1:172.21.15.63:0/170775969' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:34.193 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:33 smithi063 ceph-mon[39470]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:34.193 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:33 smithi063 ceph-mon[39470]: from='client.44141 v1:172.21.15.63:0/170775969' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:34.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:33 smithi165 ceph-mon[32696]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:33 smithi165 ceph-mon[32696]: from='client.44141 v1:172.21.15.63:0/170775969' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:36.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[29993]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1403543642' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"} : dispatch 2024-09-16T10:10:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[29993]: from='client.44147 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"} : dispatch 2024-09-16T10:10:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[29993]: from='client.44147 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"}]': finished 2024-09-16T10:10:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[29993]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T10:10:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/964920310' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:10:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[39470]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1403543642' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"} : dispatch 2024-09-16T10:10:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[39470]: from='client.44147 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"} : dispatch 2024-09-16T10:10:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[39470]: from='client.44147 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"}]': finished 2024-09-16T10:10:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[39470]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T10:10:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/964920310' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:10:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:35 smithi165 ceph-mon[32696]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1403543642' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"} : dispatch 2024-09-16T10:10:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:35 smithi165 ceph-mon[32696]: from='client.44147 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"} : dispatch 2024-09-16T10:10:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:35 smithi165 ceph-mon[32696]: from='client.44147 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6"}]': finished 2024-09-16T10:10:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:35 smithi165 ceph-mon[32696]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T10:10:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:35 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:36.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/964920310' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:10:38.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:37 smithi063 ceph-mon[29993]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:37 smithi063 ceph-mon[39470]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:38.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:37 smithi165 ceph-mon[32696]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:40.174 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:39 smithi063 ceph-mon[39470]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:40.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:39 smithi063 ceph-mon[29993]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:40.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:39 smithi165 ceph-mon[32696]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:42.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:41 smithi063 ceph-mon[29993]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:41 smithi063 ceph-mon[39470]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:42.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:41 smithi165 ceph-mon[32696]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:43.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:42 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-16T10:10:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:42 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:42 smithi063 ceph-mon[29993]: Deploying daemon osd.1 on smithi063 2024-09-16T10:10:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:42 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-16T10:10:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:42 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:42 smithi063 ceph-mon[39470]: Deploying daemon osd.1 on smithi063 2024-09-16T10:10:43.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:42 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-16T10:10:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:42 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:42 smithi165 ceph-mon[32696]: Deploying daemon osd.1 on smithi063 2024-09-16T10:10:44.050 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:43 smithi063 ceph-mon[29993]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:44.050 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:43 smithi063 ceph-mon[39470]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:43 smithi165 ceph-mon[32696]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:45 smithi063 ceph-mon[29993]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:45 smithi063 ceph-mon[39470]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:46.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:45 smithi165 ceph-mon[32696]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:48.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:47 smithi063 ceph-mon[29993]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:48.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:47 smithi063 ceph-mon[39470]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:47 smithi165 ceph-mon[32696]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:49.029 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:49.029 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:49.030 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:49.030 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:49.030 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:49.030 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:49.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:50.654 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:50 smithi063 ceph-mon[29993]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:50.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:50 smithi063 ceph-mon[39470]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:50.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:50 smithi165 ceph-mon[32696]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:52.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[29993]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:52.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:52.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:52.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[39470]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:52.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:52.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:52.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:52 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.338 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 1 on host 'smithi063' 2024-09-16T10:10:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:52 smithi165 ceph-mon[32696]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:52 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:52 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:52 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:52 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:52 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:52.927 DEBUG:teuthology.orchestra.run.smithi063:osd.1> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.1.service 2024-09-16T10:10:52.929 INFO:tasks.cephadm:Deploying osd.2 on smithi063 with /dev/vg_nvme/lv_2... 2024-09-16T10:10:52.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T10:10:53.161 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:53 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:53.161 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:53 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:53.161 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:53 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:53.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:53 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:53.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:53 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:53.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:53 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:53 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:10:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:53 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:53 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:53.702 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:10:53 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1[48525]: 2024-09-16T10:10:53.605+0000 7fe4ebc98640 -1 osd.1 0 log_to_monitors true 2024-09-16T10:10:54.275 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:10:54.608 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:54 smithi063 ceph-mon[29993]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:54.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:54 smithi063 ceph-mon[29993]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-16T10:10:54.609 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:54 smithi063 ceph-mon[39470]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:54.609 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:54 smithi063 ceph-mon[39470]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-16T10:10:54.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:54 smithi165 ceph-mon[32696]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:54.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:54 smithi165 ceph-mon[32696]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-16T10:10:55.700 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[29993]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T10:10:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[29993]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T10:10:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[29993]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:55.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[39470]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T10:10:55.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[39470]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T10:10:55.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[39470]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:55.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:55.702 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1[48525]: 2024-09-16T10:10:55.347+0000 7fe4e7c1b640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T10:10:55.702 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:10:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1[48525]: 2024-09-16T10:10:55.349+0000 7fe4e3a42640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:10:55.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:55 smithi165 ceph-mon[32696]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T10:10:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:55 smithi165 ceph-mon[32696]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T10:10:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:55 smithi165 ceph-mon[32696]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:10:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:55 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:56.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:56.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:10:56.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T10:10:56.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:56.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='osd.1 v1:172.21.15.63:6805/3717451957' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:10:56.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:56.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:56.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:10:56.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:10:57.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi063 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi063 to 8032M 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[29993]: osd.1 v1:172.21.15.63:6805/3717451957 boot 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[29993]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi063 2024-09-16T10:10:57.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi063 to 8032M 2024-09-16T10:10:57.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[39470]: osd.1 v1:172.21.15.63:6805/3717451957 boot 2024-09-16T10:10:57.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[39470]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T10:10:57.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:57 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:57.463 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:10:57.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_2 2024-09-16T10:10:57.746 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:10:57.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:57 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:10:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:57 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:10:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:57 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi063 2024-09-16T10:10:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:57 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi063 to 8032M 2024-09-16T10:10:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:57 smithi165 ceph-mon[32696]: osd.1 v1:172.21.15.63:6805/3717451957 boot 2024-09-16T10:10:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:57 smithi165 ceph-mon[32696]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T10:10:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:57 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:10:58.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:58 smithi063 ceph-mon[29993]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:58.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:58 smithi063 ceph-mon[29993]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T10:10:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:58 smithi063 ceph-mon[39470]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:58 smithi063 ceph-mon[39470]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T10:10:58.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:58 smithi165 ceph-mon[32696]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T10:10:58.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:58 smithi165 ceph-mon[32696]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T10:10:59.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[29993]: from='client.44168 v1:172.21.15.63:0/2828400262' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:59.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:59.646 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:59.646 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:59.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[39470]: from='client.44168 v1:172.21.15.63:0/2828400262' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:59.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:59.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:59.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:10:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:10:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:59 smithi165 ceph-mon[32696]: from='client.44168 v1:172.21.15.63:0/2828400262' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:10:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:10:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:10:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:10:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:00 smithi063 ceph-mon[29993]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:00 smithi063 ceph-mon[39470]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:00.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:00 smithi165 ceph-mon[32696]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:01.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2520641052' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8"} : dispatch 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2520641052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8"}]': finished 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[29993]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3184866515' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2520641052' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8"} : dispatch 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2520641052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8"}]': finished 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[39470]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3184866515' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2520641052' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8"} : dispatch 2024-09-16T10:11:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2520641052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8"}]': finished 2024-09-16T10:11:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:01 smithi165 ceph-mon[32696]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T10:11:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:01 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3184866515' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:02.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:02 smithi063 ceph-mon[29993]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:02.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:02 smithi063 ceph-mon[39470]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:02.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:02 smithi165 ceph-mon[32696]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:04.613 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:04 smithi063 ceph-mon[29993]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:04.613 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:04 smithi063 ceph-mon[39470]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:04.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:04 smithi165 ceph-mon[32696]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:06.699 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:06 smithi063 ceph-mon[29993]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:06.699 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:06 smithi063 ceph-mon[39470]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:06.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:06 smithi165 ceph-mon[32696]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:08 smithi063 ceph-mon[29993]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-16T10:11:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:08 smithi063 ceph-mon[39470]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-16T10:11:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:08.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:08 smithi165 ceph-mon[32696]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-16T10:11:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:09.554 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:09 smithi063 ceph-mon[29993]: Deploying daemon osd.2 on smithi063 2024-09-16T10:11:09.554 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:09 smithi063 ceph-mon[39470]: Deploying daemon osd.2 on smithi063 2024-09-16T10:11:09.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:09 smithi165 ceph-mon[32696]: Deploying daemon osd.2 on smithi063 2024-09-16T10:11:10.658 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:10 smithi063 ceph-mon[29993]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:10.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:10 smithi063 ceph-mon[39470]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:10.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:10 smithi165 ceph-mon[32696]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:12.575 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:12 smithi063 ceph-mon[29993]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:12.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:12 smithi063 ceph-mon[39470]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:12 smithi165 ceph-mon[32696]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[29993]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[39470]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:14 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:14 smithi165 ceph-mon[32696]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:14 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:14 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:14 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.655 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[39470]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:16.655 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.655 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.655 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:16.655 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:16.656 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.656 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[29993]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:16.656 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.656 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.656 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:16.656 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:16.656 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:16 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:16 smithi165 ceph-mon[32696]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:16 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:16 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:16 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:16 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:16 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:16.969 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 2 on host 'smithi063' 2024-09-16T10:11:17.536 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:17 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:17.536 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:17 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:17.536 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:17 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:17.536 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:17 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:17.536 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:17 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:17.537 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:17 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:17.538 DEBUG:teuthology.orchestra.run.smithi063:osd.2> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.2.service 2024-09-16T10:11:17.540 INFO:tasks.cephadm:Deploying osd.3 on smithi063 with /dev/vg_nvme/lv_1... 2024-09-16T10:11:17.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T10:11:17.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:17 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:17 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:17 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:18.602 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:11:18.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:18 smithi063 ceph-mon[39470]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:18 smithi063 ceph-mon[29993]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:18.702 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:18 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2[54013]: 2024-09-16T10:11:18.449+0000 7fee7967c640 -1 osd.2 0 log_to_monitors true 2024-09-16T10:11:18.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:18 smithi165 ceph-mon[32696]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:19.572 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:19 smithi063 ceph-mon[29993]: from='osd.2 v1:172.21.15.63:6809/196353360' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T10:11:19.573 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:19 smithi063 ceph-mon[29993]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T10:11:19.573 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:19 smithi063 ceph-mon[29993]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:19.573 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:19 smithi063 ceph-mon[39470]: from='osd.2 v1:172.21.15.63:6809/196353360' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T10:11:19.573 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:19 smithi063 ceph-mon[39470]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T10:11:19.573 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:19 smithi063 ceph-mon[39470]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:19.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:19 smithi165 ceph-mon[32696]: from='osd.2 v1:172.21.15.63:6809/196353360' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T10:11:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:19 smithi165 ceph-mon[32696]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T10:11:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:19 smithi165 ceph-mon[32696]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='osd.2 v1:172.21.15.63:6809/196353360' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi063 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.673 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi063 to 5354M 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T10:11:20.674 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='osd.2 v1:172.21.15.63:6809/196353360' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi063 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi063 to 5354M 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:20.675 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='osd.2 v1:172.21.15.63:6809/196353360' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi063 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi063 to 5354M 2024-09-16T10:11:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:20.951 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2[54013]: 2024-09-16T10:11:20.860+0000 7fee75e13640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T10:11:20.951 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:20 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2[54013]: 2024-09-16T10:11:20.863+0000 7fee6d41e640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[29993]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[29993]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[29993]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[29993]: from='osd.2 ' entity='osd.2' 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[39470]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[39470]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T10:11:21.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[39470]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:21.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[39470]: from='osd.2 ' entity='osd.2' 2024-09-16T10:11:21.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:21 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:21 smithi165 ceph-mon[32696]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:11:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:21 smithi165 ceph-mon[32696]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T10:11:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:21 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:21 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:21 smithi165 ceph-mon[32696]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T10:11:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:21 smithi165 ceph-mon[32696]: from='osd.2 ' entity='osd.2' 2024-09-16T10:11:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:21 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:22.016 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:11:22.041 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_1 2024-09-16T10:11:22.297 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:11:22.675 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[29993]: osd.2 v1:172.21.15.63:6809/196353360 boot 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[29993]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[39470]: osd.2 v1:172.21.15.63:6809/196353360 boot 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[39470]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T10:11:22.676 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:22 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:22 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:11:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:22 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:11:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:22 smithi165 ceph-mon[32696]: osd.2 v1:172.21.15.63:6809/196353360 boot 2024-09-16T10:11:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:22 smithi165 ceph-mon[32696]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T10:11:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:22 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:11:23.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[39470]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:23.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[39470]: from='client.44195 v1:172.21.15.63:0/3234491403' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:11:23.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:11:23.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:11:23.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:23.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:11:23.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[29993]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:23.680 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[29993]: from='client.44195 v1:172.21.15.63:0/3234491403' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:11:23.680 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:11:23.680 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:11:23.680 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:23.680 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:11:23.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:23 smithi165 ceph-mon[32696]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:23 smithi165 ceph-mon[32696]: from='client.44195 v1:172.21.15.63:0/3234491403' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:11:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:11:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:11:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:11:24.517 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:24 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T10:11:24.517 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:24 smithi063 ceph-mon[29993]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T10:11:24.517 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:24 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:11:24.517 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:24 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T10:11:24.517 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:24 smithi063 ceph-mon[39470]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T10:11:24.517 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:24 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:11:24.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:24 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T10:11:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:24 smithi165 ceph-mon[32696]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T10:11:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:24 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:11:25.201 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57784]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T10:11:25.201 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57784]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.202 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57784]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.202 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57784]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.202 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57787]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T10:11:25.202 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.202 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57787]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.202 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57787]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57790]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T10:11:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57790]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57790]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57790]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57793]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T10:11:25.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57793]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57793]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57793]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.203 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57759]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T10:11:25.203 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57759]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.203 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57759]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.203 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57759]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.203 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57773]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T10:11:25.203 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57773]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57773]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57773]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57752]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57752]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57752]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57752]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57756]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57756]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57756]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.204 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:11:24 smithi063 sudo[57756]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34649]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34649]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34649]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34649]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34652]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 sudo[34652]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.556 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4125724404' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"} : dispatch 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='client.44207 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"} : dispatch 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='client.44207 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"}]': finished 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57796]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57796]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57796]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57799]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57799]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57799]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 sudo[57799]: pam_unix(sudo:session): session closed for user root 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4125724404' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='client.44207 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"} : dispatch 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='client.44207 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"}]': finished 2024-09-16T10:11:25.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T10:11:25.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:25 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4125724404' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"} : dispatch 2024-09-16T10:11:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='client.44207 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"} : dispatch 2024-09-16T10:11:25.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='client.44207 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4"}]': finished 2024-09-16T10:11:25.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T10:11:25.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:25 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/936619257' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:26.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/936619257' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/936619257' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:27 smithi165 ceph-mon[32696]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:27.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:27 smithi063 ceph-mon[29993]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:27 smithi063 ceph-mon[39470]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:28 smithi165 ceph-mon[32696]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-16T10:11:28.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:28 smithi063 ceph-mon[29993]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-16T10:11:28.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:28 smithi063 ceph-mon[39470]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-16T10:11:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:29 smithi063 ceph-mon[29993]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:29.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:29 smithi063 ceph-mon[39470]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:29 smithi165 ceph-mon[32696]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:31.786 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:31 smithi063 ceph-mon[29993]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:31.786 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:31 smithi063 ceph-mon[39470]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:31 smithi165 ceph-mon[32696]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:32.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-16T10:11:32.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:32.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-16T10:11:32.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:33.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-16T10:11:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:33.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:33 smithi063 ceph-mon[29993]: Deploying daemon osd.3 on smithi063 2024-09-16T10:11:33.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:33 smithi063 ceph-mon[29993]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:33 smithi063 ceph-mon[39470]: Deploying daemon osd.3 on smithi063 2024-09-16T10:11:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:33 smithi063 ceph-mon[39470]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:33 smithi165 ceph-mon[32696]: Deploying daemon osd.3 on smithi063 2024-09-16T10:11:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:33 smithi165 ceph-mon[32696]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:35.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:35 smithi063 ceph-mon[29993]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:35 smithi063 ceph-mon[39470]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:35 smithi165 ceph-mon[32696]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:37.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:37 smithi063 ceph-mon[29993]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:37.906 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:37 smithi063 ceph-mon[39470]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:38.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:37 smithi165 ceph-mon[32696]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:38.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:38 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:38.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:38 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:38.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:38 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:38.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:38 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:38.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:38 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:38.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:38 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:38 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:38 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:38 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:40.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:40 smithi063 ceph-mon[29993]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:40.430 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:40 smithi063 ceph-mon[39470]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:40.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:40 smithi165 ceph-mon[32696]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:41.473 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 3 on host 'smithi063' 2024-09-16T10:11:42.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:42.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:42.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:42.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:42.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.061 DEBUG:teuthology.orchestra.run.smithi063:osd.3> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.3.service 2024-09-16T10:11:42.063 INFO:tasks.cephadm:Deploying osd.4 on smithi165 with /dev/vg_nvme/lv_4... 2024-09-16T10:11:42.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T10:11:42.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:42.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:42.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:42.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:11:42.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:42.318 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:11:43.223 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:11:43 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:11:43.164+0000 7fdd4f266640 -1 osd.3 0 log_to_monitors true 2024-09-16T10:11:44.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:43 smithi165 ceph-mon[32696]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:44.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:43 smithi165 ceph-mon[32696]: from='osd.3 v1:172.21.15.63:6813/249840957' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T10:11:44.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:43 smithi165 ceph-mon[32696]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T10:11:44.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:43 smithi063 ceph-mon[39470]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:44.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:43 smithi063 ceph-mon[39470]: from='osd.3 v1:172.21.15.63:6813/249840957' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T10:11:44.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:43 smithi063 ceph-mon[39470]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T10:11:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:43 smithi063 ceph-mon[29993]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:43 smithi063 ceph-mon[29993]: from='osd.3 v1:172.21.15.63:6813/249840957' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T10:11:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:43 smithi063 ceph-mon[29993]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T10:11:44.801 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:11:44.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_4 2024-09-16T10:11:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:44 smithi165 ceph-mon[32696]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T10:11:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:44 smithi165 ceph-mon[32696]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T10:11:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:44 smithi165 ceph-mon[32696]: from='osd.3 v1:172.21.15.63:6813/249840957' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:44 smithi165 ceph-mon[32696]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:45.122 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:11:45.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[29993]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T10:11:45.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[29993]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[29993]: from='osd.3 v1:172.21.15.63:6813/249840957' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[29993]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[39470]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[39470]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[39470]: from='osd.3 v1:172.21.15.63:6813/249840957' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:45.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:44 smithi063 ceph-mon[39470]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]} : dispatch 2024-09-16T10:11:45.701 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:11:45.292+0000 7fdd4b1e9640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T10:11:45.701 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:11:45.295+0000 7fdd43008640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi063 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi063 to 4016M 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='osd.3 ' entity='osd.3' 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:11:46.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:45 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:46.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi063 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi063 to 4016M 2024-09-16T10:11:46.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='osd.3 ' entity='osd.3' 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T10:11:46.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi063 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi063 to 4016M 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:11:46.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:11:46.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='osd.3 ' entity='osd.3' 2024-09-16T10:11:46.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:11:46.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:11:46.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:45 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:11:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:11:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[29993]: from='client.24238 v1:172.21.15.165:0/1547547315' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:11:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[29993]: osd.3 v1:172.21.15.63:6813/249840957 boot 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[29993]: osdmap e24: 4 total, 4 up, 4 in 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[39470]: from='client.24238 v1:172.21.15.165:0/1547547315' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[39470]: osd.3 v1:172.21.15.63:6813/249840957 boot 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[39470]: osdmap e24: 4 total, 4 up, 4 in 2024-09-16T10:11:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:46 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:47.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:46 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:11:47.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:46 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:11:47.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:46 smithi165 ceph-mon[32696]: from='client.24238 v1:172.21.15.165:0/1547547315' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:11:47.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:46 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:47.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:46 smithi165 ceph-mon[32696]: osd.3 v1:172.21.15.63:6813/249840957 boot 2024-09-16T10:11:47.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:46 smithi165 ceph-mon[32696]: osdmap e24: 4 total, 4 up, 4 in 2024-09-16T10:11:47.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:46 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:11:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:48 smithi165 ceph-mon[32696]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:48 smithi165 ceph-mon[32696]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T10:11:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/282028463' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"} : dispatch 2024-09-16T10:11:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:48 smithi165 ceph-mon[32696]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"} : dispatch 2024-09-16T10:11:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:48 smithi165 ceph-mon[32696]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"}]': finished 2024-09-16T10:11:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:48 smithi165 ceph-mon[32696]: osdmap e26: 5 total, 4 up, 5 in 2024-09-16T10:11:48.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:11:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[29993]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[29993]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T10:11:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/282028463' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"} : dispatch 2024-09-16T10:11:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[29993]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"} : dispatch 2024-09-16T10:11:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[29993]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"}]': finished 2024-09-16T10:11:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[29993]: osdmap e26: 5 total, 4 up, 5 in 2024-09-16T10:11:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:11:48.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[39470]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T10:11:48.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[39470]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T10:11:48.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/282028463' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"} : dispatch 2024-09-16T10:11:48.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[39470]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"} : dispatch 2024-09-16T10:11:48.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[39470]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7"}]': finished 2024-09-16T10:11:48.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[39470]: osdmap e26: 5 total, 4 up, 5 in 2024-09-16T10:11:48.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:11:49.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:49 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/3264224071' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:49.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:49 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/3264224071' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:49.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:49 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/3264224071' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:11:50.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:50 smithi063 ceph-mon[29993]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:50.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:50 smithi063 ceph-mon[39470]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:50.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:50 smithi165 ceph-mon[32696]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:52 smithi165 ceph-mon[32696]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:52.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:52 smithi063 ceph-mon[29993]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:52.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:52 smithi063 ceph-mon[39470]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:54.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:54 smithi165 ceph-mon[32696]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:54.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:54 smithi063 ceph-mon[29993]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:54 smithi063 ceph-mon[39470]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:11:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:11:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[29993]: osdmap e27: 5 total, 4 up, 5 in 2024-09-16T10:11:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:11:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:11:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:11:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[39470]: osdmap e27: 5 total, 4 up, 5 in 2024-09-16T10:11:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:55 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:11:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:11:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:11:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:55 smithi165 ceph-mon[32696]: osdmap e27: 5 total, 4 up, 5 in 2024-09-16T10:11:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:55 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:11:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:56 smithi165 ceph-mon[32696]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-16T10:11:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:56.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:56 smithi063 ceph-mon[29993]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:56.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-16T10:11:56.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:56.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:56 smithi063 ceph-mon[39470]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:56.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-16T10:11:56.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:11:57.383 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:57 smithi165 ceph-mon[32696]: Deploying daemon osd.4 on smithi165 2024-09-16T10:11:57.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:57 smithi063 ceph-mon[29993]: Deploying daemon osd.4 on smithi165 2024-09-16T10:11:57.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:57 smithi063 ceph-mon[39470]: Deploying daemon osd.4 on smithi165 2024-09-16T10:11:58.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:11:58 smithi165 ceph-mon[32696]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:58.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:11:58 smithi063 ceph-mon[29993]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:11:58.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:11:58 smithi063 ceph-mon[39470]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:00.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:00 smithi063 ceph-mon[29993]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:00.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:00 smithi063 ceph-mon[39470]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:00 smithi165 ceph-mon[32696]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:12:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:12:02.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T10:12:02.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:02.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:02.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:02.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:02.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:12:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:12:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T10:12:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:04.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:04 smithi165 ceph-mon[32696]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:04.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:04 smithi063 ceph-mon[29993]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:04 smithi063 ceph-mon[39470]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:05 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:05 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:05 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:05 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:05 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:05 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:05 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:05 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:05 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:05.961 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 4 on host 'smithi165' 2024-09-16T10:12:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:06 smithi165 ceph-mon[32696]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:06.583 DEBUG:teuthology.orchestra.run.smithi165:osd.4> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.4.service 2024-09-16T10:12:06.586 INFO:tasks.cephadm:Deploying osd.5 on smithi165 with /dev/vg_nvme/lv_3... 2024-09-16T10:12:06.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T10:12:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[29993]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[39470]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:07.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:07 smithi063 ceph-mon[29993]: from='osd.4 v1:172.21.15.165:6800/2373552307' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T10:12:07.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:07 smithi063 ceph-mon[29993]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T10:12:07.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:07 smithi063 ceph-mon[39470]: from='osd.4 v1:172.21.15.165:6800/2373552307' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T10:12:07.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:07 smithi063 ceph-mon[39470]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T10:12:07.738 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:07 smithi165 ceph-mon[32696]: from='osd.4 v1:172.21.15.165:6800/2373552307' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T10:12:07.738 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:07 smithi165 ceph-mon[32696]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T10:12:07.803 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:12:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-mon[32696]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-mon[32696]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T10:12:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-mon[32696]: from='osd.4 v1:172.21.15.165:6800/2373552307' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-mon[32696]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T10:12:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-mon[32696]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:08.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[29993]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[29993]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[29993]: from='osd.4 v1:172.21.15.165:6800/2373552307' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[29993]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[29993]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[39470]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[39470]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[39470]: from='osd.4 v1:172.21.15.165:6800/2373552307' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[39470]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:08 smithi063 ceph-mon[39470]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:08.908 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:12:08.807+0000 7f2b2002d640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T10:12:08.908 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:12:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:12:08.811+0000 7f2b17e4c640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:12:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:09 smithi165 ceph-mon[32696]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:12:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:09 smithi165 ceph-mon[32696]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T10:12:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:09 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:09 smithi165 ceph-mon[32696]: from='osd.4 ' entity='osd.4' 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[29993]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[29993]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[29993]: from='osd.4 ' entity='osd.4' 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[39470]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[39470]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:09 smithi063 ceph-mon[39470]: from='osd.4 ' entity='osd.4' 2024-09-16T10:12:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:12:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: osd.4 v1:172.21.15.165:6800/2373552307 boot 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:10 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: osd.4 v1:172.21.15.165:6800/2373552307 boot 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T10:12:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: osd.4 v1:172.21.15.165:6800/2373552307 boot 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:10.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:10 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:12:11.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_3 2024-09-16T10:12:11.590 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:12:11.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:11 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi165 2024-09-16T10:12:11.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:11 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi165 to 17088M 2024-09-16T10:12:11.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:11 smithi165 ceph-mon[32696]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T10:12:11.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:11 smithi165 ceph-mon[32696]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T10:12:11.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi165 2024-09-16T10:12:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi165 to 17088M 2024-09-16T10:12:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[29993]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T10:12:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[29993]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T10:12:11.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi165 2024-09-16T10:12:11.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi165 to 17088M 2024-09-16T10:12:11.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[39470]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T10:12:11.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:11 smithi063 ceph-mon[39470]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T10:12:12.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[29993]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:12:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:12:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[39470]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:12:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:12:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:12 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:12 smithi165 ceph-mon[32696]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:12 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:12:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:12 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:12:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:12 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:13.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:13 smithi063 ceph-mon[29993]: from='client.24265 v1:172.21.15.165:0/3486174881' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:12:13.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:13 smithi063 ceph-mon[39470]: from='client.24265 v1:172.21.15.165:0/3486174881' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:12:13.754 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:13 smithi165 ceph-mon[32696]: from='client.24265 v1:172.21.15.165:0/3486174881' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:12:14.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:14 smithi063 ceph-mon[29993]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 447 GiB / 447 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-09-16T10:12:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:14 smithi063 ceph-mon[39470]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 447 GiB / 447 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-09-16T10:12:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:14 smithi165 ceph-mon[32696]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 370 MiB used, 447 GiB / 447 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-09-16T10:12:15.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/2701698677' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"} : dispatch 2024-09-16T10:12:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:15 smithi165 ceph-mon[32696]: from='client.44261 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"} : dispatch 2024-09-16T10:12:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:15 smithi165 ceph-mon[32696]: from='client.44261 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"}]': finished 2024-09-16T10:12:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:15 smithi165 ceph-mon[32696]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T10:12:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:15 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/1771444738' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:12:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/2701698677' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"} : dispatch 2024-09-16T10:12:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[29993]: from='client.44261 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"} : dispatch 2024-09-16T10:12:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[29993]: from='client.44261 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"}]': finished 2024-09-16T10:12:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[29993]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T10:12:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/1771444738' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:12:15.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/2701698677' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"} : dispatch 2024-09-16T10:12:15.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[39470]: from='client.44261 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"} : dispatch 2024-09-16T10:12:15.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[39470]: from='client.44261 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713"}]': finished 2024-09-16T10:12:15.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[39470]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T10:12:15.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:15.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/1771444738' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:12:16.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:16 smithi063 ceph-mon[29993]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-16T10:12:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:16 smithi063 ceph-mon[39470]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-16T10:12:16.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:16 smithi165 ceph-mon[32696]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-16T10:12:18.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:18 smithi063 ceph-mon[29993]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-16T10:12:18.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:18 smithi063 ceph-mon[39470]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-16T10:12:18.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:18 smithi165 ceph-mon[32696]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-16T10:12:20.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:20 smithi165 ceph-mon[32696]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T10:12:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:20 smithi063 ceph-mon[29993]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T10:12:20.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:20 smithi063 ceph-mon[39470]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T10:12:22.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:22 smithi165 ceph-mon[32696]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T10:12:22.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:22 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-16T10:12:22.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:22 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:22 smithi063 ceph-mon[29993]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T10:12:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:22 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-16T10:12:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:22 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:22 smithi063 ceph-mon[39470]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T10:12:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:22 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-16T10:12:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:22 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:23.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:23 smithi063 ceph-mon[29993]: Deploying daemon osd.5 on smithi165 2024-09-16T10:12:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:23 smithi063 ceph-mon[39470]: Deploying daemon osd.5 on smithi165 2024-09-16T10:12:23.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:23 smithi165 ceph-mon[32696]: Deploying daemon osd.5 on smithi165 2024-09-16T10:12:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:24 smithi063 ceph-mon[29993]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:24 smithi063 ceph-mon[39470]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:24.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:24 smithi165 ceph-mon[32696]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:26.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:26 smithi063 ceph-mon[29993]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:26.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:26 smithi063 ceph-mon[39470]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:26.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:26 smithi165 ceph-mon[32696]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:28.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[29993]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[39470]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:28 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:28.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:28 smithi165 ceph-mon[32696]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:28.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:28 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:28.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:28 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:28.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:28 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:29.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:29 smithi165 ceph-mon[32696]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:29.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:29 smithi063 ceph-mon[29993]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:29.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:29 smithi063 ceph-mon[39470]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[39470]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[29993]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:31.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:31.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:31.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:31 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:32.231 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:31 smithi165 ceph-mon[32696]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:32.231 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:31 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:32.231 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:31 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:32.231 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:31 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:32.231 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:31 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:32.231 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:31 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:32.271 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 5 on host 'smithi165' 2024-09-16T10:12:32.877 DEBUG:teuthology.orchestra.run.smithi165:osd.5> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.5.service 2024-09-16T10:12:32.879 INFO:tasks.cephadm:Deploying osd.6 on smithi165 with /dev/vg_nvme/lv_2... 2024-09-16T10:12:32.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T10:12:33.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:33.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:33.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:32 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:33.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:33.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:33.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:32 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:33.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:32 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:33.437 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:12:33 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:12:33.127+0000 7f0966780640 -1 osd.5 0 log_to_monitors true 2024-09-16T10:12:34.141 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:12:34.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:33 smithi063 ceph-mon[29993]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:34.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:33 smithi063 ceph-mon[29993]: from='osd.5 v1:172.21.15.165:6804/783450148' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T10:12:34.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:33 smithi063 ceph-mon[29993]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T10:12:34.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:33 smithi063 ceph-mon[39470]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:34.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:33 smithi063 ceph-mon[39470]: from='osd.5 v1:172.21.15.165:6804/783450148' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T10:12:34.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:33 smithi063 ceph-mon[39470]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T10:12:34.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:33 smithi165 ceph-mon[32696]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:34.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:33 smithi165 ceph-mon[32696]: from='osd.5 v1:172.21.15.165:6804/783450148' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T10:12:34.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:33 smithi165 ceph-mon[32696]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T10:12:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[29993]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T10:12:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[29993]: from='osd.5 v1:172.21.15.165:6804/783450148' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[29993]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T10:12:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[29993]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:35.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[39470]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T10:12:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[39470]: from='osd.5 v1:172.21.15.165:6804/783450148' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[39470]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T10:12:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:34 smithi063 ceph-mon[39470]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:35.281 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:12:34 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:12:34.943+0000 7f0962703640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T10:12:35.281 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:12:34 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:12:34.950+0000 7f095e52a640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:12:35.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:34 smithi165 ceph-mon[32696]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T10:12:35.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:34 smithi165 ceph-mon[32696]: from='osd.5 v1:172.21.15.165:6804/783450148' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:35.282 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:34 smithi165 ceph-mon[32696]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T10:12:35.282 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:35.282 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:34 smithi165 ceph-mon[32696]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: osd.5 v1:172.21.15.165:6804/783450148 boot 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:36.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:12:36.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T10:12:36.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: osd.5 v1:172.21.15.165:6804/783450148 boot 2024-09-16T10:12:36.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T10:12:36.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:35 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T10:12:36.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:12:36.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T10:12:36.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:36.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: osd.5 v1:172.21.15.165:6804/783450148 boot 2024-09-16T10:12:36.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T10:12:36.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:35 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi165 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi165 to 8544M 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[29993]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi165 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi165 to 8544M 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:37.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:36 smithi063 ceph-mon[39470]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi165 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi165 to 8544M 2024-09-16T10:12:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:37.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:12:37.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:37.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:36 smithi165 ceph-mon[32696]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T10:12:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:12:37.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_2 2024-09-16T10:12:37.965 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:12:38.337 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:38 smithi165 ceph-mon[32696]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:38.338 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:38 smithi165 ceph-mon[32696]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T10:12:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:38 smithi063 ceph-mon[29993]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:38 smithi063 ceph-mon[29993]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T10:12:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:38 smithi063 ceph-mon[39470]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:38 smithi063 ceph-mon[39470]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T10:12:39.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:39 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:12:39.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:39 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:12:39.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:39 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:39.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:12:39.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:12:39.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:39 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:39.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:39 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:12:39.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:39 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:12:39.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:39 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:40.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:40 smithi165 ceph-mon[32696]: from='client.24289 v1:172.21.15.165:0/1906425208' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:12:40.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:40 smithi165 ceph-mon[32696]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:40.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:40 smithi063 ceph-mon[29993]: from='client.24289 v1:172.21.15.165:0/1906425208' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:12:40.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:40 smithi063 ceph-mon[29993]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:40 smithi063 ceph-mon[39470]: from='client.24289 v1:172.21.15.165:0/1906425208' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:12:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:40 smithi063 ceph-mon[39470]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:12:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:12:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: osdmap e40: 6 total, 6 up, 6 in 2024-09-16T10:12:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/943004651' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"} : dispatch 2024-09-16T10:12:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"} : dispatch 2024-09-16T10:12:41.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"}]': finished 2024-09-16T10:12:41.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T10:12:41.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:12:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:12:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:12:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: osdmap e40: 6 total, 6 up, 6 in 2024-09-16T10:12:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/943004651' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"} : dispatch 2024-09-16T10:12:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"} : dispatch 2024-09-16T10:12:41.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"}]': finished 2024-09-16T10:12:41.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T10:12:41.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:12:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:12:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:12:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: osdmap e40: 6 total, 6 up, 6 in 2024-09-16T10:12:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/943004651' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"} : dispatch 2024-09-16T10:12:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: from='client.24295 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"} : dispatch 2024-09-16T10:12:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa"}]': finished 2024-09-16T10:12:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T10:12:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:12:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:42 smithi165 ceph-mon[32696]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:42 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:12:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:42 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:12:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/3514438110' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:12:42.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[29993]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:12:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:12:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/3514438110' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:12:42.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[39470]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:42.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:12:42.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:12:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/3514438110' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:12:44.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:44 smithi165 ceph-mon[32696]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:44.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:44 smithi063 ceph-mon[29993]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:44.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:44 smithi063 ceph-mon[39470]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:46.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:46 smithi165 ceph-mon[32696]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:46.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:46 smithi063 ceph-mon[29993]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:46 smithi063 ceph-mon[39470]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:48.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:48 smithi165 ceph-mon[32696]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:48.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:48 smithi063 ceph-mon[29993]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:48.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:48 smithi063 ceph-mon[39470]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:49.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-16T10:12:49.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-16T10:12:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:49.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:49 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-16T10:12:49.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:49 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:12:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:50 smithi165 ceph-mon[32696]: Deploying daemon osd.6 on smithi165 2024-09-16T10:12:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:50 smithi165 ceph-mon[32696]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:50.603 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:50 smithi063 ceph-mon[29993]: Deploying daemon osd.6 on smithi165 2024-09-16T10:12:50.603 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:50 smithi063 ceph-mon[29993]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:50.603 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:50 smithi063 ceph-mon[39470]: Deploying daemon osd.6 on smithi165 2024-09-16T10:12:50.604 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:50 smithi063 ceph-mon[39470]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:52.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:52 smithi063 ceph-mon[29993]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:52.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:52 smithi063 ceph-mon[39470]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:52.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:52 smithi165 ceph-mon[32696]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:54 smithi165 ceph-mon[32696]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:54 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:54.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:54 smithi063 ceph-mon[29993]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:54 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:54 smithi063 ceph-mon[39470]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:54 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:12:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:12:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:12:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:12:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:12:55.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:55 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:55.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:55 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:55.747 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:12:55.747 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:12:56.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:56 smithi165 ceph-mon[32696]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:56.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:56 smithi165 ceph-mon[32696]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T10:12:56.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:56 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:12:56.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:56 smithi063 ceph-mon[29993]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:56.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:56 smithi063 ceph-mon[29993]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T10:12:56.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:56 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:12:56.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:56 smithi063 ceph-mon[39470]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:56.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:56 smithi063 ceph-mon[39470]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T10:12:56.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:56 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:12:57.731 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 6 on host 'smithi165' 2024-09-16T10:12:58.414 DEBUG:teuthology.orchestra.run.smithi165:osd.6> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.6.service 2024-09-16T10:12:58.416 INFO:tasks.cephadm:Deploying osd.7 on smithi165 with /dev/vg_nvme/lv_1... 2024-09-16T10:12:58.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T10:12:58.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:58 smithi063 ceph-mon[29993]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:58 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:58 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:58.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:58 smithi063 ceph-mon[39470]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:58.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:58 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:58.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:58 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:58.720 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:12:58.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:58 smithi165 ceph-mon[32696]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:58.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:58 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:58.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:58 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:59.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:59.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:59 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:59.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:12:59 smithi165 ceph-mon[32696]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:59.777 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:12:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:12:59.575+0000 7fecf7511640 -1 osd.6 0 log_to_monitors true 2024-09-16T10:12:59.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:59.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:59 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:59.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:12:59 smithi063 ceph-mon[29993]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:12:59.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:59.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:59 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:12:59.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:12:59 smithi063 ceph-mon[39470]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:13:00.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:00 smithi165 ceph-mon[32696]: from='osd.6 v1:172.21.15.165:6808/424252741' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T10:13:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:00 smithi165 ceph-mon[32696]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T10:13:00.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:00 smithi063 ceph-mon[29993]: from='osd.6 v1:172.21.15.165:6808/424252741' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T10:13:00.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:00 smithi063 ceph-mon[29993]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T10:13:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:00 smithi063 ceph-mon[39470]: from='osd.6 v1:172.21.15.165:6808/424252741' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T10:13:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:00 smithi063 ceph-mon[39470]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T10:13:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-mon[32696]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T10:13:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-mon[32696]: from='osd.6 v1:172.21.15.165:6808/424252741' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-mon[32696]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T10:13:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-mon[32696]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-mon[32696]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:13:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[29993]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T10:13:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[29993]: from='osd.6 v1:172.21.15.165:6808/424252741' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[29993]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T10:13:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[29993]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[29993]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:13:01.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[39470]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T10:13:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[39470]: from='osd.6 v1:172.21.15.165:6808/424252741' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[39470]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T10:13:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[39470]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:01 smithi063 ceph-mon[39470]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:13:02.302 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:13:01.990+0000 7fecf3494640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T10:13:02.302 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:13:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:13:01.996+0000 7feceb2b3640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:13:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-16T10:13:02.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_1 2024-09-16T10:13:02.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:02 smithi165 ceph-mon[32696]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:13:02.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:02 smithi165 ceph-mon[32696]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T10:13:02.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:02.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:02 smithi165 ceph-mon[32696]: from='osd.6 ' entity='osd.6' 2024-09-16T10:13:02.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:02.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[29993]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:13:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[29993]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T10:13:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[29993]: from='osd.6 ' entity='osd.6' 2024-09-16T10:13:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:02.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[39470]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:13:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[39470]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T10:13:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[39470]: from='osd.6 ' entity='osd.6' 2024-09-16T10:13:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:03.004 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:03 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:03 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:03 smithi165 ceph-mon[32696]: osd.6 v1:172.21.15.165:6808/424252741 boot 2024-09-16T10:13:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:03 smithi165 ceph-mon[32696]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T10:13:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:03 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:03 smithi165 ceph-mon[32696]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[29993]: osd.6 v1:172.21.15.165:6808/424252741 boot 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[29993]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[29993]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:03.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:03.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[39470]: osd.6 v1:172.21.15.165:6808/424252741 boot 2024-09-16T10:13:03.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[39470]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T10:13:03.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:13:03.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:03 smithi063 ceph-mon[39470]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: osdmap e46: 7 total, 7 up, 7 in 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='client.24316 v1:172.21.15.165:0/725151887' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi165 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi165 to 5696M 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:04 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: osdmap e46: 7 total, 7 up, 7 in 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='client.24316 v1:172.21.15.165:0/725151887' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi165 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi165 to 5696M 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: osdmap e46: 7 total, 7 up, 7 in 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='client.24316 v1:172.21.15.165:0/725151887' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi165 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi165 to 5696M 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:04.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:04.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:04.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:04.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:04.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:04 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:05.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:05 smithi165 ceph-mon[32696]: osdmap e47: 7 total, 7 up, 7 in 2024-09-16T10:13:05.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:05 smithi165 ceph-mon[32696]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:05.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:05 smithi063 ceph-mon[29993]: osdmap e47: 7 total, 7 up, 7 in 2024-09-16T10:13:05.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:05 smithi063 ceph-mon[29993]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:05 smithi063 ceph-mon[39470]: osdmap e47: 7 total, 7 up, 7 in 2024-09-16T10:13:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:05 smithi063 ceph-mon[39470]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:06.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/1211409765' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"} : dispatch 2024-09-16T10:13:06.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:06 smithi165 ceph-mon[32696]: from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"} : dispatch 2024-09-16T10:13:06.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:06 smithi165 ceph-mon[32696]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"}]': finished 2024-09-16T10:13:06.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:06 smithi165 ceph-mon[32696]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T10:13:06.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:06 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:06.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/1211409765' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"} : dispatch 2024-09-16T10:13:06.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[29993]: from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"} : dispatch 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[29993]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"}]': finished 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[29993]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/1211409765' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"} : dispatch 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[39470]: from='client.24322 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"} : dispatch 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[39470]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48"}]': finished 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[39470]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T10:13:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:06 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:07.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:07 smithi063 ceph-mon[29993]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:07 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/3180949097' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:13:07.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:07 smithi063 ceph-mon[39470]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:07.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:07 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/3180949097' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:13:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:07 smithi165 ceph-mon[32696]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:07 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/3180949097' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T10:13:09.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:09 smithi063 ceph-mon[29993]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:09.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:09 smithi063 ceph-mon[39470]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:09 smithi165 ceph-mon[32696]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:11.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:11 smithi165 ceph-mon[32696]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:11.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:11 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:11.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:11 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:11.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:11 smithi063 ceph-mon[39470]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:11.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:11 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:11.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:11 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:11.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:11 smithi063 ceph-mon[29993]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:11.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:11 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:11.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:11 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:12.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:12 smithi165 ceph-mon[32696]: osdmap e49: 8 total, 7 up, 8 in 2024-09-16T10:13:12.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:12 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:12.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:12 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:12.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:12 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:13.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[29993]: osdmap e49: 8 total, 7 up, 8 in 2024-09-16T10:13:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[39470]: osdmap e49: 8 total, 7 up, 8 in 2024-09-16T10:13:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:12 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:14.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:13 smithi165 ceph-mon[32696]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:14.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:13 smithi165 ceph-mon[32696]: osdmap e50: 8 total, 7 up, 8 in 2024-09-16T10:13:14.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:13 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:14.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:13 smithi063 ceph-mon[29993]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:13 smithi063 ceph-mon[29993]: osdmap e50: 8 total, 7 up, 8 in 2024-09-16T10:13:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:13 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:13 smithi063 ceph-mon[39470]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:13 smithi063 ceph-mon[39470]: osdmap e50: 8 total, 7 up, 8 in 2024-09-16T10:13:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:13 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:15.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:14 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-16T10:13:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:14 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:14 smithi165 ceph-mon[32696]: Deploying daemon osd.7 on smithi165 2024-09-16T10:13:15.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-16T10:13:15.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:14 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:15.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:14 smithi063 ceph-mon[29993]: Deploying daemon osd.7 on smithi165 2024-09-16T10:13:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:14 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-16T10:13:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:14 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:14 smithi063 ceph-mon[39470]: Deploying daemon osd.7 on smithi165 2024-09-16T10:13:16.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:15 smithi165 ceph-mon[32696]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:16.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:15 smithi063 ceph-mon[29993]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:16.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:15 smithi063 ceph-mon[39470]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:17.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:17 smithi165 ceph-mon[32696]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:18.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:17 smithi063 ceph-mon[29993]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:18.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:17 smithi063 ceph-mon[39470]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:19.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:18 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:19.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:18 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:19.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:18 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:19.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:18 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:18 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:18 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:19.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:19 smithi165 ceph-mon[32696]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:19.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:19 smithi165 ceph-mon[32696]: osdmap e51: 8 total, 7 up, 8 in 2024-09-16T10:13:19.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:19 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:20.103 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:19 smithi063 ceph-mon[39470]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:20.103 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:19 smithi063 ceph-mon[39470]: osdmap e51: 8 total, 7 up, 8 in 2024-09-16T10:13:20.103 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:19 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:20.103 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:19 smithi063 ceph-mon[29993]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:20.103 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:19 smithi063 ceph-mon[29993]: osdmap e51: 8 total, 7 up, 8 in 2024-09-16T10:13:20.104 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:19 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:20.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:20.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:20.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:20 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:21.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:20 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:21.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:21.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:21.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:20 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:22 smithi165 ceph-mon[32696]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:22 smithi063 ceph-mon[29993]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:22 smithi063 ceph-mon[39470]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:23.713 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 7 on host 'smithi165' 2024-09-16T10:13:24.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:24.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:24.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:24.026 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.026 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:24.026 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.026 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:23 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:23 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:24.379 DEBUG:teuthology.orchestra.run.smithi165:osd.7> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.7.service 2024-09-16T10:13:24.382 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-16T10:13:24.382 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T10:13:24.654 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:13:25.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:13:25 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-16T10:13:24.999+0000 7f3826819640 -1 osd.7 0 log_to_monitors true 2024-09-16T10:13:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:25 smithi165 ceph-mon[32696]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:25 smithi165 ceph-mon[32696]: from='osd.7 v1:172.21.15.165:6812/2484590937' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:13:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:25 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:13:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2466778593' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T10:13:26.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[29993]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:26.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[29993]: from='osd.7 v1:172.21.15.165:6812/2484590937' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:13:26.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:13:26.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2466778593' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T10:13:26.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[39470]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[39470]: from='osd.7 v1:172.21.15.165:6812/2484590937' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:13:26.128 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2466778593' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T10:13:26.156 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":51,"num_osds":8,"num_up_osds":7,"osd_up_since":1726481582,"num_in_osds":8,"osd_in_since":1726481586,"num_remapped_pgs":0} 2024-09-16T10:13:27.146 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:13:26 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-16T10:13:26.875+0000 7f382279c640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T10:13:27.146 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:13:26 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-16T10:13:26.880+0000 7f381e5c3640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:13:27.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:26 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T10:13:27.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:26 smithi165 ceph-mon[32696]: osdmap e52: 8 total, 7 up, 8 in 2024-09-16T10:13:27.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:26 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:27.147 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:26 smithi165 ceph-mon[32696]: from='osd.7 v1:172.21.15.165:6812/2484590937' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:27.147 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:26 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:27.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T10:13:27.189 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T10:13:27.189 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[29993]: osdmap e52: 8 total, 7 up, 8 in 2024-09-16T10:13:27.189 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:27.189 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[29993]: from='osd.7 v1:172.21.15.165:6812/2484590937' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:27.189 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:27.190 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T10:13:27.190 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[39470]: osdmap e52: 8 total, 7 up, 8 in 2024-09-16T10:13:27.190 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:27.190 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[39470]: from='osd.7 v1:172.21.15.165:6812/2484590937' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:27.190 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:26 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:13:27.432 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:28.083 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:13:28.083 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:28.083 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:13:28.083 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: osdmap e53: 8 total, 7 up, 8 in 2024-09-16T10:13:28.083 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:28.083 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:28.083 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: Detected new or changed devices on smithi165 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi165 to 4272M 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:28.084 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: osdmap e53: 8 total, 7 up, 8 in 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: Detected new or changed devices on smithi165 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.085 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi165 to 4272M 2024-09-16T10:13:28.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:28.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:28.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:27 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-16T10:13:28.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-16T10:13:28.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: osdmap e53: 8 total, 7 up, 8 in 2024-09-16T10:13:28.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:28.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:28.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: Detected new or changed devices on smithi165 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi165 to 4272M 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:28.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:27 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:28.885 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1726481607,"num_in_osds":8,"osd_in_since":1726481586,"num_remapped_pgs":1} 2024-09-16T10:13:28.886 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T10:13:29.185 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:29.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:29.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:29.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[29993]: osd.7 v1:172.21.15.165:6812/2484590937 boot 2024-09-16T10:13:29.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[29993]: osdmap e54: 8 total, 8 up, 8 in 2024-09-16T10:13:29.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:29.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3755744729' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T10:13:29.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:29.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:29.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[39470]: osd.7 v1:172.21.15.165:6812/2484590937 boot 2024-09-16T10:13:29.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[39470]: osdmap e54: 8 total, 8 up, 8 in 2024-09-16T10:13:29.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:29.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3755744729' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T10:13:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:28 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:28 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:28 smithi165 ceph-mon[32696]: osd.7 v1:172.21.15.165:6812/2484590937 boot 2024-09-16T10:13:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:28 smithi165 ceph-mon[32696]: osdmap e54: 8 total, 8 up, 8 in 2024-09-16T10:13:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:28 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:13:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3755744729' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T10:13:29.808 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:13:29.808 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":55,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","created":"2024-09-16T10:08:21.081928+0000","modified":"2024-09-16T10:13:28.876338+0000","last_up_change":"2024-09-16T10:13:27.869667+0000","last_in_change":"2024-09-16T10:13:06.309218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:11:23.055843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c4234e56-a2f2-467f-9616-190a52181d42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":2031112888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":2031112888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":2031112888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":2031112888}]},"public_addr":"172.21.15.63:6801/2031112888","cluster_addr":"172.21.15.63:6802/2031112888","heartbeat_back_addr":"172.21.15.63:6804/2031112888","heartbeat_front_addr":"172.21.15.63:6803/2031112888","state":["exists","up"]},{"osd":1,"uuid":"6fe9a4c9-c351-4223-8aab-6405152696c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":3717451957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":3717451957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":3717451957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":3717451957}]},"public_addr":"172.21.15.63:6805/3717451957","cluster_addr":"172.21.15.63:6806/3717451957","heartbeat_back_addr":"172.21.15.63:6808/3717451957","heartbeat_front_addr":"172.21.15.63:6807/3717451957","state":["exists","up"]},{"osd":2,"uuid":"4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":196353360}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":196353360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6812","nonce":196353360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":196353360}]},"public_addr":"172.21.15.63:6809/196353360","cluster_addr":"172.21.15.63:6810/196353360","heartbeat_back_addr":"172.21.15.63:6812/196353360","heartbeat_front_addr":"172.21.15.63:6811/196353360","state":["exists","up"]},{"osd":3,"uuid":"355833dd-7cc5-4f59-b036-ed8939892ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6813","nonce":249840957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6814","nonce":249840957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6816","nonce":249840957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6815","nonce":249840957}]},"public_addr":"172.21.15.63:6813/249840957","cluster_addr":"172.21.15.63:6814/249840957","heartbeat_back_addr":"172.21.15.63:6816/249840957","heartbeat_front_addr":"172.21.15.63:6815/249840957","state":["exists","up"]},{"osd":4,"uuid":"da2fbe06-9462-44ec-9ffb-c5563ccee4b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":2373552307}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":2373552307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":2373552307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":2373552307}]},"public_addr":"172.21.15.165:6800/2373552307","cluster_addr":"172.21.15.165:6801/2373552307","heartbeat_back_addr":"172.21.15.165:6803/2373552307","heartbeat_front_addr":"172.21.15.165:6802/2373552307","state":["exists","up"]},{"osd":5,"uuid":"b866ad81-ae05-4c8e-9e12-822440fcb713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":783450148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":783450148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":783450148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":783450148}]},"public_addr":"172.21.15.165:6804/783450148","cluster_addr":"172.21.15.165:6805/783450148","heartbeat_back_addr":"172.21.15.165:6807/783450148","heartbeat_front_addr":"172.21.15.165:6806/783450148","state":["exists","up"]},{"osd":6,"uuid":"ce05cb1a-e5c6-4ea9-a2c0-0039c874affa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":424252741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":424252741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":424252741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":424252741}]},"public_addr":"172.21.15.165:6808/424252741","cluster_addr":"172.21.15.165:6809/424252741","heartbeat_back_addr":"172.21.15.165:6811/424252741","heartbeat_front_addr":"172.21.15.165:6810/424252741","state":["exists","up"]},{"osd":7,"uuid":"1b464a8a-ad32-47ab-82fb-3097077ebc48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":2484590937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":2484590937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":2484590937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":2484590937}]},"public_addr":"172.21.15.165:6812/2484590937","cluster_addr":"172.21.15.165:6813/2484590937","heartbeat_back_addr":"172.21.15.165:6815/2484590937","heartbeat_front_addr":"172.21.15.165:6814/2484590937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:12.089802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:18.062468+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:12:40.070489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:12:54.811533+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:11.537291+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:12:34.176998+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:00.617014+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:25.978290+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/2411639026":"2024-09-17T10:09:14.602270+0000","172.21.15.63:6800/274451570":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/2625337613":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1037308168":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1387667925":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/3928610681":"2024-09-17T10:08:51.022048+0000","172.21.15.63:6800/670090510":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/455268617":"2024-09-17T10:08:51.022048+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:13:30.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:29 smithi063 ceph-mon[39470]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:30.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:29 smithi063 ceph-mon[39470]: osdmap e55: 8 total, 8 up, 8 in 2024-09-16T10:13:30.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:29 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2458364488' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:13:30.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:29 smithi063 ceph-mon[29993]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:30.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:29 smithi063 ceph-mon[29993]: osdmap e55: 8 total, 8 up, 8 in 2024-09-16T10:13:30.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:29 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2458364488' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:13:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:29 smithi165 ceph-mon[32696]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:29 smithi165 ceph-mon[32696]: osdmap e55: 8 total, 8 up, 8 in 2024-09-16T10:13:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:29 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2458364488' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:13:30.515 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T10:13:30.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-16T10:13:30.774 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:31.144 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:30 smithi063 ceph-mon[29993]: osdmap e56: 8 total, 8 up, 8 in 2024-09-16T10:13:31.145 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:30 smithi063 ceph-mon[39470]: osdmap e56: 8 total, 8 up, 8 in 2024-09-16T10:13:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:30 smithi165 ceph-mon[32696]: osdmap e56: 8 total, 8 up, 8 in 2024-09-16T10:13:31.421 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-09-16T10:13:32.090 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:31 smithi063 ceph-mon[29993]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:32.090 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3904015530' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-16T10:13:32.091 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:31 smithi063 ceph-mon[39470]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:32.091 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3904015530' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-16T10:13:32.121 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi063 2024-09-16T10:13:32.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi063=foo.a' 2024-09-16T10:13:32.150 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:31 smithi165 ceph-mon[32696]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:32.150 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3904015530' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-16T10:13:32.382 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:33.068 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled rgw.foo.a update... 2024-09-16T10:13:33.742 DEBUG:teuthology.orchestra.run.smithi063:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-16T10:13:33.745 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi165 2024-09-16T10:13:33.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-16T10:13:34.030 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='client.24367 v1:172.21.15.165:0/305691486' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi063=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: Saving service rgw.foo.a spec with placement smithi063=foo.a;count:1 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:34.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.323 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-16T10:13:34.323 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-16T10:13:34.323 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.323 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:34.323 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:34 smithi165 ceph-mon[32696]: Deploying daemon rgw.foo.a on smithi063 2024-09-16T10:13:34.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='client.24367 v1:172.21.15.165:0/305691486' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi063=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: Saving service rgw.foo.a spec with placement smithi063=foo.a;count:1 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[29993]: Deploying daemon rgw.foo.a on smithi063 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='client.24367 v1:172.21.15.165:0/305691486' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi063=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: Saving service rgw.foo.a spec with placement smithi063=foo.a;count:1 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-16T10:13:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-16T10:13:34.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:34.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:34.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:34 smithi063 ceph-mon[39470]: Deploying daemon rgw.foo.a on smithi063 2024-09-16T10:13:35.075 INFO:teuthology.orchestra.run.smithi165.stderr:pool 'datapool' created 2024-09-16T10:13:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/1570706836' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T10:13:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:35 smithi063 ceph-mon[29993]: from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T10:13:35.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/1570706836' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T10:13:35.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:35 smithi063 ceph-mon[39470]: from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T10:13:35.400 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/1570706836' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T10:13:35.400 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:35 smithi165 ceph-mon[32696]: from='client.24373 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T10:13:35.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-16T10:13:35.951 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:13:35 smithi063 systemd[1]: Started Ceph rgw.foo.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:13:35.985 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:36.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:36.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-16T10:13:36.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: osdmap e57: 8 total, 8 up, 8 in 2024-09-16T10:13:36.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: osdmap e57: 8 total, 8 up, 8 in 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.398 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:36 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:36.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:36.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: from='client.24373 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-16T10:13:36.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: osdmap e57: 8 total, 8 up, 8 in 2024-09-16T10:13:36.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:36.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:36 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:13:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: Saving service rgw.foo.a spec with placement smithi063=foo.a;count:1 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3545906739' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='client.44372 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/1262748649' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='client.24388 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='client.44372 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-16T10:13:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[29993]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: Saving service rgw.foo.a spec with placement smithi063=foo.a;count:1 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3545906739' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='client.44372 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/1262748649' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='client.24388 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='client.44372 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-16T10:13:37.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-16T10:13:37.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:37 smithi063 ceph-mon[39470]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T10:13:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: Saving service rgw.foo.a spec with placement smithi063=foo.a;count:1 2024-09-16T10:13:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T10:13:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3545906739' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T10:13:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='client.44372 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T10:13:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/1262748649' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T10:13:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='client.24388 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T10:13:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:13:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='client.44372 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-16T10:13:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-16T10:13:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:37 smithi165 ceph-mon[32696]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T10:13:38.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:38 smithi165 ceph-mon[32696]: pgmap v179: 36 pgs: 1 creating+peering, 34 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:38.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:38 smithi165 ceph-mon[32696]: Checking dashboard <-> RGW credentials 2024-09-16T10:13:38.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:38 smithi165 ceph-mon[32696]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:13:38.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:38 smithi165 ceph-mon[32696]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T10:13:38.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:38 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[29993]: pgmap v179: 36 pgs: 1 creating+peering, 34 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[29993]: Checking dashboard <-> RGW credentials 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[29993]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[29993]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[39470]: pgmap v179: 36 pgs: 1 creating+peering, 34 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[39470]: Checking dashboard <-> RGW credentials 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[39470]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[39470]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T10:13:38.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:38.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:38 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T10:13:39.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.165 --placement '1;smithi165=iscsi.a' 2024-09-16T10:13:40.013 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:40.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:40 smithi165 ceph-mon[32696]: pgmap v182: 68 pgs: 14 active+clean, 10 creating+peering, 44 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T10:13:40.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T10:13:40.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:40 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T10:13:40.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:40 smithi165 ceph-mon[32696]: osdmap e61: 8 total, 8 up, 8 in 2024-09-16T10:13:40.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:40 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:40.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[29993]: pgmap v182: 68 pgs: 14 active+clean, 10 creating+peering, 44 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T10:13:40.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[29993]: osdmap e61: 8 total, 8 up, 8 in 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[39470]: pgmap v182: 68 pgs: 14 active+clean, 10 creating+peering, 44 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[39470]: osdmap e61: 8 total, 8 up, 8 in 2024-09-16T10:13:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:40 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:40.686 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled iscsi.datapool update... 2024-09-16T10:13:41.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:41 smithi165 ceph-mon[32696]: osdmap e62: 8 total, 8 up, 8 in 2024-09-16T10:13:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:41 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:41 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:41.344 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-16T10:13:41.345 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:13:41.345 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-16T10:13:41.372 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[29993]: osdmap e62: 8 total, 8 up, 8 in 2024-09-16T10:13:41.372 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.372 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.372 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.372 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:41.372 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[39470]: osdmap e62: 8 total, 8 up, 8 in 2024-09-16T10:13:41.372 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.373 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.373 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T10:13:41.373 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:41 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:41.377 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:13:41.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-16T10:13:41.404 DEBUG:teuthology.orchestra.run.smithi165:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-16T10:13:41.447 INFO:tasks.cephadm:Adding prometheus.a on smithi165 2024-09-16T10:13:41.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi165=a' 2024-09-16T10:13:41.729 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:42.404 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled prometheus update... 2024-09-16T10:13:42.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:42 smithi165 ceph-mon[32696]: pgmap v185: 100 pgs: 38 active+clean, 10 creating+peering, 52 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-16T10:13:42.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:42 smithi165 ceph-mon[32696]: from='client.24427 v1:172.21.15.165:0/2946145815' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:42.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:42 smithi165 ceph-mon[32696]: Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-16T10:13:42.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T10:13:42.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:42 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T10:13:42.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:42 smithi165 ceph-mon[32696]: osdmap e63: 8 total, 8 up, 8 in 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[29993]: pgmap v185: 100 pgs: 38 active+clean, 10 creating+peering, 52 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[29993]: from='client.24427 v1:172.21.15.165:0/2946145815' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[29993]: Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[29993]: osdmap e63: 8 total, 8 up, 8 in 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[39470]: pgmap v185: 100 pgs: 38 active+clean, 10 creating+peering, 52 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-16T10:13:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[39470]: from='client.24427 v1:172.21.15.165:0/2946145815' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:42.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[39470]: Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-16T10:13:42.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T10:13:42.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T10:13:42.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:42 smithi063 ceph-mon[39470]: osdmap e63: 8 total, 8 up, 8 in 2024-09-16T10:13:43.041 DEBUG:teuthology.orchestra.run.smithi165:prometheus.a> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-16T10:13:43.044 INFO:tasks.cephadm:Adding node-exporter.a on smithi063 2024-09-16T10:13:43.044 INFO:tasks.cephadm:Adding node-exporter.b on smithi165 2024-09-16T10:13:43.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi063=a;smithi165=b' 2024-09-16T10:13:43.338 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: osdmap e64: 8 total, 8 up, 8 in 2024-09-16T10:13:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.24433 v1:172.21.15.165:0/4254704153' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:43.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: Saving service prometheus spec with placement smithi165=a;count:1 2024-09-16T10:13:43.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:43.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T10:13:43.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T10:13:43.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: osdmap e65: 8 total, 8 up, 8 in 2024-09-16T10:13:43.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: osdmap e64: 8 total, 8 up, 8 in 2024-09-16T10:13:43.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.24433 v1:172.21.15.165:0/4254704153' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: Saving service prometheus spec with placement smithi165=a;count:1 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: osdmap e65: 8 total, 8 up, 8 in 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:43 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: osdmap e64: 8 total, 8 up, 8 in 2024-09-16T10:13:43.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.24433 v1:172.21.15.165:0/4254704153' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: Saving service prometheus spec with placement smithi165=a;count:1 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: osdmap e65: 8 total, 8 up, 8 in 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3053947796' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:43 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T10:13:43.992 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled node-exporter update... 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: pgmap v188: 132 pgs: 78 active+clean, 13 creating+peering, 41 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: osdmap e66: 8 total, 8 up, 8 in 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-16T10:13:44.679 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-16T10:13:44.679 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:44 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:44.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: pgmap v188: 132 pgs: 78 active+clean, 13 creating+peering, 41 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T10:13:44.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: osdmap e66: 8 total, 8 up, 8 in 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: pgmap v188: 132 pgs: 78 active+clean, 13 creating+peering, 41 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T10:13:44.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2920274648' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: osdmap e66: 8 total, 8 up, 8 in 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-16T10:13:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:44 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:13:44.706 DEBUG:teuthology.orchestra.run.smithi063:node-exporter.a> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-16T10:13:44.708 DEBUG:teuthology.orchestra.run.smithi165:node-exporter.b> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-16T10:13:44.710 INFO:tasks.cephadm:Adding alertmanager.a on smithi063 2024-09-16T10:13:44.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi063=a' 2024-09-16T10:13:45.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:45 smithi063 ceph-mon[29993]: from='client.24439 v1:172.21.15.165:0/3468361099' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi063=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:45 smithi063 ceph-mon[29993]: Saving service node-exporter spec with placement smithi063=a;smithi165=b;count:2 2024-09-16T10:13:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:45 smithi063 ceph-mon[29993]: Deploying daemon iscsi.iscsi.a on smithi165 2024-09-16T10:13:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:45 smithi063 ceph-mon[39470]: from='client.24439 v1:172.21.15.165:0/3468361099' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi063=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:45 smithi063 ceph-mon[39470]: Saving service node-exporter spec with placement smithi063=a;smithi165=b;count:2 2024-09-16T10:13:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:45 smithi063 ceph-mon[39470]: Deploying daemon iscsi.iscsi.a on smithi165 2024-09-16T10:13:45.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:45 smithi165 ceph-mon[32696]: from='client.24439 v1:172.21.15.165:0/3468361099' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi063=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:45.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:45 smithi165 ceph-mon[32696]: Saving service node-exporter spec with placement smithi063=a;smithi165=b;count:2 2024-09-16T10:13:45.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:45 smithi165 ceph-mon[32696]: Deploying daemon iscsi.iscsi.a on smithi165 2024-09-16T10:13:45.904 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:46.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:46 smithi165 ceph-mon[32696]: pgmap v191: 132 pgs: 97 active+clean, 16 creating+peering, 19 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-09-16T10:13:46.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:46 smithi165 ceph-mon[32696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:13:46.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:46 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:13:46.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:46 smithi165 ceph-mon[32696]: osdmap e67: 8 total, 8 up, 8 in 2024-09-16T10:13:46.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[29993]: pgmap v191: 132 pgs: 97 active+clean, 16 creating+peering, 19 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-09-16T10:13:46.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[29993]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:13:46.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:13:46.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[29993]: osdmap e67: 8 total, 8 up, 8 in 2024-09-16T10:13:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[39470]: pgmap v191: 132 pgs: 97 active+clean, 16 creating+peering, 19 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-09-16T10:13:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[39470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:13:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:13:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:46 smithi063 ceph-mon[39470]: osdmap e67: 8 total, 8 up, 8 in 2024-09-16T10:13:46.771 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled alertmanager update... 2024-09-16T10:13:47.464 DEBUG:teuthology.orchestra.run.smithi063:alertmanager.a> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-16T10:13:47.467 INFO:tasks.cephadm:Adding grafana.a on smithi165 2024-09-16T10:13:47.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi165=a' 2024-09-16T10:13:47.863 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:47.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:47 smithi165 ceph-mon[32696]: pgmap v193: 132 pgs: 114 active+clean, 11 creating+peering, 7 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.0 KiB/s wr, 108 op/s 2024-09-16T10:13:47.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:47 smithi165 ceph-mon[32696]: from='client.24460 v1:172.21.15.165:0/481156599' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:47.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:47 smithi165 ceph-mon[32696]: Saving service alertmanager spec with placement smithi063=a;count:1 2024-09-16T10:13:47.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:47 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:48.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[29993]: pgmap v193: 132 pgs: 114 active+clean, 11 creating+peering, 7 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.0 KiB/s wr, 108 op/s 2024-09-16T10:13:48.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[29993]: from='client.24460 v1:172.21.15.165:0/481156599' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:48.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[29993]: Saving service alertmanager spec with placement smithi063=a;count:1 2024-09-16T10:13:48.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:48.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[39470]: pgmap v193: 132 pgs: 114 active+clean, 11 creating+peering, 7 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.0 KiB/s wr, 108 op/s 2024-09-16T10:13:48.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[39470]: from='client.24460 v1:172.21.15.165:0/481156599' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi063=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:48.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[39470]: Saving service alertmanager spec with placement smithi063=a;count:1 2024-09-16T10:13:48.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:47 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.122 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled grafana update... 2024-09-16T10:13:49.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:48 smithi165 ceph-mon[32696]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-16T10:13:49.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:48 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:48 smithi165 ceph-mon[32696]: Deploying daemon prometheus.a on smithi165 2024-09-16T10:13:49.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/2328546037' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-16T10:13:49.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[39470]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[39470]: Deploying daemon prometheus.a on smithi165 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/2328546037' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[29993]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-16T10:13:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:49.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[29993]: Deploying daemon prometheus.a on smithi165 2024-09-16T10:13:49.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/2328546037' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-16T10:13:49.894 DEBUG:teuthology.orchestra.run.smithi165:grafana.a> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@grafana.a.service 2024-09-16T10:13:49.897 INFO:tasks.cephadm:Setting up client nodes... 2024-09-16T10:13:49.897 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T10:13:50.168 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:50.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[39470]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 179 op/s 2024-09-16T10:13:50.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[39470]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-16T10:13:50.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[39470]: from='client.24496 v1:172.21.15.165:0/2673078403' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[39470]: Saving service grafana spec with placement smithi165=a;count:1 2024-09-16T10:13:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[29993]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 179 op/s 2024-09-16T10:13:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[29993]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-16T10:13:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[29993]: from='client.24496 v1:172.21.15.165:0/2673078403' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[29993]: Saving service grafana spec with placement smithi165=a;count:1 2024-09-16T10:13:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:49 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:50.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:49 smithi165 ceph-mon[32696]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 179 op/s 2024-09-16T10:13:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:49 smithi165 ceph-mon[32696]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-16T10:13:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:49 smithi165 ceph-mon[32696]: from='client.24496 v1:172.21.15.165:0/2673078403' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T10:13:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:49 smithi165 ceph-mon[32696]: Saving service grafana spec with placement smithi165=a;count:1 2024-09-16T10:13:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:49 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:50.936 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-09-16T10:13:50.937 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDeBOhmFauENxAABzxgiUohLX0aWWq5LFNLqg== 2024-09-16T10:13:51.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:51.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[29993]: osdmap e68: 8 total, 8 up, 8 in 2024-09-16T10:13:51.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1598165913' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:51.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[29993]: from='client.44486 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:51.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[29993]: from='client.44486 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T10:13:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[39470]: osdmap e68: 8 total, 8 up, 8 in 2024-09-16T10:13:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1598165913' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[39470]: from='client.44486 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:51 smithi063 ceph-mon[39470]: from='client.44486 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T10:13:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:51 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:51 smithi165 ceph-mon[32696]: osdmap e68: 8 total, 8 up, 8 in 2024-09-16T10:13:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1598165913' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:51 smithi165 ceph-mon[32696]: from='client.44486 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:51.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:51 smithi165 ceph-mon[32696]: from='client.44486 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T10:13:51.692 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T10:13:51.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-16T10:13:51.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T10:13:51.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T10:13:51.977 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:13:52.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:52 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:52.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:52 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:52.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:52 smithi165 ceph-mon[32696]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.6 KiB/s wr, 148 op/s 2024-09-16T10:13:52.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:52 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:52.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:52 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:52.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:52 smithi063 ceph-mon[29993]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.6 KiB/s wr, 148 op/s 2024-09-16T10:13:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:52 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:52 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:52 smithi063 ceph-mon[39470]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.6 KiB/s wr, 148 op/s 2024-09-16T10:13:52.938 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-09-16T10:13:52.938 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQDgBOhmCvWPNxAAtopRglbHK5mxpZrXswyt8A== 2024-09-16T10:13:53.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.165:0/271131992' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:53.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:53 smithi063 ceph-mon[29993]: from='client.24508 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:53.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:53 smithi063 ceph-mon[29993]: from='client.24508 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T10:13:53.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.165:0/271131992' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:53.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:53 smithi063 ceph-mon[39470]: from='client.24508 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:53.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:53 smithi063 ceph-mon[39470]: from='client.24508 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T10:13:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.165:0/271131992' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:53 smithi165 ceph-mon[32696]: from='client.24508 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T10:13:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:53 smithi165 ceph-mon[32696]: from='client.24508 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T10:13:54.258 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-16T10:13:54.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-16T10:13:54.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T10:13:54.301 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-16T10:13:54.329 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:54 smithi063 ceph-mon[29993]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.2 KiB/s wr, 109 op/s 2024-09-16T10:13:54.330 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:54 smithi063 ceph-mon[39470]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.2 KiB/s wr, 109 op/s 2024-09-16T10:13:54.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:54 smithi165 ceph-mon[32696]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.2 KiB/s wr, 109 op/s 2024-09-16T10:13:54.573 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:55.262 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:13:55.262 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:55.262 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:55.262 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:55 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:55.263 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:55.263 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:55.263 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:55 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:55 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:13:56.017 INFO:teuthology.orchestra.run.smithi063.stdout:[{"version":24,"timestamp":"2024-09-16T10:13:44.330470+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"n6pYCdE4OTAJXDDDRFj41eleH7ttHBgCT9nv7WKf"}]}] 2024-09-16T10:13:56.018 INFO:tasks.ceph_manager:config epoch is 24 2024-09-16T10:13:56.018 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T10:13:56.018 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-16T10:13:56.018 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-16T10:13:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:56.439 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[29993]: pgmap v198: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.3 KiB/s wr, 80 op/s 2024-09-16T10:13:56.439 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[29993]: osdmap e69: 8 total, 8 up, 8 in 2024-09-16T10:13:56.439 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4246400033' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-16T10:13:56.439 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[29993]: osdmap e70: 8 total, 8 up, 8 in 2024-09-16T10:13:56.440 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[39470]: pgmap v198: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.3 KiB/s wr, 80 op/s 2024-09-16T10:13:56.440 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[39470]: osdmap e69: 8 total, 8 up, 8 in 2024-09-16T10:13:56.440 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4246400033' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-16T10:13:56.440 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:56 smithi063 ceph-mon[39470]: osdmap e70: 8 total, 8 up, 8 in 2024-09-16T10:13:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:56 smithi165 ceph-mon[32696]: pgmap v198: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.3 KiB/s wr, 80 op/s 2024-09-16T10:13:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:56 smithi165 ceph-mon[32696]: osdmap e69: 8 total, 8 up, 8 in 2024-09-16T10:13:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4246400033' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-16T10:13:56.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:56 smithi165 ceph-mon[32696]: osdmap e70: 8 total, 8 up, 8 in 2024-09-16T10:13:57.024 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:13:57.395 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:57 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4229568387' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-16T10:13:57.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:57 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4229568387' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-16T10:13:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:57 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4229568387' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-16T10:13:57.787 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6800","nonce":2945123439}]},"active_addr":"172.21.15.63:6800/2945123439","active_change":"2024-09-16T10:09:14.602415+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24121,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.63:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2993184887}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":56975596}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":1121825432}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":955884223}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":812143427}]}]} 2024-09-16T10:13:57.792 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-16T10:13:57.792 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-16T10:13:57.792 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T10:13:58.054 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:13:58.142 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:58 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:13:58.143 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:58 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:13:58.143 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:58 smithi165 ceph-mon[32696]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-09-16T10:13:58.143 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:58 smithi165 ceph-mon[32696]: osdmap e71: 8 total, 8 up, 8 in 2024-09-16T10:13:58.336 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:13:58.336 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:13:58.336 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[29993]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-09-16T10:13:58.336 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[29993]: osdmap e71: 8 total, 8 up, 8 in 2024-09-16T10:13:58.337 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:13:58.337 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:13:58.337 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[39470]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-09-16T10:13:58.337 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:58 smithi063 ceph-mon[39470]: osdmap e71: 8 total, 8 up, 8 in 2024-09-16T10:13:58.728 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:13:58.728 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":71,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","created":"2024-09-16T10:08:21.081928+0000","modified":"2024-09-16T10:13:57.076423+0000","last_up_change":"2024-09-16T10:13:27.869667+0000","last_in_change":"2024-09-16T10:13:06.309218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:11:23.055843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-16T10:13:34.734896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-16T10:13:35.550974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-16T10:13:37.166765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-16T10:13:39.104673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-16T10:13:41.175068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c4234e56-a2f2-467f-9616-190a52181d42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":2031112888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":2031112888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":2031112888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":2031112888}]},"public_addr":"172.21.15.63:6801/2031112888","cluster_addr":"172.21.15.63:6802/2031112888","heartbeat_back_addr":"172.21.15.63:6804/2031112888","heartbeat_front_addr":"172.21.15.63:6803/2031112888","state":["exists","up"]},{"osd":1,"uuid":"6fe9a4c9-c351-4223-8aab-6405152696c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":3717451957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":3717451957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":3717451957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":3717451957}]},"public_addr":"172.21.15.63:6805/3717451957","cluster_addr":"172.21.15.63:6806/3717451957","heartbeat_back_addr":"172.21.15.63:6808/3717451957","heartbeat_front_addr":"172.21.15.63:6807/3717451957","state":["exists","up"]},{"osd":2,"uuid":"4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":196353360}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":196353360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6812","nonce":196353360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":196353360}]},"public_addr":"172.21.15.63:6809/196353360","cluster_addr":"172.21.15.63:6810/196353360","heartbeat_back_addr":"172.21.15.63:6812/196353360","heartbeat_front_addr":"172.21.15.63:6811/196353360","state":["exists","up"]},{"osd":3,"uuid":"355833dd-7cc5-4f59-b036-ed8939892ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6813","nonce":249840957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6814","nonce":249840957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6816","nonce":249840957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6815","nonce":249840957}]},"public_addr":"172.21.15.63:6813/249840957","cluster_addr":"172.21.15.63:6814/249840957","heartbeat_back_addr":"172.21.15.63:6816/249840957","heartbeat_front_addr":"172.21.15.63:6815/249840957","state":["exists","up"]},{"osd":4,"uuid":"da2fbe06-9462-44ec-9ffb-c5563ccee4b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":2373552307}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":2373552307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":2373552307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":2373552307}]},"public_addr":"172.21.15.165:6800/2373552307","cluster_addr":"172.21.15.165:6801/2373552307","heartbeat_back_addr":"172.21.15.165:6803/2373552307","heartbeat_front_addr":"172.21.15.165:6802/2373552307","state":["exists","up"]},{"osd":5,"uuid":"b866ad81-ae05-4c8e-9e12-822440fcb713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":783450148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":783450148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":783450148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":783450148}]},"public_addr":"172.21.15.165:6804/783450148","cluster_addr":"172.21.15.165:6805/783450148","heartbeat_back_addr":"172.21.15.165:6807/783450148","heartbeat_front_addr":"172.21.15.165:6806/783450148","state":["exists","up"]},{"osd":6,"uuid":"ce05cb1a-e5c6-4ea9-a2c0-0039c874affa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":424252741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":424252741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":424252741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":424252741}]},"public_addr":"172.21.15.165:6808/424252741","cluster_addr":"172.21.15.165:6809/424252741","heartbeat_back_addr":"172.21.15.165:6811/424252741","heartbeat_front_addr":"172.21.15.165:6810/424252741","state":["exists","up"]},{"osd":7,"uuid":"1b464a8a-ad32-47ab-82fb-3097077ebc48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":2484590937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":2484590937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":2484590937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":2484590937}]},"public_addr":"172.21.15.165:6812/2484590937","cluster_addr":"172.21.15.165:6813/2484590937","heartbeat_back_addr":"172.21.15.165:6815/2484590937","heartbeat_front_addr":"172.21.15.165:6814/2484590937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:12.089802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:18.062468+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:54.321118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:56.626506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:11.537291+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:50.556231+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:00.617014+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:25.978290+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/2411639026":"2024-09-17T10:09:14.602270+0000","172.21.15.63:6800/274451570":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/2625337613":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1037308168":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1387667925":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/3928610681":"2024-09-17T10:08:51.022048+0000","172.21.15.63:6800/670090510":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/455268617":"2024-09-17T10:08:51.022048+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:13:59.243 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:13:59.243 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:13:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3999099105' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:13:59.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:13:59.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:13:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3999099105' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:13:59.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:13:59.431 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:13:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3999099105' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:13:59.462 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-16T10:13:59.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T10:13:59.722 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:00.121 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-mon[32696]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-09-16T10:14:00.121 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 podman[58106]: 2024-09-16 10:13:59.995335158 +0000 UTC m=+0.022717343 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-16T10:14:00.122 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 podman[58106]: 2024-09-16 10:14:00.119871778 +0000 UTC m=+0.147253950 container create 927c0e755445f4680ea9f5561e62ef4cae6d4d2fdde2252cb73feacc44ee09f4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:00.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:00 smithi063 ceph-mon[29993]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-09-16T10:14:00.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:00 smithi063 ceph-mon[39470]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-09-16T10:14:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:14:00.345 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":71,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","created":"2024-09-16T10:08:21.081928+0000","modified":"2024-09-16T10:13:57.076423+0000","last_up_change":"2024-09-16T10:13:27.869667+0000","last_in_change":"2024-09-16T10:13:06.309218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:11:23.055843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-16T10:13:34.734896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-16T10:13:35.550974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-16T10:13:37.166765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-16T10:13:39.104673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-16T10:13:41.175068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c4234e56-a2f2-467f-9616-190a52181d42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":2031112888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":2031112888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":2031112888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":2031112888}]},"public_addr":"172.21.15.63:6801/2031112888","cluster_addr":"172.21.15.63:6802/2031112888","heartbeat_back_addr":"172.21.15.63:6804/2031112888","heartbeat_front_addr":"172.21.15.63:6803/2031112888","state":["exists","up"]},{"osd":1,"uuid":"6fe9a4c9-c351-4223-8aab-6405152696c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":3717451957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":3717451957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":3717451957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":3717451957}]},"public_addr":"172.21.15.63:6805/3717451957","cluster_addr":"172.21.15.63:6806/3717451957","heartbeat_back_addr":"172.21.15.63:6808/3717451957","heartbeat_front_addr":"172.21.15.63:6807/3717451957","state":["exists","up"]},{"osd":2,"uuid":"4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":196353360}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":196353360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6812","nonce":196353360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":196353360}]},"public_addr":"172.21.15.63:6809/196353360","cluster_addr":"172.21.15.63:6810/196353360","heartbeat_back_addr":"172.21.15.63:6812/196353360","heartbeat_front_addr":"172.21.15.63:6811/196353360","state":["exists","up"]},{"osd":3,"uuid":"355833dd-7cc5-4f59-b036-ed8939892ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6813","nonce":249840957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6814","nonce":249840957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6816","nonce":249840957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6815","nonce":249840957}]},"public_addr":"172.21.15.63:6813/249840957","cluster_addr":"172.21.15.63:6814/249840957","heartbeat_back_addr":"172.21.15.63:6816/249840957","heartbeat_front_addr":"172.21.15.63:6815/249840957","state":["exists","up"]},{"osd":4,"uuid":"da2fbe06-9462-44ec-9ffb-c5563ccee4b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":2373552307}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":2373552307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":2373552307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":2373552307}]},"public_addr":"172.21.15.165:6800/2373552307","cluster_addr":"172.21.15.165:6801/2373552307","heartbeat_back_addr":"172.21.15.165:6803/2373552307","heartbeat_front_addr":"172.21.15.165:6802/2373552307","state":["exists","up"]},{"osd":5,"uuid":"b866ad81-ae05-4c8e-9e12-822440fcb713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":783450148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":783450148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":783450148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":783450148}]},"public_addr":"172.21.15.165:6804/783450148","cluster_addr":"172.21.15.165:6805/783450148","heartbeat_back_addr":"172.21.15.165:6807/783450148","heartbeat_front_addr":"172.21.15.165:6806/783450148","state":["exists","up"]},{"osd":6,"uuid":"ce05cb1a-e5c6-4ea9-a2c0-0039c874affa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":424252741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":424252741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":424252741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":424252741}]},"public_addr":"172.21.15.165:6808/424252741","cluster_addr":"172.21.15.165:6809/424252741","heartbeat_back_addr":"172.21.15.165:6811/424252741","heartbeat_front_addr":"172.21.15.165:6810/424252741","state":["exists","up"]},{"osd":7,"uuid":"1b464a8a-ad32-47ab-82fb-3097077ebc48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":2484590937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":2484590937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":2484590937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":2484590937}]},"public_addr":"172.21.15.165:6812/2484590937","cluster_addr":"172.21.15.165:6813/2484590937","heartbeat_back_addr":"172.21.15.165:6815/2484590937","heartbeat_front_addr":"172.21.15.165:6814/2484590937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:12.089802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:18.062468+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:54.321118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:56.626506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:11.537291+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:50.556231+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:00.617014+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:13:25.978290+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/2411639026":"2024-09-17T10:09:14.602270+0000","172.21.15.63:6800/274451570":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/2625337613":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1037308168":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1387667925":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/3928610681":"2024-09-17T10:08:51.022048+0000","172.21.15.63:6800/670090510":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/455268617":"2024-09-17T10:08:51.022048+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:14:00.802 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 podman[58106]: 2024-09-16 10:14:00.380430062 +0000 UTC m=+0.407812234 container init 927c0e755445f4680ea9f5561e62ef4cae6d4d2fdde2252cb73feacc44ee09f4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 podman[58106]: 2024-09-16 10:14:00.38446649 +0000 UTC m=+0.411848662 container start 927c0e755445f4680ea9f5561e62ef4cae6d4d2fdde2252cb73feacc44ee09f4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.404Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.404Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.404Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi165 (none))" 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.404Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.404Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.406Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.406Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.408Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.408Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.410Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.410Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.379µs 2024-09-16T10:14:00.803 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.410Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.410Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.410Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.09µs wal_replay_duration=164.516µs wbl_replay_duration=127ns total_replay_duration=223.024µs 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.411Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.411Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.411Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.427Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.602972ms db_storage=810ns remote_storage=1.504µs web_handler=321ns query_engine=665ns scrape=391.039µs scrape_sd=87.824µs notify=901ns notify_sd=1.109µs rules=14.794774ms tracing=6.989µs 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.427Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:14:00.427Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 bash[58106]: 927c0e755445f4680ea9f5561e62ef4cae6d4d2fdde2252cb73feacc44ee09f4 2024-09-16T10:14:00.804 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:14:00 smithi165 systemd[1]: Started Ceph prometheus.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:14:01.087 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-16T10:14:01.088 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-16T10:14:01.088 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-16T10:14:01.089 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-16T10:14:01.089 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-16T10:14:01.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-16T10:14:01.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-16T10:14:01.091 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-16T10:14:01.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2603665071' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:14:01.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-16T10:14:01.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-mon[32696]: osdmap e72: 8 total, 8 up, 8 in 2024-09-16T10:14:01.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2603665071' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[39470]: osdmap e72: 8 total, 8 up, 8 in 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2603665071' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' 2024-09-16T10:14:01.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-16T10:14:01.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-mon[29993]: osdmap e72: 8 total, 8 up, 8 in 2024-09-16T10:14:01.590 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:01.595 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:01.599 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:01.599 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:01.883 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:01.892 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ignoring --setuser ceph since I am not root 2024-09-16T10:14:01.892 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ignoring --setgroup ceph since I am not root 2024-09-16T10:14:02.052 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: ignoring --setuser ceph since I am not root 2024-09-16T10:14:02.052 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: ignoring --setgroup ceph since I am not root 2024-09-16T10:14:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:02.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:02 smithi063 ceph-mon[39470]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:02.158 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:02 smithi063 ceph-mon[39470]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-16T10:14:02.158 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:02 smithi063 ceph-mon[39470]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-16T10:14:02.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:02 smithi063 ceph-mon[29993]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:02.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:02 smithi063 ceph-mon[29993]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-16T10:14:02.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:02 smithi063 ceph-mon[29993]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-16T10:14:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:02 smithi165 ceph-mon[32696]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:02 smithi165 ceph-mon[32696]: from='mgr.14150 v1:172.21.15.63:0/3130147487' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-16T10:14:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:02 smithi165 ceph-mon[32696]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-16T10:14:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:04.781 INFO:teuthology.orchestra.run.smithi063.stdout:231928233993 2024-09-16T10:14:04.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-16T10:14:05.510 INFO:teuthology.orchestra.run.smithi063.stdout:55834574886 2024-09-16T10:14:05.510 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-16T10:14:06.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[29993]: Standby manager daemon x restarted 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[29993]: Standby manager daemon x started 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[29993]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[39470]: Standby manager daemon x restarted 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[39470]: Standby manager daemon x started 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T10:14:06.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T10:14:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:06 smithi063 ceph-mon[39470]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T10:14:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-mon[32696]: Standby manager daemon x restarted 2024-09-16T10:14:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-mon[32696]: Standby manager daemon x started 2024-09-16T10:14:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T10:14:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T10:14:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T10:14:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-mon[32696]: from='mgr.? v1:172.21.15.165:0/2043122469' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T10:14:07.052 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: [16/Sep/2024:10:14:06] ENGINE Bus STARTING 2024-09-16T10:14:07.053 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: CherryPy Checker: 2024-09-16T10:14:07.053 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: The Application mounted at '' has an empty config. 2024-09-16T10:14:07.053 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: 2024-09-16T10:14:07.053 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: [16/Sep/2024:10:14:06] ENGINE Serving on http://:::9283 2024-09-16T10:14:07.053 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:14:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x[34202]: [16/Sep/2024:10:14:06] ENGINE Bus STARTED 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: Active manager daemon y restarted 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: Activating manager daemon y 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: osdmap e73: 8 total, 8 up, 8 in 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: mgrmap e19: y(active, starting, since 0.0103164s), standbys: x 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:14:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T10:14:07.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: Manager daemon y is now available 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:14:07.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: Active manager daemon y restarted 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: Activating manager daemon y 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: osdmap e73: 8 total, 8 up, 8 in 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: mgrmap e19: y(active, starting, since 0.0103164s), standbys: x 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:14:07.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: Manager daemon y is now available 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:07.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:14:07.957 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:14:07.957 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:14:07.957 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:14:08.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-16T10:14:08.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: Active manager daemon y restarted 2024-09-16T10:14:08.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: Activating manager daemon y 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: osdmap e73: 8 total, 8 up, 8 in 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: mgrmap e19: y(active, starting, since 0.0103164s), standbys: x 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T10:14:08.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: Manager daemon y is now available 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T10:14:08.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:14:08.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T10:14:08.085 INFO:teuthology.orchestra.run.smithi063.stdout:34359738412 2024-09-16T10:14:08.086 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-16T10:14:08.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:14:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:14:08.302 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:08 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:08] ENGINE Bus STARTING 2024-09-16T10:14:08.302 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:08 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: CherryPy Checker: 2024-09-16T10:14:08.302 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:08 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: The Application mounted at '' has an empty config. 2024-09-16T10:14:08.302 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:08 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:14:08.302 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:08 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:08] ENGINE Serving on http://:::9283 2024-09-16T10:14:08.303 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:08 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:08] ENGINE Bus STARTED 2024-09-16T10:14:09.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[29993]: mgrmap e20: y(active, since 1.01091s), standbys: x 2024-09-16T10:14:09.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[29993]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:09.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:09.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:09.172 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[39470]: mgrmap e20: y(active, since 1.01091s), standbys: x 2024-09-16T10:14:09.172 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[39470]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:09.172 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:09.172 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:09.198 INFO:teuthology.orchestra.run.smithi063.stdout:193273528334 2024-09-16T10:14:09.198 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-16T10:14:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:09 smithi165 ceph-mon[32696]: mgrmap e20: y(active, since 1.01091s), standbys: x 2024-09-16T10:14:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:09 smithi165 ceph-mon[32696]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:09 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:09.362 INFO:teuthology.orchestra.run.smithi063.stdout:158913789971 2024-09-16T10:14:09.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-16T10:14:09.993 INFO:teuthology.orchestra.run.smithi063.stdout:77309411363 2024-09-16T10:14:09.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-16T10:14:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:10 smithi165 ceph-mon[32696]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:10 smithi165 ceph-mon[32696]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-16T10:14:10.438 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-mon[29993]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:10.438 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-mon[29993]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-16T10:14:10.438 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-mon[39470]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:10.438 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-mon[39470]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-16T10:14:10.683 INFO:teuthology.orchestra.run.smithi063.stdout:103079215133 2024-09-16T10:14:10.683 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:14:10.691+0000 7f70d49ec640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:10:14:10] ENGINE Error in HTTPServer.serve 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: Traceback (most recent call last): 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._connections.run(self.expiration_interval) 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._run(expiration_interval) 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:14:10.952 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: s = self.context.wrap_socket( 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: return self.sslsocket_class._create( 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self.do_handshake() 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: self._sslobj.do_handshake() 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:14:10.953 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:14:11.092 INFO:teuthology.orchestra.run.smithi063.stdout:133143986200 2024-09-16T10:14:11.093 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-16T10:14:11.180 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:11.181 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:11.182 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:11.247 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:11.252 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:11.253 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: [16/Sep/2024:10:14:10] ENGINE Bus STARTING 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: osdmap e74: 8 total, 8 up, 8 in 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: [16/Sep/2024:10:14:10] ENGINE Bus STARTING 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: osdmap e74: 8 total, 8 up, 8 in 2024-09-16T10:14:11.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: [16/Sep/2024:10:14:10] ENGINE Bus STARTING 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: osdmap e74: 8 total, 8 up, 8 in 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:12.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[29993]: [16/Sep/2024:10:14:10] ENGINE Serving on https://172.21.15.63:7150 2024-09-16T10:14:12.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[29993]: [16/Sep/2024:10:14:10] ENGINE Error in HTTPServer.serve 2024-09-16T10:14:12.545 INFO:journalctl@ceph.mon.a.smithi063.stdout: Traceback (most recent call last): 2024-09-16T10:14:12.545 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:14:12.545 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._connections.run(self.expiration_interval) 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._run(expiration_interval) 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: s = self.context.wrap_socket( 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: return self.sslsocket_class._create( 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:14:12.546 INFO:journalctl@ceph.mon.a.smithi063.stdout: self.do_handshake() 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout: self._sslobj.do_handshake() 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[29993]: [16/Sep/2024:10:14:10] ENGINE Serving on http://172.21.15.63:8765 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[29993]: [16/Sep/2024:10:14:10] ENGINE Bus STARTED 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi165 to 3760M 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[29993]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[29993]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[39470]: [16/Sep/2024:10:14:10] ENGINE Serving on https://172.21.15.63:7150 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[39470]: [16/Sep/2024:10:14:10] ENGINE Error in HTTPServer.serve 2024-09-16T10:14:12.547 INFO:journalctl@ceph.mon.c.smithi063.stdout: Traceback (most recent call last): 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: self._connections.run(self.expiration_interval) 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: self._run(expiration_interval) 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: s = self.context.wrap_socket( 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: return self.sslsocket_class._create( 2024-09-16T10:14:12.548 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout: self.do_handshake() 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout: self._sslobj.do_handshake() 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[39470]: [16/Sep/2024:10:14:10] ENGINE Serving on http://172.21.15.63:8765 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[39470]: [16/Sep/2024:10:14:10] ENGINE Bus STARTED 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi165 to 3760M 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[39470]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:12.549 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:12 smithi063 ceph-mon[39470]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-16T10:14:12.726 INFO:teuthology.orchestra.run.smithi063.stdout:158913789972 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:12 smithi165 ceph-mon[32696]: [16/Sep/2024:10:14:10] ENGINE Serving on https://172.21.15.63:7150 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:12 smithi165 ceph-mon[32696]: [16/Sep/2024:10:14:10] ENGINE Error in HTTPServer.serve 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: Traceback (most recent call last): 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._connections.run(self.expiration_interval) 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._run(expiration_interval) 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T10:14:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: s = self.context.wrap_socket( 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: return self.sslsocket_class._create( 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: self.do_handshake() 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._sslobj.do_handshake() 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:12 smithi165 ceph-mon[32696]: [16/Sep/2024:10:14:10] ENGINE Serving on http://172.21.15.63:8765 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:12 smithi165 ceph-mon[32696]: [16/Sep/2024:10:14:10] ENGINE Bus STARTED 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:12 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi165 to 3760M 2024-09-16T10:14:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:12 smithi165 ceph-mon[32696]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:12.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:12 smithi165 ceph-mon[32696]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-16T10:14:12.941 INFO:teuthology.orchestra.run.smithi063.stdout:193273528334 2024-09-16T10:14:13.633 INFO:teuthology.orchestra.run.smithi063.stdout:55834574888 2024-09-16T10:14:13.633 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1035028143' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T10:14:13.633 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/334880093' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T10:14:13.634 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1035028143' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T10:14:13.634 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/334880093' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T10:14:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1035028143' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T10:14:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/334880093' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T10:14:13.867 INFO:teuthology.orchestra.run.smithi063.stdout:103079215134 2024-09-16T10:14:14.084 INFO:teuthology.orchestra.run.smithi063.stdout:231928233995 2024-09-16T10:14:14.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[29993]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-16T10:14:14.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3265034550' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T10:14:14.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2617846597' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T10:14:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2699127446' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T10:14:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[39470]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-16T10:14:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3265034550' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T10:14:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2617846597' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T10:14:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2699127446' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T10:14:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:14 smithi165 ceph-mon[32696]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-16T10:14:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3265034550' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T10:14:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2617846597' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T10:14:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2699127446' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T10:14:14.841 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789971 got 158913789972 for osd.5 2024-09-16T10:14:14.841 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:15.563 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:15 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:14:15.563 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:15 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:14:15.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:15 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:14:15.563 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:15 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:14:15.598 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233993 got 231928233995 for osd.7 2024-09-16T10:14:15.598 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:15.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:15 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:14:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:15 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:14:16.003 INFO:teuthology.orchestra.run.smithi063.stdout:34359738414 2024-09-16T10:14:16.218 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528334 got 193273528334 for osd.6 2024-09-16T10:14:16.218 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:16.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[29993]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T10:14:16.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:14:16.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:14:16.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3503561593' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T10:14:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[39470]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T10:14:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:14:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:14:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:16 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3503561593' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T10:14:16.782 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738412 got 34359738414 for osd.0 2024-09-16T10:14:16.782 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:16 smithi165 ceph-mon[32696]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T10:14:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:16 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:14:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:16 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:14:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:16 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3503561593' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T10:14:17.383 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:14:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:14:17.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574886 got 55834574888 for osd.1 2024-09-16T10:14:17.436 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:17.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:17 smithi063 ceph-mon[29993]: osdmap e75: 8 total, 8 up, 8 in 2024-09-16T10:14:17.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:17 smithi063 ceph-mon[39470]: osdmap e75: 8 total, 8 up, 8 in 2024-09-16T10:14:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:17 smithi165 ceph-mon[32696]: osdmap e75: 8 total, 8 up, 8 in 2024-09-16T10:14:17.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215133 got 103079215134 for osd.3 2024-09-16T10:14:17.852 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:17.984 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:17.985 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:18.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:14:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:14:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:14:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:14:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[29993]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T10:14:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[29993]: osdmap e76: 8 total, 8 up, 8 in 2024-09-16T10:14:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:14:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:14:18.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:14:18.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:14:18.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[39470]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T10:14:18.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[39470]: osdmap e76: 8 total, 8 up, 8 in 2024-09-16T10:14:18.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:14:18.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:18 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:14:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:18 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:14:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:18 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:14:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:18 smithi165 ceph-mon[32696]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T10:14:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:18 smithi165 ceph-mon[32696]: osdmap e76: 8 total, 8 up, 8 in 2024-09-16T10:14:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:18 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:14:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:18 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:14:19.242 INFO:teuthology.orchestra.run.smithi063.stdout:133143986202 2024-09-16T10:14:19.486 INFO:teuthology.orchestra.run.smithi063.stdout:77309411366 2024-09-16T10:14:19.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:19 smithi063 ceph-mon[29993]: osdmap e77: 8 total, 8 up, 8 in 2024-09-16T10:14:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1027134296' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T10:14:19.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:19.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:19 smithi063 ceph-mon[39470]: osdmap e77: 8 total, 8 up, 8 in 2024-09-16T10:14:19.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1027134296' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T10:14:19.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:19 smithi165 ceph-mon[32696]: osdmap e77: 8 total, 8 up, 8 in 2024-09-16T10:14:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1027134296' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T10:14:20.026 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986200 got 133143986202 for osd.4 2024-09-16T10:14:20.026 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:20.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:20 smithi063 ceph-mon[39470]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:20.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/822925021' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T10:14:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:20 smithi063 ceph-mon[29993]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/822925021' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T10:14:20.751 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411366 for osd.2 2024-09-16T10:14:20.752 DEBUG:teuthology.parallel:result is None 2024-09-16T10:14:20.752 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T10:14:20.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:14:20.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:20 smithi165 ceph-mon[32696]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/822925021' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T10:14:21.038 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:21.821 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:14:21.823 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:14:21.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[39470]: osdmap e78: 8 total, 8 up, 8 in 2024-09-16T10:14:21.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[39470]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:14:21.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:21.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:21.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[29993]: osdmap e78: 8 total, 8 up, 8 in 2024-09-16T10:14:21.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[29993]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:14:21.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:21.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:21 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:21 smithi165 ceph-mon[32696]: osdmap e78: 8 total, 8 up, 8 in 2024-09-16T10:14:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:21 smithi165 ceph-mon[32696]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:14:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:21 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:21 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.679 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-16T10:14:21.008884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":883,"num_read_kb":740,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240632,"kb_used_data":7764,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720584,"statfs":{"total":767960285184,"available":767713878016,"internally_reserved":0,"allocated":7950336,"data_stored":4226653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.636917"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428258+0000","last_change":"2024-09-16T10:13:37.096999+0000","last_active":"2024-09-16T10:14:16.428258+0000","last_peered":"2024-09-16T10:14:16.428258+0000","last_clean":"2024-09-16T10:14:16.428258+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:14:16.428258+0000","last_undegraded":"2024-09-16T10:14:16.428258+0000","last_fullsized":"2024-09-16T10:14:16.428258+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383251+0000","last_change":"2024-09-16T10:13:39.093832+0000","last_active":"2024-09-16T10:14:17.383251+0000","last_peered":"2024-09-16T10:14:17.383251+0000","last_clean":"2024-09-16T10:14:17.383251+0000","last_became_active":"2024-09-16T10:13:39.093430+0000","last_became_peered":"2024-09-16T10:13:39.093430+0000","last_unstale":"2024-09-16T10:14:17.383251+0000","last_undegraded":"2024-09-16T10:14:17.383251+0000","last_fullsized":"2024-09-16T10:14:17.383251+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.941021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383719+0000","last_change":"2024-09-16T10:13:41.108286+0000","last_active":"2024-09-16T10:14:17.383719+0000","last_peered":"2024-09-16T10:14:17.383719+0000","last_clean":"2024-09-16T10:14:17.383719+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:14:17.383719+0000","last_undegraded":"2024-09-16T10:14:17.383719+0000","last_fullsized":"2024-09-16T10:14:17.383719+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.804825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383411+0000","last_change":"2024-09-16T10:13:43.113148+0000","last_active":"2024-09-16T10:14:17.383411+0000","last_peered":"2024-09-16T10:14:17.383411+0000","last_clean":"2024-09-16T10:14:17.383411+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:14:17.383411+0000","last_undegraded":"2024-09-16T10:14:17.383411+0000","last_fullsized":"2024-09-16T10:14:17.383411+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.390079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887558+0000","last_change":"2024-09-16T10:13:43.287323+0000","last_active":"2024-09-16T10:14:17.887558+0000","last_peered":"2024-09-16T10:14:17.887558+0000","last_clean":"2024-09-16T10:14:17.887558+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:14:17.887558+0000","last_undegraded":"2024-09-16T10:14:17.887558+0000","last_fullsized":"2024-09-16T10:14:17.887558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.723245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888312+0000","last_change":"2024-09-16T10:13:37.092457+0000","last_active":"2024-09-16T10:14:17.888312+0000","last_peered":"2024-09-16T10:14:17.888312+0000","last_clean":"2024-09-16T10:14:17.888312+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:14:17.888312+0000","last_undegraded":"2024-09-16T10:14:17.888312+0000","last_fullsized":"2024-09-16T10:14:17.888312+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.920553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887715+0000","last_change":"2024-09-16T10:13:39.095829+0000","last_active":"2024-09-16T10:14:17.887715+0000","last_peered":"2024-09-16T10:14:17.887715+0000","last_clean":"2024-09-16T10:14:17.887715+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:14:17.887715+0000","last_undegraded":"2024-09-16T10:14:17.887715+0000","last_fullsized":"2024-09-16T10:14:17.887715+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.084583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383241+0000","last_change":"2024-09-16T10:13:41.121501+0000","last_active":"2024-09-16T10:14:17.383241+0000","last_peered":"2024-09-16T10:14:17.383241+0000","last_clean":"2024-09-16T10:14:17.383241+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:14:17.383241+0000","last_undegraded":"2024-09-16T10:14:17.383241+0000","last_fullsized":"2024-09-16T10:14:17.383241+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.386931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429359+0000","last_change":"2024-09-16T10:13:37.092847+0000","last_active":"2024-09-16T10:14:16.429359+0000","last_peered":"2024-09-16T10:14:16.429359+0000","last_clean":"2024-09-16T10:14:16.429359+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:14:16.429359+0000","last_undegraded":"2024-09-16T10:14:16.429359+0000","last_fullsized":"2024-09-16T10:14:16.429359+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.507863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383808+0000","last_change":"2024-09-16T10:13:39.092562+0000","last_active":"2024-09-16T10:14:17.383808+0000","last_peered":"2024-09-16T10:14:17.383808+0000","last_clean":"2024-09-16T10:14:17.383808+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:14:17.383808+0000","last_undegraded":"2024-09-16T10:14:17.383808+0000","last_fullsized":"2024-09-16T10:14:17.383808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.237966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428882+0000","last_change":"2024-09-16T10:13:41.112242+0000","last_active":"2024-09-16T10:14:16.428882+0000","last_peered":"2024-09-16T10:14:16.428882+0000","last_clean":"2024-09-16T10:14:16.428882+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:14:16.428882+0000","last_undegraded":"2024-09-16T10:14:16.428882+0000","last_fullsized":"2024-09-16T10:14:16.428882+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.338353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427492+0000","last_change":"2024-09-16T10:13:43.115780+0000","last_active":"2024-09-16T10:14:16.427492+0000","last_peered":"2024-09-16T10:14:16.427492+0000","last_clean":"2024-09-16T10:14:16.427492+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:14:16.427492+0000","last_undegraded":"2024-09-16T10:14:16.427492+0000","last_fullsized":"2024-09-16T10:14:16.427492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.438318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429062+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:14:16.429062+0000","last_peered":"2024-09-16T10:14:16.429062+0000","last_clean":"2024-09-16T10:14:16.429062+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:14:16.429062+0000","last_undegraded":"2024-09-16T10:14:16.429062+0000","last_fullsized":"2024-09-16T10:14:16.429062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.318174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383482+0000","last_change":"2024-09-16T10:13:39.092696+0000","last_active":"2024-09-16T10:14:17.383482+0000","last_peered":"2024-09-16T10:14:17.383482+0000","last_clean":"2024-09-16T10:14:17.383482+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:14:17.383482+0000","last_undegraded":"2024-09-16T10:14:17.383482+0000","last_fullsized":"2024-09-16T10:14:17.383482+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.518434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428343+0000","last_change":"2024-09-16T10:13:41.111640+0000","last_active":"2024-09-16T10:14:16.428343+0000","last_peered":"2024-09-16T10:14:16.428343+0000","last_clean":"2024-09-16T10:14:16.428343+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:14:16.428343+0000","last_undegraded":"2024-09-16T10:14:16.428343+0000","last_fullsized":"2024-09-16T10:14:16.428343+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.280028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428141+0000","last_change":"2024-09-16T10:13:43.121701+0000","last_active":"2024-09-16T10:14:16.428141+0000","last_peered":"2024-09-16T10:14:16.428141+0000","last_clean":"2024-09-16T10:14:16.428141+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:14:16.428141+0000","last_undegraded":"2024-09-16T10:14:16.428141+0000","last_fullsized":"2024-09-16T10:14:16.428141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.188718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384100+0000","last_change":"2024-09-16T10:13:43.284208+0000","last_active":"2024-09-16T10:14:17.384100+0000","last_peered":"2024-09-16T10:14:17.384100+0000","last_clean":"2024-09-16T10:14:17.384100+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:14:17.384100+0000","last_undegraded":"2024-09-16T10:14:17.384100+0000","last_fullsized":"2024-09-16T10:14:17.384100+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.053189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428963+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:14:16.428963+0000","last_peered":"2024-09-16T10:14:16.428963+0000","last_clean":"2024-09-16T10:14:16.428963+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:14:16.428963+0000","last_undegraded":"2024-09-16T10:14:16.428963+0000","last_fullsized":"2024-09-16T10:14:16.428963+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.063039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.385605+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:14:17.385605+0000","last_peered":"2024-09-16T10:14:17.385605+0000","last_clean":"2024-09-16T10:14:17.385605+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:14:17.385605+0000","last_undegraded":"2024-09-16T10:14:17.385605+0000","last_fullsized":"2024-09-16T10:14:17.385605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.478889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384440+0000","last_change":"2024-09-16T10:13:41.120296+0000","last_active":"2024-09-16T10:14:17.384440+0000","last_peered":"2024-09-16T10:14:17.384440+0000","last_clean":"2024-09-16T10:14:17.384440+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:14:17.384440+0000","last_undegraded":"2024-09-16T10:14:17.384440+0000","last_fullsized":"2024-09-16T10:14:17.384440+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.391104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888196+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:14:17.888196+0000","last_peered":"2024-09-16T10:14:17.888196+0000","last_clean":"2024-09-16T10:14:17.888196+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:14:17.888196+0000","last_undegraded":"2024-09-16T10:14:17.888196+0000","last_fullsized":"2024-09-16T10:14:17.888196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.407109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384045+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:14:17.384045+0000","last_peered":"2024-09-16T10:14:17.384045+0000","last_clean":"2024-09-16T10:14:17.384045+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:14:17.384045+0000","last_undegraded":"2024-09-16T10:14:17.384045+0000","last_fullsized":"2024-09-16T10:14:17.384045+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.861513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888066+0000","last_change":"2024-09-16T10:13:39.100013+0000","last_active":"2024-09-16T10:14:17.888066+0000","last_peered":"2024-09-16T10:14:17.888066+0000","last_clean":"2024-09-16T10:14:17.888066+0000","last_became_active":"2024-09-16T10:13:39.099675+0000","last_became_peered":"2024-09-16T10:13:39.099675+0000","last_unstale":"2024-09-16T10:14:17.888066+0000","last_undegraded":"2024-09-16T10:14:17.888066+0000","last_fullsized":"2024-09-16T10:14:17.888066+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.654393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383664+0000","last_change":"2024-09-16T10:13:41.112993+0000","last_active":"2024-09-16T10:14:17.383664+0000","last_peered":"2024-09-16T10:14:17.383664+0000","last_clean":"2024-09-16T10:14:17.383664+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:14:17.383664+0000","last_undegraded":"2024-09-16T10:14:17.383664+0000","last_fullsized":"2024-09-16T10:14:17.383664+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.116156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":33,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428636+0000","last_change":"2024-09-16T10:13:43.119664+0000","last_active":"2024-09-16T10:14:16.428636+0000","last_peered":"2024-09-16T10:14:16.428636+0000","last_clean":"2024-09-16T10:14:16.428636+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:14:16.428636+0000","last_undegraded":"2024-09-16T10:14:16.428636+0000","last_fullsized":"2024-09-16T10:14:16.428636+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.178009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384040+0000","last_change":"2024-09-16T10:13:37.082594+0000","last_active":"2024-09-16T10:14:17.384040+0000","last_peered":"2024-09-16T10:14:17.384040+0000","last_clean":"2024-09-16T10:14:17.384040+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:14:17.384040+0000","last_undegraded":"2024-09-16T10:14:17.384040+0000","last_fullsized":"2024-09-16T10:14:17.384040+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.835656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":55,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428683+0000","last_change":"2024-09-16T10:13:39.108076+0000","last_active":"2024-09-16T10:14:16.428683+0000","last_peered":"2024-09-16T10:14:16.428683+0000","last_clean":"2024-09-16T10:14:16.428683+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:14:16.428683+0000","last_undegraded":"2024-09-16T10:14:16.428683+0000","last_fullsized":"2024-09-16T10:14:16.428683+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.726907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.430129+0000","last_change":"2024-09-16T10:13:41.116955+0000","last_active":"2024-09-16T10:14:16.430129+0000","last_peered":"2024-09-16T10:14:16.430129+0000","last_clean":"2024-09-16T10:14:16.430129+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:14:16.430129+0000","last_undegraded":"2024-09-16T10:14:16.430129+0000","last_fullsized":"2024-09-16T10:14:16.430129+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.203272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384749+0000","last_change":"2024-09-16T10:13:39.096753+0000","last_active":"2024-09-16T10:14:17.384749+0000","last_peered":"2024-09-16T10:14:17.384749+0000","last_clean":"2024-09-16T10:14:17.384749+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:14:17.384749+0000","last_undegraded":"2024-09-16T10:14:17.384749+0000","last_fullsized":"2024-09-16T10:14:17.384749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.363335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887635+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:14:17.887635+0000","last_peered":"2024-09-16T10:14:17.887635+0000","last_clean":"2024-09-16T10:14:17.887635+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:14:17.887635+0000","last_undegraded":"2024-09-16T10:14:17.887635+0000","last_fullsized":"2024-09-16T10:14:17.887635+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.355614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384205+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:14:17.384205+0000","last_peered":"2024-09-16T10:14:17.384205+0000","last_clean":"2024-09-16T10:14:17.384205+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:14:17.384205+0000","last_undegraded":"2024-09-16T10:14:17.384205+0000","last_fullsized":"2024-09-16T10:14:17.384205+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.675105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428417+0000","last_change":"2024-09-16T10:13:43.114886+0000","last_active":"2024-09-16T10:14:16.428417+0000","last_peered":"2024-09-16T10:14:16.428417+0000","last_clean":"2024-09-16T10:14:16.428417+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:14:16.428417+0000","last_undegraded":"2024-09-16T10:14:16.428417+0000","last_fullsized":"2024-09-16T10:14:16.428417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.234184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":74,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887416+0000","last_change":"2024-09-16T10:13:39.106135+0000","last_active":"2024-09-16T10:14:17.887416+0000","last_peered":"2024-09-16T10:14:17.887416+0000","last_clean":"2024-09-16T10:14:17.887416+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:14:17.887416+0000","last_undegraded":"2024-09-16T10:14:17.887416+0000","last_fullsized":"2024-09-16T10:14:17.887416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.184956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888414+0000","last_change":"2024-09-16T10:13:37.093660+0000","last_active":"2024-09-16T10:14:17.888414+0000","last_peered":"2024-09-16T10:14:17.888414+0000","last_clean":"2024-09-16T10:14:17.888414+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:14:17.888414+0000","last_undegraded":"2024-09-16T10:14:17.888414+0000","last_fullsized":"2024-09-16T10:14:17.888414+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.816759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383308+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:14:17.383308+0000","last_peered":"2024-09-16T10:14:17.383308+0000","last_clean":"2024-09-16T10:14:17.383308+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:14:17.383308+0000","last_undegraded":"2024-09-16T10:14:17.383308+0000","last_fullsized":"2024-09-16T10:14:17.383308+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.743002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383873+0000","last_change":"2024-09-16T10:13:43.122864+0000","last_active":"2024-09-16T10:14:17.383873+0000","last_peered":"2024-09-16T10:14:17.383873+0000","last_clean":"2024-09-16T10:14:17.383873+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:14:17.383873+0000","last_undegraded":"2024-09-16T10:14:17.383873+0000","last_fullsized":"2024-09-16T10:14:17.383873+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.235493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383318+0000","last_change":"2024-09-16T10:13:39.094074+0000","last_active":"2024-09-16T10:14:17.383318+0000","last_peered":"2024-09-16T10:14:17.383318+0000","last_clean":"2024-09-16T10:14:17.383318+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:14:17.383318+0000","last_undegraded":"2024-09-16T10:14:17.383318+0000","last_fullsized":"2024-09-16T10:14:17.383318+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.532032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":45,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428973+0000","last_change":"2024-09-16T10:13:37.096046+0000","last_active":"2024-09-16T10:14:16.428973+0000","last_peered":"2024-09-16T10:14:16.428973+0000","last_clean":"2024-09-16T10:14:16.428973+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:14:16.428973+0000","last_undegraded":"2024-09-16T10:14:16.428973+0000","last_fullsized":"2024-09-16T10:14:16.428973+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.593585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887292+0000","last_change":"2024-09-16T10:13:41.112759+0000","last_active":"2024-09-16T10:14:17.887292+0000","last_peered":"2024-09-16T10:14:17.887292+0000","last_clean":"2024-09-16T10:14:17.887292+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:14:17.887292+0000","last_undegraded":"2024-09-16T10:14:17.887292+0000","last_fullsized":"2024-09-16T10:14:17.887292+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.600065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428730+0000","last_change":"2024-09-16T10:13:43.284631+0000","last_active":"2024-09-16T10:14:16.428730+0000","last_peered":"2024-09-16T10:14:16.428730+0000","last_clean":"2024-09-16T10:14:16.428730+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:14:16.428730+0000","last_undegraded":"2024-09-16T10:14:16.428730+0000","last_fullsized":"2024-09-16T10:14:16.428730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.200363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384064+0000","last_change":"2024-09-16T10:13:39.095801+0000","last_active":"2024-09-16T10:14:17.384064+0000","last_peered":"2024-09-16T10:14:17.384064+0000","last_clean":"2024-09-16T10:14:17.384064+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:14:17.384064+0000","last_undegraded":"2024-09-16T10:14:17.384064+0000","last_fullsized":"2024-09-16T10:14:17.384064+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.537230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429442+0000","last_change":"2024-09-16T10:13:37.093423+0000","last_active":"2024-09-16T10:14:16.429442+0000","last_peered":"2024-09-16T10:14:16.429442+0000","last_clean":"2024-09-16T10:14:16.429442+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:14:16.429442+0000","last_undegraded":"2024-09-16T10:14:16.429442+0000","last_fullsized":"2024-09-16T10:14:16.429442+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.178620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428654+0000","last_change":"2024-09-16T10:13:41.115177+0000","last_active":"2024-09-16T10:14:16.428654+0000","last_peered":"2024-09-16T10:14:16.428654+0000","last_clean":"2024-09-16T10:14:16.428654+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:14:16.428654+0000","last_undegraded":"2024-09-16T10:14:16.428654+0000","last_fullsized":"2024-09-16T10:14:16.428654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.803669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427590+0000","last_change":"2024-09-16T10:13:43.119055+0000","last_active":"2024-09-16T10:14:16.427590+0000","last_peered":"2024-09-16T10:14:16.427590+0000","last_clean":"2024-09-16T10:14:16.427590+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:14:16.427590+0000","last_undegraded":"2024-09-16T10:14:16.427590+0000","last_fullsized":"2024-09-16T10:14:16.427590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:54.513879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":76,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428819+0000","last_change":"2024-09-16T10:13:39.106153+0000","last_active":"2024-09-16T10:14:16.428819+0000","last_peered":"2024-09-16T10:14:16.428819+0000","last_clean":"2024-09-16T10:14:16.428819+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:14:16.428819+0000","last_undegraded":"2024-09-16T10:14:16.428819+0000","last_fullsized":"2024-09-16T10:14:16.428819+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.043307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384185+0000","last_change":"2024-09-16T10:13:37.092754+0000","last_active":"2024-09-16T10:14:17.384185+0000","last_peered":"2024-09-16T10:14:17.384185+0000","last_clean":"2024-09-16T10:14:17.384185+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:14:17.384185+0000","last_undegraded":"2024-09-16T10:14:17.384185+0000","last_fullsized":"2024-09-16T10:14:17.384185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.036476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428010+0000","last_change":"2024-09-16T10:13:41.114703+0000","last_active":"2024-09-16T10:14:16.428010+0000","last_peered":"2024-09-16T10:14:16.428010+0000","last_clean":"2024-09-16T10:14:16.428010+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:14:16.428010+0000","last_undegraded":"2024-09-16T10:14:16.428010+0000","last_fullsized":"2024-09-16T10:14:16.428010+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.640025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383606+0000","last_change":"2024-09-16T10:13:43.283556+0000","last_active":"2024-09-16T10:14:17.383606+0000","last_peered":"2024-09-16T10:14:17.383606+0000","last_clean":"2024-09-16T10:14:17.383606+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:14:17.383606+0000","last_undegraded":"2024-09-16T10:14:17.383606+0000","last_fullsized":"2024-09-16T10:14:17.383606+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:52.096829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":99,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384036+0000","last_change":"2024-09-16T10:13:39.098696+0000","last_active":"2024-09-16T10:14:17.384036+0000","last_peered":"2024-09-16T10:14:17.384036+0000","last_clean":"2024-09-16T10:14:17.384036+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:14:17.384036+0000","last_undegraded":"2024-09-16T10:14:17.384036+0000","last_fullsized":"2024-09-16T10:14:17.384036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.501760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383756+0000","last_change":"2024-09-16T10:13:37.092585+0000","last_active":"2024-09-16T10:14:17.383756+0000","last_peered":"2024-09-16T10:14:17.383756+0000","last_clean":"2024-09-16T10:14:17.383756+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:14:17.383756+0000","last_undegraded":"2024-09-16T10:14:17.383756+0000","last_fullsized":"2024-09-16T10:14:17.383756+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.123070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":54,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428823+0000","last_change":"2024-09-16T10:13:39.091269+0000","last_active":"2024-09-16T10:14:16.428823+0000","last_peered":"2024-09-16T10:14:16.428823+0000","last_clean":"2024-09-16T10:14:16.428823+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:14:16.428823+0000","last_undegraded":"2024-09-16T10:14:16.428823+0000","last_fullsized":"2024-09-16T10:14:16.428823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.638736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427962+0000","last_change":"2024-09-16T10:13:41.118142+0000","last_active":"2024-09-16T10:14:16.427962+0000","last_peered":"2024-09-16T10:14:16.427962+0000","last_clean":"2024-09-16T10:14:16.427962+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:14:16.427962+0000","last_undegraded":"2024-09-16T10:14:16.427962+0000","last_fullsized":"2024-09-16T10:14:16.427962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.463063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888019+0000","last_change":"2024-09-16T10:13:43.124755+0000","last_active":"2024-09-16T10:14:17.888019+0000","last_peered":"2024-09-16T10:14:17.888019+0000","last_clean":"2024-09-16T10:14:17.888019+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:14:17.888019+0000","last_undegraded":"2024-09-16T10:14:17.888019+0000","last_fullsized":"2024-09-16T10:14:17.888019+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.479324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":69,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384860+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:14:17.384860+0000","last_peered":"2024-09-16T10:14:17.384860+0000","last_clean":"2024-09-16T10:14:17.384860+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:14:17.384860+0000","last_undegraded":"2024-09-16T10:14:17.384860+0000","last_fullsized":"2024-09-16T10:14:17.384860+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.987015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384042+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:14:17.384042+0000","last_peered":"2024-09-16T10:14:17.384042+0000","last_clean":"2024-09-16T10:14:17.384042+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:14:17.384042+0000","last_undegraded":"2024-09-16T10:14:17.384042+0000","last_fullsized":"2024-09-16T10:14:17.384042+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384919+0000","last_change":"2024-09-16T10:13:39.090453+0000","last_active":"2024-09-16T10:14:17.384919+0000","last_peered":"2024-09-16T10:14:17.384919+0000","last_clean":"2024-09-16T10:14:17.384919+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:14:17.384919+0000","last_undegraded":"2024-09-16T10:14:17.384919+0000","last_fullsized":"2024-09-16T10:14:17.384919+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.048019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.385343+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:14:17.385343+0000","last_peered":"2024-09-16T10:14:17.385343+0000","last_clean":"2024-09-16T10:14:17.385343+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:14:17.385343+0000","last_undegraded":"2024-09-16T10:14:17.385343+0000","last_fullsized":"2024-09-16T10:14:17.385343+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.485969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429194+0000","last_change":"2024-09-16T10:13:43.283490+0000","last_active":"2024-09-16T10:14:16.429194+0000","last_peered":"2024-09-16T10:14:16.429194+0000","last_clean":"2024-09-16T10:14:16.429194+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:14:16.429194+0000","last_undegraded":"2024-09-16T10:14:16.429194+0000","last_fullsized":"2024-09-16T10:14:16.429194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.489088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":58,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428774+0000","last_change":"2024-09-16T10:13:39.091817+0000","last_active":"2024-09-16T10:14:16.428774+0000","last_peered":"2024-09-16T10:14:16.428774+0000","last_clean":"2024-09-16T10:14:16.428774+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:14:16.428774+0000","last_undegraded":"2024-09-16T10:14:16.428774+0000","last_fullsized":"2024-09-16T10:14:16.428774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.559454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427899+0000","last_change":"2024-09-16T10:13:37.096771+0000","last_active":"2024-09-16T10:14:16.427899+0000","last_peered":"2024-09-16T10:14:16.427899+0000","last_clean":"2024-09-16T10:14:16.427899+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:14:16.427899+0000","last_undegraded":"2024-09-16T10:14:16.427899+0000","last_fullsized":"2024-09-16T10:14:16.427899+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.458372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":90,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429223+0000","last_change":"2024-09-16T10:13:39.105362+0000","last_active":"2024-09-16T10:14:16.429223+0000","last_peered":"2024-09-16T10:14:16.429223+0000","last_clean":"2024-09-16T10:14:16.429223+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:14:16.429223+0000","last_undegraded":"2024-09-16T10:14:16.429223+0000","last_fullsized":"2024-09-16T10:14:16.429223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:35.154849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429616+0000","last_change":"2024-09-16T10:13:41.112756+0000","last_active":"2024-09-16T10:14:16.429616+0000","last_peered":"2024-09-16T10:14:16.429616+0000","last_clean":"2024-09-16T10:14:16.429616+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:14:16.429616+0000","last_undegraded":"2024-09-16T10:14:16.429616+0000","last_fullsized":"2024-09-16T10:14:16.429616+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.034690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383698+0000","last_change":"2024-09-16T10:13:43.123048+0000","last_active":"2024-09-16T10:14:17.383698+0000","last_peered":"2024-09-16T10:14:17.383698+0000","last_clean":"2024-09-16T10:14:17.383698+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:14:17.383698+0000","last_undegraded":"2024-09-16T10:14:17.383698+0000","last_fullsized":"2024-09-16T10:14:17.383698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.340209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":71,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.430188+0000","last_change":"2024-09-16T10:13:39.106378+0000","last_active":"2024-09-16T10:14:16.430188+0000","last_peered":"2024-09-16T10:14:16.430188+0000","last_clean":"2024-09-16T10:14:16.430188+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:14:16.430188+0000","last_undegraded":"2024-09-16T10:14:16.430188+0000","last_fullsized":"2024-09-16T10:14:16.430188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.582337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887735+0000","last_change":"2024-09-16T10:13:37.088320+0000","last_active":"2024-09-16T10:14:17.887735+0000","last_peered":"2024-09-16T10:14:17.887735+0000","last_clean":"2024-09-16T10:14:17.887735+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:14:17.887735+0000","last_undegraded":"2024-09-16T10:14:17.887735+0000","last_fullsized":"2024-09-16T10:14:17.887735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:37.142506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":89,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429377+0000","last_change":"2024-09-16T10:14:14.911115+0000","last_active":"2024-09-16T10:14:16.429377+0000","last_peered":"2024-09-16T10:14:16.429377+0000","last_clean":"2024-09-16T10:14:16.429377+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:14:16.429377+0000","last_undegraded":"2024-09-16T10:14:16.429377+0000","last_fullsized":"2024-09-16T10:14:16.429377+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:14:14.911041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:14:14.911041+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:17.171811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428788+0000","last_change":"2024-09-16T10:13:41.108176+0000","last_active":"2024-09-16T10:14:16.428788+0000","last_peered":"2024-09-16T10:14:16.428788+0000","last_clean":"2024-09-16T10:14:16.428788+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:14:16.428788+0000","last_undegraded":"2024-09-16T10:14:16.428788+0000","last_fullsized":"2024-09-16T10:14:16.428788+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.887140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429604+0000","last_change":"2024-09-16T10:13:43.121658+0000","last_active":"2024-09-16T10:14:16.429604+0000","last_peered":"2024-09-16T10:14:16.429604+0000","last_clean":"2024-09-16T10:14:16.429604+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:14:16.429604+0000","last_undegraded":"2024-09-16T10:14:16.429604+0000","last_fullsized":"2024-09-16T10:14:16.429604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:11.237055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384244+0000","last_change":"2024-09-16T10:13:39.094951+0000","last_active":"2024-09-16T10:14:17.384244+0000","last_peered":"2024-09-16T10:14:17.384244+0000","last_clean":"2024-09-16T10:14:17.384244+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:14:17.384244+0000","last_undegraded":"2024-09-16T10:14:17.384244+0000","last_fullsized":"2024-09-16T10:14:17.384244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.179189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383637+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:14:17.383637+0000","last_peered":"2024-09-16T10:14:17.383637+0000","last_clean":"2024-09-16T10:14:17.383637+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:14:17.383637+0000","last_undegraded":"2024-09-16T10:14:17.383637+0000","last_fullsized":"2024-09-16T10:14:17.383637+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.958295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429380+0000","last_change":"2024-09-16T10:13:41.116069+0000","last_active":"2024-09-16T10:14:16.429380+0000","last_peered":"2024-09-16T10:14:16.429380+0000","last_clean":"2024-09-16T10:14:16.429380+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:14:16.429380+0000","last_undegraded":"2024-09-16T10:14:16.429380+0000","last_fullsized":"2024-09-16T10:14:16.429380+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.901053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887760+0000","last_change":"2024-09-16T10:13:43.283629+0000","last_active":"2024-09-16T10:14:17.887760+0000","last_peered":"2024-09-16T10:14:17.887760+0000","last_clean":"2024-09-16T10:14:17.887760+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:14:17.887760+0000","last_undegraded":"2024-09-16T10:14:17.887760+0000","last_fullsized":"2024-09-16T10:14:17.887760+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.176307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":70,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384135+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:14:17.384135+0000","last_peered":"2024-09-16T10:14:17.384135+0000","last_clean":"2024-09-16T10:14:17.384135+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:14:17.384135+0000","last_undegraded":"2024-09-16T10:14:17.384135+0000","last_fullsized":"2024-09-16T10:14:17.384135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.476261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428416+0000","last_change":"2024-09-16T10:13:37.089527+0000","last_active":"2024-09-16T10:14:16.428416+0000","last_peered":"2024-09-16T10:14:16.428416+0000","last_clean":"2024-09-16T10:14:16.428416+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:14:16.428416+0000","last_undegraded":"2024-09-16T10:14:16.428416+0000","last_fullsized":"2024-09-16T10:14:16.428416+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.605092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383799+0000","last_change":"2024-09-16T10:13:41.121389+0000","last_active":"2024-09-16T10:14:17.383799+0000","last_peered":"2024-09-16T10:14:17.383799+0000","last_clean":"2024-09-16T10:14:17.383799+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:14:17.383799+0000","last_undegraded":"2024-09-16T10:14:17.383799+0000","last_fullsized":"2024-09-16T10:14:17.383799+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.974031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384861+0000","last_change":"2024-09-16T10:13:43.125890+0000","last_active":"2024-09-16T10:14:17.384861+0000","last_peered":"2024-09-16T10:14:17.384861+0000","last_clean":"2024-09-16T10:14:17.384861+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:14:17.384861+0000","last_undegraded":"2024-09-16T10:14:17.384861+0000","last_fullsized":"2024-09-16T10:14:17.384861+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383763+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:14:17.383763+0000","last_peered":"2024-09-16T10:14:17.383763+0000","last_clean":"2024-09-16T10:14:17.383763+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:14:17.383763+0000","last_undegraded":"2024-09-16T10:14:17.383763+0000","last_fullsized":"2024-09-16T10:14:17.383763+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.021437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887956+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:14:17.887956+0000","last_peered":"2024-09-16T10:14:17.887956+0000","last_clean":"2024-09-16T10:14:17.887956+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:14:17.887956+0000","last_undegraded":"2024-09-16T10:14:17.887956+0000","last_fullsized":"2024-09-16T10:14:17.887956+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.609140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384588+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:14:17.384588+0000","last_peered":"2024-09-16T10:14:17.384588+0000","last_clean":"2024-09-16T10:14:17.384588+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:14:17.384588+0000","last_undegraded":"2024-09-16T10:14:17.384588+0000","last_fullsized":"2024-09-16T10:14:17.384588+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384091+0000","last_change":"2024-09-16T10:13:43.119606+0000","last_active":"2024-09-16T10:14:17.384091+0000","last_peered":"2024-09-16T10:14:17.384091+0000","last_clean":"2024-09-16T10:14:17.384091+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:14:17.384091+0000","last_undegraded":"2024-09-16T10:14:17.384091+0000","last_fullsized":"2024-09-16T10:14:17.384091+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.164444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428712+0000","last_change":"2024-09-16T10:13:39.091847+0000","last_active":"2024-09-16T10:14:16.428712+0000","last_peered":"2024-09-16T10:14:16.428712+0000","last_clean":"2024-09-16T10:14:16.428712+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:14:16.428712+0000","last_undegraded":"2024-09-16T10:14:16.428712+0000","last_fullsized":"2024-09-16T10:14:16.428712+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.954882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429127+0000","last_change":"2024-09-16T10:13:37.090259+0000","last_active":"2024-09-16T10:14:16.429127+0000","last_peered":"2024-09-16T10:14:16.429127+0000","last_clean":"2024-09-16T10:14:16.429127+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:14:16.429127+0000","last_undegraded":"2024-09-16T10:14:16.429127+0000","last_fullsized":"2024-09-16T10:14:16.429127+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.290444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.385077+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:14:17.385077+0000","last_peered":"2024-09-16T10:14:17.385077+0000","last_clean":"2024-09-16T10:14:17.385077+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:14:17.385077+0000","last_undegraded":"2024-09-16T10:14:17.385077+0000","last_fullsized":"2024-09-16T10:14:17.385077+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":33,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427797+0000","last_change":"2024-09-16T10:13:43.122627+0000","last_active":"2024-09-16T10:14:16.427797+0000","last_peered":"2024-09-16T10:14:16.427797+0000","last_clean":"2024-09-16T10:14:16.427797+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:14:16.427797+0000","last_undegraded":"2024-09-16T10:14:16.427797+0000","last_fullsized":"2024-09-16T10:14:16.427797+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":71,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428141+0000","last_change":"2024-09-16T10:13:39.105379+0000","last_active":"2024-09-16T10:14:16.428141+0000","last_peered":"2024-09-16T10:14:16.428141+0000","last_clean":"2024-09-16T10:14:16.428141+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:14:16.428141+0000","last_undegraded":"2024-09-16T10:14:16.428141+0000","last_fullsized":"2024-09-16T10:14:16.428141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.713974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429570+0000","last_change":"2024-09-16T10:13:37.093687+0000","last_active":"2024-09-16T10:14:16.429570+0000","last_peered":"2024-09-16T10:14:16.429570+0000","last_clean":"2024-09-16T10:14:16.429570+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:14:16.429570+0000","last_undegraded":"2024-09-16T10:14:16.429570+0000","last_fullsized":"2024-09-16T10:14:16.429570+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.984174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384426+0000","last_change":"2024-09-16T10:13:41.113511+0000","last_active":"2024-09-16T10:14:17.384426+0000","last_peered":"2024-09-16T10:14:17.384426+0000","last_clean":"2024-09-16T10:14:17.384426+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:14:17.384426+0000","last_undegraded":"2024-09-16T10:14:17.384426+0000","last_fullsized":"2024-09-16T10:14:17.384426+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.162031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429280+0000","last_change":"2024-09-16T10:13:43.122094+0000","last_active":"2024-09-16T10:14:16.429280+0000","last_peered":"2024-09-16T10:14:16.429280+0000","last_clean":"2024-09-16T10:14:16.429280+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:14:16.429280+0000","last_undegraded":"2024-09-16T10:14:16.429280+0000","last_fullsized":"2024-09-16T10:14:16.429280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.948353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383516+0000","last_change":"2024-09-16T10:13:39.095588+0000","last_active":"2024-09-16T10:14:17.383516+0000","last_peered":"2024-09-16T10:14:17.383516+0000","last_clean":"2024-09-16T10:14:17.383516+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:14:17.383516+0000","last_undegraded":"2024-09-16T10:14:17.383516+0000","last_fullsized":"2024-09-16T10:14:17.383516+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.533870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429047+0000","last_change":"2024-09-16T10:13:37.097257+0000","last_active":"2024-09-16T10:14:16.429047+0000","last_peered":"2024-09-16T10:14:16.429047+0000","last_clean":"2024-09-16T10:14:16.429047+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:14:16.429047+0000","last_undegraded":"2024-09-16T10:14:16.429047+0000","last_fullsized":"2024-09-16T10:14:16.429047+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.017319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384207+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:14:17.384207+0000","last_peered":"2024-09-16T10:14:17.384207+0000","last_clean":"2024-09-16T10:14:17.384207+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:14:17.384207+0000","last_undegraded":"2024-09-16T10:14:17.384207+0000","last_fullsized":"2024-09-16T10:14:17.384207+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.545235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887634+0000","last_change":"2024-09-16T10:13:43.125823+0000","last_active":"2024-09-16T10:14:17.887634+0000","last_peered":"2024-09-16T10:14:17.887634+0000","last_clean":"2024-09-16T10:14:17.887634+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:14:17.887634+0000","last_undegraded":"2024-09-16T10:14:17.887634+0000","last_fullsized":"2024-09-16T10:14:17.887634+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.646167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":65,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427991+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:14:16.427991+0000","last_peered":"2024-09-16T10:14:16.427991+0000","last_clean":"2024-09-16T10:14:16.427991+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:14:16.427991+0000","last_undegraded":"2024-09-16T10:14:16.427991+0000","last_fullsized":"2024-09-16T10:14:16.427991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.689518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429080+0000","last_change":"2024-09-16T10:13:37.092779+0000","last_active":"2024-09-16T10:14:16.429080+0000","last_peered":"2024-09-16T10:14:16.429080+0000","last_clean":"2024-09-16T10:14:16.429080+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:14:16.429080+0000","last_undegraded":"2024-09-16T10:14:16.429080+0000","last_fullsized":"2024-09-16T10:14:16.429080+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.367150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428556+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:16.428556+0000","last_peered":"2024-09-16T10:14:16.428556+0000","last_clean":"2024-09-16T10:14:16.428556+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:14:16.428556+0000","last_undegraded":"2024-09-16T10:14:16.428556+0000","last_fullsized":"2024-09-16T10:14:16.428556+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.594260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428476+0000","last_change":"2024-09-16T10:13:43.283434+0000","last_active":"2024-09-16T10:14:16.428476+0000","last_peered":"2024-09-16T10:14:16.428476+0000","last_clean":"2024-09-16T10:14:16.428476+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:14:16.428476+0000","last_undegraded":"2024-09-16T10:14:16.428476+0000","last_fullsized":"2024-09-16T10:14:16.428476+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.845991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427994+0000","last_change":"2024-09-16T10:13:37.089575+0000","last_active":"2024-09-16T10:14:16.427994+0000","last_peered":"2024-09-16T10:14:16.427994+0000","last_clean":"2024-09-16T10:14:16.427994+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:14:16.427994+0000","last_undegraded":"2024-09-16T10:14:16.427994+0000","last_fullsized":"2024-09-16T10:14:16.427994+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.178781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888338+0000","last_change":"2024-09-16T10:13:39.107923+0000","last_active":"2024-09-16T10:14:17.888338+0000","last_peered":"2024-09-16T10:14:17.888338+0000","last_clean":"2024-09-16T10:14:17.888338+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:14:17.888338+0000","last_undegraded":"2024-09-16T10:14:17.888338+0000","last_fullsized":"2024-09-16T10:14:17.888338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.475348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429457+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:14:16.429457+0000","last_peered":"2024-09-16T10:14:16.429457+0000","last_clean":"2024-09-16T10:14:16.429457+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:14:16.429457+0000","last_undegraded":"2024-09-16T10:14:16.429457+0000","last_fullsized":"2024-09-16T10:14:16.429457+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.978048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429090+0000","last_change":"2024-09-16T10:13:43.283644+0000","last_active":"2024-09-16T10:14:16.429090+0000","last_peered":"2024-09-16T10:14:16.429090+0000","last_clean":"2024-09-16T10:14:16.429090+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:14:16.429090+0000","last_undegraded":"2024-09-16T10:14:16.429090+0000","last_fullsized":"2024-09-16T10:14:16.429090+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.605473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428889+0000","last_change":"2024-09-16T10:13:39.107919+0000","last_active":"2024-09-16T10:14:16.428889+0000","last_peered":"2024-09-16T10:14:16.428889+0000","last_clean":"2024-09-16T10:14:16.428889+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:14:16.428889+0000","last_undegraded":"2024-09-16T10:14:16.428889+0000","last_fullsized":"2024-09-16T10:14:16.428889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.504540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429670+0000","last_change":"2024-09-16T10:13:37.095220+0000","last_active":"2024-09-16T10:14:16.429670+0000","last_peered":"2024-09-16T10:14:16.429670+0000","last_clean":"2024-09-16T10:14:16.429670+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:14:16.429670+0000","last_undegraded":"2024-09-16T10:14:16.429670+0000","last_fullsized":"2024-09-16T10:14:16.429670+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.601224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888268+0000","last_change":"2024-09-16T10:13:41.118905+0000","last_active":"2024-09-16T10:14:17.888268+0000","last_peered":"2024-09-16T10:14:17.888268+0000","last_clean":"2024-09-16T10:14:17.888268+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:14:17.888268+0000","last_undegraded":"2024-09-16T10:14:17.888268+0000","last_fullsized":"2024-09-16T10:14:17.888268+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.674048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.385238+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:14:17.385238+0000","last_peered":"2024-09-16T10:14:17.385238+0000","last_clean":"2024-09-16T10:14:17.385238+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:14:17.385238+0000","last_undegraded":"2024-09-16T10:14:17.385238+0000","last_fullsized":"2024-09-16T10:14:17.385238+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.053913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429336+0000","last_change":"2024-09-16T10:13:39.105916+0000","last_active":"2024-09-16T10:14:16.429336+0000","last_peered":"2024-09-16T10:14:16.429336+0000","last_clean":"2024-09-16T10:14:16.429336+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:14:16.429336+0000","last_undegraded":"2024-09-16T10:14:16.429336+0000","last_fullsized":"2024-09-16T10:14:16.429336+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.209227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427640+0000","last_change":"2024-09-16T10:13:37.097837+0000","last_active":"2024-09-16T10:14:16.427640+0000","last_peered":"2024-09-16T10:14:16.427640+0000","last_clean":"2024-09-16T10:14:16.427640+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:14:16.427640+0000","last_undegraded":"2024-09-16T10:14:16.427640+0000","last_fullsized":"2024-09-16T10:14:16.427640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:36.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887879+0000","last_change":"2024-09-16T10:13:41.119310+0000","last_active":"2024-09-16T10:14:17.887879+0000","last_peered":"2024-09-16T10:14:17.887879+0000","last_clean":"2024-09-16T10:14:17.887879+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:14:17.887879+0000","last_undegraded":"2024-09-16T10:14:17.887879+0000","last_fullsized":"2024-09-16T10:14:17.887879+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.607866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384353+0000","last_change":"2024-09-16T10:13:43.119772+0000","last_active":"2024-09-16T10:14:17.384353+0000","last_peered":"2024-09-16T10:14:17.384353+0000","last_clean":"2024-09-16T10:14:17.384353+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:14:17.384353+0000","last_undegraded":"2024-09-16T10:14:17.384353+0000","last_fullsized":"2024-09-16T10:14:17.384353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.088506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888130+0000","last_change":"2024-09-16T10:13:39.105939+0000","last_active":"2024-09-16T10:14:17.888130+0000","last_peered":"2024-09-16T10:14:17.888130+0000","last_clean":"2024-09-16T10:14:17.888130+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:14:17.888130+0000","last_undegraded":"2024-09-16T10:14:17.888130+0000","last_fullsized":"2024-09-16T10:14:17.888130+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.788054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429987+0000","last_change":"2024-09-16T10:13:37.094640+0000","last_active":"2024-09-16T10:14:16.429987+0000","last_peered":"2024-09-16T10:14:16.429987+0000","last_clean":"2024-09-16T10:14:16.429987+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:14:16.429987+0000","last_undegraded":"2024-09-16T10:14:16.429987+0000","last_fullsized":"2024-09-16T10:14:16.429987+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.045517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429280+0000","last_change":"2024-09-16T10:13:41.116455+0000","last_active":"2024-09-16T10:14:16.429280+0000","last_peered":"2024-09-16T10:14:16.429280+0000","last_clean":"2024-09-16T10:14:16.429280+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:14:16.429280+0000","last_undegraded":"2024-09-16T10:14:16.429280+0000","last_fullsized":"2024-09-16T10:14:16.429280+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.665680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428456+0000","last_change":"2024-09-16T10:13:43.120971+0000","last_active":"2024-09-16T10:14:16.428456+0000","last_peered":"2024-09-16T10:14:16.428456+0000","last_clean":"2024-09-16T10:14:16.428456+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:14:16.428456+0000","last_undegraded":"2024-09-16T10:14:16.428456+0000","last_fullsized":"2024-09-16T10:14:16.428456+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.791625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383525+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:14:17.383525+0000","last_peered":"2024-09-16T10:14:17.383525+0000","last_clean":"2024-09-16T10:14:17.383525+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:14:17.383525+0000","last_undegraded":"2024-09-16T10:14:17.383525+0000","last_fullsized":"2024-09-16T10:14:17.383525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.996979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383965+0000","last_change":"2024-09-16T10:13:37.090129+0000","last_active":"2024-09-16T10:14:17.383965+0000","last_peered":"2024-09-16T10:14:17.383965+0000","last_clean":"2024-09-16T10:14:17.383965+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:14:17.383965+0000","last_undegraded":"2024-09-16T10:14:17.383965+0000","last_fullsized":"2024-09-16T10:14:17.383965+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.823652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383765+0000","last_change":"2024-09-16T10:13:41.112912+0000","last_active":"2024-09-16T10:14:17.383765+0000","last_peered":"2024-09-16T10:14:17.383765+0000","last_clean":"2024-09-16T10:14:17.383765+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:14:17.383765+0000","last_undegraded":"2024-09-16T10:14:17.383765+0000","last_fullsized":"2024-09-16T10:14:17.383765+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.606633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887373+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:14:17.887373+0000","last_peered":"2024-09-16T10:14:17.887373+0000","last_clean":"2024-09-16T10:14:17.887373+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:14:17.887373+0000","last_undegraded":"2024-09-16T10:14:17.887373+0000","last_fullsized":"2024-09-16T10:14:17.887373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.223461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.384197+0000","last_change":"2024-09-16T10:13:39.098286+0000","last_active":"2024-09-16T10:14:17.384197+0000","last_peered":"2024-09-16T10:14:17.384197+0000","last_clean":"2024-09-16T10:14:17.384197+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:14:17.384197+0000","last_undegraded":"2024-09-16T10:14:17.384197+0000","last_fullsized":"2024-09-16T10:14:17.384197+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.259110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429419+0000","last_change":"2024-09-16T10:13:37.091081+0000","last_active":"2024-09-16T10:14:16.429419+0000","last_peered":"2024-09-16T10:14:16.429419+0000","last_clean":"2024-09-16T10:14:16.429419+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:14:16.429419+0000","last_undegraded":"2024-09-16T10:14:16.429419+0000","last_fullsized":"2024-09-16T10:14:16.429419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.952981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887757+0000","last_change":"2024-09-16T10:13:41.114260+0000","last_active":"2024-09-16T10:14:17.887757+0000","last_peered":"2024-09-16T10:14:17.887757+0000","last_clean":"2024-09-16T10:14:17.887757+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:14:17.887757+0000","last_undegraded":"2024-09-16T10:14:17.887757+0000","last_fullsized":"2024-09-16T10:14:17.887757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.627978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427406+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:14:16.427406+0000","last_peered":"2024-09-16T10:14:16.427406+0000","last_clean":"2024-09-16T10:14:16.427406+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:14:16.427406+0000","last_undegraded":"2024-09-16T10:14:16.427406+0000","last_fullsized":"2024-09-16T10:14:16.427406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887530+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:14:17.887530+0000","last_peered":"2024-09-16T10:14:17.887530+0000","last_clean":"2024-09-16T10:14:17.887530+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:14:17.887530+0000","last_undegraded":"2024-09-16T10:14:17.887530+0000","last_fullsized":"2024-09-16T10:14:17.887530+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.847239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.429519+0000","last_change":"2024-09-16T10:13:37.088453+0000","last_active":"2024-09-16T10:14:16.429519+0000","last_peered":"2024-09-16T10:14:16.429519+0000","last_clean":"2024-09-16T10:14:16.429519+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:14:16.429519+0000","last_undegraded":"2024-09-16T10:14:16.429519+0000","last_fullsized":"2024-09-16T10:14:16.429519+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.596318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427922+0000","last_change":"2024-09-16T10:13:41.112830+0000","last_active":"2024-09-16T10:14:16.427922+0000","last_peered":"2024-09-16T10:14:16.427922+0000","last_clean":"2024-09-16T10:14:16.427922+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:14:16.427922+0000","last_undegraded":"2024-09-16T10:14:16.427922+0000","last_fullsized":"2024-09-16T10:14:16.427922+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:41.842458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887901+0000","last_change":"2024-09-16T10:13:43.286716+0000","last_active":"2024-09-16T10:14:17.887901+0000","last_peered":"2024-09-16T10:14:17.887901+0000","last_clean":"2024-09-16T10:14:17.887901+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:14:17.887901+0000","last_undegraded":"2024-09-16T10:14:17.887901+0000","last_fullsized":"2024-09-16T10:14:17.887901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.000117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887375+0000","last_change":"2024-09-16T10:13:39.101213+0000","last_active":"2024-09-16T10:14:17.887375+0000","last_peered":"2024-09-16T10:14:17.887375+0000","last_clean":"2024-09-16T10:14:17.887375+0000","last_became_active":"2024-09-16T10:13:39.100788+0000","last_became_peered":"2024-09-16T10:13:39.100788+0000","last_unstale":"2024-09-16T10:14:17.887375+0000","last_undegraded":"2024-09-16T10:14:17.887375+0000","last_fullsized":"2024-09-16T10:14:17.887375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.518507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":50,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428339+0000","last_change":"2024-09-16T10:13:37.094533+0000","last_active":"2024-09-16T10:14:16.428339+0000","last_peered":"2024-09-16T10:14:16.428339+0000","last_clean":"2024-09-16T10:14:16.428339+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:14:16.428339+0000","last_undegraded":"2024-09-16T10:14:16.428339+0000","last_fullsized":"2024-09-16T10:14:16.428339+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.183923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427965+0000","last_change":"2024-09-16T10:13:41.114472+0000","last_active":"2024-09-16T10:14:16.427965+0000","last_peered":"2024-09-16T10:14:16.427965+0000","last_clean":"2024-09-16T10:14:16.427965+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:14:16.427965+0000","last_undegraded":"2024-09-16T10:14:16.427965+0000","last_fullsized":"2024-09-16T10:14:16.427965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.243642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428236+0000","last_change":"2024-09-16T10:13:43.119917+0000","last_active":"2024-09-16T10:14:16.428236+0000","last_peered":"2024-09-16T10:14:16.428236+0000","last_clean":"2024-09-16T10:14:16.428236+0000","last_became_active":"2024-09-16T10:13:43.119612+0000","last_became_peered":"2024-09-16T10:13:43.119612+0000","last_unstale":"2024-09-16T10:14:16.428236+0000","last_undegraded":"2024-09-16T10:14:16.428236+0000","last_fullsized":"2024-09-16T10:14:16.428236+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.539099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.383716+0000","last_change":"2024-09-16T10:13:43.114761+0000","last_active":"2024-09-16T10:14:17.383716+0000","last_peered":"2024-09-16T10:14:17.383716+0000","last_clean":"2024-09-16T10:14:17.383716+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:14:17.383716+0000","last_undegraded":"2024-09-16T10:14:17.383716+0000","last_fullsized":"2024-09-16T10:14:17.383716+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.210195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887548+0000","last_change":"2024-09-16T10:13:37.089832+0000","last_active":"2024-09-16T10:14:17.887548+0000","last_peered":"2024-09-16T10:14:17.887548+0000","last_clean":"2024-09-16T10:14:17.887548+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:14:17.887548+0000","last_undegraded":"2024-09-16T10:14:17.887548+0000","last_fullsized":"2024-09-16T10:14:17.887548+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.587954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":59,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.430132+0000","last_change":"2024-09-16T10:13:39.098801+0000","last_active":"2024-09-16T10:14:16.430132+0000","last_peered":"2024-09-16T10:14:16.430132+0000","last_clean":"2024-09-16T10:14:16.430132+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:14:16.430132+0000","last_undegraded":"2024-09-16T10:14:16.430132+0000","last_fullsized":"2024-09-16T10:14:16.430132+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.353803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428600+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:16.428600+0000","last_peered":"2024-09-16T10:14:16.428600+0000","last_clean":"2024-09-16T10:14:16.428600+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:14:16.428600+0000","last_undegraded":"2024-09-16T10:14:16.428600+0000","last_fullsized":"2024-09-16T10:14:16.428600+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.860042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233996,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1359872,"data_stored":895555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528336,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32492,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712660,"statfs":{"total":95995035648,"available":95961763840,"internally_reserved":0,"allocated":1335296,"data_stored":894457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789974,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27760,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":749568,"data_stored":305441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986203,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":786432,"data_stored":311552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215136,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":786432,"data_stored":306409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31924,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":753664,"data_stored":310672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":811008,"data_stored":306310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1368064,"data_stored":896257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:14:22.685 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi063 to 3760M 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi063:/etc/ceph/ceph.conf 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='client.14775 v1:172.21.15.63:0/4155669053' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:14:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: Deploying daemon alertmanager.a on smithi063 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi063 to 3760M 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi063:/etc/ceph/ceph.conf 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='client.14775 v1:172.21.15.63:0/4155669053' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:22.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: Deploying daemon alertmanager.a on smithi063 2024-09-16T10:14:22.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi063 to 3760M 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi063:/etc/ceph/ceph.conf 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-16T10:14:23.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='client.14775 v1:172.21.15.63:0/4155669053' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Updating smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: Deploying daemon alertmanager.a on smithi063 2024-09-16T10:14:23.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:23.743 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:14:23.745 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:14:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:23 smithi165 ceph-mon[32696]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T10:14:24.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:23 smithi063 ceph-mon[29993]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T10:14:24.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:23 smithi063 ceph-mon[39470]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T10:14:25.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:24 smithi063 ceph-mon[29993]: from='client.44657 v1:172.21.15.63:0/2997019565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:25.019 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:24 smithi063 ceph-mon[39470]: from='client.44657 v1:172.21.15.63:0/2997019565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:25.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:24 smithi165 ceph-mon[32696]: from='client.44657 v1:172.21.15.63:0/2997019565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:25.206 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-16T10:14:23.009274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":888,"num_read_kb":745,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240824,"kb_used_data":7956,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720392,"statfs":{"total":767960285184,"available":767713681408,"internally_reserved":0,"allocated":8146944,"data_stored":4329105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001030"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428258+0000","last_change":"2024-09-16T10:13:37.096999+0000","last_active":"2024-09-16T10:14:16.428258+0000","last_peered":"2024-09-16T10:14:16.428258+0000","last_clean":"2024-09-16T10:14:16.428258+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:14:16.428258+0000","last_undegraded":"2024-09-16T10:14:16.428258+0000","last_fullsized":"2024-09-16T10:14:16.428258+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677037+0000","last_change":"2024-09-16T10:13:39.093832+0000","last_active":"2024-09-16T10:14:21.677037+0000","last_peered":"2024-09-16T10:14:21.677037+0000","last_clean":"2024-09-16T10:14:21.677037+0000","last_became_active":"2024-09-16T10:13:39.093430+0000","last_became_peered":"2024-09-16T10:13:39.093430+0000","last_unstale":"2024-09-16T10:14:21.677037+0000","last_undegraded":"2024-09-16T10:14:21.677037+0000","last_fullsized":"2024-09-16T10:14:21.677037+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.941021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.707049+0000","last_change":"2024-09-16T10:13:41.108286+0000","last_active":"2024-09-16T10:14:20.707049+0000","last_peered":"2024-09-16T10:14:20.707049+0000","last_clean":"2024-09-16T10:14:20.707049+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:14:20.707049+0000","last_undegraded":"2024-09-16T10:14:20.707049+0000","last_fullsized":"2024-09-16T10:14:20.707049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.804825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677284+0000","last_change":"2024-09-16T10:13:43.113148+0000","last_active":"2024-09-16T10:14:21.677284+0000","last_peered":"2024-09-16T10:14:21.677284+0000","last_clean":"2024-09-16T10:14:21.677284+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:14:21.677284+0000","last_undegraded":"2024-09-16T10:14:21.677284+0000","last_fullsized":"2024-09-16T10:14:21.677284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.390079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887558+0000","last_change":"2024-09-16T10:13:43.287323+0000","last_active":"2024-09-16T10:14:17.887558+0000","last_peered":"2024-09-16T10:14:17.887558+0000","last_clean":"2024-09-16T10:14:17.887558+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:14:17.887558+0000","last_undegraded":"2024-09-16T10:14:17.887558+0000","last_fullsized":"2024-09-16T10:14:17.887558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.723245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888312+0000","last_change":"2024-09-16T10:13:37.092457+0000","last_active":"2024-09-16T10:14:17.888312+0000","last_peered":"2024-09-16T10:14:17.888312+0000","last_clean":"2024-09-16T10:14:17.888312+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:14:17.888312+0000","last_undegraded":"2024-09-16T10:14:17.888312+0000","last_fullsized":"2024-09-16T10:14:17.888312+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.920553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887715+0000","last_change":"2024-09-16T10:13:39.095829+0000","last_active":"2024-09-16T10:14:17.887715+0000","last_peered":"2024-09-16T10:14:17.887715+0000","last_clean":"2024-09-16T10:14:17.887715+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:14:17.887715+0000","last_undegraded":"2024-09-16T10:14:17.887715+0000","last_fullsized":"2024-09-16T10:14:17.887715+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.084583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676405+0000","last_change":"2024-09-16T10:13:41.121501+0000","last_active":"2024-09-16T10:14:21.676405+0000","last_peered":"2024-09-16T10:14:21.676405+0000","last_clean":"2024-09-16T10:14:21.676405+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:14:21.676405+0000","last_undegraded":"2024-09-16T10:14:21.676405+0000","last_fullsized":"2024-09-16T10:14:21.676405+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.386931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710516+0000","last_change":"2024-09-16T10:13:37.092847+0000","last_active":"2024-09-16T10:14:20.710516+0000","last_peered":"2024-09-16T10:14:20.710516+0000","last_clean":"2024-09-16T10:14:20.710516+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:14:20.710516+0000","last_undegraded":"2024-09-16T10:14:20.710516+0000","last_fullsized":"2024-09-16T10:14:20.710516+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.507863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677767+0000","last_change":"2024-09-16T10:13:39.092562+0000","last_active":"2024-09-16T10:14:21.677767+0000","last_peered":"2024-09-16T10:14:21.677767+0000","last_clean":"2024-09-16T10:14:21.677767+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:14:21.677767+0000","last_undegraded":"2024-09-16T10:14:21.677767+0000","last_fullsized":"2024-09-16T10:14:21.677767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.237966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710040+0000","last_change":"2024-09-16T10:13:41.112242+0000","last_active":"2024-09-16T10:14:20.710040+0000","last_peered":"2024-09-16T10:14:20.710040+0000","last_clean":"2024-09-16T10:14:20.710040+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:14:20.710040+0000","last_undegraded":"2024-09-16T10:14:20.710040+0000","last_fullsized":"2024-09-16T10:14:20.710040+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.338353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427492+0000","last_change":"2024-09-16T10:13:43.115780+0000","last_active":"2024-09-16T10:14:16.427492+0000","last_peered":"2024-09-16T10:14:16.427492+0000","last_clean":"2024-09-16T10:14:16.427492+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:14:16.427492+0000","last_undegraded":"2024-09-16T10:14:16.427492+0000","last_fullsized":"2024-09-16T10:14:16.427492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.438318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710216+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:14:20.710216+0000","last_peered":"2024-09-16T10:14:20.710216+0000","last_clean":"2024-09-16T10:14:20.710216+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:14:20.710216+0000","last_undegraded":"2024-09-16T10:14:20.710216+0000","last_fullsized":"2024-09-16T10:14:20.710216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.318174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677688+0000","last_change":"2024-09-16T10:13:39.092696+0000","last_active":"2024-09-16T10:14:21.677688+0000","last_peered":"2024-09-16T10:14:21.677688+0000","last_clean":"2024-09-16T10:14:21.677688+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:14:21.677688+0000","last_undegraded":"2024-09-16T10:14:21.677688+0000","last_fullsized":"2024-09-16T10:14:21.677688+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.518434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676622+0000","last_change":"2024-09-16T10:13:41.111640+0000","last_active":"2024-09-16T10:14:21.676622+0000","last_peered":"2024-09-16T10:14:21.676622+0000","last_clean":"2024-09-16T10:14:21.676622+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:14:21.676622+0000","last_undegraded":"2024-09-16T10:14:21.676622+0000","last_fullsized":"2024-09-16T10:14:21.676622+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.280028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.709887+0000","last_change":"2024-09-16T10:13:43.121701+0000","last_active":"2024-09-16T10:14:20.709887+0000","last_peered":"2024-09-16T10:14:20.709887+0000","last_clean":"2024-09-16T10:14:20.709887+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:14:20.709887+0000","last_undegraded":"2024-09-16T10:14:20.709887+0000","last_fullsized":"2024-09-16T10:14:20.709887+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.188718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678381+0000","last_change":"2024-09-16T10:13:43.284208+0000","last_active":"2024-09-16T10:14:21.678381+0000","last_peered":"2024-09-16T10:14:21.678381+0000","last_clean":"2024-09-16T10:14:21.678381+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:14:21.678381+0000","last_undegraded":"2024-09-16T10:14:21.678381+0000","last_fullsized":"2024-09-16T10:14:21.678381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.053189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428963+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:14:16.428963+0000","last_peered":"2024-09-16T10:14:16.428963+0000","last_clean":"2024-09-16T10:14:16.428963+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:14:16.428963+0000","last_undegraded":"2024-09-16T10:14:16.428963+0000","last_fullsized":"2024-09-16T10:14:16.428963+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.063039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":71,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.586248+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:14:21.586248+0000","last_peered":"2024-09-16T10:14:21.586248+0000","last_clean":"2024-09-16T10:14:21.586248+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:14:21.586248+0000","last_undegraded":"2024-09-16T10:14:21.586248+0000","last_fullsized":"2024-09-16T10:14:21.586248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.478889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.707534+0000","last_change":"2024-09-16T10:13:41.120296+0000","last_active":"2024-09-16T10:14:20.707534+0000","last_peered":"2024-09-16T10:14:20.707534+0000","last_clean":"2024-09-16T10:14:20.707534+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:14:20.707534+0000","last_undegraded":"2024-09-16T10:14:20.707534+0000","last_fullsized":"2024-09-16T10:14:20.707534+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.391104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888196+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:14:17.888196+0000","last_peered":"2024-09-16T10:14:17.888196+0000","last_clean":"2024-09-16T10:14:17.888196+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:14:17.888196+0000","last_undegraded":"2024-09-16T10:14:17.888196+0000","last_fullsized":"2024-09-16T10:14:17.888196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.407109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677932+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:14:21.677932+0000","last_peered":"2024-09-16T10:14:21.677932+0000","last_clean":"2024-09-16T10:14:21.677932+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:14:21.677932+0000","last_undegraded":"2024-09-16T10:14:21.677932+0000","last_fullsized":"2024-09-16T10:14:21.677932+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.861513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888066+0000","last_change":"2024-09-16T10:13:39.100013+0000","last_active":"2024-09-16T10:14:17.888066+0000","last_peered":"2024-09-16T10:14:17.888066+0000","last_clean":"2024-09-16T10:14:17.888066+0000","last_became_active":"2024-09-16T10:13:39.099675+0000","last_became_peered":"2024-09-16T10:13:39.099675+0000","last_unstale":"2024-09-16T10:14:17.888066+0000","last_undegraded":"2024-09-16T10:14:17.888066+0000","last_fullsized":"2024-09-16T10:14:17.888066+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.654393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676490+0000","last_change":"2024-09-16T10:13:41.112993+0000","last_active":"2024-09-16T10:14:21.676490+0000","last_peered":"2024-09-16T10:14:21.676490+0000","last_clean":"2024-09-16T10:14:21.676490+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:14:21.676490+0000","last_undegraded":"2024-09-16T10:14:21.676490+0000","last_fullsized":"2024-09-16T10:14:21.676490+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.116156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677899+0000","last_change":"2024-09-16T10:13:43.119664+0000","last_active":"2024-09-16T10:14:21.677899+0000","last_peered":"2024-09-16T10:14:21.677899+0000","last_clean":"2024-09-16T10:14:21.677899+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:14:21.677899+0000","last_undegraded":"2024-09-16T10:14:21.677899+0000","last_fullsized":"2024-09-16T10:14:21.677899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.178009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.706957+0000","last_change":"2024-09-16T10:13:37.082594+0000","last_active":"2024-09-16T10:14:20.706957+0000","last_peered":"2024-09-16T10:14:20.706957+0000","last_clean":"2024-09-16T10:14:20.706957+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:14:20.706957+0000","last_undegraded":"2024-09-16T10:14:20.706957+0000","last_fullsized":"2024-09-16T10:14:20.706957+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.835656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":55,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428683+0000","last_change":"2024-09-16T10:13:39.108076+0000","last_active":"2024-09-16T10:14:16.428683+0000","last_peered":"2024-09-16T10:14:16.428683+0000","last_clean":"2024-09-16T10:14:16.428683+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:14:16.428683+0000","last_undegraded":"2024-09-16T10:14:16.428683+0000","last_fullsized":"2024-09-16T10:14:16.428683+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.726907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.489352+0000","last_change":"2024-09-16T10:13:41.116955+0000","last_active":"2024-09-16T10:14:21.489352+0000","last_peered":"2024-09-16T10:14:21.489352+0000","last_clean":"2024-09-16T10:14:21.489352+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:14:21.489352+0000","last_undegraded":"2024-09-16T10:14:21.489352+0000","last_fullsized":"2024-09-16T10:14:21.489352+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.203272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":71,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.707937+0000","last_change":"2024-09-16T10:13:39.096753+0000","last_active":"2024-09-16T10:14:20.707937+0000","last_peered":"2024-09-16T10:14:20.707937+0000","last_clean":"2024-09-16T10:14:20.707937+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:14:20.707937+0000","last_undegraded":"2024-09-16T10:14:20.707937+0000","last_fullsized":"2024-09-16T10:14:20.707937+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.363335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887635+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:14:17.887635+0000","last_peered":"2024-09-16T10:14:17.887635+0000","last_clean":"2024-09-16T10:14:17.887635+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:14:17.887635+0000","last_undegraded":"2024-09-16T10:14:17.887635+0000","last_fullsized":"2024-09-16T10:14:17.887635+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.355614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676754+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:14:21.676754+0000","last_peered":"2024-09-16T10:14:21.676754+0000","last_clean":"2024-09-16T10:14:21.676754+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:14:21.676754+0000","last_undegraded":"2024-09-16T10:14:21.676754+0000","last_fullsized":"2024-09-16T10:14:21.676754+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.675105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710185+0000","last_change":"2024-09-16T10:13:43.114886+0000","last_active":"2024-09-16T10:14:20.710185+0000","last_peered":"2024-09-16T10:14:20.710185+0000","last_clean":"2024-09-16T10:14:20.710185+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:14:20.710185+0000","last_undegraded":"2024-09-16T10:14:20.710185+0000","last_fullsized":"2024-09-16T10:14:20.710185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.234184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":74,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887416+0000","last_change":"2024-09-16T10:13:39.106135+0000","last_active":"2024-09-16T10:14:17.887416+0000","last_peered":"2024-09-16T10:14:17.887416+0000","last_clean":"2024-09-16T10:14:17.887416+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:14:17.887416+0000","last_undegraded":"2024-09-16T10:14:17.887416+0000","last_fullsized":"2024-09-16T10:14:17.887416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.184956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888414+0000","last_change":"2024-09-16T10:13:37.093660+0000","last_active":"2024-09-16T10:14:17.888414+0000","last_peered":"2024-09-16T10:14:17.888414+0000","last_clean":"2024-09-16T10:14:17.888414+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:14:17.888414+0000","last_undegraded":"2024-09-16T10:14:17.888414+0000","last_fullsized":"2024-09-16T10:14:17.888414+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.816759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676419+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:14:21.676419+0000","last_peered":"2024-09-16T10:14:21.676419+0000","last_clean":"2024-09-16T10:14:21.676419+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:14:21.676419+0000","last_undegraded":"2024-09-16T10:14:21.676419+0000","last_fullsized":"2024-09-16T10:14:21.676419+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.743002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677667+0000","last_change":"2024-09-16T10:13:43.122864+0000","last_active":"2024-09-16T10:14:21.677667+0000","last_peered":"2024-09-16T10:14:21.677667+0000","last_clean":"2024-09-16T10:14:21.677667+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:14:21.677667+0000","last_undegraded":"2024-09-16T10:14:21.677667+0000","last_fullsized":"2024-09-16T10:14:21.677667+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.235493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677143+0000","last_change":"2024-09-16T10:13:39.094074+0000","last_active":"2024-09-16T10:14:21.677143+0000","last_peered":"2024-09-16T10:14:21.677143+0000","last_clean":"2024-09-16T10:14:21.677143+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:14:21.677143+0000","last_undegraded":"2024-09-16T10:14:21.677143+0000","last_fullsized":"2024-09-16T10:14:21.677143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.532032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":45,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428973+0000","last_change":"2024-09-16T10:13:37.096046+0000","last_active":"2024-09-16T10:14:16.428973+0000","last_peered":"2024-09-16T10:14:16.428973+0000","last_clean":"2024-09-16T10:14:16.428973+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:14:16.428973+0000","last_undegraded":"2024-09-16T10:14:16.428973+0000","last_fullsized":"2024-09-16T10:14:16.428973+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.593585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887292+0000","last_change":"2024-09-16T10:13:41.112759+0000","last_active":"2024-09-16T10:14:17.887292+0000","last_peered":"2024-09-16T10:14:17.887292+0000","last_clean":"2024-09-16T10:14:17.887292+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:14:17.887292+0000","last_undegraded":"2024-09-16T10:14:17.887292+0000","last_fullsized":"2024-09-16T10:14:17.887292+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.600065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677584+0000","last_change":"2024-09-16T10:13:43.284631+0000","last_active":"2024-09-16T10:14:21.677584+0000","last_peered":"2024-09-16T10:14:21.677584+0000","last_clean":"2024-09-16T10:14:21.677584+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:14:21.677584+0000","last_undegraded":"2024-09-16T10:14:21.677584+0000","last_fullsized":"2024-09-16T10:14:21.677584+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.200363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.706944+0000","last_change":"2024-09-16T10:13:39.095801+0000","last_active":"2024-09-16T10:14:20.706944+0000","last_peered":"2024-09-16T10:14:20.706944+0000","last_clean":"2024-09-16T10:14:20.706944+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:14:20.706944+0000","last_undegraded":"2024-09-16T10:14:20.706944+0000","last_fullsized":"2024-09-16T10:14:20.706944+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.537230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710572+0000","last_change":"2024-09-16T10:13:37.093423+0000","last_active":"2024-09-16T10:14:20.710572+0000","last_peered":"2024-09-16T10:14:20.710572+0000","last_clean":"2024-09-16T10:14:20.710572+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:14:20.710572+0000","last_undegraded":"2024-09-16T10:14:20.710572+0000","last_fullsized":"2024-09-16T10:14:20.710572+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.178620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":53,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428654+0000","last_change":"2024-09-16T10:13:41.115177+0000","last_active":"2024-09-16T10:14:16.428654+0000","last_peered":"2024-09-16T10:14:16.428654+0000","last_clean":"2024-09-16T10:14:16.428654+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:14:16.428654+0000","last_undegraded":"2024-09-16T10:14:16.428654+0000","last_fullsized":"2024-09-16T10:14:16.428654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.803669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427590+0000","last_change":"2024-09-16T10:13:43.119055+0000","last_active":"2024-09-16T10:14:16.427590+0000","last_peered":"2024-09-16T10:14:16.427590+0000","last_clean":"2024-09-16T10:14:16.427590+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:14:16.427590+0000","last_undegraded":"2024-09-16T10:14:16.427590+0000","last_fullsized":"2024-09-16T10:14:16.427590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:54.513879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":76,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428819+0000","last_change":"2024-09-16T10:13:39.106153+0000","last_active":"2024-09-16T10:14:16.428819+0000","last_peered":"2024-09-16T10:14:16.428819+0000","last_clean":"2024-09-16T10:14:16.428819+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:14:16.428819+0000","last_undegraded":"2024-09-16T10:14:16.428819+0000","last_fullsized":"2024-09-16T10:14:16.428819+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.043307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.708523+0000","last_change":"2024-09-16T10:13:37.092754+0000","last_active":"2024-09-16T10:14:20.708523+0000","last_peered":"2024-09-16T10:14:20.708523+0000","last_clean":"2024-09-16T10:14:20.708523+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:14:20.708523+0000","last_undegraded":"2024-09-16T10:14:20.708523+0000","last_fullsized":"2024-09-16T10:14:20.708523+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.036476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":41,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.488440+0000","last_change":"2024-09-16T10:13:41.114703+0000","last_active":"2024-09-16T10:14:21.488440+0000","last_peered":"2024-09-16T10:14:21.488440+0000","last_clean":"2024-09-16T10:14:21.488440+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:14:21.488440+0000","last_undegraded":"2024-09-16T10:14:21.488440+0000","last_fullsized":"2024-09-16T10:14:21.488440+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.640025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.706562+0000","last_change":"2024-09-16T10:13:43.283556+0000","last_active":"2024-09-16T10:14:20.706562+0000","last_peered":"2024-09-16T10:14:20.706562+0000","last_clean":"2024-09-16T10:14:20.706562+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:14:20.706562+0000","last_undegraded":"2024-09-16T10:14:20.706562+0000","last_fullsized":"2024-09-16T10:14:20.706562+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:52.096829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":104,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.708416+0000","last_change":"2024-09-16T10:13:39.098696+0000","last_active":"2024-09-16T10:14:20.708416+0000","last_peered":"2024-09-16T10:14:20.708416+0000","last_clean":"2024-09-16T10:14:20.708416+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:14:20.708416+0000","last_undegraded":"2024-09-16T10:14:20.708416+0000","last_fullsized":"2024-09-16T10:14:20.708416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.501760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678194+0000","last_change":"2024-09-16T10:13:37.092585+0000","last_active":"2024-09-16T10:14:21.678194+0000","last_peered":"2024-09-16T10:14:21.678194+0000","last_clean":"2024-09-16T10:14:21.678194+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:14:21.678194+0000","last_undegraded":"2024-09-16T10:14:21.678194+0000","last_fullsized":"2024-09-16T10:14:21.678194+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.123070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710201+0000","last_change":"2024-09-16T10:13:39.091269+0000","last_active":"2024-09-16T10:14:20.710201+0000","last_peered":"2024-09-16T10:14:20.710201+0000","last_clean":"2024-09-16T10:14:20.710201+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:14:20.710201+0000","last_undegraded":"2024-09-16T10:14:20.710201+0000","last_fullsized":"2024-09-16T10:14:20.710201+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.638736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427962+0000","last_change":"2024-09-16T10:13:41.118142+0000","last_active":"2024-09-16T10:14:16.427962+0000","last_peered":"2024-09-16T10:14:16.427962+0000","last_clean":"2024-09-16T10:14:16.427962+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:14:16.427962+0000","last_undegraded":"2024-09-16T10:14:16.427962+0000","last_fullsized":"2024-09-16T10:14:16.427962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.463063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888019+0000","last_change":"2024-09-16T10:13:43.124755+0000","last_active":"2024-09-16T10:14:17.888019+0000","last_peered":"2024-09-16T10:14:17.888019+0000","last_clean":"2024-09-16T10:14:17.888019+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:14:17.888019+0000","last_undegraded":"2024-09-16T10:14:17.888019+0000","last_fullsized":"2024-09-16T10:14:17.888019+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.479324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.707957+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:14:20.707957+0000","last_peered":"2024-09-16T10:14:20.707957+0000","last_clean":"2024-09-16T10:14:20.707957+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:14:20.707957+0000","last_undegraded":"2024-09-16T10:14:20.707957+0000","last_fullsized":"2024-09-16T10:14:20.707957+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.987015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.706922+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:14:20.706922+0000","last_peered":"2024-09-16T10:14:20.706922+0000","last_clean":"2024-09-16T10:14:20.706922+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:14:20.706922+0000","last_undegraded":"2024-09-16T10:14:20.706922+0000","last_fullsized":"2024-09-16T10:14:20.706922+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.585993+0000","last_change":"2024-09-16T10:13:39.090453+0000","last_active":"2024-09-16T10:14:21.585993+0000","last_peered":"2024-09-16T10:14:21.585993+0000","last_clean":"2024-09-16T10:14:21.585993+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:14:21.585993+0000","last_undegraded":"2024-09-16T10:14:21.585993+0000","last_fullsized":"2024-09-16T10:14:21.585993+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.048019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.585651+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:14:21.585651+0000","last_peered":"2024-09-16T10:14:21.585651+0000","last_clean":"2024-09-16T10:14:21.585651+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:14:21.585651+0000","last_undegraded":"2024-09-16T10:14:21.585651+0000","last_fullsized":"2024-09-16T10:14:21.585651+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.485969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677619+0000","last_change":"2024-09-16T10:13:43.283490+0000","last_active":"2024-09-16T10:14:21.677619+0000","last_peered":"2024-09-16T10:14:21.677619+0000","last_clean":"2024-09-16T10:14:21.677619+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:14:21.677619+0000","last_undegraded":"2024-09-16T10:14:21.677619+0000","last_fullsized":"2024-09-16T10:14:21.677619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.489088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":58,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428774+0000","last_change":"2024-09-16T10:13:39.091817+0000","last_active":"2024-09-16T10:14:16.428774+0000","last_peered":"2024-09-16T10:14:16.428774+0000","last_clean":"2024-09-16T10:14:16.428774+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:14:16.428774+0000","last_undegraded":"2024-09-16T10:14:16.428774+0000","last_fullsized":"2024-09-16T10:14:16.428774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.559454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427899+0000","last_change":"2024-09-16T10:13:37.096771+0000","last_active":"2024-09-16T10:14:16.427899+0000","last_peered":"2024-09-16T10:14:16.427899+0000","last_clean":"2024-09-16T10:14:16.427899+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:14:16.427899+0000","last_undegraded":"2024-09-16T10:14:16.427899+0000","last_fullsized":"2024-09-16T10:14:16.427899+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.458372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":101,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677040+0000","last_change":"2024-09-16T10:13:39.105362+0000","last_active":"2024-09-16T10:14:21.677040+0000","last_peered":"2024-09-16T10:14:21.677040+0000","last_clean":"2024-09-16T10:14:21.677040+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:14:21.677040+0000","last_undegraded":"2024-09-16T10:14:21.677040+0000","last_fullsized":"2024-09-16T10:14:21.677040+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:35.154849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710755+0000","last_change":"2024-09-16T10:13:41.112756+0000","last_active":"2024-09-16T10:14:20.710755+0000","last_peered":"2024-09-16T10:14:20.710755+0000","last_clean":"2024-09-16T10:14:20.710755+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:14:20.710755+0000","last_undegraded":"2024-09-16T10:14:20.710755+0000","last_fullsized":"2024-09-16T10:14:20.710755+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.034690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.708110+0000","last_change":"2024-09-16T10:13:43.123048+0000","last_active":"2024-09-16T10:14:20.708110+0000","last_peered":"2024-09-16T10:14:20.708110+0000","last_clean":"2024-09-16T10:14:20.708110+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:14:20.708110+0000","last_undegraded":"2024-09-16T10:14:20.708110+0000","last_fullsized":"2024-09-16T10:14:20.708110+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.340209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":77,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.489251+0000","last_change":"2024-09-16T10:13:39.106378+0000","last_active":"2024-09-16T10:14:21.489251+0000","last_peered":"2024-09-16T10:14:21.489251+0000","last_clean":"2024-09-16T10:14:21.489251+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:14:21.489251+0000","last_undegraded":"2024-09-16T10:14:21.489251+0000","last_fullsized":"2024-09-16T10:14:21.489251+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.582337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887735+0000","last_change":"2024-09-16T10:13:37.088320+0000","last_active":"2024-09-16T10:14:17.887735+0000","last_peered":"2024-09-16T10:14:17.887735+0000","last_clean":"2024-09-16T10:14:17.887735+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:14:17.887735+0000","last_undegraded":"2024-09-16T10:14:17.887735+0000","last_fullsized":"2024-09-16T10:14:17.887735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:37.142506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":95,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677223+0000","last_change":"2024-09-16T10:14:14.911115+0000","last_active":"2024-09-16T10:14:21.677223+0000","last_peered":"2024-09-16T10:14:21.677223+0000","last_clean":"2024-09-16T10:14:21.677223+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:14:21.677223+0000","last_undegraded":"2024-09-16T10:14:21.677223+0000","last_fullsized":"2024-09-16T10:14:21.677223+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:14:14.911041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:14:14.911041+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:17.171811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676492+0000","last_change":"2024-09-16T10:13:41.108176+0000","last_active":"2024-09-16T10:14:21.676492+0000","last_peered":"2024-09-16T10:14:21.676492+0000","last_clean":"2024-09-16T10:14:21.676492+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:14:21.676492+0000","last_undegraded":"2024-09-16T10:14:21.676492+0000","last_fullsized":"2024-09-16T10:14:21.676492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.887140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710607+0000","last_change":"2024-09-16T10:13:43.121658+0000","last_active":"2024-09-16T10:14:20.710607+0000","last_peered":"2024-09-16T10:14:20.710607+0000","last_clean":"2024-09-16T10:14:20.710607+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:14:20.710607+0000","last_undegraded":"2024-09-16T10:14:20.710607+0000","last_fullsized":"2024-09-16T10:14:20.710607+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:11.237055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678481+0000","last_change":"2024-09-16T10:13:39.094951+0000","last_active":"2024-09-16T10:14:21.678481+0000","last_peered":"2024-09-16T10:14:21.678481+0000","last_clean":"2024-09-16T10:14:21.678481+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:14:21.678481+0000","last_undegraded":"2024-09-16T10:14:21.678481+0000","last_fullsized":"2024-09-16T10:14:21.678481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.179189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677392+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:14:21.677392+0000","last_peered":"2024-09-16T10:14:21.677392+0000","last_clean":"2024-09-16T10:14:21.677392+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:14:21.677392+0000","last_undegraded":"2024-09-16T10:14:21.677392+0000","last_fullsized":"2024-09-16T10:14:21.677392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.958295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710579+0000","last_change":"2024-09-16T10:13:41.116069+0000","last_active":"2024-09-16T10:14:20.710579+0000","last_peered":"2024-09-16T10:14:20.710579+0000","last_clean":"2024-09-16T10:14:20.710579+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:14:20.710579+0000","last_undegraded":"2024-09-16T10:14:20.710579+0000","last_fullsized":"2024-09-16T10:14:20.710579+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.901053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887760+0000","last_change":"2024-09-16T10:13:43.283629+0000","last_active":"2024-09-16T10:14:17.887760+0000","last_peered":"2024-09-16T10:14:17.887760+0000","last_clean":"2024-09-16T10:14:17.887760+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:14:17.887760+0000","last_undegraded":"2024-09-16T10:14:17.887760+0000","last_fullsized":"2024-09-16T10:14:17.887760+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.176307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678415+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:14:21.678415+0000","last_peered":"2024-09-16T10:14:21.678415+0000","last_clean":"2024-09-16T10:14:21.678415+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:14:21.678415+0000","last_undegraded":"2024-09-16T10:14:21.678415+0000","last_fullsized":"2024-09-16T10:14:21.678415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.476261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.489065+0000","last_change":"2024-09-16T10:13:37.089527+0000","last_active":"2024-09-16T10:14:21.489065+0000","last_peered":"2024-09-16T10:14:21.489065+0000","last_clean":"2024-09-16T10:14:21.489065+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:14:21.489065+0000","last_undegraded":"2024-09-16T10:14:21.489065+0000","last_fullsized":"2024-09-16T10:14:21.489065+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.605092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.708120+0000","last_change":"2024-09-16T10:13:41.121389+0000","last_active":"2024-09-16T10:14:20.708120+0000","last_peered":"2024-09-16T10:14:20.708120+0000","last_clean":"2024-09-16T10:14:20.708120+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:14:20.708120+0000","last_undegraded":"2024-09-16T10:14:20.708120+0000","last_fullsized":"2024-09-16T10:14:20.708120+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.974031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.586587+0000","last_change":"2024-09-16T10:13:43.125890+0000","last_active":"2024-09-16T10:14:21.586587+0000","last_peered":"2024-09-16T10:14:21.586587+0000","last_clean":"2024-09-16T10:14:21.586587+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:14:21.586587+0000","last_undegraded":"2024-09-16T10:14:21.586587+0000","last_fullsized":"2024-09-16T10:14:21.586587+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678127+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:14:21.678127+0000","last_peered":"2024-09-16T10:14:21.678127+0000","last_clean":"2024-09-16T10:14:21.678127+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:14:21.678127+0000","last_undegraded":"2024-09-16T10:14:21.678127+0000","last_fullsized":"2024-09-16T10:14:21.678127+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.021437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887956+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:14:17.887956+0000","last_peered":"2024-09-16T10:14:17.887956+0000","last_clean":"2024-09-16T10:14:17.887956+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:14:17.887956+0000","last_undegraded":"2024-09-16T10:14:17.887956+0000","last_fullsized":"2024-09-16T10:14:17.887956+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.609140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.585634+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:14:21.585634+0000","last_peered":"2024-09-16T10:14:21.585634+0000","last_clean":"2024-09-16T10:14:21.585634+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:14:21.585634+0000","last_undegraded":"2024-09-16T10:14:21.585634+0000","last_fullsized":"2024-09-16T10:14:21.585634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678318+0000","last_change":"2024-09-16T10:13:43.119606+0000","last_active":"2024-09-16T10:14:21.678318+0000","last_peered":"2024-09-16T10:14:21.678318+0000","last_clean":"2024-09-16T10:14:21.678318+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:14:21.678318+0000","last_undegraded":"2024-09-16T10:14:21.678318+0000","last_fullsized":"2024-09-16T10:14:21.678318+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.164444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428712+0000","last_change":"2024-09-16T10:13:39.091847+0000","last_active":"2024-09-16T10:14:16.428712+0000","last_peered":"2024-09-16T10:14:16.428712+0000","last_clean":"2024-09-16T10:14:16.428712+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:14:16.428712+0000","last_undegraded":"2024-09-16T10:14:16.428712+0000","last_fullsized":"2024-09-16T10:14:16.428712+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.954882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710364+0000","last_change":"2024-09-16T10:13:37.090259+0000","last_active":"2024-09-16T10:14:20.710364+0000","last_peered":"2024-09-16T10:14:20.710364+0000","last_clean":"2024-09-16T10:14:20.710364+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:14:20.710364+0000","last_undegraded":"2024-09-16T10:14:20.710364+0000","last_fullsized":"2024-09-16T10:14:20.710364+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.290444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.585485+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:14:21.585485+0000","last_peered":"2024-09-16T10:14:21.585485+0000","last_clean":"2024-09-16T10:14:21.585485+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:14:21.585485+0000","last_undegraded":"2024-09-16T10:14:21.585485+0000","last_fullsized":"2024-09-16T10:14:21.585485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":33,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427797+0000","last_change":"2024-09-16T10:13:43.122627+0000","last_active":"2024-09-16T10:14:16.427797+0000","last_peered":"2024-09-16T10:14:16.427797+0000","last_clean":"2024-09-16T10:14:16.427797+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:14:16.427797+0000","last_undegraded":"2024-09-16T10:14:16.427797+0000","last_fullsized":"2024-09-16T10:14:16.427797+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":76,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.488820+0000","last_change":"2024-09-16T10:13:39.105379+0000","last_active":"2024-09-16T10:14:21.488820+0000","last_peered":"2024-09-16T10:14:21.488820+0000","last_clean":"2024-09-16T10:14:21.488820+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:14:21.488820+0000","last_undegraded":"2024-09-16T10:14:21.488820+0000","last_fullsized":"2024-09-16T10:14:21.488820+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.713974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678092+0000","last_change":"2024-09-16T10:13:37.093687+0000","last_active":"2024-09-16T10:14:21.678092+0000","last_peered":"2024-09-16T10:14:21.678092+0000","last_clean":"2024-09-16T10:14:21.678092+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:14:21.678092+0000","last_undegraded":"2024-09-16T10:14:21.678092+0000","last_fullsized":"2024-09-16T10:14:21.678092+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.984174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.585485+0000","last_change":"2024-09-16T10:13:41.113511+0000","last_active":"2024-09-16T10:14:21.585485+0000","last_peered":"2024-09-16T10:14:21.585485+0000","last_clean":"2024-09-16T10:14:21.585485+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:14:21.585485+0000","last_undegraded":"2024-09-16T10:14:21.585485+0000","last_fullsized":"2024-09-16T10:14:21.585485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.162031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677087+0000","last_change":"2024-09-16T10:13:43.122094+0000","last_active":"2024-09-16T10:14:21.677087+0000","last_peered":"2024-09-16T10:14:21.677087+0000","last_clean":"2024-09-16T10:14:21.677087+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:14:21.677087+0000","last_undegraded":"2024-09-16T10:14:21.677087+0000","last_fullsized":"2024-09-16T10:14:21.677087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.948353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677303+0000","last_change":"2024-09-16T10:13:39.095588+0000","last_active":"2024-09-16T10:14:21.677303+0000","last_peered":"2024-09-16T10:14:21.677303+0000","last_clean":"2024-09-16T10:14:21.677303+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:14:21.677303+0000","last_undegraded":"2024-09-16T10:14:21.677303+0000","last_fullsized":"2024-09-16T10:14:21.677303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.533870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678418+0000","last_change":"2024-09-16T10:13:37.097257+0000","last_active":"2024-09-16T10:14:21.678418+0000","last_peered":"2024-09-16T10:14:21.678418+0000","last_clean":"2024-09-16T10:14:21.678418+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:14:21.678418+0000","last_undegraded":"2024-09-16T10:14:21.678418+0000","last_fullsized":"2024-09-16T10:14:21.678418+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.017319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.585485+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:14:21.585485+0000","last_peered":"2024-09-16T10:14:21.585485+0000","last_clean":"2024-09-16T10:14:21.585485+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:14:21.585485+0000","last_undegraded":"2024-09-16T10:14:21.585485+0000","last_fullsized":"2024-09-16T10:14:21.585485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.545235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887634+0000","last_change":"2024-09-16T10:13:43.125823+0000","last_active":"2024-09-16T10:14:17.887634+0000","last_peered":"2024-09-16T10:14:17.887634+0000","last_clean":"2024-09-16T10:14:17.887634+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:14:17.887634+0000","last_undegraded":"2024-09-16T10:14:17.887634+0000","last_fullsized":"2024-09-16T10:14:17.887634+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.646167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":71,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.709708+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:14:20.709708+0000","last_peered":"2024-09-16T10:14:20.709708+0000","last_clean":"2024-09-16T10:14:20.709708+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:14:20.709708+0000","last_undegraded":"2024-09-16T10:14:20.709708+0000","last_fullsized":"2024-09-16T10:14:20.709708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.689518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678402+0000","last_change":"2024-09-16T10:13:37.092779+0000","last_active":"2024-09-16T10:14:21.678402+0000","last_peered":"2024-09-16T10:14:21.678402+0000","last_clean":"2024-09-16T10:14:21.678402+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:14:21.678402+0000","last_undegraded":"2024-09-16T10:14:21.678402+0000","last_fullsized":"2024-09-16T10:14:21.678402+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.367150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676920+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:21.676920+0000","last_peered":"2024-09-16T10:14:21.676920+0000","last_clean":"2024-09-16T10:14:21.676920+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:14:21.676920+0000","last_undegraded":"2024-09-16T10:14:21.676920+0000","last_fullsized":"2024-09-16T10:14:21.676920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.594260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.709876+0000","last_change":"2024-09-16T10:13:43.283434+0000","last_active":"2024-09-16T10:14:20.709876+0000","last_peered":"2024-09-16T10:14:20.709876+0000","last_clean":"2024-09-16T10:14:20.709876+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:14:20.709876+0000","last_undegraded":"2024-09-16T10:14:20.709876+0000","last_fullsized":"2024-09-16T10:14:20.709876+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.845991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.488741+0000","last_change":"2024-09-16T10:13:37.089575+0000","last_active":"2024-09-16T10:14:21.488741+0000","last_peered":"2024-09-16T10:14:21.488741+0000","last_clean":"2024-09-16T10:14:21.488741+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:14:21.488741+0000","last_undegraded":"2024-09-16T10:14:21.488741+0000","last_fullsized":"2024-09-16T10:14:21.488741+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.178781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888338+0000","last_change":"2024-09-16T10:13:39.107923+0000","last_active":"2024-09-16T10:14:17.888338+0000","last_peered":"2024-09-16T10:14:17.888338+0000","last_clean":"2024-09-16T10:14:17.888338+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:14:17.888338+0000","last_undegraded":"2024-09-16T10:14:17.888338+0000","last_fullsized":"2024-09-16T10:14:17.888338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.475348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710520+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:14:20.710520+0000","last_peered":"2024-09-16T10:14:20.710520+0000","last_clean":"2024-09-16T10:14:20.710520+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:14:20.710520+0000","last_undegraded":"2024-09-16T10:14:20.710520+0000","last_fullsized":"2024-09-16T10:14:20.710520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.978048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677715+0000","last_change":"2024-09-16T10:13:43.283644+0000","last_active":"2024-09-16T10:14:21.677715+0000","last_peered":"2024-09-16T10:14:21.677715+0000","last_clean":"2024-09-16T10:14:21.677715+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:14:21.677715+0000","last_undegraded":"2024-09-16T10:14:21.677715+0000","last_fullsized":"2024-09-16T10:14:21.677715+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.605473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428889+0000","last_change":"2024-09-16T10:13:39.107919+0000","last_active":"2024-09-16T10:14:16.428889+0000","last_peered":"2024-09-16T10:14:16.428889+0000","last_clean":"2024-09-16T10:14:16.428889+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:14:16.428889+0000","last_undegraded":"2024-09-16T10:14:16.428889+0000","last_fullsized":"2024-09-16T10:14:16.428889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.504540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677073+0000","last_change":"2024-09-16T10:13:37.095220+0000","last_active":"2024-09-16T10:14:21.677073+0000","last_peered":"2024-09-16T10:14:21.677073+0000","last_clean":"2024-09-16T10:14:21.677073+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:14:21.677073+0000","last_undegraded":"2024-09-16T10:14:21.677073+0000","last_fullsized":"2024-09-16T10:14:21.677073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.601224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888268+0000","last_change":"2024-09-16T10:13:41.118905+0000","last_active":"2024-09-16T10:14:17.888268+0000","last_peered":"2024-09-16T10:14:17.888268+0000","last_clean":"2024-09-16T10:14:17.888268+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:14:17.888268+0000","last_undegraded":"2024-09-16T10:14:17.888268+0000","last_fullsized":"2024-09-16T10:14:17.888268+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.674048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.586068+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:14:21.586068+0000","last_peered":"2024-09-16T10:14:21.586068+0000","last_clean":"2024-09-16T10:14:21.586068+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:14:21.586068+0000","last_undegraded":"2024-09-16T10:14:21.586068+0000","last_fullsized":"2024-09-16T10:14:21.586068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.053913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710440+0000","last_change":"2024-09-16T10:13:39.105916+0000","last_active":"2024-09-16T10:14:20.710440+0000","last_peered":"2024-09-16T10:14:20.710440+0000","last_clean":"2024-09-16T10:14:20.710440+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:14:20.710440+0000","last_undegraded":"2024-09-16T10:14:20.710440+0000","last_fullsized":"2024-09-16T10:14:20.710440+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.209227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427640+0000","last_change":"2024-09-16T10:13:37.097837+0000","last_active":"2024-09-16T10:14:16.427640+0000","last_peered":"2024-09-16T10:14:16.427640+0000","last_clean":"2024-09-16T10:14:16.427640+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:14:16.427640+0000","last_undegraded":"2024-09-16T10:14:16.427640+0000","last_fullsized":"2024-09-16T10:14:16.427640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:36.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887879+0000","last_change":"2024-09-16T10:13:41.119310+0000","last_active":"2024-09-16T10:14:17.887879+0000","last_peered":"2024-09-16T10:14:17.887879+0000","last_clean":"2024-09-16T10:14:17.887879+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:14:17.887879+0000","last_undegraded":"2024-09-16T10:14:17.887879+0000","last_fullsized":"2024-09-16T10:14:17.887879+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.607866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678685+0000","last_change":"2024-09-16T10:13:43.119772+0000","last_active":"2024-09-16T10:14:21.678685+0000","last_peered":"2024-09-16T10:14:21.678685+0000","last_clean":"2024-09-16T10:14:21.678685+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:14:21.678685+0000","last_undegraded":"2024-09-16T10:14:21.678685+0000","last_fullsized":"2024-09-16T10:14:21.678685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.088506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.888130+0000","last_change":"2024-09-16T10:13:39.105939+0000","last_active":"2024-09-16T10:14:17.888130+0000","last_peered":"2024-09-16T10:14:17.888130+0000","last_clean":"2024-09-16T10:14:17.888130+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:14:17.888130+0000","last_undegraded":"2024-09-16T10:14:17.888130+0000","last_fullsized":"2024-09-16T10:14:17.888130+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.788054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677929+0000","last_change":"2024-09-16T10:13:37.094640+0000","last_active":"2024-09-16T10:14:21.677929+0000","last_peered":"2024-09-16T10:14:21.677929+0000","last_clean":"2024-09-16T10:14:21.677929+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:14:21.677929+0000","last_undegraded":"2024-09-16T10:14:21.677929+0000","last_fullsized":"2024-09-16T10:14:21.677929+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.045517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710376+0000","last_change":"2024-09-16T10:13:41.116455+0000","last_active":"2024-09-16T10:14:20.710376+0000","last_peered":"2024-09-16T10:14:20.710376+0000","last_clean":"2024-09-16T10:14:20.710376+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:14:20.710376+0000","last_undegraded":"2024-09-16T10:14:20.710376+0000","last_fullsized":"2024-09-16T10:14:20.710376+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.665680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428456+0000","last_change":"2024-09-16T10:13:43.120971+0000","last_active":"2024-09-16T10:14:16.428456+0000","last_peered":"2024-09-16T10:14:16.428456+0000","last_clean":"2024-09-16T10:14:16.428456+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:14:16.428456+0000","last_undegraded":"2024-09-16T10:14:16.428456+0000","last_fullsized":"2024-09-16T10:14:16.428456+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.791625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677780+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:14:21.677780+0000","last_peered":"2024-09-16T10:14:21.677780+0000","last_clean":"2024-09-16T10:14:21.677780+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:14:21.677780+0000","last_undegraded":"2024-09-16T10:14:21.677780+0000","last_fullsized":"2024-09-16T10:14:21.677780+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.996979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.678648+0000","last_change":"2024-09-16T10:13:37.090129+0000","last_active":"2024-09-16T10:14:21.678648+0000","last_peered":"2024-09-16T10:14:21.678648+0000","last_clean":"2024-09-16T10:14:21.678648+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:14:21.678648+0000","last_undegraded":"2024-09-16T10:14:21.678648+0000","last_fullsized":"2024-09-16T10:14:21.678648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.823652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.706829+0000","last_change":"2024-09-16T10:13:41.112912+0000","last_active":"2024-09-16T10:14:20.706829+0000","last_peered":"2024-09-16T10:14:20.706829+0000","last_clean":"2024-09-16T10:14:20.706829+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:14:20.706829+0000","last_undegraded":"2024-09-16T10:14:20.706829+0000","last_fullsized":"2024-09-16T10:14:20.706829+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.606633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887373+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:14:17.887373+0000","last_peered":"2024-09-16T10:14:17.887373+0000","last_clean":"2024-09-16T10:14:17.887373+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:14:17.887373+0000","last_undegraded":"2024-09-16T10:14:17.887373+0000","last_fullsized":"2024-09-16T10:14:17.887373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.223461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.707026+0000","last_change":"2024-09-16T10:13:39.098286+0000","last_active":"2024-09-16T10:14:20.707026+0000","last_peered":"2024-09-16T10:14:20.707026+0000","last_clean":"2024-09-16T10:14:20.707026+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:14:20.707026+0000","last_undegraded":"2024-09-16T10:14:20.707026+0000","last_fullsized":"2024-09-16T10:14:20.707026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.259110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677925+0000","last_change":"2024-09-16T10:13:37.091081+0000","last_active":"2024-09-16T10:14:21.677925+0000","last_peered":"2024-09-16T10:14:21.677925+0000","last_clean":"2024-09-16T10:14:21.677925+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:14:21.677925+0000","last_undegraded":"2024-09-16T10:14:21.677925+0000","last_fullsized":"2024-09-16T10:14:21.677925+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.952981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887757+0000","last_change":"2024-09-16T10:13:41.114260+0000","last_active":"2024-09-16T10:14:17.887757+0000","last_peered":"2024-09-16T10:14:17.887757+0000","last_clean":"2024-09-16T10:14:17.887757+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:14:17.887757+0000","last_undegraded":"2024-09-16T10:14:17.887757+0000","last_fullsized":"2024-09-16T10:14:17.887757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.627978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":32,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.427406+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:14:16.427406+0000","last_peered":"2024-09-16T10:14:16.427406+0000","last_clean":"2024-09-16T10:14:16.427406+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:14:16.427406+0000","last_undegraded":"2024-09-16T10:14:16.427406+0000","last_fullsized":"2024-09-16T10:14:16.427406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887530+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:14:17.887530+0000","last_peered":"2024-09-16T10:14:17.887530+0000","last_clean":"2024-09-16T10:14:17.887530+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:14:17.887530+0000","last_undegraded":"2024-09-16T10:14:17.887530+0000","last_fullsized":"2024-09-16T10:14:17.887530+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.847239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.710535+0000","last_change":"2024-09-16T10:13:37.088453+0000","last_active":"2024-09-16T10:14:20.710535+0000","last_peered":"2024-09-16T10:14:20.710535+0000","last_clean":"2024-09-16T10:14:20.710535+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:14:20.710535+0000","last_undegraded":"2024-09-16T10:14:20.710535+0000","last_fullsized":"2024-09-16T10:14:20.710535+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.596318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.677318+0000","last_change":"2024-09-16T10:13:41.112830+0000","last_active":"2024-09-16T10:14:21.677318+0000","last_peered":"2024-09-16T10:14:21.677318+0000","last_clean":"2024-09-16T10:14:21.677318+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:14:21.677318+0000","last_undegraded":"2024-09-16T10:14:21.677318+0000","last_fullsized":"2024-09-16T10:14:21.677318+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:41.842458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887901+0000","last_change":"2024-09-16T10:13:43.286716+0000","last_active":"2024-09-16T10:14:17.887901+0000","last_peered":"2024-09-16T10:14:17.887901+0000","last_clean":"2024-09-16T10:14:17.887901+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:14:17.887901+0000","last_undegraded":"2024-09-16T10:14:17.887901+0000","last_fullsized":"2024-09-16T10:14:17.887901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.000117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887375+0000","last_change":"2024-09-16T10:13:39.101213+0000","last_active":"2024-09-16T10:14:17.887375+0000","last_peered":"2024-09-16T10:14:17.887375+0000","last_clean":"2024-09-16T10:14:17.887375+0000","last_became_active":"2024-09-16T10:13:39.100788+0000","last_became_peered":"2024-09-16T10:13:39.100788+0000","last_unstale":"2024-09-16T10:14:17.887375+0000","last_undegraded":"2024-09-16T10:14:17.887375+0000","last_fullsized":"2024-09-16T10:14:17.887375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.518507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":50,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428339+0000","last_change":"2024-09-16T10:13:37.094533+0000","last_active":"2024-09-16T10:14:16.428339+0000","last_peered":"2024-09-16T10:14:16.428339+0000","last_clean":"2024-09-16T10:14:16.428339+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:14:16.428339+0000","last_undegraded":"2024-09-16T10:14:16.428339+0000","last_fullsized":"2024-09-16T10:14:16.428339+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.183923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":41,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.488677+0000","last_change":"2024-09-16T10:13:41.114472+0000","last_active":"2024-09-16T10:14:21.488677+0000","last_peered":"2024-09-16T10:14:21.488677+0000","last_clean":"2024-09-16T10:14:21.488677+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:14:21.488677+0000","last_undegraded":"2024-09-16T10:14:21.488677+0000","last_fullsized":"2024-09-16T10:14:21.488677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.243642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.676678+0000","last_change":"2024-09-16T10:13:43.119917+0000","last_active":"2024-09-16T10:14:21.676678+0000","last_peered":"2024-09-16T10:14:21.676678+0000","last_clean":"2024-09-16T10:14:21.676678+0000","last_became_active":"2024-09-16T10:13:43.119612+0000","last_became_peered":"2024-09-16T10:13:43.119612+0000","last_unstale":"2024-09-16T10:14:21.676678+0000","last_undegraded":"2024-09-16T10:14:21.676678+0000","last_fullsized":"2024-09-16T10:14:21.676678+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.539099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:20.707181+0000","last_change":"2024-09-16T10:13:43.114761+0000","last_active":"2024-09-16T10:14:20.707181+0000","last_peered":"2024-09-16T10:14:20.707181+0000","last_clean":"2024-09-16T10:14:20.707181+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:14:20.707181+0000","last_undegraded":"2024-09-16T10:14:20.707181+0000","last_fullsized":"2024-09-16T10:14:20.707181+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.210195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-16T10:14:17.887548+0000","last_change":"2024-09-16T10:13:37.089832+0000","last_active":"2024-09-16T10:14:17.887548+0000","last_peered":"2024-09-16T10:14:17.887548+0000","last_clean":"2024-09-16T10:14:17.887548+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:14:17.887548+0000","last_undegraded":"2024-09-16T10:14:17.887548+0000","last_fullsized":"2024-09-16T10:14:17.887548+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.587954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":64,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-16T10:14:21.489338+0000","last_change":"2024-09-16T10:13:39.098801+0000","last_active":"2024-09-16T10:14:21.489338+0000","last_peered":"2024-09-16T10:14:21.489338+0000","last_clean":"2024-09-16T10:14:21.489338+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:14:21.489338+0000","last_undegraded":"2024-09-16T10:14:21.489338+0000","last_fullsized":"2024-09-16T10:14:21.489338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.353803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-16T10:14:16.428600+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:16.428600+0000","last_peered":"2024-09-16T10:14:16.428600+0000","last_clean":"2024-09-16T10:14:16.428600+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:14:16.428600+0000","last_undegraded":"2024-09-16T10:14:16.428600+0000","last_fullsized":"2024-09-16T10:14:16.428600+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.860042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233997,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1396736,"data_stored":914771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528337,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32528,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712624,"statfs":{"total":95995035648,"available":95961726976,"internally_reserved":0,"allocated":1372160,"data_stored":913673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789975,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":786432,"data_stored":324657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":811008,"data_stored":324346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215136,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":786432,"data_stored":306409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":778240,"data_stored":323466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":847872,"data_stored":325526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1368064,"data_stored":896257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:14:25.214 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T10:14:25.214 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T10:14:25.214 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-16T10:14:25.214 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-16T10:14:25.786 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:25 smithi165 ceph-mon[32696]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T10:14:26.104 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:25 smithi063 ceph-mon[29993]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T10:14:26.104 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:25 smithi063 ceph-mon[39470]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T10:14:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:14:26.580 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T10:14:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/840613328' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-16T10:14:27.073 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/840613328' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-16T10:14:27.073 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/840613328' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-16T10:14:27.390 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:14:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:14:28.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:27 smithi063 ceph-mon[29993]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-09-16T10:14:28.035 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:27 smithi063 ceph-mon[39470]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-09-16T10:14:28.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:27 smithi165 ceph-mon[32696]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-09-16T10:14:28.071 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-16T10:14:28.072 INFO:tasks.cephadm:Setup complete, yielding 2024-09-16T10:14:28.072 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T10:14:28.085 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T10:14:28.085 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-athakkar-testing-2024-09-12-1857'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-athakkar-testing-2024-09-12-1857', 'sha1': '66e33d6af1496291b0602ee0a859e68145f17dad'}} 2024-09-16T10:14:28.085 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T10:14:28.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T10:14:28.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:14:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:14:28.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:28 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:29.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:28 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:29.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:28 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:29.929 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:29 smithi063 ceph-mon[29993]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:14:29.932 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:29 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:14:29.932 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:29 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:14:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:29 smithi063 ceph-mon[39470]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:14:29.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:29 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:14:29.933 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:29 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:14:30.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:29 smithi165 ceph-mon[32696]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:14:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:29 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:14:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:29 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:14:30.803 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:30 smithi063 ceph-mon[29993]: osdmap e79: 8 total, 8 up, 8 in 2024-09-16T10:14:30.803 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:30 smithi063 ceph-mon[29993]: osdmap e80: 8 total, 8 up, 8 in 2024-09-16T10:14:30.804 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:30 smithi063 ceph-mon[39470]: osdmap e79: 8 total, 8 up, 8 in 2024-09-16T10:14:30.804 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:30 smithi063 ceph-mon[39470]: osdmap e80: 8 total, 8 up, 8 in 2024-09-16T10:14:30.957 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:30 smithi165 ceph-mon[32696]: osdmap e79: 8 total, 8 up, 8 in 2024-09-16T10:14:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:30 smithi165 ceph-mon[32696]: osdmap e80: 8 total, 8 up, 8 in 2024-09-16T10:14:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:31 smithi165 ceph-mon[32696]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:14:32.081 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:31 smithi063 ceph-mon[29993]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:14:32.081 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:31 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:31] ENGINE Bus STOPPING 2024-09-16T10:14:32.081 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T10:14:32.082 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:31 smithi063 ceph-mon[39470]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:14:32.082 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:31 smithi063 podman[70717]: 2024-09-16 10:14:31.842127451 +0000 UTC m=+0.024428899 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-16T10:14:32.391 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 podman[70717]: 2024-09-16 10:14:32.217051817 +0000 UTC m=+0.399353297 container create 0f853ece66bf260487e9fe535ef7777d55832ff43ac5fb18b6e85c3581cee928 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:32.391 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:32] ENGINE Bus STOPPED 2024-09-16T10:14:32.391 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:32] ENGINE Bus STARTING 2024-09-16T10:14:32.391 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:32] ENGINE Serving on http://:::9283 2024-09-16T10:14:32.391 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:32] ENGINE Bus STARTED 2024-09-16T10:14:32.761 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 podman[70717]: 2024-09-16 10:14:32.48605824 +0000 UTC m=+0.668359679 container init 0f853ece66bf260487e9fe535ef7777d55832ff43ac5fb18b6e85c3581cee928 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 podman[70717]: 2024-09-16 10:14:32.489121942 +0000 UTC m=+0.671423390 container start 0f853ece66bf260487e9fe535ef7777d55832ff43ac5fb18b6e85c3581cee928 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.499Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.499Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.500Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.63 port=9094 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.501Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.522Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.523Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.524Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:32.524Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 bash[70717]: 0f853ece66bf260487e9fe535ef7777d55832ff43ac5fb18b6e85c3581cee928 2024-09-16T10:14:32.762 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:32 smithi063 systemd[1]: Started Ceph alertmanager.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:14:33.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/31391559' entity='client.admin' 2024-09-16T10:14:33.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:33.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:33.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/31391559' entity='client.admin' 2024-09-16T10:14:33.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:33.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:33.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.073 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T10:14:33.073 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T10:14:33.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:14:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/31391559' entity='client.admin' 2024-09-16T10:14:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:32 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:33.337 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:34.037 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:14:34.038 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[29993]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-16T10:14:34.038 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[29993]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:34.038 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.038 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.038 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-16T10:14:34.038 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.038 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:33] ENGINE Bus STOPPING 2024-09-16T10:14:34.039 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[39470]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-16T10:14:34.039 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[39470]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:34.039 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.039 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.039 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-16T10:14:34.039 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:33 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.045 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:14:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:33 smithi165 ceph-mon[32696]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-16T10:14:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:33 smithi165 ceph-mon[32696]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:33 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:33 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:33 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-16T10:14:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:33 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:34.336 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T10:14:34.336 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:34] ENGINE Bus STOPPED 2024-09-16T10:14:34.336 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:34] ENGINE Bus STARTING 2024-09-16T10:14:34.702 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:34.502Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000808974s 2024-09-16T10:14:34.702 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:34] ENGINE Serving on http://:::9283 2024-09-16T10:14:34.702 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:14:34] ENGINE Bus STARTED 2024-09-16T10:14:34.751 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T10:14:33.012109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":898,"num_read_kb":755,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241088,"kb_used_data":8208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720128,"statfs":{"total":767960285184,"available":767713411072,"internally_reserved":0,"allocated":8404992,"data_stored":4463517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002401"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070691+0000","last_change":"2024-09-16T10:13:37.096999+0000","last_active":"2024-09-16T10:14:31.070691+0000","last_peered":"2024-09-16T10:14:31.070691+0000","last_clean":"2024-09-16T10:14:31.070691+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:14:31.070691+0000","last_undegraded":"2024-09-16T10:14:31.070691+0000","last_fullsized":"2024-09-16T10:14:31.070691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710732+0000","last_change":"2024-09-16T10:13:39.093832+0000","last_active":"2024-09-16T10:14:30.710732+0000","last_peered":"2024-09-16T10:14:30.710732+0000","last_clean":"2024-09-16T10:14:30.710732+0000","last_became_active":"2024-09-16T10:13:39.093430+0000","last_became_peered":"2024-09-16T10:13:39.093430+0000","last_unstale":"2024-09-16T10:14:30.710732+0000","last_undegraded":"2024-09-16T10:14:30.710732+0000","last_fullsized":"2024-09-16T10:14:30.710732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.941021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715030+0000","last_change":"2024-09-16T10:13:41.108286+0000","last_active":"2024-09-16T10:14:30.715030+0000","last_peered":"2024-09-16T10:14:30.715030+0000","last_clean":"2024-09-16T10:14:30.715030+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:14:30.715030+0000","last_undegraded":"2024-09-16T10:14:30.715030+0000","last_fullsized":"2024-09-16T10:14:30.715030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.804825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710853+0000","last_change":"2024-09-16T10:13:43.113148+0000","last_active":"2024-09-16T10:14:30.710853+0000","last_peered":"2024-09-16T10:14:30.710853+0000","last_clean":"2024-09-16T10:14:30.710853+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:14:30.710853+0000","last_undegraded":"2024-09-16T10:14:30.710853+0000","last_fullsized":"2024-09-16T10:14:30.710853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.390079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344767+0000","last_change":"2024-09-16T10:13:43.287323+0000","last_active":"2024-09-16T10:14:31.344767+0000","last_peered":"2024-09-16T10:14:31.344767+0000","last_clean":"2024-09-16T10:14:31.344767+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:14:31.344767+0000","last_undegraded":"2024-09-16T10:14:31.344767+0000","last_fullsized":"2024-09-16T10:14:31.344767+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.723245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345790+0000","last_change":"2024-09-16T10:13:37.092457+0000","last_active":"2024-09-16T10:14:31.345790+0000","last_peered":"2024-09-16T10:14:31.345790+0000","last_clean":"2024-09-16T10:14:31.345790+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:14:31.345790+0000","last_undegraded":"2024-09-16T10:14:31.345790+0000","last_fullsized":"2024-09-16T10:14:31.345790+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.920553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345322+0000","last_change":"2024-09-16T10:13:39.095829+0000","last_active":"2024-09-16T10:14:31.345322+0000","last_peered":"2024-09-16T10:14:31.345322+0000","last_clean":"2024-09-16T10:14:31.345322+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:14:31.345322+0000","last_undegraded":"2024-09-16T10:14:31.345322+0000","last_fullsized":"2024-09-16T10:14:31.345322+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.084583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710720+0000","last_change":"2024-09-16T10:13:41.121501+0000","last_active":"2024-09-16T10:14:30.710720+0000","last_peered":"2024-09-16T10:14:30.710720+0000","last_clean":"2024-09-16T10:14:30.710720+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:14:30.710720+0000","last_undegraded":"2024-09-16T10:14:30.710720+0000","last_fullsized":"2024-09-16T10:14:30.710720+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.386931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106410+0000","last_change":"2024-09-16T10:13:37.092847+0000","last_active":"2024-09-16T10:14:32.106410+0000","last_peered":"2024-09-16T10:14:32.106410+0000","last_clean":"2024-09-16T10:14:32.106410+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:14:32.106410+0000","last_undegraded":"2024-09-16T10:14:32.106410+0000","last_fullsized":"2024-09-16T10:14:32.106410+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.507863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711156+0000","last_change":"2024-09-16T10:13:39.092562+0000","last_active":"2024-09-16T10:14:30.711156+0000","last_peered":"2024-09-16T10:14:30.711156+0000","last_clean":"2024-09-16T10:14:30.711156+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:14:30.711156+0000","last_undegraded":"2024-09-16T10:14:30.711156+0000","last_fullsized":"2024-09-16T10:14:30.711156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.237966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106150+0000","last_change":"2024-09-16T10:13:41.112242+0000","last_active":"2024-09-16T10:14:32.106150+0000","last_peered":"2024-09-16T10:14:32.106150+0000","last_clean":"2024-09-16T10:14:32.106150+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:14:32.106150+0000","last_undegraded":"2024-09-16T10:14:32.106150+0000","last_fullsized":"2024-09-16T10:14:32.106150+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.338353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.069948+0000","last_change":"2024-09-16T10:13:43.115780+0000","last_active":"2024-09-16T10:14:31.069948+0000","last_peered":"2024-09-16T10:14:31.069948+0000","last_clean":"2024-09-16T10:14:31.069948+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:14:31.069948+0000","last_undegraded":"2024-09-16T10:14:31.069948+0000","last_fullsized":"2024-09-16T10:14:31.069948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.438318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106301+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:14:32.106301+0000","last_peered":"2024-09-16T10:14:32.106301+0000","last_clean":"2024-09-16T10:14:32.106301+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:14:32.106301+0000","last_undegraded":"2024-09-16T10:14:32.106301+0000","last_fullsized":"2024-09-16T10:14:32.106301+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.318174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711001+0000","last_change":"2024-09-16T10:13:39.092696+0000","last_active":"2024-09-16T10:14:30.711001+0000","last_peered":"2024-09-16T10:14:30.711001+0000","last_clean":"2024-09-16T10:14:30.711001+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:14:30.711001+0000","last_undegraded":"2024-09-16T10:14:30.711001+0000","last_fullsized":"2024-09-16T10:14:30.711001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.518434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807838+0000","last_change":"2024-09-16T10:13:41.111640+0000","last_active":"2024-09-16T10:14:31.807838+0000","last_peered":"2024-09-16T10:14:31.807838+0000","last_clean":"2024-09-16T10:14:31.807838+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:14:31.807838+0000","last_undegraded":"2024-09-16T10:14:31.807838+0000","last_fullsized":"2024-09-16T10:14:31.807838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.280028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105300+0000","last_change":"2024-09-16T10:13:43.121701+0000","last_active":"2024-09-16T10:14:32.105300+0000","last_peered":"2024-09-16T10:14:32.105300+0000","last_clean":"2024-09-16T10:14:32.105300+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:14:32.105300+0000","last_undegraded":"2024-09-16T10:14:32.105300+0000","last_fullsized":"2024-09-16T10:14:32.105300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.188718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711596+0000","last_change":"2024-09-16T10:13:43.284208+0000","last_active":"2024-09-16T10:14:30.711596+0000","last_peered":"2024-09-16T10:14:30.711596+0000","last_clean":"2024-09-16T10:14:30.711596+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:14:30.711596+0000","last_undegraded":"2024-09-16T10:14:30.711596+0000","last_fullsized":"2024-09-16T10:14:30.711596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.053189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070607+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:14:31.070607+0000","last_peered":"2024-09-16T10:14:31.070607+0000","last_clean":"2024-09-16T10:14:31.070607+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:14:31.070607+0000","last_undegraded":"2024-09-16T10:14:31.070607+0000","last_fullsized":"2024-09-16T10:14:31.070607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.063039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714875+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:14:30.714875+0000","last_peered":"2024-09-16T10:14:30.714875+0000","last_clean":"2024-09-16T10:14:30.714875+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:14:30.714875+0000","last_undegraded":"2024-09-16T10:14:30.714875+0000","last_fullsized":"2024-09-16T10:14:30.714875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.478889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715963+0000","last_change":"2024-09-16T10:13:41.120296+0000","last_active":"2024-09-16T10:14:30.715963+0000","last_peered":"2024-09-16T10:14:30.715963+0000","last_clean":"2024-09-16T10:14:30.715963+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:14:30.715963+0000","last_undegraded":"2024-09-16T10:14:30.715963+0000","last_fullsized":"2024-09-16T10:14:30.715963+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.391104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344582+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:14:31.344582+0000","last_peered":"2024-09-16T10:14:31.344582+0000","last_clean":"2024-09-16T10:14:31.344582+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:14:31.344582+0000","last_undegraded":"2024-09-16T10:14:31.344582+0000","last_fullsized":"2024-09-16T10:14:31.344582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.407109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711359+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:14:30.711359+0000","last_peered":"2024-09-16T10:14:30.711359+0000","last_clean":"2024-09-16T10:14:30.711359+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:14:30.711359+0000","last_undegraded":"2024-09-16T10:14:30.711359+0000","last_fullsized":"2024-09-16T10:14:30.711359+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.861513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":73,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345647+0000","last_change":"2024-09-16T10:13:39.100013+0000","last_active":"2024-09-16T10:14:31.345647+0000","last_peered":"2024-09-16T10:14:31.345647+0000","last_clean":"2024-09-16T10:14:31.345647+0000","last_became_active":"2024-09-16T10:13:39.099675+0000","last_became_peered":"2024-09-16T10:13:39.099675+0000","last_unstale":"2024-09-16T10:14:31.345647+0000","last_undegraded":"2024-09-16T10:14:31.345647+0000","last_fullsized":"2024-09-16T10:14:31.345647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.654393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711399+0000","last_change":"2024-09-16T10:13:41.112993+0000","last_active":"2024-09-16T10:14:30.711399+0000","last_peered":"2024-09-16T10:14:30.711399+0000","last_clean":"2024-09-16T10:14:30.711399+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:14:30.711399+0000","last_undegraded":"2024-09-16T10:14:30.711399+0000","last_fullsized":"2024-09-16T10:14:30.711399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.116156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807329+0000","last_change":"2024-09-16T10:13:43.119664+0000","last_active":"2024-09-16T10:14:31.807329+0000","last_peered":"2024-09-16T10:14:31.807329+0000","last_clean":"2024-09-16T10:14:31.807329+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:14:31.807329+0000","last_undegraded":"2024-09-16T10:14:31.807329+0000","last_fullsized":"2024-09-16T10:14:31.807329+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.178009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715210+0000","last_change":"2024-09-16T10:13:37.082594+0000","last_active":"2024-09-16T10:14:30.715210+0000","last_peered":"2024-09-16T10:14:30.715210+0000","last_clean":"2024-09-16T10:14:30.715210+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:14:30.715210+0000","last_undegraded":"2024-09-16T10:14:30.715210+0000","last_fullsized":"2024-09-16T10:14:30.715210+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.835656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070684+0000","last_change":"2024-09-16T10:13:39.108076+0000","last_active":"2024-09-16T10:14:31.070684+0000","last_peered":"2024-09-16T10:14:31.070684+0000","last_clean":"2024-09-16T10:14:31.070684+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:14:31.070684+0000","last_undegraded":"2024-09-16T10:14:31.070684+0000","last_fullsized":"2024-09-16T10:14:31.070684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.726907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.791198+0000","last_change":"2024-09-16T10:13:41.116955+0000","last_active":"2024-09-16T10:14:30.791198+0000","last_peered":"2024-09-16T10:14:30.791198+0000","last_clean":"2024-09-16T10:14:30.791198+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:14:30.791198+0000","last_undegraded":"2024-09-16T10:14:30.791198+0000","last_fullsized":"2024-09-16T10:14:30.791198+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.203272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716397+0000","last_change":"2024-09-16T10:13:39.096753+0000","last_active":"2024-09-16T10:14:30.716397+0000","last_peered":"2024-09-16T10:14:30.716397+0000","last_clean":"2024-09-16T10:14:30.716397+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:14:30.716397+0000","last_undegraded":"2024-09-16T10:14:30.716397+0000","last_fullsized":"2024-09-16T10:14:30.716397+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.363335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345133+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:14:31.345133+0000","last_peered":"2024-09-16T10:14:31.345133+0000","last_clean":"2024-09-16T10:14:31.345133+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:14:31.345133+0000","last_undegraded":"2024-09-16T10:14:31.345133+0000","last_fullsized":"2024-09-16T10:14:31.345133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.355614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711919+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:14:30.711919+0000","last_peered":"2024-09-16T10:14:30.711919+0000","last_clean":"2024-09-16T10:14:30.711919+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:14:30.711919+0000","last_undegraded":"2024-09-16T10:14:30.711919+0000","last_fullsized":"2024-09-16T10:14:30.711919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.675105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105564+0000","last_change":"2024-09-16T10:13:43.114886+0000","last_active":"2024-09-16T10:14:32.105564+0000","last_peered":"2024-09-16T10:14:32.105564+0000","last_clean":"2024-09-16T10:14:32.105564+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:14:32.105564+0000","last_undegraded":"2024-09-16T10:14:32.105564+0000","last_fullsized":"2024-09-16T10:14:32.105564+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.234184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":82,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345001+0000","last_change":"2024-09-16T10:13:39.106135+0000","last_active":"2024-09-16T10:14:31.345001+0000","last_peered":"2024-09-16T10:14:31.345001+0000","last_clean":"2024-09-16T10:14:31.345001+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:14:31.345001+0000","last_undegraded":"2024-09-16T10:14:31.345001+0000","last_fullsized":"2024-09-16T10:14:31.345001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.184956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345931+0000","last_change":"2024-09-16T10:13:37.093660+0000","last_active":"2024-09-16T10:14:31.345931+0000","last_peered":"2024-09-16T10:14:31.345931+0000","last_clean":"2024-09-16T10:14:31.345931+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:14:31.345931+0000","last_undegraded":"2024-09-16T10:14:31.345931+0000","last_fullsized":"2024-09-16T10:14:31.345931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.816759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710725+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:14:30.710725+0000","last_peered":"2024-09-16T10:14:30.710725+0000","last_clean":"2024-09-16T10:14:30.710725+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:14:30.710725+0000","last_undegraded":"2024-09-16T10:14:30.710725+0000","last_fullsized":"2024-09-16T10:14:30.710725+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.743002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711089+0000","last_change":"2024-09-16T10:13:43.122864+0000","last_active":"2024-09-16T10:14:30.711089+0000","last_peered":"2024-09-16T10:14:30.711089+0000","last_clean":"2024-09-16T10:14:30.711089+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:14:30.711089+0000","last_undegraded":"2024-09-16T10:14:30.711089+0000","last_fullsized":"2024-09-16T10:14:30.711089+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.235493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710707+0000","last_change":"2024-09-16T10:13:39.094074+0000","last_active":"2024-09-16T10:14:30.710707+0000","last_peered":"2024-09-16T10:14:30.710707+0000","last_clean":"2024-09-16T10:14:30.710707+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:14:30.710707+0000","last_undegraded":"2024-09-16T10:14:30.710707+0000","last_fullsized":"2024-09-16T10:14:30.710707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.532032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070847+0000","last_change":"2024-09-16T10:13:37.096046+0000","last_active":"2024-09-16T10:14:31.070847+0000","last_peered":"2024-09-16T10:14:31.070847+0000","last_clean":"2024-09-16T10:14:31.070847+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:14:31.070847+0000","last_undegraded":"2024-09-16T10:14:31.070847+0000","last_fullsized":"2024-09-16T10:14:31.070847+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.593585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344949+0000","last_change":"2024-09-16T10:13:41.112759+0000","last_active":"2024-09-16T10:14:31.344949+0000","last_peered":"2024-09-16T10:14:31.344949+0000","last_clean":"2024-09-16T10:14:31.344949+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:14:31.344949+0000","last_undegraded":"2024-09-16T10:14:31.344949+0000","last_fullsized":"2024-09-16T10:14:31.344949+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.600065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807287+0000","last_change":"2024-09-16T10:13:43.284631+0000","last_active":"2024-09-16T10:14:31.807287+0000","last_peered":"2024-09-16T10:14:31.807287+0000","last_clean":"2024-09-16T10:14:31.807287+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:14:31.807287+0000","last_undegraded":"2024-09-16T10:14:31.807287+0000","last_fullsized":"2024-09-16T10:14:31.807287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.200363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715658+0000","last_change":"2024-09-16T10:13:39.095801+0000","last_active":"2024-09-16T10:14:30.715658+0000","last_peered":"2024-09-16T10:14:30.715658+0000","last_clean":"2024-09-16T10:14:30.715658+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:14:30.715658+0000","last_undegraded":"2024-09-16T10:14:30.715658+0000","last_fullsized":"2024-09-16T10:14:30.715658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.537230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106522+0000","last_change":"2024-09-16T10:13:37.093423+0000","last_active":"2024-09-16T10:14:32.106522+0000","last_peered":"2024-09-16T10:14:32.106522+0000","last_clean":"2024-09-16T10:14:32.106522+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:14:32.106522+0000","last_undegraded":"2024-09-16T10:14:32.106522+0000","last_fullsized":"2024-09-16T10:14:32.106522+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.178620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070387+0000","last_change":"2024-09-16T10:13:41.115177+0000","last_active":"2024-09-16T10:14:31.070387+0000","last_peered":"2024-09-16T10:14:31.070387+0000","last_clean":"2024-09-16T10:14:31.070387+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:14:31.070387+0000","last_undegraded":"2024-09-16T10:14:31.070387+0000","last_fullsized":"2024-09-16T10:14:31.070387+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.803669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070057+0000","last_change":"2024-09-16T10:13:43.119055+0000","last_active":"2024-09-16T10:14:31.070057+0000","last_peered":"2024-09-16T10:14:31.070057+0000","last_clean":"2024-09-16T10:14:31.070057+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:14:31.070057+0000","last_undegraded":"2024-09-16T10:14:31.070057+0000","last_fullsized":"2024-09-16T10:14:31.070057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:54.513879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":85,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070453+0000","last_change":"2024-09-16T10:13:39.106153+0000","last_active":"2024-09-16T10:14:31.070453+0000","last_peered":"2024-09-16T10:14:31.070453+0000","last_clean":"2024-09-16T10:14:31.070453+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:14:31.070453+0000","last_undegraded":"2024-09-16T10:14:31.070453+0000","last_fullsized":"2024-09-16T10:14:31.070453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.043307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716254+0000","last_change":"2024-09-16T10:13:37.092754+0000","last_active":"2024-09-16T10:14:30.716254+0000","last_peered":"2024-09-16T10:14:30.716254+0000","last_clean":"2024-09-16T10:14:30.716254+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:14:30.716254+0000","last_undegraded":"2024-09-16T10:14:30.716254+0000","last_fullsized":"2024-09-16T10:14:30.716254+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.036476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790757+0000","last_change":"2024-09-16T10:13:41.114703+0000","last_active":"2024-09-16T10:14:30.790757+0000","last_peered":"2024-09-16T10:14:30.790757+0000","last_clean":"2024-09-16T10:14:30.790757+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:14:30.790757+0000","last_undegraded":"2024-09-16T10:14:30.790757+0000","last_fullsized":"2024-09-16T10:14:30.790757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.640025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714930+0000","last_change":"2024-09-16T10:13:43.283556+0000","last_active":"2024-09-16T10:14:30.714930+0000","last_peered":"2024-09-16T10:14:30.714930+0000","last_clean":"2024-09-16T10:14:30.714930+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:14:30.714930+0000","last_undegraded":"2024-09-16T10:14:30.714930+0000","last_fullsized":"2024-09-16T10:14:30.714930+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:52.096829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":110,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716149+0000","last_change":"2024-09-16T10:13:39.098696+0000","last_active":"2024-09-16T10:14:30.716149+0000","last_peered":"2024-09-16T10:14:30.716149+0000","last_clean":"2024-09-16T10:14:30.716149+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:14:30.716149+0000","last_undegraded":"2024-09-16T10:14:30.716149+0000","last_fullsized":"2024-09-16T10:14:30.716149+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.501760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711529+0000","last_change":"2024-09-16T10:13:37.092585+0000","last_active":"2024-09-16T10:14:30.711529+0000","last_peered":"2024-09-16T10:14:30.711529+0000","last_clean":"2024-09-16T10:14:30.711529+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:14:30.711529+0000","last_undegraded":"2024-09-16T10:14:30.711529+0000","last_fullsized":"2024-09-16T10:14:30.711529+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.123070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105364+0000","last_change":"2024-09-16T10:13:39.091269+0000","last_active":"2024-09-16T10:14:32.105364+0000","last_peered":"2024-09-16T10:14:32.105364+0000","last_clean":"2024-09-16T10:14:32.105364+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:14:32.105364+0000","last_undegraded":"2024-09-16T10:14:32.105364+0000","last_fullsized":"2024-09-16T10:14:32.105364+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.638736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070481+0000","last_change":"2024-09-16T10:13:41.118142+0000","last_active":"2024-09-16T10:14:31.070481+0000","last_peered":"2024-09-16T10:14:31.070481+0000","last_clean":"2024-09-16T10:14:31.070481+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:14:31.070481+0000","last_undegraded":"2024-09-16T10:14:31.070481+0000","last_fullsized":"2024-09-16T10:14:31.070481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.463063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344939+0000","last_change":"2024-09-16T10:13:43.124755+0000","last_active":"2024-09-16T10:14:31.344939+0000","last_peered":"2024-09-16T10:14:31.344939+0000","last_clean":"2024-09-16T10:14:31.344939+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:14:31.344939+0000","last_undegraded":"2024-09-16T10:14:31.344939+0000","last_fullsized":"2024-09-16T10:14:31.344939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.479324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716471+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:14:30.716471+0000","last_peered":"2024-09-16T10:14:30.716471+0000","last_clean":"2024-09-16T10:14:30.716471+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:14:30.716471+0000","last_undegraded":"2024-09-16T10:14:30.716471+0000","last_fullsized":"2024-09-16T10:14:30.716471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.987015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714742+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:14:30.714742+0000","last_peered":"2024-09-16T10:14:30.714742+0000","last_clean":"2024-09-16T10:14:30.714742+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:14:30.714742+0000","last_undegraded":"2024-09-16T10:14:30.714742+0000","last_fullsized":"2024-09-16T10:14:30.714742+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":63,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714702+0000","last_change":"2024-09-16T10:13:39.090453+0000","last_active":"2024-09-16T10:14:30.714702+0000","last_peered":"2024-09-16T10:14:30.714702+0000","last_clean":"2024-09-16T10:14:30.714702+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:14:30.714702+0000","last_undegraded":"2024-09-16T10:14:30.714702+0000","last_fullsized":"2024-09-16T10:14:30.714702+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.048019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714604+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:14:30.714604+0000","last_peered":"2024-09-16T10:14:30.714604+0000","last_clean":"2024-09-16T10:14:30.714604+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:14:30.714604+0000","last_undegraded":"2024-09-16T10:14:30.714604+0000","last_fullsized":"2024-09-16T10:14:30.714604+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.485969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807421+0000","last_change":"2024-09-16T10:13:43.283490+0000","last_active":"2024-09-16T10:14:31.807421+0000","last_peered":"2024-09-16T10:14:31.807421+0000","last_clean":"2024-09-16T10:14:31.807421+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:14:31.807421+0000","last_undegraded":"2024-09-16T10:14:31.807421+0000","last_fullsized":"2024-09-16T10:14:31.807421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.489088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070740+0000","last_change":"2024-09-16T10:13:39.091817+0000","last_active":"2024-09-16T10:14:31.070740+0000","last_peered":"2024-09-16T10:14:31.070740+0000","last_clean":"2024-09-16T10:14:31.070740+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:14:31.070740+0000","last_undegraded":"2024-09-16T10:14:31.070740+0000","last_fullsized":"2024-09-16T10:14:31.070740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.559454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070030+0000","last_change":"2024-09-16T10:13:37.096771+0000","last_active":"2024-09-16T10:14:31.070030+0000","last_peered":"2024-09-16T10:14:31.070030+0000","last_clean":"2024-09-16T10:14:31.070030+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:14:31.070030+0000","last_undegraded":"2024-09-16T10:14:31.070030+0000","last_fullsized":"2024-09-16T10:14:31.070030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.458372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":115,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807849+0000","last_change":"2024-09-16T10:13:39.105362+0000","last_active":"2024-09-16T10:14:31.807849+0000","last_peered":"2024-09-16T10:14:31.807849+0000","last_clean":"2024-09-16T10:14:31.807849+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:14:31.807849+0000","last_undegraded":"2024-09-16T10:14:31.807849+0000","last_fullsized":"2024-09-16T10:14:31.807849+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:35.154849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106689+0000","last_change":"2024-09-16T10:13:41.112756+0000","last_active":"2024-09-16T10:14:32.106689+0000","last_peered":"2024-09-16T10:14:32.106689+0000","last_clean":"2024-09-16T10:14:32.106689+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:14:32.106689+0000","last_undegraded":"2024-09-16T10:14:32.106689+0000","last_fullsized":"2024-09-16T10:14:32.106689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.034690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715465+0000","last_change":"2024-09-16T10:13:43.123048+0000","last_active":"2024-09-16T10:14:30.715465+0000","last_peered":"2024-09-16T10:14:30.715465+0000","last_clean":"2024-09-16T10:14:30.715465+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:14:30.715465+0000","last_undegraded":"2024-09-16T10:14:30.715465+0000","last_fullsized":"2024-09-16T10:14:30.715465+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.340209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":83,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790998+0000","last_change":"2024-09-16T10:13:39.106378+0000","last_active":"2024-09-16T10:14:30.790998+0000","last_peered":"2024-09-16T10:14:30.790998+0000","last_clean":"2024-09-16T10:14:30.790998+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:14:30.790998+0000","last_undegraded":"2024-09-16T10:14:30.790998+0000","last_fullsized":"2024-09-16T10:14:30.790998+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.582337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345430+0000","last_change":"2024-09-16T10:13:37.088320+0000","last_active":"2024-09-16T10:14:31.345430+0000","last_peered":"2024-09-16T10:14:31.345430+0000","last_clean":"2024-09-16T10:14:31.345430+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:14:31.345430+0000","last_undegraded":"2024-09-16T10:14:31.345430+0000","last_fullsized":"2024-09-16T10:14:31.345430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:37.142506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":99,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807848+0000","last_change":"2024-09-16T10:14:14.911115+0000","last_active":"2024-09-16T10:14:31.807848+0000","last_peered":"2024-09-16T10:14:31.807848+0000","last_clean":"2024-09-16T10:14:31.807848+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:14:31.807848+0000","last_undegraded":"2024-09-16T10:14:31.807848+0000","last_fullsized":"2024-09-16T10:14:31.807848+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:14:14.911041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:14:14.911041+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:17.171811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808111+0000","last_change":"2024-09-16T10:13:41.108176+0000","last_active":"2024-09-16T10:14:31.808111+0000","last_peered":"2024-09-16T10:14:31.808111+0000","last_clean":"2024-09-16T10:14:31.808111+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:14:31.808111+0000","last_undegraded":"2024-09-16T10:14:31.808111+0000","last_fullsized":"2024-09-16T10:14:31.808111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.887140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105383+0000","last_change":"2024-09-16T10:13:43.121658+0000","last_active":"2024-09-16T10:14:32.105383+0000","last_peered":"2024-09-16T10:14:32.105383+0000","last_clean":"2024-09-16T10:14:32.105383+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:14:32.105383+0000","last_undegraded":"2024-09-16T10:14:32.105383+0000","last_fullsized":"2024-09-16T10:14:32.105383+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:11.237055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711888+0000","last_change":"2024-09-16T10:13:39.094951+0000","last_active":"2024-09-16T10:14:30.711888+0000","last_peered":"2024-09-16T10:14:30.711888+0000","last_clean":"2024-09-16T10:14:30.711888+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:14:30.711888+0000","last_undegraded":"2024-09-16T10:14:30.711888+0000","last_fullsized":"2024-09-16T10:14:30.711888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.179189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711199+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:14:30.711199+0000","last_peered":"2024-09-16T10:14:30.711199+0000","last_clean":"2024-09-16T10:14:30.711199+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:14:30.711199+0000","last_undegraded":"2024-09-16T10:14:30.711199+0000","last_fullsized":"2024-09-16T10:14:30.711199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.958295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106526+0000","last_change":"2024-09-16T10:13:41.116069+0000","last_active":"2024-09-16T10:14:32.106526+0000","last_peered":"2024-09-16T10:14:32.106526+0000","last_clean":"2024-09-16T10:14:32.106526+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:14:32.106526+0000","last_undegraded":"2024-09-16T10:14:32.106526+0000","last_fullsized":"2024-09-16T10:14:32.106526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.901053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344553+0000","last_change":"2024-09-16T10:13:43.283629+0000","last_active":"2024-09-16T10:14:31.344553+0000","last_peered":"2024-09-16T10:14:31.344553+0000","last_clean":"2024-09-16T10:14:31.344553+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:14:31.344553+0000","last_undegraded":"2024-09-16T10:14:31.344553+0000","last_fullsized":"2024-09-16T10:14:31.344553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.176307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711759+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:14:30.711759+0000","last_peered":"2024-09-16T10:14:30.711759+0000","last_clean":"2024-09-16T10:14:30.711759+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:14:30.711759+0000","last_undegraded":"2024-09-16T10:14:30.711759+0000","last_fullsized":"2024-09-16T10:14:30.711759+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.476261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.791073+0000","last_change":"2024-09-16T10:13:37.089527+0000","last_active":"2024-09-16T10:14:30.791073+0000","last_peered":"2024-09-16T10:14:30.791073+0000","last_clean":"2024-09-16T10:14:30.791073+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:14:30.791073+0000","last_undegraded":"2024-09-16T10:14:30.791073+0000","last_fullsized":"2024-09-16T10:14:30.791073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.605092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715526+0000","last_change":"2024-09-16T10:13:41.121389+0000","last_active":"2024-09-16T10:14:30.715526+0000","last_peered":"2024-09-16T10:14:30.715526+0000","last_clean":"2024-09-16T10:14:30.715526+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:14:30.715526+0000","last_undegraded":"2024-09-16T10:14:30.715526+0000","last_fullsized":"2024-09-16T10:14:30.715526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.974031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716031+0000","last_change":"2024-09-16T10:13:43.125890+0000","last_active":"2024-09-16T10:14:30.716031+0000","last_peered":"2024-09-16T10:14:30.716031+0000","last_clean":"2024-09-16T10:14:30.716031+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:14:30.716031+0000","last_undegraded":"2024-09-16T10:14:30.716031+0000","last_fullsized":"2024-09-16T10:14:30.716031+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711837+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:14:30.711837+0000","last_peered":"2024-09-16T10:14:30.711837+0000","last_clean":"2024-09-16T10:14:30.711837+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:14:30.711837+0000","last_undegraded":"2024-09-16T10:14:30.711837+0000","last_fullsized":"2024-09-16T10:14:30.711837+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.021437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345373+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:14:31.345373+0000","last_peered":"2024-09-16T10:14:31.345373+0000","last_clean":"2024-09-16T10:14:31.345373+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:14:31.345373+0000","last_undegraded":"2024-09-16T10:14:31.345373+0000","last_fullsized":"2024-09-16T10:14:31.345373+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.609140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715273+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:14:30.715273+0000","last_peered":"2024-09-16T10:14:30.715273+0000","last_clean":"2024-09-16T10:14:30.715273+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:14:30.715273+0000","last_undegraded":"2024-09-16T10:14:30.715273+0000","last_fullsized":"2024-09-16T10:14:30.715273+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711626+0000","last_change":"2024-09-16T10:13:43.119606+0000","last_active":"2024-09-16T10:14:30.711626+0000","last_peered":"2024-09-16T10:14:30.711626+0000","last_clean":"2024-09-16T10:14:30.711626+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:14:30.711626+0000","last_undegraded":"2024-09-16T10:14:30.711626+0000","last_fullsized":"2024-09-16T10:14:30.711626+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.164444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070344+0000","last_change":"2024-09-16T10:13:39.091847+0000","last_active":"2024-09-16T10:14:31.070344+0000","last_peered":"2024-09-16T10:14:31.070344+0000","last_clean":"2024-09-16T10:14:31.070344+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:14:31.070344+0000","last_undegraded":"2024-09-16T10:14:31.070344+0000","last_fullsized":"2024-09-16T10:14:31.070344+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.954882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106328+0000","last_change":"2024-09-16T10:13:37.090259+0000","last_active":"2024-09-16T10:14:32.106328+0000","last_peered":"2024-09-16T10:14:32.106328+0000","last_clean":"2024-09-16T10:14:32.106328+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:14:32.106328+0000","last_undegraded":"2024-09-16T10:14:32.106328+0000","last_fullsized":"2024-09-16T10:14:32.106328+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.290444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715017+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:14:30.715017+0000","last_peered":"2024-09-16T10:14:30.715017+0000","last_clean":"2024-09-16T10:14:30.715017+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:14:30.715017+0000","last_undegraded":"2024-09-16T10:14:30.715017+0000","last_fullsized":"2024-09-16T10:14:30.715017+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.069978+0000","last_change":"2024-09-16T10:13:43.122627+0000","last_active":"2024-09-16T10:14:31.069978+0000","last_peered":"2024-09-16T10:14:31.069978+0000","last_clean":"2024-09-16T10:14:31.069978+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:14:31.069978+0000","last_undegraded":"2024-09-16T10:14:31.069978+0000","last_fullsized":"2024-09-16T10:14:31.069978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":80,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790875+0000","last_change":"2024-09-16T10:13:39.105379+0000","last_active":"2024-09-16T10:14:30.790875+0000","last_peered":"2024-09-16T10:14:30.790875+0000","last_clean":"2024-09-16T10:14:30.790875+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:14:30.790875+0000","last_undegraded":"2024-09-16T10:14:30.790875+0000","last_fullsized":"2024-09-16T10:14:30.790875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.713974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808351+0000","last_change":"2024-09-16T10:13:37.093687+0000","last_active":"2024-09-16T10:14:31.808351+0000","last_peered":"2024-09-16T10:14:31.808351+0000","last_clean":"2024-09-16T10:14:31.808351+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:14:31.808351+0000","last_undegraded":"2024-09-16T10:14:31.808351+0000","last_fullsized":"2024-09-16T10:14:31.808351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.984174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715071+0000","last_change":"2024-09-16T10:13:41.113511+0000","last_active":"2024-09-16T10:14:30.715071+0000","last_peered":"2024-09-16T10:14:30.715071+0000","last_clean":"2024-09-16T10:14:30.715071+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:14:30.715071+0000","last_undegraded":"2024-09-16T10:14:30.715071+0000","last_fullsized":"2024-09-16T10:14:30.715071+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.162031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807268+0000","last_change":"2024-09-16T10:13:43.122094+0000","last_active":"2024-09-16T10:14:31.807268+0000","last_peered":"2024-09-16T10:14:31.807268+0000","last_clean":"2024-09-16T10:14:31.807268+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:14:31.807268+0000","last_undegraded":"2024-09-16T10:14:31.807268+0000","last_fullsized":"2024-09-16T10:14:31.807268+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.948353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":72,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711031+0000","last_change":"2024-09-16T10:13:39.095588+0000","last_active":"2024-09-16T10:14:30.711031+0000","last_peered":"2024-09-16T10:14:30.711031+0000","last_clean":"2024-09-16T10:14:30.711031+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:14:30.711031+0000","last_undegraded":"2024-09-16T10:14:30.711031+0000","last_fullsized":"2024-09-16T10:14:30.711031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.533870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808510+0000","last_change":"2024-09-16T10:13:37.097257+0000","last_active":"2024-09-16T10:14:31.808510+0000","last_peered":"2024-09-16T10:14:31.808510+0000","last_clean":"2024-09-16T10:14:31.808510+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:14:31.808510+0000","last_undegraded":"2024-09-16T10:14:31.808510+0000","last_fullsized":"2024-09-16T10:14:31.808510+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.017319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714213+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:14:30.714213+0000","last_peered":"2024-09-16T10:14:30.714213+0000","last_clean":"2024-09-16T10:14:30.714213+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:14:30.714213+0000","last_undegraded":"2024-09-16T10:14:30.714213+0000","last_fullsized":"2024-09-16T10:14:30.714213+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.545235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344658+0000","last_change":"2024-09-16T10:13:43.125823+0000","last_active":"2024-09-16T10:14:31.344658+0000","last_peered":"2024-09-16T10:14:31.344658+0000","last_clean":"2024-09-16T10:14:31.344658+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:14:31.344658+0000","last_undegraded":"2024-09-16T10:14:31.344658+0000","last_fullsized":"2024-09-16T10:14:31.344658+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.646167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105704+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:14:32.105704+0000","last_peered":"2024-09-16T10:14:32.105704+0000","last_clean":"2024-09-16T10:14:32.105704+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:14:32.105704+0000","last_undegraded":"2024-09-16T10:14:32.105704+0000","last_fullsized":"2024-09-16T10:14:32.105704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.689518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808509+0000","last_change":"2024-09-16T10:13:37.092779+0000","last_active":"2024-09-16T10:14:31.808509+0000","last_peered":"2024-09-16T10:14:31.808509+0000","last_clean":"2024-09-16T10:14:31.808509+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:14:31.808509+0000","last_undegraded":"2024-09-16T10:14:31.808509+0000","last_fullsized":"2024-09-16T10:14:31.808509+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.367150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807856+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:31.807856+0000","last_peered":"2024-09-16T10:14:31.807856+0000","last_clean":"2024-09-16T10:14:31.807856+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:14:31.807856+0000","last_undegraded":"2024-09-16T10:14:31.807856+0000","last_fullsized":"2024-09-16T10:14:31.807856+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.594260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105552+0000","last_change":"2024-09-16T10:13:43.283434+0000","last_active":"2024-09-16T10:14:32.105552+0000","last_peered":"2024-09-16T10:14:32.105552+0000","last_clean":"2024-09-16T10:14:32.105552+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:14:32.105552+0000","last_undegraded":"2024-09-16T10:14:32.105552+0000","last_fullsized":"2024-09-16T10:14:32.105552+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.845991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790155+0000","last_change":"2024-09-16T10:13:37.089575+0000","last_active":"2024-09-16T10:14:30.790155+0000","last_peered":"2024-09-16T10:14:30.790155+0000","last_clean":"2024-09-16T10:14:30.790155+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:14:30.790155+0000","last_undegraded":"2024-09-16T10:14:30.790155+0000","last_fullsized":"2024-09-16T10:14:30.790155+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.178781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345859+0000","last_change":"2024-09-16T10:13:39.107923+0000","last_active":"2024-09-16T10:14:31.345859+0000","last_peered":"2024-09-16T10:14:31.345859+0000","last_clean":"2024-09-16T10:14:31.345859+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:14:31.345859+0000","last_undegraded":"2024-09-16T10:14:31.345859+0000","last_fullsized":"2024-09-16T10:14:31.345859+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.475348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106474+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:14:32.106474+0000","last_peered":"2024-09-16T10:14:32.106474+0000","last_clean":"2024-09-16T10:14:32.106474+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:14:32.106474+0000","last_undegraded":"2024-09-16T10:14:32.106474+0000","last_fullsized":"2024-09-16T10:14:32.106474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.978048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807518+0000","last_change":"2024-09-16T10:13:43.283644+0000","last_active":"2024-09-16T10:14:31.807518+0000","last_peered":"2024-09-16T10:14:31.807518+0000","last_clean":"2024-09-16T10:14:31.807518+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:14:31.807518+0000","last_undegraded":"2024-09-16T10:14:31.807518+0000","last_fullsized":"2024-09-16T10:14:31.807518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.605473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070788+0000","last_change":"2024-09-16T10:13:39.107919+0000","last_active":"2024-09-16T10:14:31.070788+0000","last_peered":"2024-09-16T10:14:31.070788+0000","last_clean":"2024-09-16T10:14:31.070788+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:14:31.070788+0000","last_undegraded":"2024-09-16T10:14:31.070788+0000","last_fullsized":"2024-09-16T10:14:31.070788+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.504540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807926+0000","last_change":"2024-09-16T10:13:37.095220+0000","last_active":"2024-09-16T10:14:31.807926+0000","last_peered":"2024-09-16T10:14:31.807926+0000","last_clean":"2024-09-16T10:14:31.807926+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:14:31.807926+0000","last_undegraded":"2024-09-16T10:14:31.807926+0000","last_fullsized":"2024-09-16T10:14:31.807926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.601224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345781+0000","last_change":"2024-09-16T10:13:41.118905+0000","last_active":"2024-09-16T10:14:31.345781+0000","last_peered":"2024-09-16T10:14:31.345781+0000","last_clean":"2024-09-16T10:14:31.345781+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:14:31.345781+0000","last_undegraded":"2024-09-16T10:14:31.345781+0000","last_fullsized":"2024-09-16T10:14:31.345781+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.674048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714468+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:14:30.714468+0000","last_peered":"2024-09-16T10:14:30.714468+0000","last_clean":"2024-09-16T10:14:30.714468+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:14:30.714468+0000","last_undegraded":"2024-09-16T10:14:30.714468+0000","last_fullsized":"2024-09-16T10:14:30.714468+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.053913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106413+0000","last_change":"2024-09-16T10:13:39.105916+0000","last_active":"2024-09-16T10:14:32.106413+0000","last_peered":"2024-09-16T10:14:32.106413+0000","last_clean":"2024-09-16T10:14:32.106413+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:14:32.106413+0000","last_undegraded":"2024-09-16T10:14:32.106413+0000","last_fullsized":"2024-09-16T10:14:32.106413+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.209227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070194+0000","last_change":"2024-09-16T10:13:37.097837+0000","last_active":"2024-09-16T10:14:31.070194+0000","last_peered":"2024-09-16T10:14:31.070194+0000","last_clean":"2024-09-16T10:14:31.070194+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:14:31.070194+0000","last_undegraded":"2024-09-16T10:14:31.070194+0000","last_fullsized":"2024-09-16T10:14:31.070194+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:36.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345480+0000","last_change":"2024-09-16T10:13:41.119310+0000","last_active":"2024-09-16T10:14:31.345480+0000","last_peered":"2024-09-16T10:14:31.345480+0000","last_clean":"2024-09-16T10:14:31.345480+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:14:31.345480+0000","last_undegraded":"2024-09-16T10:14:31.345480+0000","last_fullsized":"2024-09-16T10:14:31.345480+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.607866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711889+0000","last_change":"2024-09-16T10:13:43.119772+0000","last_active":"2024-09-16T10:14:30.711889+0000","last_peered":"2024-09-16T10:14:30.711889+0000","last_clean":"2024-09-16T10:14:30.711889+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:14:30.711889+0000","last_undegraded":"2024-09-16T10:14:30.711889+0000","last_fullsized":"2024-09-16T10:14:30.711889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.088506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345574+0000","last_change":"2024-09-16T10:13:39.105939+0000","last_active":"2024-09-16T10:14:31.345574+0000","last_peered":"2024-09-16T10:14:31.345574+0000","last_clean":"2024-09-16T10:14:31.345574+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:14:31.345574+0000","last_undegraded":"2024-09-16T10:14:31.345574+0000","last_fullsized":"2024-09-16T10:14:31.345574+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.788054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808121+0000","last_change":"2024-09-16T10:13:37.094640+0000","last_active":"2024-09-16T10:14:31.808121+0000","last_peered":"2024-09-16T10:14:31.808121+0000","last_clean":"2024-09-16T10:14:31.808121+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:14:31.808121+0000","last_undegraded":"2024-09-16T10:14:31.808121+0000","last_fullsized":"2024-09-16T10:14:31.808121+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.045517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106302+0000","last_change":"2024-09-16T10:13:41.116455+0000","last_active":"2024-09-16T10:14:32.106302+0000","last_peered":"2024-09-16T10:14:32.106302+0000","last_clean":"2024-09-16T10:14:32.106302+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:14:32.106302+0000","last_undegraded":"2024-09-16T10:14:32.106302+0000","last_fullsized":"2024-09-16T10:14:32.106302+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.665680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070445+0000","last_change":"2024-09-16T10:13:43.120971+0000","last_active":"2024-09-16T10:14:31.070445+0000","last_peered":"2024-09-16T10:14:31.070445+0000","last_clean":"2024-09-16T10:14:31.070445+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:14:31.070445+0000","last_undegraded":"2024-09-16T10:14:31.070445+0000","last_fullsized":"2024-09-16T10:14:31.070445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.791625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711070+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:14:30.711070+0000","last_peered":"2024-09-16T10:14:30.711070+0000","last_clean":"2024-09-16T10:14:30.711070+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:14:30.711070+0000","last_undegraded":"2024-09-16T10:14:30.711070+0000","last_fullsized":"2024-09-16T10:14:30.711070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.996979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711840+0000","last_change":"2024-09-16T10:13:37.090129+0000","last_active":"2024-09-16T10:14:30.711840+0000","last_peered":"2024-09-16T10:14:30.711840+0000","last_clean":"2024-09-16T10:14:30.711840+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:14:30.711840+0000","last_undegraded":"2024-09-16T10:14:30.711840+0000","last_fullsized":"2024-09-16T10:14:30.711840+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.823652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715397+0000","last_change":"2024-09-16T10:13:41.112912+0000","last_active":"2024-09-16T10:14:30.715397+0000","last_peered":"2024-09-16T10:14:30.715397+0000","last_clean":"2024-09-16T10:14:30.715397+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:14:30.715397+0000","last_undegraded":"2024-09-16T10:14:30.715397+0000","last_fullsized":"2024-09-16T10:14:30.715397+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.606633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344470+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:14:31.344470+0000","last_peered":"2024-09-16T10:14:31.344470+0000","last_clean":"2024-09-16T10:14:31.344470+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:14:31.344470+0000","last_undegraded":"2024-09-16T10:14:31.344470+0000","last_fullsized":"2024-09-16T10:14:31.344470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.223461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715520+0000","last_change":"2024-09-16T10:13:39.098286+0000","last_active":"2024-09-16T10:14:30.715520+0000","last_peered":"2024-09-16T10:14:30.715520+0000","last_clean":"2024-09-16T10:14:30.715520+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:14:30.715520+0000","last_undegraded":"2024-09-16T10:14:30.715520+0000","last_fullsized":"2024-09-16T10:14:30.715520+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.259110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808247+0000","last_change":"2024-09-16T10:13:37.091081+0000","last_active":"2024-09-16T10:14:31.808247+0000","last_peered":"2024-09-16T10:14:31.808247+0000","last_clean":"2024-09-16T10:14:31.808247+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:14:31.808247+0000","last_undegraded":"2024-09-16T10:14:31.808247+0000","last_fullsized":"2024-09-16T10:14:31.808247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.952981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345276+0000","last_change":"2024-09-16T10:13:41.114260+0000","last_active":"2024-09-16T10:14:31.345276+0000","last_peered":"2024-09-16T10:14:31.345276+0000","last_clean":"2024-09-16T10:14:31.345276+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:14:31.345276+0000","last_undegraded":"2024-09-16T10:14:31.345276+0000","last_fullsized":"2024-09-16T10:14:31.345276+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.627978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.069883+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:14:31.069883+0000","last_peered":"2024-09-16T10:14:31.069883+0000","last_clean":"2024-09-16T10:14:31.069883+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:14:31.069883+0000","last_undegraded":"2024-09-16T10:14:31.069883+0000","last_fullsized":"2024-09-16T10:14:31.069883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345182+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:14:31.345182+0000","last_peered":"2024-09-16T10:14:31.345182+0000","last_clean":"2024-09-16T10:14:31.345182+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:14:31.345182+0000","last_undegraded":"2024-09-16T10:14:31.345182+0000","last_fullsized":"2024-09-16T10:14:31.345182+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.847239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106476+0000","last_change":"2024-09-16T10:13:37.088453+0000","last_active":"2024-09-16T10:14:32.106476+0000","last_peered":"2024-09-16T10:14:32.106476+0000","last_clean":"2024-09-16T10:14:32.106476+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:14:32.106476+0000","last_undegraded":"2024-09-16T10:14:32.106476+0000","last_fullsized":"2024-09-16T10:14:32.106476+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.596318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807553+0000","last_change":"2024-09-16T10:13:41.112830+0000","last_active":"2024-09-16T10:14:31.807553+0000","last_peered":"2024-09-16T10:14:31.807553+0000","last_clean":"2024-09-16T10:14:31.807553+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:14:31.807553+0000","last_undegraded":"2024-09-16T10:14:31.807553+0000","last_fullsized":"2024-09-16T10:14:31.807553+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:41.842458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344517+0000","last_change":"2024-09-16T10:13:43.286716+0000","last_active":"2024-09-16T10:14:31.344517+0000","last_peered":"2024-09-16T10:14:31.344517+0000","last_clean":"2024-09-16T10:14:31.344517+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:14:31.344517+0000","last_undegraded":"2024-09-16T10:14:31.344517+0000","last_fullsized":"2024-09-16T10:14:31.344517+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.000117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344971+0000","last_change":"2024-09-16T10:13:39.101213+0000","last_active":"2024-09-16T10:14:31.344971+0000","last_peered":"2024-09-16T10:14:31.344971+0000","last_clean":"2024-09-16T10:14:31.344971+0000","last_became_active":"2024-09-16T10:13:39.100788+0000","last_became_peered":"2024-09-16T10:13:39.100788+0000","last_unstale":"2024-09-16T10:14:31.344971+0000","last_undegraded":"2024-09-16T10:14:31.344971+0000","last_fullsized":"2024-09-16T10:14:31.344971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.518507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070727+0000","last_change":"2024-09-16T10:13:37.094533+0000","last_active":"2024-09-16T10:14:31.070727+0000","last_peered":"2024-09-16T10:14:31.070727+0000","last_clean":"2024-09-16T10:14:31.070727+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:14:31.070727+0000","last_undegraded":"2024-09-16T10:14:31.070727+0000","last_fullsized":"2024-09-16T10:14:31.070727+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.183923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790283+0000","last_change":"2024-09-16T10:13:41.114472+0000","last_active":"2024-09-16T10:14:30.790283+0000","last_peered":"2024-09-16T10:14:30.790283+0000","last_clean":"2024-09-16T10:14:30.790283+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:14:30.790283+0000","last_undegraded":"2024-09-16T10:14:30.790283+0000","last_fullsized":"2024-09-16T10:14:30.790283+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.243642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807285+0000","last_change":"2024-09-16T10:13:43.119917+0000","last_active":"2024-09-16T10:14:31.807285+0000","last_peered":"2024-09-16T10:14:31.807285+0000","last_clean":"2024-09-16T10:14:31.807285+0000","last_became_active":"2024-09-16T10:13:43.119612+0000","last_became_peered":"2024-09-16T10:13:43.119612+0000","last_unstale":"2024-09-16T10:14:31.807285+0000","last_undegraded":"2024-09-16T10:14:31.807285+0000","last_fullsized":"2024-09-16T10:14:31.807285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.539099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715721+0000","last_change":"2024-09-16T10:13:43.114761+0000","last_active":"2024-09-16T10:14:30.715721+0000","last_peered":"2024-09-16T10:14:30.715721+0000","last_clean":"2024-09-16T10:14:30.715721+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:14:30.715721+0000","last_undegraded":"2024-09-16T10:14:30.715721+0000","last_fullsized":"2024-09-16T10:14:30.715721+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.210195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345096+0000","last_change":"2024-09-16T10:13:37.089832+0000","last_active":"2024-09-16T10:14:31.345096+0000","last_peered":"2024-09-16T10:14:31.345096+0000","last_clean":"2024-09-16T10:14:31.345096+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:14:31.345096+0000","last_undegraded":"2024-09-16T10:14:31.345096+0000","last_fullsized":"2024-09-16T10:14:31.345096+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.587954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.791167+0000","last_change":"2024-09-16T10:13:39.098801+0000","last_active":"2024-09-16T10:14:30.791167+0000","last_peered":"2024-09-16T10:14:30.791167+0000","last_clean":"2024-09-16T10:14:30.791167+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:14:30.791167+0000","last_undegraded":"2024-09-16T10:14:30.791167+0000","last_fullsized":"2024-09-16T10:14:30.791167+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.353803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070623+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:31.070623+0000","last_peered":"2024-09-16T10:14:31.070623+0000","last_clean":"2024-09-16T10:14:31.070623+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:14:31.070623+0000","last_undegraded":"2024-09-16T10:14:31.070623+0000","last_fullsized":"2024-09-16T10:14:31.070623+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.860042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233999,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28416,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716736,"statfs":{"total":95995035648,"available":95965937664,"internally_reserved":0,"allocated":1421312,"data_stored":927565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528339,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32552,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712600,"statfs":{"total":95995035648,"available":95961702400,"internally_reserved":0,"allocated":1396736,"data_stored":926467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789977,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":798720,"data_stored":331079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32004,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713148,"statfs":{"total":95995035648,"available":95962263552,"internally_reserved":0,"allocated":835584,"data_stored":337140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215139,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":847872,"data_stored":338419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":802816,"data_stored":336260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":872448,"data_stored":338320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28428,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716724,"statfs":{"total":95995035648,"available":95965925376,"internally_reserved":0,"allocated":1429504,"data_stored":928267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:14:34.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:14:35.016 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[29993]: Deploying daemon grafana.a on smithi165 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[29993]: from='client.44675 v1:172.21.15.63:0/2512835317' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[39470]: Deploying daemon grafana.a on smithi165 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[39470]: from='client.44675 v1:172.21.15.63:0/2512835317' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:35.138 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:35.139 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:34 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:34 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-16T10:14:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:34 smithi165 ceph-mon[32696]: Deploying daemon grafana.a on smithi165 2024-09-16T10:14:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:34 smithi165 ceph-mon[32696]: from='client.44675 v1:172.21.15.63:0/2512835317' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:34 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:14:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:34 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:14:35.679 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:14:35.681 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:14:35.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:35 smithi063 ceph-mon[29993]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:35 smithi063 ceph-mon[39470]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:36.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:35 smithi165 ceph-mon[32696]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:36.881 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T10:14:35.012626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":898,"num_read_kb":755,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241088,"kb_used_data":8208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720128,"statfs":{"total":767960285184,"available":767713411072,"internally_reserved":0,"allocated":8404992,"data_stored":4463517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002341"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070691+0000","last_change":"2024-09-16T10:13:37.096999+0000","last_active":"2024-09-16T10:14:31.070691+0000","last_peered":"2024-09-16T10:14:31.070691+0000","last_clean":"2024-09-16T10:14:31.070691+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:14:31.070691+0000","last_undegraded":"2024-09-16T10:14:31.070691+0000","last_fullsized":"2024-09-16T10:14:31.070691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710732+0000","last_change":"2024-09-16T10:13:39.093832+0000","last_active":"2024-09-16T10:14:30.710732+0000","last_peered":"2024-09-16T10:14:30.710732+0000","last_clean":"2024-09-16T10:14:30.710732+0000","last_became_active":"2024-09-16T10:13:39.093430+0000","last_became_peered":"2024-09-16T10:13:39.093430+0000","last_unstale":"2024-09-16T10:14:30.710732+0000","last_undegraded":"2024-09-16T10:14:30.710732+0000","last_fullsized":"2024-09-16T10:14:30.710732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.941021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715030+0000","last_change":"2024-09-16T10:13:41.108286+0000","last_active":"2024-09-16T10:14:30.715030+0000","last_peered":"2024-09-16T10:14:30.715030+0000","last_clean":"2024-09-16T10:14:30.715030+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:14:30.715030+0000","last_undegraded":"2024-09-16T10:14:30.715030+0000","last_fullsized":"2024-09-16T10:14:30.715030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.804825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710853+0000","last_change":"2024-09-16T10:13:43.113148+0000","last_active":"2024-09-16T10:14:30.710853+0000","last_peered":"2024-09-16T10:14:30.710853+0000","last_clean":"2024-09-16T10:14:30.710853+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:14:30.710853+0000","last_undegraded":"2024-09-16T10:14:30.710853+0000","last_fullsized":"2024-09-16T10:14:30.710853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.390079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344767+0000","last_change":"2024-09-16T10:13:43.287323+0000","last_active":"2024-09-16T10:14:31.344767+0000","last_peered":"2024-09-16T10:14:31.344767+0000","last_clean":"2024-09-16T10:14:31.344767+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:14:31.344767+0000","last_undegraded":"2024-09-16T10:14:31.344767+0000","last_fullsized":"2024-09-16T10:14:31.344767+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.723245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345790+0000","last_change":"2024-09-16T10:13:37.092457+0000","last_active":"2024-09-16T10:14:31.345790+0000","last_peered":"2024-09-16T10:14:31.345790+0000","last_clean":"2024-09-16T10:14:31.345790+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:14:31.345790+0000","last_undegraded":"2024-09-16T10:14:31.345790+0000","last_fullsized":"2024-09-16T10:14:31.345790+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.920553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345322+0000","last_change":"2024-09-16T10:13:39.095829+0000","last_active":"2024-09-16T10:14:31.345322+0000","last_peered":"2024-09-16T10:14:31.345322+0000","last_clean":"2024-09-16T10:14:31.345322+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:14:31.345322+0000","last_undegraded":"2024-09-16T10:14:31.345322+0000","last_fullsized":"2024-09-16T10:14:31.345322+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.084583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710720+0000","last_change":"2024-09-16T10:13:41.121501+0000","last_active":"2024-09-16T10:14:30.710720+0000","last_peered":"2024-09-16T10:14:30.710720+0000","last_clean":"2024-09-16T10:14:30.710720+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:14:30.710720+0000","last_undegraded":"2024-09-16T10:14:30.710720+0000","last_fullsized":"2024-09-16T10:14:30.710720+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.386931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106410+0000","last_change":"2024-09-16T10:13:37.092847+0000","last_active":"2024-09-16T10:14:32.106410+0000","last_peered":"2024-09-16T10:14:32.106410+0000","last_clean":"2024-09-16T10:14:32.106410+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:14:32.106410+0000","last_undegraded":"2024-09-16T10:14:32.106410+0000","last_fullsized":"2024-09-16T10:14:32.106410+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.507863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711156+0000","last_change":"2024-09-16T10:13:39.092562+0000","last_active":"2024-09-16T10:14:30.711156+0000","last_peered":"2024-09-16T10:14:30.711156+0000","last_clean":"2024-09-16T10:14:30.711156+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:14:30.711156+0000","last_undegraded":"2024-09-16T10:14:30.711156+0000","last_fullsized":"2024-09-16T10:14:30.711156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.237966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106150+0000","last_change":"2024-09-16T10:13:41.112242+0000","last_active":"2024-09-16T10:14:32.106150+0000","last_peered":"2024-09-16T10:14:32.106150+0000","last_clean":"2024-09-16T10:14:32.106150+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:14:32.106150+0000","last_undegraded":"2024-09-16T10:14:32.106150+0000","last_fullsized":"2024-09-16T10:14:32.106150+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.338353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.069948+0000","last_change":"2024-09-16T10:13:43.115780+0000","last_active":"2024-09-16T10:14:31.069948+0000","last_peered":"2024-09-16T10:14:31.069948+0000","last_clean":"2024-09-16T10:14:31.069948+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:14:31.069948+0000","last_undegraded":"2024-09-16T10:14:31.069948+0000","last_fullsized":"2024-09-16T10:14:31.069948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.438318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106301+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:14:32.106301+0000","last_peered":"2024-09-16T10:14:32.106301+0000","last_clean":"2024-09-16T10:14:32.106301+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:14:32.106301+0000","last_undegraded":"2024-09-16T10:14:32.106301+0000","last_fullsized":"2024-09-16T10:14:32.106301+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.318174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711001+0000","last_change":"2024-09-16T10:13:39.092696+0000","last_active":"2024-09-16T10:14:30.711001+0000","last_peered":"2024-09-16T10:14:30.711001+0000","last_clean":"2024-09-16T10:14:30.711001+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:14:30.711001+0000","last_undegraded":"2024-09-16T10:14:30.711001+0000","last_fullsized":"2024-09-16T10:14:30.711001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.518434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807838+0000","last_change":"2024-09-16T10:13:41.111640+0000","last_active":"2024-09-16T10:14:31.807838+0000","last_peered":"2024-09-16T10:14:31.807838+0000","last_clean":"2024-09-16T10:14:31.807838+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:14:31.807838+0000","last_undegraded":"2024-09-16T10:14:31.807838+0000","last_fullsized":"2024-09-16T10:14:31.807838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.280028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105300+0000","last_change":"2024-09-16T10:13:43.121701+0000","last_active":"2024-09-16T10:14:32.105300+0000","last_peered":"2024-09-16T10:14:32.105300+0000","last_clean":"2024-09-16T10:14:32.105300+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:14:32.105300+0000","last_undegraded":"2024-09-16T10:14:32.105300+0000","last_fullsized":"2024-09-16T10:14:32.105300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.188718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711596+0000","last_change":"2024-09-16T10:13:43.284208+0000","last_active":"2024-09-16T10:14:30.711596+0000","last_peered":"2024-09-16T10:14:30.711596+0000","last_clean":"2024-09-16T10:14:30.711596+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:14:30.711596+0000","last_undegraded":"2024-09-16T10:14:30.711596+0000","last_fullsized":"2024-09-16T10:14:30.711596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:55.053189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070607+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:14:31.070607+0000","last_peered":"2024-09-16T10:14:31.070607+0000","last_clean":"2024-09-16T10:14:31.070607+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:14:31.070607+0000","last_undegraded":"2024-09-16T10:14:31.070607+0000","last_fullsized":"2024-09-16T10:14:31.070607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.063039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714875+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:14:30.714875+0000","last_peered":"2024-09-16T10:14:30.714875+0000","last_clean":"2024-09-16T10:14:30.714875+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:14:30.714875+0000","last_undegraded":"2024-09-16T10:14:30.714875+0000","last_fullsized":"2024-09-16T10:14:30.714875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.478889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715963+0000","last_change":"2024-09-16T10:13:41.120296+0000","last_active":"2024-09-16T10:14:30.715963+0000","last_peered":"2024-09-16T10:14:30.715963+0000","last_clean":"2024-09-16T10:14:30.715963+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:14:30.715963+0000","last_undegraded":"2024-09-16T10:14:30.715963+0000","last_fullsized":"2024-09-16T10:14:30.715963+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.391104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344582+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:14:31.344582+0000","last_peered":"2024-09-16T10:14:31.344582+0000","last_clean":"2024-09-16T10:14:31.344582+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:14:31.344582+0000","last_undegraded":"2024-09-16T10:14:31.344582+0000","last_fullsized":"2024-09-16T10:14:31.344582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.407109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711359+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:14:30.711359+0000","last_peered":"2024-09-16T10:14:30.711359+0000","last_clean":"2024-09-16T10:14:30.711359+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:14:30.711359+0000","last_undegraded":"2024-09-16T10:14:30.711359+0000","last_fullsized":"2024-09-16T10:14:30.711359+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.861513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":73,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345647+0000","last_change":"2024-09-16T10:13:39.100013+0000","last_active":"2024-09-16T10:14:31.345647+0000","last_peered":"2024-09-16T10:14:31.345647+0000","last_clean":"2024-09-16T10:14:31.345647+0000","last_became_active":"2024-09-16T10:13:39.099675+0000","last_became_peered":"2024-09-16T10:13:39.099675+0000","last_unstale":"2024-09-16T10:14:31.345647+0000","last_undegraded":"2024-09-16T10:14:31.345647+0000","last_fullsized":"2024-09-16T10:14:31.345647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.654393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711399+0000","last_change":"2024-09-16T10:13:41.112993+0000","last_active":"2024-09-16T10:14:30.711399+0000","last_peered":"2024-09-16T10:14:30.711399+0000","last_clean":"2024-09-16T10:14:30.711399+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:14:30.711399+0000","last_undegraded":"2024-09-16T10:14:30.711399+0000","last_fullsized":"2024-09-16T10:14:30.711399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.116156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807329+0000","last_change":"2024-09-16T10:13:43.119664+0000","last_active":"2024-09-16T10:14:31.807329+0000","last_peered":"2024-09-16T10:14:31.807329+0000","last_clean":"2024-09-16T10:14:31.807329+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:14:31.807329+0000","last_undegraded":"2024-09-16T10:14:31.807329+0000","last_fullsized":"2024-09-16T10:14:31.807329+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.178009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715210+0000","last_change":"2024-09-16T10:13:37.082594+0000","last_active":"2024-09-16T10:14:30.715210+0000","last_peered":"2024-09-16T10:14:30.715210+0000","last_clean":"2024-09-16T10:14:30.715210+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:14:30.715210+0000","last_undegraded":"2024-09-16T10:14:30.715210+0000","last_fullsized":"2024-09-16T10:14:30.715210+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.835656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070684+0000","last_change":"2024-09-16T10:13:39.108076+0000","last_active":"2024-09-16T10:14:31.070684+0000","last_peered":"2024-09-16T10:14:31.070684+0000","last_clean":"2024-09-16T10:14:31.070684+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:14:31.070684+0000","last_undegraded":"2024-09-16T10:14:31.070684+0000","last_fullsized":"2024-09-16T10:14:31.070684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.726907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.791198+0000","last_change":"2024-09-16T10:13:41.116955+0000","last_active":"2024-09-16T10:14:30.791198+0000","last_peered":"2024-09-16T10:14:30.791198+0000","last_clean":"2024-09-16T10:14:30.791198+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:14:30.791198+0000","last_undegraded":"2024-09-16T10:14:30.791198+0000","last_fullsized":"2024-09-16T10:14:30.791198+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.203272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716397+0000","last_change":"2024-09-16T10:13:39.096753+0000","last_active":"2024-09-16T10:14:30.716397+0000","last_peered":"2024-09-16T10:14:30.716397+0000","last_clean":"2024-09-16T10:14:30.716397+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:14:30.716397+0000","last_undegraded":"2024-09-16T10:14:30.716397+0000","last_fullsized":"2024-09-16T10:14:30.716397+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.363335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345133+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:14:31.345133+0000","last_peered":"2024-09-16T10:14:31.345133+0000","last_clean":"2024-09-16T10:14:31.345133+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:14:31.345133+0000","last_undegraded":"2024-09-16T10:14:31.345133+0000","last_fullsized":"2024-09-16T10:14:31.345133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.355614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711919+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:14:30.711919+0000","last_peered":"2024-09-16T10:14:30.711919+0000","last_clean":"2024-09-16T10:14:30.711919+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:14:30.711919+0000","last_undegraded":"2024-09-16T10:14:30.711919+0000","last_fullsized":"2024-09-16T10:14:30.711919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.675105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105564+0000","last_change":"2024-09-16T10:13:43.114886+0000","last_active":"2024-09-16T10:14:32.105564+0000","last_peered":"2024-09-16T10:14:32.105564+0000","last_clean":"2024-09-16T10:14:32.105564+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:14:32.105564+0000","last_undegraded":"2024-09-16T10:14:32.105564+0000","last_fullsized":"2024-09-16T10:14:32.105564+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.234184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":82,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345001+0000","last_change":"2024-09-16T10:13:39.106135+0000","last_active":"2024-09-16T10:14:31.345001+0000","last_peered":"2024-09-16T10:14:31.345001+0000","last_clean":"2024-09-16T10:14:31.345001+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:14:31.345001+0000","last_undegraded":"2024-09-16T10:14:31.345001+0000","last_fullsized":"2024-09-16T10:14:31.345001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.184956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345931+0000","last_change":"2024-09-16T10:13:37.093660+0000","last_active":"2024-09-16T10:14:31.345931+0000","last_peered":"2024-09-16T10:14:31.345931+0000","last_clean":"2024-09-16T10:14:31.345931+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:14:31.345931+0000","last_undegraded":"2024-09-16T10:14:31.345931+0000","last_fullsized":"2024-09-16T10:14:31.345931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.816759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710725+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:14:30.710725+0000","last_peered":"2024-09-16T10:14:30.710725+0000","last_clean":"2024-09-16T10:14:30.710725+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:14:30.710725+0000","last_undegraded":"2024-09-16T10:14:30.710725+0000","last_fullsized":"2024-09-16T10:14:30.710725+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.743002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711089+0000","last_change":"2024-09-16T10:13:43.122864+0000","last_active":"2024-09-16T10:14:30.711089+0000","last_peered":"2024-09-16T10:14:30.711089+0000","last_clean":"2024-09-16T10:14:30.711089+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:14:30.711089+0000","last_undegraded":"2024-09-16T10:14:30.711089+0000","last_fullsized":"2024-09-16T10:14:30.711089+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.235493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.710707+0000","last_change":"2024-09-16T10:13:39.094074+0000","last_active":"2024-09-16T10:14:30.710707+0000","last_peered":"2024-09-16T10:14:30.710707+0000","last_clean":"2024-09-16T10:14:30.710707+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:14:30.710707+0000","last_undegraded":"2024-09-16T10:14:30.710707+0000","last_fullsized":"2024-09-16T10:14:30.710707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:07.532032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070847+0000","last_change":"2024-09-16T10:13:37.096046+0000","last_active":"2024-09-16T10:14:31.070847+0000","last_peered":"2024-09-16T10:14:31.070847+0000","last_clean":"2024-09-16T10:14:31.070847+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:14:31.070847+0000","last_undegraded":"2024-09-16T10:14:31.070847+0000","last_fullsized":"2024-09-16T10:14:31.070847+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.593585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344949+0000","last_change":"2024-09-16T10:13:41.112759+0000","last_active":"2024-09-16T10:14:31.344949+0000","last_peered":"2024-09-16T10:14:31.344949+0000","last_clean":"2024-09-16T10:14:31.344949+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:14:31.344949+0000","last_undegraded":"2024-09-16T10:14:31.344949+0000","last_fullsized":"2024-09-16T10:14:31.344949+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.600065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807287+0000","last_change":"2024-09-16T10:13:43.284631+0000","last_active":"2024-09-16T10:14:31.807287+0000","last_peered":"2024-09-16T10:14:31.807287+0000","last_clean":"2024-09-16T10:14:31.807287+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:14:31.807287+0000","last_undegraded":"2024-09-16T10:14:31.807287+0000","last_fullsized":"2024-09-16T10:14:31.807287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.200363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715658+0000","last_change":"2024-09-16T10:13:39.095801+0000","last_active":"2024-09-16T10:14:30.715658+0000","last_peered":"2024-09-16T10:14:30.715658+0000","last_clean":"2024-09-16T10:14:30.715658+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:14:30.715658+0000","last_undegraded":"2024-09-16T10:14:30.715658+0000","last_fullsized":"2024-09-16T10:14:30.715658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.537230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106522+0000","last_change":"2024-09-16T10:13:37.093423+0000","last_active":"2024-09-16T10:14:32.106522+0000","last_peered":"2024-09-16T10:14:32.106522+0000","last_clean":"2024-09-16T10:14:32.106522+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:14:32.106522+0000","last_undegraded":"2024-09-16T10:14:32.106522+0000","last_fullsized":"2024-09-16T10:14:32.106522+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.178620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070387+0000","last_change":"2024-09-16T10:13:41.115177+0000","last_active":"2024-09-16T10:14:31.070387+0000","last_peered":"2024-09-16T10:14:31.070387+0000","last_clean":"2024-09-16T10:14:31.070387+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:14:31.070387+0000","last_undegraded":"2024-09-16T10:14:31.070387+0000","last_fullsized":"2024-09-16T10:14:31.070387+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.803669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070057+0000","last_change":"2024-09-16T10:13:43.119055+0000","last_active":"2024-09-16T10:14:31.070057+0000","last_peered":"2024-09-16T10:14:31.070057+0000","last_clean":"2024-09-16T10:14:31.070057+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:14:31.070057+0000","last_undegraded":"2024-09-16T10:14:31.070057+0000","last_fullsized":"2024-09-16T10:14:31.070057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:54.513879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":85,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070453+0000","last_change":"2024-09-16T10:13:39.106153+0000","last_active":"2024-09-16T10:14:31.070453+0000","last_peered":"2024-09-16T10:14:31.070453+0000","last_clean":"2024-09-16T10:14:31.070453+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:14:31.070453+0000","last_undegraded":"2024-09-16T10:14:31.070453+0000","last_fullsized":"2024-09-16T10:14:31.070453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:57.043307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716254+0000","last_change":"2024-09-16T10:13:37.092754+0000","last_active":"2024-09-16T10:14:30.716254+0000","last_peered":"2024-09-16T10:14:30.716254+0000","last_clean":"2024-09-16T10:14:30.716254+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:14:30.716254+0000","last_undegraded":"2024-09-16T10:14:30.716254+0000","last_fullsized":"2024-09-16T10:14:30.716254+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.036476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790757+0000","last_change":"2024-09-16T10:13:41.114703+0000","last_active":"2024-09-16T10:14:30.790757+0000","last_peered":"2024-09-16T10:14:30.790757+0000","last_clean":"2024-09-16T10:14:30.790757+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:14:30.790757+0000","last_undegraded":"2024-09-16T10:14:30.790757+0000","last_fullsized":"2024-09-16T10:14:30.790757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.640025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714930+0000","last_change":"2024-09-16T10:13:43.283556+0000","last_active":"2024-09-16T10:14:30.714930+0000","last_peered":"2024-09-16T10:14:30.714930+0000","last_clean":"2024-09-16T10:14:30.714930+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:14:30.714930+0000","last_undegraded":"2024-09-16T10:14:30.714930+0000","last_fullsized":"2024-09-16T10:14:30.714930+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:52.096829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":110,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716149+0000","last_change":"2024-09-16T10:13:39.098696+0000","last_active":"2024-09-16T10:14:30.716149+0000","last_peered":"2024-09-16T10:14:30.716149+0000","last_clean":"2024-09-16T10:14:30.716149+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:14:30.716149+0000","last_undegraded":"2024-09-16T10:14:30.716149+0000","last_fullsized":"2024-09-16T10:14:30.716149+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.501760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711529+0000","last_change":"2024-09-16T10:13:37.092585+0000","last_active":"2024-09-16T10:14:30.711529+0000","last_peered":"2024-09-16T10:14:30.711529+0000","last_clean":"2024-09-16T10:14:30.711529+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:14:30.711529+0000","last_undegraded":"2024-09-16T10:14:30.711529+0000","last_fullsized":"2024-09-16T10:14:30.711529+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.123070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105364+0000","last_change":"2024-09-16T10:13:39.091269+0000","last_active":"2024-09-16T10:14:32.105364+0000","last_peered":"2024-09-16T10:14:32.105364+0000","last_clean":"2024-09-16T10:14:32.105364+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:14:32.105364+0000","last_undegraded":"2024-09-16T10:14:32.105364+0000","last_fullsized":"2024-09-16T10:14:32.105364+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.638736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070481+0000","last_change":"2024-09-16T10:13:41.118142+0000","last_active":"2024-09-16T10:14:31.070481+0000","last_peered":"2024-09-16T10:14:31.070481+0000","last_clean":"2024-09-16T10:14:31.070481+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:14:31.070481+0000","last_undegraded":"2024-09-16T10:14:31.070481+0000","last_fullsized":"2024-09-16T10:14:31.070481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.463063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344939+0000","last_change":"2024-09-16T10:13:43.124755+0000","last_active":"2024-09-16T10:14:31.344939+0000","last_peered":"2024-09-16T10:14:31.344939+0000","last_clean":"2024-09-16T10:14:31.344939+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:14:31.344939+0000","last_undegraded":"2024-09-16T10:14:31.344939+0000","last_fullsized":"2024-09-16T10:14:31.344939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.479324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716471+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:14:30.716471+0000","last_peered":"2024-09-16T10:14:30.716471+0000","last_clean":"2024-09-16T10:14:30.716471+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:14:30.716471+0000","last_undegraded":"2024-09-16T10:14:30.716471+0000","last_fullsized":"2024-09-16T10:14:30.716471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.987015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714742+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:14:30.714742+0000","last_peered":"2024-09-16T10:14:30.714742+0000","last_clean":"2024-09-16T10:14:30.714742+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:14:30.714742+0000","last_undegraded":"2024-09-16T10:14:30.714742+0000","last_fullsized":"2024-09-16T10:14:30.714742+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.434426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":63,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714702+0000","last_change":"2024-09-16T10:13:39.090453+0000","last_active":"2024-09-16T10:14:30.714702+0000","last_peered":"2024-09-16T10:14:30.714702+0000","last_clean":"2024-09-16T10:14:30.714702+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:14:30.714702+0000","last_undegraded":"2024-09-16T10:14:30.714702+0000","last_fullsized":"2024-09-16T10:14:30.714702+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.048019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714604+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:14:30.714604+0000","last_peered":"2024-09-16T10:14:30.714604+0000","last_clean":"2024-09-16T10:14:30.714604+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:14:30.714604+0000","last_undegraded":"2024-09-16T10:14:30.714604+0000","last_fullsized":"2024-09-16T10:14:30.714604+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.485969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807421+0000","last_change":"2024-09-16T10:13:43.283490+0000","last_active":"2024-09-16T10:14:31.807421+0000","last_peered":"2024-09-16T10:14:31.807421+0000","last_clean":"2024-09-16T10:14:31.807421+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:14:31.807421+0000","last_undegraded":"2024-09-16T10:14:31.807421+0000","last_fullsized":"2024-09-16T10:14:31.807421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.489088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070740+0000","last_change":"2024-09-16T10:13:39.091817+0000","last_active":"2024-09-16T10:14:31.070740+0000","last_peered":"2024-09-16T10:14:31.070740+0000","last_clean":"2024-09-16T10:14:31.070740+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:14:31.070740+0000","last_undegraded":"2024-09-16T10:14:31.070740+0000","last_fullsized":"2024-09-16T10:14:31.070740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.559454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070030+0000","last_change":"2024-09-16T10:13:37.096771+0000","last_active":"2024-09-16T10:14:31.070030+0000","last_peered":"2024-09-16T10:14:31.070030+0000","last_clean":"2024-09-16T10:14:31.070030+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:14:31.070030+0000","last_undegraded":"2024-09-16T10:14:31.070030+0000","last_fullsized":"2024-09-16T10:14:31.070030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.458372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":115,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807849+0000","last_change":"2024-09-16T10:13:39.105362+0000","last_active":"2024-09-16T10:14:31.807849+0000","last_peered":"2024-09-16T10:14:31.807849+0000","last_clean":"2024-09-16T10:14:31.807849+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:14:31.807849+0000","last_undegraded":"2024-09-16T10:14:31.807849+0000","last_fullsized":"2024-09-16T10:14:31.807849+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:35.154849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106689+0000","last_change":"2024-09-16T10:13:41.112756+0000","last_active":"2024-09-16T10:14:32.106689+0000","last_peered":"2024-09-16T10:14:32.106689+0000","last_clean":"2024-09-16T10:14:32.106689+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:14:32.106689+0000","last_undegraded":"2024-09-16T10:14:32.106689+0000","last_fullsized":"2024-09-16T10:14:32.106689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.034690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715465+0000","last_change":"2024-09-16T10:13:43.123048+0000","last_active":"2024-09-16T10:14:30.715465+0000","last_peered":"2024-09-16T10:14:30.715465+0000","last_clean":"2024-09-16T10:14:30.715465+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:14:30.715465+0000","last_undegraded":"2024-09-16T10:14:30.715465+0000","last_fullsized":"2024-09-16T10:14:30.715465+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.340209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":83,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790998+0000","last_change":"2024-09-16T10:13:39.106378+0000","last_active":"2024-09-16T10:14:30.790998+0000","last_peered":"2024-09-16T10:14:30.790998+0000","last_clean":"2024-09-16T10:14:30.790998+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:14:30.790998+0000","last_undegraded":"2024-09-16T10:14:30.790998+0000","last_fullsized":"2024-09-16T10:14:30.790998+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.582337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345430+0000","last_change":"2024-09-16T10:13:37.088320+0000","last_active":"2024-09-16T10:14:31.345430+0000","last_peered":"2024-09-16T10:14:31.345430+0000","last_clean":"2024-09-16T10:14:31.345430+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:14:31.345430+0000","last_undegraded":"2024-09-16T10:14:31.345430+0000","last_fullsized":"2024-09-16T10:14:31.345430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:37.142506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":99,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807848+0000","last_change":"2024-09-16T10:14:14.911115+0000","last_active":"2024-09-16T10:14:31.807848+0000","last_peered":"2024-09-16T10:14:31.807848+0000","last_clean":"2024-09-16T10:14:31.807848+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:14:31.807848+0000","last_undegraded":"2024-09-16T10:14:31.807848+0000","last_fullsized":"2024-09-16T10:14:31.807848+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:14:14.911041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:14:14.911041+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:17.171811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808111+0000","last_change":"2024-09-16T10:13:41.108176+0000","last_active":"2024-09-16T10:14:31.808111+0000","last_peered":"2024-09-16T10:14:31.808111+0000","last_clean":"2024-09-16T10:14:31.808111+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:14:31.808111+0000","last_undegraded":"2024-09-16T10:14:31.808111+0000","last_fullsized":"2024-09-16T10:14:31.808111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.887140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105383+0000","last_change":"2024-09-16T10:13:43.121658+0000","last_active":"2024-09-16T10:14:32.105383+0000","last_peered":"2024-09-16T10:14:32.105383+0000","last_clean":"2024-09-16T10:14:32.105383+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:14:32.105383+0000","last_undegraded":"2024-09-16T10:14:32.105383+0000","last_fullsized":"2024-09-16T10:14:32.105383+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:11.237055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711888+0000","last_change":"2024-09-16T10:13:39.094951+0000","last_active":"2024-09-16T10:14:30.711888+0000","last_peered":"2024-09-16T10:14:30.711888+0000","last_clean":"2024-09-16T10:14:30.711888+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:14:30.711888+0000","last_undegraded":"2024-09-16T10:14:30.711888+0000","last_fullsized":"2024-09-16T10:14:30.711888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.179189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711199+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:14:30.711199+0000","last_peered":"2024-09-16T10:14:30.711199+0000","last_clean":"2024-09-16T10:14:30.711199+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:14:30.711199+0000","last_undegraded":"2024-09-16T10:14:30.711199+0000","last_fullsized":"2024-09-16T10:14:30.711199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.958295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106526+0000","last_change":"2024-09-16T10:13:41.116069+0000","last_active":"2024-09-16T10:14:32.106526+0000","last_peered":"2024-09-16T10:14:32.106526+0000","last_clean":"2024-09-16T10:14:32.106526+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:14:32.106526+0000","last_undegraded":"2024-09-16T10:14:32.106526+0000","last_fullsized":"2024-09-16T10:14:32.106526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.901053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344553+0000","last_change":"2024-09-16T10:13:43.283629+0000","last_active":"2024-09-16T10:14:31.344553+0000","last_peered":"2024-09-16T10:14:31.344553+0000","last_clean":"2024-09-16T10:14:31.344553+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:14:31.344553+0000","last_undegraded":"2024-09-16T10:14:31.344553+0000","last_fullsized":"2024-09-16T10:14:31.344553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.176307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711759+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:14:30.711759+0000","last_peered":"2024-09-16T10:14:30.711759+0000","last_clean":"2024-09-16T10:14:30.711759+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:14:30.711759+0000","last_undegraded":"2024-09-16T10:14:30.711759+0000","last_fullsized":"2024-09-16T10:14:30.711759+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.476261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.791073+0000","last_change":"2024-09-16T10:13:37.089527+0000","last_active":"2024-09-16T10:14:30.791073+0000","last_peered":"2024-09-16T10:14:30.791073+0000","last_clean":"2024-09-16T10:14:30.791073+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:14:30.791073+0000","last_undegraded":"2024-09-16T10:14:30.791073+0000","last_fullsized":"2024-09-16T10:14:30.791073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.605092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715526+0000","last_change":"2024-09-16T10:13:41.121389+0000","last_active":"2024-09-16T10:14:30.715526+0000","last_peered":"2024-09-16T10:14:30.715526+0000","last_clean":"2024-09-16T10:14:30.715526+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:14:30.715526+0000","last_undegraded":"2024-09-16T10:14:30.715526+0000","last_fullsized":"2024-09-16T10:14:30.715526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.974031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.716031+0000","last_change":"2024-09-16T10:13:43.125890+0000","last_active":"2024-09-16T10:14:30.716031+0000","last_peered":"2024-09-16T10:14:30.716031+0000","last_clean":"2024-09-16T10:14:30.716031+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:14:30.716031+0000","last_undegraded":"2024-09-16T10:14:30.716031+0000","last_fullsized":"2024-09-16T10:14:30.716031+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711837+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:14:30.711837+0000","last_peered":"2024-09-16T10:14:30.711837+0000","last_clean":"2024-09-16T10:14:30.711837+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:14:30.711837+0000","last_undegraded":"2024-09-16T10:14:30.711837+0000","last_fullsized":"2024-09-16T10:14:30.711837+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.021437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345373+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:14:31.345373+0000","last_peered":"2024-09-16T10:14:31.345373+0000","last_clean":"2024-09-16T10:14:31.345373+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:14:31.345373+0000","last_undegraded":"2024-09-16T10:14:31.345373+0000","last_fullsized":"2024-09-16T10:14:31.345373+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.609140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715273+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:14:30.715273+0000","last_peered":"2024-09-16T10:14:30.715273+0000","last_clean":"2024-09-16T10:14:30.715273+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:14:30.715273+0000","last_undegraded":"2024-09-16T10:14:30.715273+0000","last_fullsized":"2024-09-16T10:14:30.715273+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.589846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711626+0000","last_change":"2024-09-16T10:13:43.119606+0000","last_active":"2024-09-16T10:14:30.711626+0000","last_peered":"2024-09-16T10:14:30.711626+0000","last_clean":"2024-09-16T10:14:30.711626+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:14:30.711626+0000","last_undegraded":"2024-09-16T10:14:30.711626+0000","last_fullsized":"2024-09-16T10:14:30.711626+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.164444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070344+0000","last_change":"2024-09-16T10:13:39.091847+0000","last_active":"2024-09-16T10:14:31.070344+0000","last_peered":"2024-09-16T10:14:31.070344+0000","last_clean":"2024-09-16T10:14:31.070344+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:14:31.070344+0000","last_undegraded":"2024-09-16T10:14:31.070344+0000","last_fullsized":"2024-09-16T10:14:31.070344+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:01.954882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106328+0000","last_change":"2024-09-16T10:13:37.090259+0000","last_active":"2024-09-16T10:14:32.106328+0000","last_peered":"2024-09-16T10:14:32.106328+0000","last_clean":"2024-09-16T10:14:32.106328+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:14:32.106328+0000","last_undegraded":"2024-09-16T10:14:32.106328+0000","last_fullsized":"2024-09-16T10:14:32.106328+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.290444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715017+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:14:30.715017+0000","last_peered":"2024-09-16T10:14:30.715017+0000","last_clean":"2024-09-16T10:14:30.715017+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:14:30.715017+0000","last_undegraded":"2024-09-16T10:14:30.715017+0000","last_fullsized":"2024-09-16T10:14:30.715017+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.452904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.069978+0000","last_change":"2024-09-16T10:13:43.122627+0000","last_active":"2024-09-16T10:14:31.069978+0000","last_peered":"2024-09-16T10:14:31.069978+0000","last_clean":"2024-09-16T10:14:31.069978+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:14:31.069978+0000","last_undegraded":"2024-09-16T10:14:31.069978+0000","last_fullsized":"2024-09-16T10:14:31.069978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":80,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790875+0000","last_change":"2024-09-16T10:13:39.105379+0000","last_active":"2024-09-16T10:14:30.790875+0000","last_peered":"2024-09-16T10:14:30.790875+0000","last_clean":"2024-09-16T10:14:30.790875+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:14:30.790875+0000","last_undegraded":"2024-09-16T10:14:30.790875+0000","last_fullsized":"2024-09-16T10:14:30.790875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.713974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808351+0000","last_change":"2024-09-16T10:13:37.093687+0000","last_active":"2024-09-16T10:14:31.808351+0000","last_peered":"2024-09-16T10:14:31.808351+0000","last_clean":"2024-09-16T10:14:31.808351+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:14:31.808351+0000","last_undegraded":"2024-09-16T10:14:31.808351+0000","last_fullsized":"2024-09-16T10:14:31.808351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.984174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715071+0000","last_change":"2024-09-16T10:13:41.113511+0000","last_active":"2024-09-16T10:14:30.715071+0000","last_peered":"2024-09-16T10:14:30.715071+0000","last_clean":"2024-09-16T10:14:30.715071+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:14:30.715071+0000","last_undegraded":"2024-09-16T10:14:30.715071+0000","last_fullsized":"2024-09-16T10:14:30.715071+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:42.162031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807268+0000","last_change":"2024-09-16T10:13:43.122094+0000","last_active":"2024-09-16T10:14:31.807268+0000","last_peered":"2024-09-16T10:14:31.807268+0000","last_clean":"2024-09-16T10:14:31.807268+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:14:31.807268+0000","last_undegraded":"2024-09-16T10:14:31.807268+0000","last_fullsized":"2024-09-16T10:14:31.807268+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.948353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":72,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711031+0000","last_change":"2024-09-16T10:13:39.095588+0000","last_active":"2024-09-16T10:14:30.711031+0000","last_peered":"2024-09-16T10:14:30.711031+0000","last_clean":"2024-09-16T10:14:30.711031+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:14:30.711031+0000","last_undegraded":"2024-09-16T10:14:30.711031+0000","last_fullsized":"2024-09-16T10:14:30.711031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:05.533870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808510+0000","last_change":"2024-09-16T10:13:37.097257+0000","last_active":"2024-09-16T10:14:31.808510+0000","last_peered":"2024-09-16T10:14:31.808510+0000","last_clean":"2024-09-16T10:14:31.808510+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:14:31.808510+0000","last_undegraded":"2024-09-16T10:14:31.808510+0000","last_fullsized":"2024-09-16T10:14:31.808510+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.017319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714213+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:14:30.714213+0000","last_peered":"2024-09-16T10:14:30.714213+0000","last_clean":"2024-09-16T10:14:30.714213+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:14:30.714213+0000","last_undegraded":"2024-09-16T10:14:30.714213+0000","last_fullsized":"2024-09-16T10:14:30.714213+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.545235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344658+0000","last_change":"2024-09-16T10:13:43.125823+0000","last_active":"2024-09-16T10:14:31.344658+0000","last_peered":"2024-09-16T10:14:31.344658+0000","last_clean":"2024-09-16T10:14:31.344658+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:14:31.344658+0000","last_undegraded":"2024-09-16T10:14:31.344658+0000","last_fullsized":"2024-09-16T10:14:31.344658+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.646167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105704+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:14:32.105704+0000","last_peered":"2024-09-16T10:14:32.105704+0000","last_clean":"2024-09-16T10:14:32.105704+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:14:32.105704+0000","last_undegraded":"2024-09-16T10:14:32.105704+0000","last_fullsized":"2024-09-16T10:14:32.105704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.689518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808509+0000","last_change":"2024-09-16T10:13:37.092779+0000","last_active":"2024-09-16T10:14:31.808509+0000","last_peered":"2024-09-16T10:14:31.808509+0000","last_clean":"2024-09-16T10:14:31.808509+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:14:31.808509+0000","last_undegraded":"2024-09-16T10:14:31.808509+0000","last_fullsized":"2024-09-16T10:14:31.808509+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.367150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807856+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:31.807856+0000","last_peered":"2024-09-16T10:14:31.807856+0000","last_clean":"2024-09-16T10:14:31.807856+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:14:31.807856+0000","last_undegraded":"2024-09-16T10:14:31.807856+0000","last_fullsized":"2024-09-16T10:14:31.807856+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:48.594260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.105552+0000","last_change":"2024-09-16T10:13:43.283434+0000","last_active":"2024-09-16T10:14:32.105552+0000","last_peered":"2024-09-16T10:14:32.105552+0000","last_clean":"2024-09-16T10:14:32.105552+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:14:32.105552+0000","last_undegraded":"2024-09-16T10:14:32.105552+0000","last_fullsized":"2024-09-16T10:14:32.105552+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.845991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790155+0000","last_change":"2024-09-16T10:13:37.089575+0000","last_active":"2024-09-16T10:14:30.790155+0000","last_peered":"2024-09-16T10:14:30.790155+0000","last_clean":"2024-09-16T10:14:30.790155+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:14:30.790155+0000","last_undegraded":"2024-09-16T10:14:30.790155+0000","last_fullsized":"2024-09-16T10:14:30.790155+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:51.178781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345859+0000","last_change":"2024-09-16T10:13:39.107923+0000","last_active":"2024-09-16T10:14:31.345859+0000","last_peered":"2024-09-16T10:14:31.345859+0000","last_clean":"2024-09-16T10:14:31.345859+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:14:31.345859+0000","last_undegraded":"2024-09-16T10:14:31.345859+0000","last_fullsized":"2024-09-16T10:14:31.345859+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.475348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106474+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:14:32.106474+0000","last_peered":"2024-09-16T10:14:32.106474+0000","last_clean":"2024-09-16T10:14:32.106474+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:14:32.106474+0000","last_undegraded":"2024-09-16T10:14:32.106474+0000","last_fullsized":"2024-09-16T10:14:32.106474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.978048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807518+0000","last_change":"2024-09-16T10:13:43.283644+0000","last_active":"2024-09-16T10:14:31.807518+0000","last_peered":"2024-09-16T10:14:31.807518+0000","last_clean":"2024-09-16T10:14:31.807518+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:14:31.807518+0000","last_undegraded":"2024-09-16T10:14:31.807518+0000","last_fullsized":"2024-09-16T10:14:31.807518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.605473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070788+0000","last_change":"2024-09-16T10:13:39.107919+0000","last_active":"2024-09-16T10:14:31.070788+0000","last_peered":"2024-09-16T10:14:31.070788+0000","last_clean":"2024-09-16T10:14:31.070788+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:14:31.070788+0000","last_undegraded":"2024-09-16T10:14:31.070788+0000","last_fullsized":"2024-09-16T10:14:31.070788+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:39.504540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807926+0000","last_change":"2024-09-16T10:13:37.095220+0000","last_active":"2024-09-16T10:14:31.807926+0000","last_peered":"2024-09-16T10:14:31.807926+0000","last_clean":"2024-09-16T10:14:31.807926+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:14:31.807926+0000","last_undegraded":"2024-09-16T10:14:31.807926+0000","last_fullsized":"2024-09-16T10:14:31.807926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.601224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345781+0000","last_change":"2024-09-16T10:13:41.118905+0000","last_active":"2024-09-16T10:14:31.345781+0000","last_peered":"2024-09-16T10:14:31.345781+0000","last_clean":"2024-09-16T10:14:31.345781+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:14:31.345781+0000","last_undegraded":"2024-09-16T10:14:31.345781+0000","last_fullsized":"2024-09-16T10:14:31.345781+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.674048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.714468+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:14:30.714468+0000","last_peered":"2024-09-16T10:14:30.714468+0000","last_clean":"2024-09-16T10:14:30.714468+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:14:30.714468+0000","last_undegraded":"2024-09-16T10:14:30.714468+0000","last_fullsized":"2024-09-16T10:14:30.714468+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.053913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106413+0000","last_change":"2024-09-16T10:13:39.105916+0000","last_active":"2024-09-16T10:14:32.106413+0000","last_peered":"2024-09-16T10:14:32.106413+0000","last_clean":"2024-09-16T10:14:32.106413+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:14:32.106413+0000","last_undegraded":"2024-09-16T10:14:32.106413+0000","last_fullsized":"2024-09-16T10:14:32.106413+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.209227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070194+0000","last_change":"2024-09-16T10:13:37.097837+0000","last_active":"2024-09-16T10:14:31.070194+0000","last_peered":"2024-09-16T10:14:31.070194+0000","last_clean":"2024-09-16T10:14:31.070194+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:14:31.070194+0000","last_undegraded":"2024-09-16T10:14:31.070194+0000","last_fullsized":"2024-09-16T10:14:31.070194+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:36.558315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345480+0000","last_change":"2024-09-16T10:13:41.119310+0000","last_active":"2024-09-16T10:14:31.345480+0000","last_peered":"2024-09-16T10:14:31.345480+0000","last_clean":"2024-09-16T10:14:31.345480+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:14:31.345480+0000","last_undegraded":"2024-09-16T10:14:31.345480+0000","last_fullsized":"2024-09-16T10:14:31.345480+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:00.607866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711889+0000","last_change":"2024-09-16T10:13:43.119772+0000","last_active":"2024-09-16T10:14:30.711889+0000","last_peered":"2024-09-16T10:14:30.711889+0000","last_clean":"2024-09-16T10:14:30.711889+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:14:30.711889+0000","last_undegraded":"2024-09-16T10:14:30.711889+0000","last_fullsized":"2024-09-16T10:14:30.711889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:50.088506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345574+0000","last_change":"2024-09-16T10:13:39.105939+0000","last_active":"2024-09-16T10:14:31.345574+0000","last_peered":"2024-09-16T10:14:31.345574+0000","last_clean":"2024-09-16T10:14:31.345574+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:14:31.345574+0000","last_undegraded":"2024-09-16T10:14:31.345574+0000","last_fullsized":"2024-09-16T10:14:31.345574+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.788054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808121+0000","last_change":"2024-09-16T10:13:37.094640+0000","last_active":"2024-09-16T10:14:31.808121+0000","last_peered":"2024-09-16T10:14:31.808121+0000","last_clean":"2024-09-16T10:14:31.808121+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:14:31.808121+0000","last_undegraded":"2024-09-16T10:14:31.808121+0000","last_fullsized":"2024-09-16T10:14:31.808121+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:59.045517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106302+0000","last_change":"2024-09-16T10:13:41.116455+0000","last_active":"2024-09-16T10:14:32.106302+0000","last_peered":"2024-09-16T10:14:32.106302+0000","last_clean":"2024-09-16T10:14:32.106302+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:14:32.106302+0000","last_undegraded":"2024-09-16T10:14:32.106302+0000","last_fullsized":"2024-09-16T10:14:32.106302+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:44.665680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070445+0000","last_change":"2024-09-16T10:13:43.120971+0000","last_active":"2024-09-16T10:14:31.070445+0000","last_peered":"2024-09-16T10:14:31.070445+0000","last_clean":"2024-09-16T10:14:31.070445+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:14:31.070445+0000","last_undegraded":"2024-09-16T10:14:31.070445+0000","last_fullsized":"2024-09-16T10:14:31.070445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:09.791625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711070+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:14:30.711070+0000","last_peered":"2024-09-16T10:14:30.711070+0000","last_clean":"2024-09-16T10:14:30.711070+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:14:30.711070+0000","last_undegraded":"2024-09-16T10:14:30.711070+0000","last_fullsized":"2024-09-16T10:14:30.711070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.996979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.711840+0000","last_change":"2024-09-16T10:13:37.090129+0000","last_active":"2024-09-16T10:14:30.711840+0000","last_peered":"2024-09-16T10:14:30.711840+0000","last_clean":"2024-09-16T10:14:30.711840+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:14:30.711840+0000","last_undegraded":"2024-09-16T10:14:30.711840+0000","last_fullsized":"2024-09-16T10:14:30.711840+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.823652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715397+0000","last_change":"2024-09-16T10:13:41.112912+0000","last_active":"2024-09-16T10:14:30.715397+0000","last_peered":"2024-09-16T10:14:30.715397+0000","last_clean":"2024-09-16T10:14:30.715397+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:14:30.715397+0000","last_undegraded":"2024-09-16T10:14:30.715397+0000","last_fullsized":"2024-09-16T10:14:30.715397+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:47.606633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344470+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:14:31.344470+0000","last_peered":"2024-09-16T10:14:31.344470+0000","last_clean":"2024-09-16T10:14:31.344470+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:14:31.344470+0000","last_undegraded":"2024-09-16T10:14:31.344470+0000","last_fullsized":"2024-09-16T10:14:31.344470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:08.223461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715520+0000","last_change":"2024-09-16T10:13:39.098286+0000","last_active":"2024-09-16T10:14:30.715520+0000","last_peered":"2024-09-16T10:14:30.715520+0000","last_clean":"2024-09-16T10:14:30.715520+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:14:30.715520+0000","last_undegraded":"2024-09-16T10:14:30.715520+0000","last_fullsized":"2024-09-16T10:14:30.715520+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:03.259110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.808247+0000","last_change":"2024-09-16T10:13:37.091081+0000","last_active":"2024-09-16T10:14:31.808247+0000","last_peered":"2024-09-16T10:14:31.808247+0000","last_clean":"2024-09-16T10:14:31.808247+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:14:31.808247+0000","last_undegraded":"2024-09-16T10:14:31.808247+0000","last_fullsized":"2024-09-16T10:14:31.808247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:49.952981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345276+0000","last_change":"2024-09-16T10:13:41.114260+0000","last_active":"2024-09-16T10:14:31.345276+0000","last_peered":"2024-09-16T10:14:31.345276+0000","last_clean":"2024-09-16T10:14:31.345276+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:14:31.345276+0000","last_undegraded":"2024-09-16T10:14:31.345276+0000","last_fullsized":"2024-09-16T10:14:31.345276+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:45.627978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.069883+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:14:31.069883+0000","last_peered":"2024-09-16T10:14:31.069883+0000","last_clean":"2024-09-16T10:14:31.069883+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:14:31.069883+0000","last_undegraded":"2024-09-16T10:14:31.069883+0000","last_fullsized":"2024-09-16T10:14:31.069883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:10.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345182+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:14:31.345182+0000","last_peered":"2024-09-16T10:14:31.345182+0000","last_clean":"2024-09-16T10:14:31.345182+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:14:31.345182+0000","last_undegraded":"2024-09-16T10:14:31.345182+0000","last_fullsized":"2024-09-16T10:14:31.345182+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.847239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:32.106476+0000","last_change":"2024-09-16T10:13:37.088453+0000","last_active":"2024-09-16T10:14:32.106476+0000","last_peered":"2024-09-16T10:14:32.106476+0000","last_clean":"2024-09-16T10:14:32.106476+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:14:32.106476+0000","last_undegraded":"2024-09-16T10:14:32.106476+0000","last_fullsized":"2024-09-16T10:14:32.106476+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:43.596318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807553+0000","last_change":"2024-09-16T10:13:41.112830+0000","last_active":"2024-09-16T10:14:31.807553+0000","last_peered":"2024-09-16T10:14:31.807553+0000","last_clean":"2024-09-16T10:14:31.807553+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:14:31.807553+0000","last_undegraded":"2024-09-16T10:14:31.807553+0000","last_fullsized":"2024-09-16T10:14:31.807553+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:41.842458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344517+0000","last_change":"2024-09-16T10:13:43.286716+0000","last_active":"2024-09-16T10:14:31.344517+0000","last_peered":"2024-09-16T10:14:31.344517+0000","last_clean":"2024-09-16T10:14:31.344517+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:14:31.344517+0000","last_undegraded":"2024-09-16T10:14:31.344517+0000","last_fullsized":"2024-09-16T10:14:31.344517+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.000117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.344971+0000","last_change":"2024-09-16T10:13:39.101213+0000","last_active":"2024-09-16T10:14:31.344971+0000","last_peered":"2024-09-16T10:14:31.344971+0000","last_clean":"2024-09-16T10:14:31.344971+0000","last_became_active":"2024-09-16T10:13:39.100788+0000","last_became_peered":"2024-09-16T10:13:39.100788+0000","last_unstale":"2024-09-16T10:14:31.344971+0000","last_undegraded":"2024-09-16T10:14:31.344971+0000","last_fullsized":"2024-09-16T10:14:31.344971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.518507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070727+0000","last_change":"2024-09-16T10:13:37.094533+0000","last_active":"2024-09-16T10:14:31.070727+0000","last_peered":"2024-09-16T10:14:31.070727+0000","last_clean":"2024-09-16T10:14:31.070727+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:14:31.070727+0000","last_undegraded":"2024-09-16T10:14:31.070727+0000","last_fullsized":"2024-09-16T10:14:31.070727+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.183923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.790283+0000","last_change":"2024-09-16T10:13:41.114472+0000","last_active":"2024-09-16T10:14:30.790283+0000","last_peered":"2024-09-16T10:14:30.790283+0000","last_clean":"2024-09-16T10:14:30.790283+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:14:30.790283+0000","last_undegraded":"2024-09-16T10:14:30.790283+0000","last_fullsized":"2024-09-16T10:14:30.790283+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:02.243642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.807285+0000","last_change":"2024-09-16T10:13:43.119917+0000","last_active":"2024-09-16T10:14:31.807285+0000","last_peered":"2024-09-16T10:14:31.807285+0000","last_clean":"2024-09-16T10:14:31.807285+0000","last_became_active":"2024-09-16T10:13:43.119612+0000","last_became_peered":"2024-09-16T10:13:43.119612+0000","last_unstale":"2024-09-16T10:14:31.807285+0000","last_undegraded":"2024-09-16T10:14:31.807285+0000","last_fullsized":"2024-09-16T10:14:31.807285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:46.539099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.715721+0000","last_change":"2024-09-16T10:13:43.114761+0000","last_active":"2024-09-16T10:14:30.715721+0000","last_peered":"2024-09-16T10:14:30.715721+0000","last_clean":"2024-09-16T10:14:30.715721+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:14:30.715721+0000","last_undegraded":"2024-09-16T10:14:30.715721+0000","last_fullsized":"2024-09-16T10:14:30.715721+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:04.210195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.345096+0000","last_change":"2024-09-16T10:13:37.089832+0000","last_active":"2024-09-16T10:14:31.345096+0000","last_peered":"2024-09-16T10:14:31.345096+0000","last_clean":"2024-09-16T10:14:31.345096+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:14:31.345096+0000","last_undegraded":"2024-09-16T10:14:31.345096+0000","last_fullsized":"2024-09-16T10:14:31.345096+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:58.587954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:30.791167+0000","last_change":"2024-09-16T10:13:39.098801+0000","last_active":"2024-09-16T10:14:30.791167+0000","last_peered":"2024-09-16T10:14:30.791167+0000","last_clean":"2024-09-16T10:14:30.791167+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:14:30.791167+0000","last_undegraded":"2024-09-16T10:14:30.791167+0000","last_fullsized":"2024-09-16T10:14:30.791167+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:40.353803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-16T10:14:31.070623+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:14:31.070623+0000","last_peered":"2024-09-16T10:14:31.070623+0000","last_clean":"2024-09-16T10:14:31.070623+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:14:31.070623+0000","last_undegraded":"2024-09-16T10:14:31.070623+0000","last_fullsized":"2024-09-16T10:14:31.070623+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:56.860042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233999,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28416,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716736,"statfs":{"total":95995035648,"available":95965937664,"internally_reserved":0,"allocated":1421312,"data_stored":927565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528339,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32552,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712600,"statfs":{"total":95995035648,"available":95961702400,"internally_reserved":0,"allocated":1396736,"data_stored":926467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789977,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":798720,"data_stored":331079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32004,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713148,"statfs":{"total":95995035648,"available":95962263552,"internally_reserved":0,"allocated":835584,"data_stored":337140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215139,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":847872,"data_stored":338419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":802816,"data_stored":336260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":872448,"data_stored":338320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28428,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716724,"statfs":{"total":95995035648,"available":95965925376,"internally_reserved":0,"allocated":1429504,"data_stored":928267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:14:36.887 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T10:14:36.887 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-16T10:14:37.151 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:37.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:36 smithi063 ceph-mon[29993]: from='client.14805 v1:172.21.15.63:0/4129309446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:37.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:14:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:14:37.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:36 smithi063 ceph-mon[39470]: from='client.14805 v1:172.21.15.63:0/4129309446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:37.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:36 smithi165 ceph-mon[32696]: from='client.14805 v1:172.21.15.63:0/4129309446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:14:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 80 2024-09-16T10:14:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:14:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-09-16T10:08:21.081928+0000 2024-09-16T10:14:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-09-16T10:14:30.706271+0000 2024-09-16T10:14:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T10:14:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:crush_version 18 2024-09-16T10:14:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:full_ratio 0.95 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:backfillfull_ratio 0.9 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:nearfull_ratio 0.85 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:require_min_compat_client luminous 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:min_compat_client jewel 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:require_osd_release squid 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:stretch_mode_enabled false 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 61 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-16T10:14:37.840 INFO:teuthology.orchestra.run.smithi063.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:max_osd 8 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 up in weight 1 up_from 8 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6801/2031112888 v1:172.21.15.63:6802/2031112888 exists,up c4234e56-a2f2-467f-9616-190a52181d42 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6805/3717451957 v1:172.21.15.63:6806/3717451957 exists,up 6fe9a4c9-c351-4223-8aab-6405152696c6 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 up in weight 1 up_from 18 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6809/196353360 v1:172.21.15.63:6810/196353360 exists,up 4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 up in weight 1 up_from 24 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6813/249840957 v1:172.21.15.63:6814/249840957 exists,up 355833dd-7cc5-4f59-b036-ed8939892ac4 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 up in weight 1 up_from 31 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6800/2373552307 v1:172.21.15.165:6801/2373552307 exists,up da2fbe06-9462-44ec-9ffb-c5563ccee4b7 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 up in weight 1 up_from 37 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6804/783450148 v1:172.21.15.165:6805/783450148 exists,up b866ad81-ae05-4c8e-9e12-822440fcb713 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6 up in weight 1 up_from 45 up_thru 62 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6808/424252741 v1:172.21.15.165:6809/424252741 exists,up ce05cb1a-e5c6-4ea9-a2c0-0039c874affa 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7 up in weight 1 up_from 54 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6812/2484590937 v1:172.21.15.165:6813/2484590937 exists,up 1b464a8a-ad32-47ab-82fb-3097077ebc48 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/56975596 expires 2024-09-17T10:14:06.991973+0000 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/455268617 expires 2024-09-17T10:08:51.022048+0000 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/3928610681 expires 2024-09-17T10:08:51.022048+0000 2024-09-16T10:14:37.841 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/955884223 expires 2024-09-17T10:14:06.991973+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/1387667925 expires 2024-09-17T10:08:51.022048+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:6800/670090510 expires 2024-09-17T10:08:51.022048+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/1037308168 expires 2024-09-17T10:09:14.602270+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/2411639026 expires 2024-09-17T10:09:14.602270+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/812143427 expires 2024-09-17T10:14:06.991973+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:6800/274451570 expires 2024-09-17T10:09:14.602270+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/1121825432 expires 2024-09-17T10:14:06.991973+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:6800/2945123439 expires 2024-09-17T10:14:06.991973+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/2625337613 expires 2024-09-17T10:09:14.602270+0000 2024-09-16T10:14:37.842 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/2993184887 expires 2024-09-17T10:14:06.991973+0000 2024-09-16T10:14:38.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[29993]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3261148733' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[39470]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:38.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3261148733' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-16T10:14:38.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:14:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:14:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:37 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:14:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:37 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:14:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:37 smithi165 ceph-mon[32696]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:14:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:37 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:37 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3261148733' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-16T10:14:38.580 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6801/2031112888 v1:172.21.15.63:6802/2031112888 exists,up c4234e56-a2f2-467f-9616-190a52181d42', 'osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6805/3717451957 v1:172.21.15.63:6806/3717451957 exists,up 6fe9a4c9-c351-4223-8aab-6405152696c6', 'osd.2 up in weight 1 up_from 18 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6809/196353360 v1:172.21.15.63:6810/196353360 exists,up 4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'osd.3 up in weight 1 up_from 24 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.63:6813/249840957 v1:172.21.15.63:6814/249840957 exists,up 355833dd-7cc5-4f59-b036-ed8939892ac4', 'osd.4 up in weight 1 up_from 31 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6800/2373552307 v1:172.21.15.165:6801/2373552307 exists,up da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'osd.5 up in weight 1 up_from 37 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6804/783450148 v1:172.21.15.165:6805/783450148 exists,up b866ad81-ae05-4c8e-9e12-822440fcb713', 'osd.6 up in weight 1 up_from 45 up_thru 62 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6808/424252741 v1:172.21.15.165:6809/424252741 exists,up ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'osd.7 up in weight 1 up_from 54 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6812/2484590937 v1:172.21.15.165:6813/2484590937 exists,up 1b464a8a-ad32-47ab-82fb-3097077ebc48'] 2024-09-16T10:14:38.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-16T10:14:38.839 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:39.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:14:39.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:14:39.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[29993]: 3.12 scrub starts 2024-09-16T10:14:39.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[29993]: 3.12 scrub ok 2024-09-16T10:14:39.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:39.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:14:39.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:14:39.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[39470]: 3.12 scrub starts 2024-09-16T10:14:39.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[39470]: 3.12 scrub ok 2024-09-16T10:14:39.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:38 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:39.276 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mon.a", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster": "ceph", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "admin_socket_mode": "", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_client_required": "cephx, none", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_cluster_required": "cephx", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_debug": "true", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_service_required": "cephx", 2024-09-16T10:14:39.277 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "auth_supported": "", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_aio": "true", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_aio_reap_max": "16", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_async_discard_threads": "0", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_block_size": "4096", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_debug_aio": "false", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_enable_discard": "false", 2024-09-16T10:14:39.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_flock_retry": "3", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_inject_crash": "0", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_ioring": "false", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_ioring_hipri": "false", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bdev_type": "", 2024-09-16T10:14:39.279 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_allocator": "hybrid", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_buffered_io": "true", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T10:14:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_replay_recovery": "false", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_sync_write": "false", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T10:14:39.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_allocator": "hybrid", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_async_db_compaction": "true", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T10:14:39.282 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_create": "true", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_db_create": "false", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_db_path": "", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_db_size": "0", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_path": "", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_size": "107374182400", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_wal_create": "false", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_wal_path": "", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bluefs": "true", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T10:14:39.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_autotune": "true", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T10:14:39.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_size": "0", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_type": "2q", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_clone_cow": "true", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T10:14:39.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T10:14:39.286 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T10:14:39.286 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_mode": "none", 2024-09-16T10:14:39.286 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T10:14:39.286 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_freelist": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T10:14:39.287 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T10:14:39.288 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fail_eio": "false", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T10:14:39.289 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T10:14:39.290 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_max_blob_size": "0", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T10:14:39.291 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T10:14:39.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T10:14:39.293 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T10:14:39.293 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T10:14:39.293 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T10:14:39.293 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T10:14:39.293 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_spdk_mem": "512", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T10:14:39.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_tracing": "false", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_write_v2": "false", 2024-09-16T10:14:39.295 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_write_v2_random": "false", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T10:14:39.296 INFO:teuthology.orchestra.run.smithi063.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephx_cluster_require_version": "2", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephx_require_signatures": "false", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephx_require_version": "2", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephx_service_require_signatures": "false", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephx_service_require_version": "2", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "cephx_sign_messages": "true", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "chdir": "", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "client_acl_type": "", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "client_asio_thread_count": "2", 2024-09-16T10:14:39.297 INFO:teuthology.orchestra.run.smithi063.stdout: "client_cache_mid": "0.750000", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_cache_size": "16384", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_caps_release_delay": "5", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_check_pool_perm": "true", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_debug_force_sync_read": "false", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_debug_getattr_caps": "false", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_debug_inject_features": "", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_die_on_failed_remount": "false", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_dirsize_rbytes": "true", 2024-09-16T10:14:39.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_force_lazyio": "false", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_fs": "", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_inject_release_failure": "false", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_max_inline_size": "4096", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_mds_namespace": "", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_metadata": "", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_mount_gid": "-1", 2024-09-16T10:14:39.299 INFO:teuthology.orchestra.run.smithi063.stdout: "client_mount_timeout": "300", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_mount_uid": "-1", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_mountpoint": "/", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_notify_timeout": "10", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_oc": "true", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_oc_max_objects": "1000", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_oc_size": "209715200", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_permissions": "true", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_quota": "true", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_quota_df": "true", 2024-09-16T10:14:39.300 INFO:teuthology.orchestra.run.smithi063.stdout: "client_readahead_max_bytes": "0", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_readahead_max_periods": "4", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_readahead_min": "131072", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_reconnect_stale": "false", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_shutdown_timeout": "30", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_snapdir": ".snap", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_tick_interval": "1", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_trace": "", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_use_faked_inos": "false", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_use_random_mds": "false", 2024-09-16T10:14:39.301 INFO:teuthology.orchestra.run.smithi063.stdout: "clog_to_graylog": "false", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "clog_to_graylog_port": "12201", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "clog_to_monitors": "default=true", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "clog_to_syslog": "false", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "clog_to_syslog_level": "info", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "-", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_network": "", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_network_interface": "", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "compressor_zlib_isal": "false", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "compressor_zlib_level": "5", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "compressor_zstd_level": "1", 2024-09-16T10:14:39.302 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location_hook": "", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location_hook_timeout": "10", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "daemonize": "false", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "daos_pool": "tank", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T10:14:39.303 INFO:teuthology.orchestra.run.smithi063.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_alienstore": "0/5", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_asok": "1/5", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_asok_assert_abort": "false", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_auth": "1/5", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_bdev": "1/3", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_bluefs": "1/5", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_bluestore": "1/5", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_buffer": "0/1", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_ceph_dedup": "0/5", 2024-09-16T10:14:39.304 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_cephsqlite": "0/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_client": "0/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_compressor": "1/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_context": "0/1", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_crush": "1/1", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_crypto": "1/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_cyanstore": "0/5", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T10:14:39.305 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_dpdk": "1/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_eventtrace": "1/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_filer": "0/1", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_filestore": "1/3", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_finisher": "1/1", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_fuse": "1/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_javaclient": "1/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_journal": "1/3", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_journaler": "0/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_kstore": "1/5", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_lockdep": "0/1", 2024-09-16T10:14:39.306 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mclock": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mds": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mds_balancer": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mds_locker": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mds_log": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mds_migrator": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_memstore": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mgr": "2/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mgrc": "1/5", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_mon": "20/20", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_monc": "0/10", 2024-09-16T10:14:39.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_ms": "1/1", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_none": "0/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_objclass": "0/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_objectcacher": "0/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_objecter": "0/1", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_optracker": "0/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_osd": "1/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_paxos": "20/20", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_perfcounter": "1/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_prioritycache": "1/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rados": "0/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rbd": "0/5", 2024-09-16T10:14:39.308 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rbd_replay": "0/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_refs": "0/0", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_reserver": "1/1", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw_access": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw_flight": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw_notification": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rgw_sync": "1/5", 2024-09-16T10:14:39.309 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_rocksdb": "4/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_backref": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_cache": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_device": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_epm": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_journal": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_lba": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_odata": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_omap": "0/5", 2024-09-16T10:14:39.310 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_onode": "0/5", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_t": "0/5", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_seastore_tm": "0/5", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_striper": "0/1", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_test": "0/5", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_throttle": "1/1", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_timer": "0/1", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_tp": "0/5", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "debug_trace": "1/5", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "device_failure_prediction_mode": "none", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-16T10:14:39.311 INFO:teuthology.orchestra.run.smithi063.stdout: "err_to_graylog": "false", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "err_to_journald": "false", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "err_to_stderr": "true", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "err_to_syslog": "false", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event_tracing": "false", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_cert_file": "", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_http_port": "9926", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_key_file": "", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_prio_limit": "5", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_sort_metrics": "true", 2024-09-16T10:14:39.312 INFO:teuthology.orchestra.run.smithi063.stdout: "exporter_stats_period": "5", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "fake_statfs_for_testing": "0", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "fatal_signal_handlers": "true", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filer_max_purge_ops": "10", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filer_max_truncate_ops": "128", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_blackhole": "false", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_btrfs_snap": "true", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_caller_concurrency": "10", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T10:14:39.313 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_debug_omap_check": "false", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_debug_verify_split": "false", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_dump_file": "", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_fadvise": "true", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_fail_eio": "true", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_fd_cache_size": "128", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_fiemap": "false", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T10:14:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_inject_stall": "0", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_journal_parallel": "false", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_journal_trailing": "false", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_journal_writeahead": "false", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_kill_at": "0", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T10:14:39.315 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_merge_threshold": "-10", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_odsync_write": "false", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T10:14:39.316 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_omap_backend_path": "", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_op_threads": "2", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_punch_hole": "false", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T10:14:39.317 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_queue_max_ops": "50", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_seek_data_hole": "false", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_sloppy_crc": "false", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_splice": "false", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_split_multiple": "2", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_split_rand_factor": "20", 2024-09-16T10:14:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_update_to": "1000", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T10:14:39.319 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_xfs_extsize": "false", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "filestore_zfs_snap": "false", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fio_dir": "/tmp/fio", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_allow_other": "true", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_big_writes": "true", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_debug": "false", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_default_permissions": "false", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_disable_pagecache": "false", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_max_write": "0", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_multithreaded": "true", 2024-09-16T10:14:39.320 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_require_active_mds": "true", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_set_user_groups": "true", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_splice_move": "true", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_splice_read": "true", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_splice_write": "true", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "gss_target_name": "ceph", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_file": "", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_inject_failure": "0", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_interval": "5", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "host": "smithi063", 2024-09-16T10:14:39.321 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "inject_early_sigterm": "false", 2024-09-16T10:14:39.322 INFO:teuthology.orchestra.run.smithi063.stdout: "jaeger_agent_port": "6799", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "jaeger_tracing_enable": "false", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_aio": "true", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_align_min_size": "65536", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_block_align": "true", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_block_size": "4096", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_dio": "true", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_discard": "false", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_force_aio": "false", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_ignore_corruption": "false", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_max_write_entries": "100", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_replay_from": "0", 2024-09-16T10:14:39.323 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_write_header_frequency": "0", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journal_zero_on_create": "false", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journaler_prefetch_periods": "10", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journaler_prezero_periods": "5", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "journaler_write_head_interval": "15", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "keyfile": "", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_backend": "rocksdb", 2024-09-16T10:14:39.324 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_max_bytes": "67108864", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_max_ops": "512", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_onode_map_size": "1024", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "kstore_sync_transaction": "false", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "librados_thread_count": "2", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "lockdep": "false", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "lockdep_force_backtrace": "false", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "log_coarse_timestamps": "true", 2024-09-16T10:14:39.325 INFO:teuthology.orchestra.run.smithi063.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_flush_on_exit": "false", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_graylog_port": "12201", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_max_new": "1000", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_max_recent": "10000", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_stderr_prefix": "", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": "true", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_graylog": "false", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_journald": "false", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_stderr": "false", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_syslog": "false", 2024-09-16T10:14:39.326 INFO:teuthology.orchestra.run.smithi063.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_action_on_write_error": "1", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_alternate_name_max": "8192", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_asio_thread_count": "2", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_export_pin": "true", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T10:14:39.327 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_interval": "10", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_max": "-1", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_max_until": "-1", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_merge_size": "50", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_mode": "0", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T10:14:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_split_bits": "3", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_split_size": "10000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_mid": "0.700000", 2024-09-16T10:14:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_trim_interval": "1", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T10:14:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_debug_auth_pins": "false", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_debug_frag": "false", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_debug_scatterstat": "false", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_debug_subtrees": "false", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_default_dir_hash": "2", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_defer_session_stale": "true", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T10:14:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dir_max_entries": "0", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dir_prefetch": "true", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_early_reply": "true", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_enable_op_tracker": "true", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_enforce_unique_name": "true", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T10:14:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_inject_health_dummy": "false", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T10:14:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_join_fs": "", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_journal_format": "1", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_create_at": "0", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_dirfrag_at": "0", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_export_at": "0", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_import_at": "0", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_journal_at": "0", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T10:14:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_link_at": "0", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_openc_at": "0", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_rename_at": "0", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_max_events": "-1", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_max_segments": "128", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_pause": "false", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_segment_size": "0", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T10:14:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_trim_threshold": "128", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_completed_requests": "100000", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_export_size": "20971520", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_file_recover": "32", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_purge_files": "64", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_purge_ops": "8192", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T10:14:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_metrics_update_interval": "2", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_min_caps_per_client": "100", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_numa_node": "-1", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_op_history_duration": "600", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_op_history_size": "20", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_op_log_threshold": "5", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_ping_grace": "15", 2024-09-16T10:14:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_ping_interval": "5", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_recall_max_caps": "30000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_replay_interval": "1.000000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_root_ino_gid": "0", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_root_ino_uid": "0", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T10:14:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_scrub_stats_review_period": "1", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_shutdown_check": "0", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_skip_ino": "0", 2024-09-16T10:14:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_snap_min_uid": "0", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_snap_rstat": "false", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_symlink_recovery": "true", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_thrash_exports": "0", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_thrash_fragments": "0", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_tick_interval": "5.000000", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_valgrind_exit": "false", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_verify_backtrace": "1", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_verify_scatter": "false", 2024-09-16T10:14:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_wipe_sessions": "false", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool_debug": "false", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "memstore_page_set": "false", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "memstore_page_size": "65536", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_client_bytes": "134217728", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_client_messages": "512", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T10:14:39.341 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_disabled_modules": "", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_max_pg_creating": "1024", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_mds_messages": "128", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_mon_messages": "128", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_op_history_duration": "600", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_op_history_size": "20", 2024-09-16T10:14:39.342 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_op_log_threshold": "5", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_osd_messages": "8192", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_pool": "true", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_standby_modules": "true", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_stats_period": "5", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_stats_threshold": "5", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_tick_period": "2", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_allow_pool_delete": "true", 2024-09-16T10:14:39.343 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_allow_pool_size_one": "false", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_bytes": "104857600", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_directed_command_retry": "2", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T10:14:39.344 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_client_target_rank": "-1", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_level": "debug", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_journald": "true", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T10:14:39.345 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_compact_on_start": "false", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_compact_on_trim": "true", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_cpu_threads": "4", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_data_avail_crit": "5", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_data_avail_warn": "5", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T10:14:39.346 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_dump_json": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_extra_checks": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_down_mkfs_grace": "60", 2024-09-16T10:14:39.347 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_down_uptime_grace": "60", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_election_default_strategy": "1", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_election_timeout": "5.000000", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_enable_op_tracker": "true", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_fake_pool_delete": "false", 2024-09-16T10:14:39.348 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_force_quorum_join": "false", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_health_detail_to_clog": "true", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_health_log_update_period": "5", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_health_max_detail": "50", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_health_to_clog": "true", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_host": "172.21.15.63:6789/0", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_host_override": "", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_initial_members": "", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T10:14:39.349 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_lease": "5.000000", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_log_full_interval": "50", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_log_max": "10000", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_log_max_summary": "50", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_log_epochs": "500", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-16T10:14:39.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_osd": "10000", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_memory_autotune": "true", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_memory_target": "2147483648", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mgr_digest_period": "5", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T10:14:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_min_osdmap_epochs": "50", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_op_complaint_time": "30", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_op_history_duration": "600", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_op_history_size": "20", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_op_log_threshold": "5", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T10:14:39.352 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_cache_size": "500", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T10:14:39.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T10:14:39.354 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_report_timeout": "900", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T10:14:39.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T10:14:39.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:38 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:14:39.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:38 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:14:39.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:38 smithi165 ceph-mon[32696]: 3.12 scrub starts 2024-09-16T10:14:39.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:38 smithi165 ceph-mon[32696]: 3.12 scrub ok 2024-09-16T10:14:39.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:38 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:39.357 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T10:14:39.358 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_reweight_max_osds": "4", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T10:14:39.359 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_scrub_interval": "86400", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_scrub_max_keys": "100", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_scrub_timeout": "300", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_session_timeout": "300", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_smart_report_timeout": "5", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_stretch_pool_size": "4", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_sync_debug": "false", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T10:14:39.360 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_tick_interval": "5", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T10:14:39.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_older_version": "true", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T10:14:39.362 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": "", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "motr_tracing_enabled": "false", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_op_threads": "3", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T10:14:39.363 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_cm": "false", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_device_name": "", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_sl": "3", 2024-09-16T10:14:39.364 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_rdma_type": "ib", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_async_reap_threshold": "5", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_before_connect": "false", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_ipv4": "true", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_ipv6": "false", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_msgr1": "true", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_msgr2": "false", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_port_max": "7568", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_port_min": "6800", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_retry_count": "3", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_bind_retry_delay": "5", 2024-09-16T10:14:39.365 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_blackhole_client": "false", 2024-09-16T10:14:39.366 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_blackhole_mds": "false", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_blackhole_mgr": "false", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_blackhole_mon": "false", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_blackhole_osd": "false", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_client_mode": "crc secure", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_cluster_type": "", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_compress_secure": "false", 2024-09-16T10:14:39.368 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_crc_data": "true", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_crc_header": "true", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_die_on_bug": "true", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_die_on_old_message": "true", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T10:14:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_hugepages": "", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_lro": "true", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_pmd": "", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_port_id": "0", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T10:14:39.370 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_dump_on_send": "false", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_inject_delay_type": "", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_inject_network_congestion": "0", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_inject_socket_failures": "0", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_max_accept_failures": "4", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_max_backoff": "15.000000", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T10:14:39.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_osd_compress_mode": "none", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_pq_min_cost": "65536", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_public_type": "", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_service_mode": "crc secure", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_tcp_nodelay": "true", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ms_type": "async+posix", 2024-09-16T10:14:39.372 INFO:teuthology.orchestra.run.smithi063.stdout: "no_config_file": "false", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_inflight_ops": "1024", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter_timeout": "10.000000", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objectstore_blackhole": "false", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T10:14:39.373 INFO:teuthology.orchestra.run.smithi063.stdout: "openssl_engine_opts": "", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_agent_max_ops": "4", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_agent_slop": "0.020000", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_asio_thread_count": "2", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_backfill_scan_max": "512", 2024-09-16T10:14:39.374 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_backfill_scan_min": "64", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_backoff_on_peering": "false", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_beacon_report_interval": "300", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bench_duration": "30", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_blkin_trace_all": "false", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T10:14:39.375 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_class_update_on_start": "true", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_client_message_cap": "256", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_client_op_priority": "63", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_client_watch_timeout": "30", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_command_max_records": "256", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_compact_on_start": "false", 2024-09-16T10:14:39.376 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_crush_update_on_start": "true", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T10:14:39.377 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_op_order": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_shutdown": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_trim_objects": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T10:14:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T10:14:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_default_notify_timeout": "30", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_ec_partial_reads": "true", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_enable_op_tracker": "true", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T10:14:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_fast_info": "true", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_fast_shutdown": "true", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_function_tracing": "false", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_heartbeat_grace": "20", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_heartbeat_interval": "6", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T10:14:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_heartbeat_stale": "600", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_journal_size": "5120", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_kill_backfill_at": "0", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T10:14:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_map_cache_size": "50", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_map_dedup": "true", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_map_message_max": "40", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_attr_name_len": "100", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_attr_size": "0", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_backfills": "1", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_markdown_count": "5", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_markdown_period": "600", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_object_name_len": "2048", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_object_size": "134217728", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T10:14:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_pgls": "1024", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_push_cost": "8388608", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_push_objects": "10", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_scrubs": "3", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_max_write_size": "90", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T10:14:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_profile": "balanced", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T10:14:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_memory_base": "805306368", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_memory_target": "4294967296", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_memory_target_autotune": "false", 2024-09-16T10:14:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_min_recovery_priority": "0", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mon_report_interval": "5", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_num_cache_shards": "32", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_numa_node": "-1", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-16T10:14:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_objecter_finishers": "1", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_objectstore": "bluestore", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_objectstore_fuse": "false", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_objectstore_tracing": "false", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_history_duration": "600", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_history_size": "20", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T10:14:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_log_threshold": "5", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_num_shards": "0", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_op_thread_timeout": "15", 2024-09-16T10:14:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_open_classes_on_start": "true", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_os_flags": "0", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_peering_op_priority": "255", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_delete_priority": "5", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T10:14:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_crimson": "false", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T10:14:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_flags": "0", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_min_size": "0", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_size": "3", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_default_type": "replicated", 2024-09-16T10:14:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_cost": "20971520", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_max_active": "0", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T10:14:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_op_priority": "3", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_priority": "5", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_repair_during_recovery": "false", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T10:14:39.394 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_cost": "52428800", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_end_hour": "0", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T10:14:39.395 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_priority": "5", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_retry_delay": "30", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_retry_trimming": "10", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T10:14:39.396 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_skip_data_digest": "false", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_smart_report_timeout": "5", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_snap_trim_priority": "5", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_target_transaction_size": "30", 2024-09-16T10:14:39.397 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_tracing": "false", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_use_stale_snap": "false", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T10:14:39.398 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_kill_at": "0", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_max_join_drift": "10", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_min": "500", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_min_wait": "0.050000", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_service_trim_max": "500", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_service_trim_min": "10", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_stash_full_interval": "25", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_trim_max": "500", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "paxos_trim_min": "250", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "perf": "true", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "pid_file": "", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T10:14:39.399 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "-", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrv": "", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "public_bind_addr": "-", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "public_network": "", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "public_network_interface": "", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "qat_compressor_enabled": "false", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "rados_mon_op_timeout": "0", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "rados_osd_op_timeout": "0", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "rados_tracing": "false", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_atime_update_interval": "60", 2024-09-16T10:14:39.400 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache": "true", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_policy": "writearound", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_size": "33554432", 2024-09-16T10:14:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_compression_hint": "none", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_clone_format": "auto", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_data_pool": "", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_features": "61", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_format": "2", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_map_options": "", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_order": "22", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_pool": "rbd", 2024-09-16T10:14:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_stripe_count": "0", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_io_scheduler": "simple", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T10:14:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_order": "24", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_pool": "", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_journal_splay_width": "4", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T10:14:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T10:14:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T10:14:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_op_threads": "1", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_plugins": "", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T10:14:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T10:14:39.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T10:14:39.409 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_tracing": "false", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_validate_names": "true", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_validate_pool": "true", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "restapi_base_url": "", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "restapi_log_level": "", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_admin_entry": "admin", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_asio_assert_yielding": "false", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_backend_store": "rados", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_barbican_url": "", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_beast_enable_async": "true", 2024-09-16T10:14:39.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_cache_enabled": "true", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_config_store": "rados", 2024-09-16T10:14:39.411 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_content_length_compat": "false", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_cross_domain_policy": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T10:14:39.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T10:14:39.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T10:14:39.414 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T10:14:39.415 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data_log_window": "30", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T10:14:39.416 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T10:14:39.417 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_disable_s3select": "false", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dns_name": "", 2024-09-16T10:14:39.418 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dns_s3website_name": "", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_mdsearch": "true", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_ops_log": "false", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_static_website": "false", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enable_usage_log": "false", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T10:14:39.419 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_expose_bucket": "false", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_extended_http_attrs": "", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_filter": "none", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_max_objs": "32", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T10:14:39.420 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_init_timeout": "300", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T10:14:39.421 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_password": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_project": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_token": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_admin_user": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T10:14:39.422 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_url": "", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lc_max_objs": "32", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lc_max_worker": "3", 2024-09-16T10:14:39.423 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ldap_uri": "", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T10:14:39.424 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_log_http_headers": "", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_attr_size": "0", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T10:14:39.425 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_listing_results": "5000", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_notify_retries": "10", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T10:14:39.426 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T10:14:39.427 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_num_control_oids": "8", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_numa_node": "-1", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T10:14:39.428 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_op_tracing": "false", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_opa_token": "", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_opa_url": "", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ops_log_rados": "false", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T10:14:39.429 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_print_continue": "true", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T10:14:39.430 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_rados_tracing": "false", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_realm": "", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_realm_id": "", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_region": "", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_request_uri": "", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T10:14:39.431 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_resolve_cname": "false", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_run_sync_thread": "true", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T10:14:39.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_script_uri": "", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_service_provider_name": "", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sts_client_id": "", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sts_client_secret": "", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sts_entry": "sts", 2024-09-16T10:14:39.433 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sts_key": "sts", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_auth_url": "", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_custom_header": "", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_need_stats": "true", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_tenant_name": "", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_url": "", 2024-09-16T10:14:39.434 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_lease_period": "120", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T10:14:39.435 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_thread_pool_size": "512", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_comment": "", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_createby": "", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_encoding": "", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_flag": "false", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_origin": "", 2024-09-16T10:14:39.436 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_torrent_tracker": "", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_usage_max_shards": "32", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_use_opa_authz": "false", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_counters_cache": "false", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T10:14:39.437 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_user_unique_email": "true", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_verify_ssl": "true", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_zone": "", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_zone_id": "", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_zonegroup": "", 2024-09-16T10:14:39.438 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_zonegroup_id": "", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_block_size": "4096", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T10:14:39.439 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_partition_filters": "false", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_perf": "false", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T10:14:39.440 INFO:teuthology.orchestra.run.smithi063.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "run_dir": "/var/run/ceph", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_block_create": "true", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_device_size": "53687091200", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_full_integrity_check": "false", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T10:14:39.441 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_main_device_type": "SSD", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "seastore_segment_size": "67108864", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "set_keepcaps": "false", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "setgroup": "ceph", 2024-09-16T10:14:39.442 INFO:teuthology.orchestra.run.smithi063.stdout: "setuser": "ceph", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "setuser_match_path": "", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "thp": "false", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "threadpool_default_timeout": "60", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "throttler_perf_counter": "true", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "tmp_dir": "/tmp", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "uadk_compressor_enabled": "false", 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-16T10:14:39.443 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:14:40.116 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:39 smithi063 ceph-mon[29993]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:14:40.117 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:39 smithi063 ceph-mon[39470]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:14:40.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T10:14:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:39 smithi165 ceph-mon[32696]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:14:40.400 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:40.879 INFO:teuthology.orchestra.run.smithi063.stdout:mon.a: {} 2024-09-16T10:14:40.879 INFO:teuthology.orchestra.run.smithi063.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T10:14:40.888 INFO:teuthology.orchestra.run.smithi063.stdout:mon.b: {} 2024-09-16T10:14:40.889 INFO:teuthology.orchestra.run.smithi063.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T10:14:40.897 INFO:teuthology.orchestra.run.smithi063.stdout:mon.c: {} 2024-09-16T10:14:40.897 INFO:teuthology.orchestra.run.smithi063.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T10:14:41.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:40 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:14:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:40 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:14:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2690542267' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:14:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:40 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:14:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:40 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:14:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2690542267' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:14:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:40 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:14:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:40 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:14:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2690542267' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:14:41.536 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-16T10:14:41.546 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-16T10:14:41.568 INFO:tasks.radosbench:Beginning radosbench... 2024-09-16T10:14:41.569 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T10:14:41.569 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-16T10:14:41.570 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T10:14:41.570 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:14:41.570 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T10:14:41.570 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T10:14:41.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:41.571 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T10:14:41.571 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:14:41.571 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T10:14:41.572 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:14:41.572 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T10:14:41.572 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:14:41.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:41.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:41.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:41.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:41.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:41.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:41.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:41.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:41.870 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:41.875 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:41.879 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:41.922 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[29993]: 4.16 scrub starts 2024-09-16T10:14:41.922 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[29993]: 4.16 scrub ok 2024-09-16T10:14:41.922 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[29993]: 4.e scrub starts 2024-09-16T10:14:41.922 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[29993]: 4.e scrub ok 2024-09-16T10:14:41.922 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[29993]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-16T10:14:41.922 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:14:41.922 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:14:41.946 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:41.968 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:41.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:42.070 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:42.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:42.194 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:42.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[39470]: 4.16 scrub starts 2024-09-16T10:14:42.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[39470]: 4.16 scrub ok 2024-09-16T10:14:42.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[39470]: 4.e scrub starts 2024-09-16T10:14:42.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[39470]: 4.e scrub ok 2024-09-16T10:14:42.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[39470]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-16T10:14:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:14:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:41 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:14:42.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:41 smithi165 ceph-mon[32696]: 4.16 scrub starts 2024-09-16T10:14:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:41 smithi165 ceph-mon[32696]: 4.16 scrub ok 2024-09-16T10:14:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:41 smithi165 ceph-mon[32696]: 4.e scrub starts 2024-09-16T10:14:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:41 smithi165 ceph-mon[32696]: 4.e scrub ok 2024-09-16T10:14:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:41 smithi165 ceph-mon[32696]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-16T10:14:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:41 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:14:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:41 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:14:42.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.342 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:42.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:42.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:42.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:42.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:42.581 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:14:42.504Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002446268s 2024-09-16T10:14:42.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:42.690 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:42.712 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:42.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:42.836 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:42.934 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_0' created 2024-09-16T10:14:42.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:42.964 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:42.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:43.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:43.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:43.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:43.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[29993]: 4.17 scrub starts 2024-09-16T10:14:43.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[29993]: 4.17 scrub ok 2024-09-16T10:14:43.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[29993]: 4.2 scrub starts 2024-09-16T10:14:43.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[29993]: 4.2 scrub ok 2024-09-16T10:14:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2683807611' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-16T10:14:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[39470]: 4.17 scrub starts 2024-09-16T10:14:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[39470]: 4.17 scrub ok 2024-09-16T10:14:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[39470]: 4.2 scrub starts 2024-09-16T10:14:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[39470]: 4.2 scrub ok 2024-09-16T10:14:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2683807611' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-16T10:14:43.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:43.290 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:42 smithi165 ceph-mon[32696]: 4.17 scrub starts 2024-09-16T10:14:43.290 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:42 smithi165 ceph-mon[32696]: 4.17 scrub ok 2024-09-16T10:14:43.290 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:42 smithi165 ceph-mon[32696]: 4.2 scrub starts 2024-09-16T10:14:43.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:42 smithi165 ceph-mon[32696]: 4.2 scrub ok 2024-09-16T10:14:43.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2683807611' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-16T10:14:43.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:43.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:43.437 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:43.540 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:14:43.541 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:14:43.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:14:43.557 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:14:43.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:43.573 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:14:43.573 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:14:43.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:43.589 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:14:43.589 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:14:43.606 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:14:43.606 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:14:43.623 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:14:43.623 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:14:43.640 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:14:43.640 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:14:43.657 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:14:43.657 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:14:43.685 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.709 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:43.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:43.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:43.832 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:43.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:43.947 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T10:14:43.964 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:43.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:44.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:44.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:44.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2683807611' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: osdmap e81: 8 total, 8 up, 8 in 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: pgmap v29: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/621066791' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: from='client.44711 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1274316393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2683807611' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: osdmap e81: 8 total, 8 up, 8 in 2024-09-16T10:14:44.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: pgmap v29: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:14:44.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/621066791' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:14:44.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: from='client.44711 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:14:44.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:14:44.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:14:44.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1274316393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:14:44.211 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:44.233 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:44.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2683807611' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-16T10:14:44.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: osdmap e81: 8 total, 8 up, 8 in 2024-09-16T10:14:44.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: pgmap v29: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:14:44.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/621066791' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:14:44.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: from='client.44711 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:14:44.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:14:44.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:14:44.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1274316393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:14:44.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:44.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:44.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:44.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:44.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:44.603 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:44.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:44.728 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:44.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:44.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:44.941 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:44 smithi165 systemd[1]: Starting Ceph grafana.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:14:44.953 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:44.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T10:14:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:44.978 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:44.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:45.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:45.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:45.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:45.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[29993]: 4.10 scrub starts 2024-09-16T10:14:45.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[29993]: 4.10 scrub ok 2024-09-16T10:14:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[29993]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:14:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[29993]: from='client.44711 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:14:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[29993]: osdmap e82: 8 total, 8 up, 8 in 2024-09-16T10:14:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:14:45.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:14:45.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[39470]: 4.10 scrub starts 2024-09-16T10:14:45.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[39470]: 4.10 scrub ok 2024-09-16T10:14:45.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[39470]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:14:45.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[39470]: from='client.44711 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:14:45.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[39470]: osdmap e82: 8 total, 8 up, 8 in 2024-09-16T10:14:45.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:14:45.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:44 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:14:45.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:45.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:44 smithi165 ceph-mon[32696]: 4.10 scrub starts 2024-09-16T10:14:45.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:44 smithi165 ceph-mon[32696]: 4.10 scrub ok 2024-09-16T10:14:45.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:44 smithi165 ceph-mon[32696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:14:45.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:44 smithi165 ceph-mon[32696]: from='client.44711 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:14:45.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:44 smithi165 ceph-mon[32696]: osdmap e82: 8 total, 8 up, 8 in 2024-09-16T10:14:45.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:44 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:14:45.226 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:44 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:14:45.226 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:44 smithi165 podman[59805]: 2024-09-16 10:14:44.84936229 +0000 UTC m=+0.016592752 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-16T10:14:45.226 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:44 smithi165 podman[59805]: 2024-09-16 10:14:44.971260937 +0000 UTC m=+0.138491391 container create ccf521d17e6e50bb3b84daff580f744c6f8045c225c4eca6a9a637ef2f821a47 (image=quay.io/ceph/grafana:10.4.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T10:14:45.231 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:45.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:45.333 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:45.358 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:45.358 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:45.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:45.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T10:14:45.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:45.476 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 podman[59805]: 2024-09-16 10:14:45.224252047 +0000 UTC m=+0.391482499 container init ccf521d17e6e50bb3b84daff580f744c6f8045c225c4eca6a9a637ef2f821a47 (image=quay.io/ceph/grafana:10.4.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T10:14:45.476 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 podman[59805]: 2024-09-16 10:14:45.227448943 +0000 UTC m=+0.394679404 container start ccf521d17e6e50bb3b84daff580f744c6f8045c225c4eca6a9a637ef2f821a47 (image=quay.io/ceph/grafana:10.4.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T10:14:45.477 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 bash[59805]: ccf521d17e6e50bb3b84daff580f744c6f8045c225c4eca6a9a637ef2f821a47 2024-09-16T10:14:45.477 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 systemd[1]: Started Ceph grafana.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:14:45.477 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352564655Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-16T10:14:45Z 2024-09-16T10:14:45.477 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352777096Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-16T10:14:45.477 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352785309Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-16T10:14:45.477 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352788756Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-16T10:14:45.478 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352796847Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-16T10:14:45.478 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352800019Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-16T10:14:45.478 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352802717Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-16T10:14:45.478 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352805374Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-16T10:14:45.478 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352809709Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-16T10:14:45.479 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.35281306Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-16T10:14:45.479 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352815646Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-16T10:14:45.479 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352818328Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-16T10:14:45.479 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352821206Z level=info msg=Target target=[all] 2024-09-16T10:14:45.479 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352827094Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-16T10:14:45.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352829687Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-16T10:14:45.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352832621Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-16T10:14:45.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352835416Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-16T10:14:45.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352838416Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-16T10:14:45.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=settings t=2024-09-16T10:14:45.352841059Z level=info msg="App mode production" 2024-09-16T10:14:45.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore t=2024-09-16T10:14:45.353090681Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-16T10:14:45.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore t=2024-09-16T10:14:45.353103404Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-16T10:14:45.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.353546001Z level=info msg="Starting DB migrations" 2024-09-16T10:14:45.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.354396764Z level=info msg="Executing migration" id="create migration_log table" 2024-09-16T10:14:45.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.355163785Z level=info msg="Migration successfully executed" id="create migration_log table" duration=766.05µs 2024-09-16T10:14:45.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.35567535Z level=info msg="Executing migration" id="create user table" 2024-09-16T10:14:45.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.356213999Z level=info msg="Migration successfully executed" id="create user table" duration=538.505µs 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.356624248Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.357133917Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=511.47µs 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.359234009Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.359765116Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=531.389µs 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.362002326Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.362552177Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=550.901µs 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.364606636Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.365184225Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=578.125µs 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.36727403Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.36897338Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=1.698686ms 2024-09-16T10:14:45.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.369984844Z level=info msg="Executing migration" id="create user table v2" 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.370546974Z level=info msg="Migration successfully executed" id="create user table v2" duration=562.374µs 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.37243929Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.372962361Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=523.414µs 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.37505885Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.375531777Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=472.913µs 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.37796308Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.378294039Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=331.755µs 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.380570281Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.38100618Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=435.955µs 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.383473536Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.384279785Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=808.375µs 2024-09-16T10:14:45.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.386018537Z level=info msg="Executing migration" id="Update user table charset" 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.386037364Z level=info msg="Migration successfully executed" id="Update user table charset" duration=19.794µs 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.386461514Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.387166716Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=705.187µs 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.387505375Z level=info msg="Executing migration" id="Add missing user data" 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.387650008Z level=info msg="Migration successfully executed" id="Add missing user data" duration=144.596µs 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.388351715Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.389070199Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=718.71µs 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.389426314Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.389910028Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=483.543µs 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.390300356Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.391027358Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=726.72µs 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.391372483Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-16T10:14:45.486 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.396546023Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.172724ms 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.396924068Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.397648786Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=725.945µs 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.397996611Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.398133762Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=137.183µs 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.398966244Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.399573975Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=611.445µs 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.400004624Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.400560373Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=556.469µs 2024-09-16T10:14:45.487 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.40099381Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.401465176Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=471.343µs 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.401852764Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.40232409Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=472.434µs 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.402694735Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.403167898Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=472.908µs 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.403572033Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.404046339Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=474.098µs 2024-09-16T10:14:45.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.404400839Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.404417731Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=17.116µs 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.4052807Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.405728903Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=448.077µs 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.406130357Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.406567312Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=436.926µs 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.406991709Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.407438541Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=445.13µs 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.407853414Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.408277022Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=423.554µs 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.408693766Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-16T10:14:45.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.410668543Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=1.974352ms 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.41104693Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.41156935Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=522.248µs 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.411916122Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.412376726Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=460.583µs 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.412799379Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.413263924Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=464.803µs 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.413678652Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.414148979Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=470.056µs 2024-09-16T10:14:45.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.414530663Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.41498521Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=454.337µs 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.415388014Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.415642822Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=254.489µs 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.416224061Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.416585863Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=361.375µs 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.416905348Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.41714047Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=235.009µs 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.417776393Z level=info msg="Executing migration" id="create star table" 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.418212004Z level=info msg="Migration successfully executed" id="create star table" duration=435.212µs 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.418630844Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.419108813Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=477.852µs 2024-09-16T10:14:45.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.419479285Z level=info msg="Executing migration" id="create org table v1" 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.419947162Z level=info msg="Migration successfully executed" id="create org table v1" duration=468.257µs 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.420274486Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.42073475Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=460.19µs 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.421148096Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.421573057Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=425.616µs 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.421989689Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.422473341Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=483.614µs 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.422870161Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.423344913Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=474.448µs 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.42374131Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-16T10:14:45.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.424215563Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=475.247µs 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.426547079Z level=info msg="Executing migration" id="Update org table charset" 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.426579919Z level=info msg="Migration successfully executed" id="Update org table charset" duration=33.585µs 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.4293979Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.429413332Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=15.04µs 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.432274647Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.432411219Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=135.712µs 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.432845754Z level=info msg="Executing migration" id="create dashboard table" 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.433370059Z level=info msg="Migration successfully executed" id="create dashboard table" duration=522.452µs 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.435675377Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.43625321Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=577.911µs 2024-09-16T10:14:45.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.438377711Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.438933263Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=555.161µs 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.441013102Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.441469124Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=456.892µs 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.443842744Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.444397634Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=554.681µs 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.446434637Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.446956191Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=521.433µs 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.449330025Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.452870969Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.543028ms 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.453272668Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.453826756Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=553.699µs 2024-09-16T10:14:45.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.456013329Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.456506329Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=478.262µs 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.456928363Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.457420597Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=492.086µs 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.457804205Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.458041371Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=237.094µs 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.45867947Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.459312164Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=633.129µs 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.459673738Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.45971207Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=38.631µs 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.4606957Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.461850897Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.158077ms 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.462215375Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-16T10:14:45.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.463310588Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.094592ms 2024-09-16T10:14:45.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.463673186Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-16T10:14:45.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.464723538Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.050045ms 2024-09-16T10:14:45.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.465087033Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-16T10:14:45.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.465560779Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=473.553µs 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.46594168Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.467052236Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.110099ms 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.46742817Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.467922031Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=493.629µs 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.468308847Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.468768542Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=459.243µs 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.469164696Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.469180147Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=15.902µs 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.469940447Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.46995679Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=16.979µs 2024-09-16T10:14:45.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.470367992Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.471528734Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.16053ms 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.471898441Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.473051136Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.152359ms 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.473426467Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.474576909Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.150203ms 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.474975049Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.476148618Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.173249ms 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.476498657Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.476641461Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=142.682µs 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.477352669Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.477845989Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=493.008µs 2024-09-16T10:14:45.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.478242432Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.478716679Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=474.002µs 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.479087919Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.479105111Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=16.317µs 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.479962337Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.480464946Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=502.321µs 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.480858019Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.481312983Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=455.155µs 2024-09-16T10:14:45.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.481735274Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-16T10:14:45.501 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:45.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:45.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:45.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:45.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:45.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:45.730 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.484853535Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.117735ms 2024-09-16T10:14:45.730 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.485210899Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-16T10:14:45.730 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.485657835Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=446.589µs 2024-09-16T10:14:45.730 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.486049945Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-16T10:14:45.730 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.486544921Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=494.714µs 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.486939876Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.487439686Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=499.973µs 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.487843633Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.4880595Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=216.21µs 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.488687709Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.489098123Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=410.293µs 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.489547889Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.490749049Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.201998ms 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.491111328Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.491591833Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=480.897µs 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.491950447Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-16T10:14:45.731 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.492065066Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=115.077µs 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.492814292Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.492927066Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=112.814µs 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.493811381Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.494343585Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=551.347µs 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.494688026Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.49598055Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.292114ms 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.496363703Z level=info msg="Executing migration" id="create data_source table" 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.497046636Z level=info msg="Migration successfully executed" id="create data_source table" duration=682.579µs 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.499241838Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.499736566Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=494.534µs 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.502165313Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-16T10:14:45.732 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.502675877Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=510.348µs 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.505282657Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.505877604Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=597.109µs 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.508072813Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.508550252Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=477.386µs 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.511016388Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.514410349Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.393705ms 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.514781877Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.515404905Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=622.848µs 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.517753139Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.518305644Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=552.287µs 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.520665826Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.521340715Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=676.847µs 2024-09-16T10:14:45.733 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.523537705Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.523933355Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=395.338µs 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.526349137Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.527756588Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.407086ms 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.529174205Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.530578119Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.403837ms 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.532240382Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.532293297Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=52.853µs 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.534943598Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-16T10:14:45.734 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.535072895Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=129.197µs 2024-09-16T10:14:45.735 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.537843146Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-16T10:14:45.735 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.539244688Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.401386ms 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.540621247Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.540874666Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=256.012µs 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.54335659Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.543548411Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=194.294µs 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.54632072Z level=info msg="Executing migration" id="Add uid column" 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.547689693Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.369665ms 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.548995054Z level=info msg="Executing migration" id="Update uid value" 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.549150183Z level=info msg="Migration successfully executed" id="Update uid value" duration=155.498µs 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.551630261Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.552197342Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=566.828µs 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.554227402Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-16T10:14:45.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.554743747Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=516.152µs 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.557111326Z level=info msg="Executing migration" id="create api_key table" 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.557670428Z level=info msg="Migration successfully executed" id="create api_key table" duration=558.746µs 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.559782272Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.560441639Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=661.675µs 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.562233377Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.562743629Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=510.022µs 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.565146655Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.565735683Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=590.641µs 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.567586065Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.568287663Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=704.268µs 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.568651249Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.569143277Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=492.32µs 2024-09-16T10:14:45.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.569487656Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.569985145Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=497.398µs 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.570330115Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.574063014Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.732534ms 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.57448764Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.574996993Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=509.175µs 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.575383443Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.575924724Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=541.052µs 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.576309561Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.576840907Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=530.99µs 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.577185677Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.577675466Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=489.506µs 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.578049635Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-16T10:14:45.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.578315633Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=265.82µs 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.578681488Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.579088363Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=407.344µs 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.579423228Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.579441869Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=18.784µs 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.579835794Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.581317773Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.480662ms 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.581656667Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.583178217Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.521354ms 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.583539246Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.58365193Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=113.713µs 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.584073098Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.585592795Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.51927ms 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.585985042Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-16T10:14:45.739 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.587490891Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.506565ms 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.587879541Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.588398747Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=517.718µs 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.58876021Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.589161161Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=400.951µs 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.58953298Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.590119108Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=585.907µs 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.590485062Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.591037821Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=551.693µs 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.591427313Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.592015607Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=588.151µs 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.592397631Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.592953943Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=556.138µs 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.593342627Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-16T10:14:45.740 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.593390755Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=48.192µs 2024-09-16T10:14:45.741 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.593822626Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-16T10:14:45.741 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.593841131Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=18.759µs 2024-09-16T10:14:45.741 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.59425837Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-16T10:14:45.741 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.595951965Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.693217ms 2024-09-16T10:14:45.741 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.596341126Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-16T10:14:45.741 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.597989718Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.648347ms 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.598445221Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.598494802Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=50.839µs 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.598901393Z level=info msg="Executing migration" id="create quota table v1" 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.599368655Z level=info msg="Migration successfully executed" id="create quota table v1" duration=467.2µs 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.599721561Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.600227759Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=505.888µs 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.600570523Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.600590679Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=20.371µs 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.601011542Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.601519511Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=507.695µs 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.601900105Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-16T10:14:45.743 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.602436208Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=535.988µs 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.602811619Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.604519056Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.707119ms 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.604917811Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.604939289Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=22.201µs 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.605368698Z level=info msg="Executing migration" id="create session table" 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.605966795Z level=info msg="Migration successfully executed" id="create session table" duration=598.034µs 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.606338601Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.606399899Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=61.529µs 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.609033946Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.609106096Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=71.326µs 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.609510206Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.610033658Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=523.112µs 2024-09-16T10:14:45.744 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.610440097Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.610929768Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=489.455µs 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.611284827Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.6113041Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=19.305µs 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.611715117Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.611733258Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=18.234µs 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.61215761Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.614021168Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.862947ms 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.614383022Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.616220287Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.836838ms 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.616573911Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.616644441Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=70.534µs 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.617052785Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-16T10:14:45.745 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.617121253Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=68.596µs 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.617519852Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.618044274Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=524.202µs 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.618430513Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.618449284Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=18.948µs 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.618854158Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.620789492Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.934326ms 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.621203397Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.621311714Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=109.541µs 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.621716455Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.623615542Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.897221ms 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.623990769Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.625904336Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.913206ms 2024-09-16T10:14:45.746 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.626274701Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.626321721Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=47.982µs 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.626779546Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.6274261Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=646.676µs 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.627789408Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.62835588Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=566.875µs 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.628739355Z level=info msg="Executing migration" id="create alert table v1" 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.629512488Z level=info msg="Migration successfully executed" id="create alert table v1" duration=773.4µs 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.630480537Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.631121015Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=640.299µs 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.633138198Z level=info msg="Executing migration" id="add index alert state" 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.633680361Z level=info msg="Migration successfully executed" id="add index alert state" duration=541.837µs 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.635885779Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.636432121Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=546.477µs 2024-09-16T10:14:45.747 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.636814525Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.637259225Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=445.866µs 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.637635563Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.638216799Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=580.93µs 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.638586543Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.63913452Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=546.673µs 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.639494781Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.645238955Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.743931ms 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.645616126Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.646106778Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=490.599µs 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.646480656Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.647024435Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=542.555µs 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.647416498Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-16T10:14:45.748 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.647608816Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=192.238µs 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.648258049Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.648620294Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=361.677µs 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.649109Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.649592284Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=482.973µs 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.649949512Z level=info msg="Executing migration" id="Add column is_default" 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.652183601Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.2355ms 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.652863135Z level=info msg="Executing migration" id="Add column frequency" 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.655080797Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.217422ms 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.655544472Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.657776728Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.232011ms 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.658290521Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.66037711Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.087281ms 2024-09-16T10:14:45.749 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.660752709Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.661294334Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=541.481µs 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.663563223Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.663596996Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=34.068µs 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.66647318Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.666517018Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=46.815µs 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.669170086Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.669651994Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=482.053µs 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.670045974Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.670587764Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=541.925µs 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.672900909Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.673417483Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=516.468µs 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.675888631Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.676454527Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=569.391µs 2024-09-16T10:14:45.750 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.678385258Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.678935402Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=549.903µs 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.679303993Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.681423833Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.119466ms 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.68223417Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.684518593Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.284066ms 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.685026142Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.685143433Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=117.004µs 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.688036048Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.688635875Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=603.105µs 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.689000769Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.689534108Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=533.148µs 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.691934804Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.69409809Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.163465ms 2024-09-16T10:14:45.751 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.694954272Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.69499641Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=44.068µs 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.697393139Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.697944519Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=550.002µs 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.698289615Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.698887335Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=597.431µs 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.700993484Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.70105739Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=64.004µs 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.703824198Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.704395648Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=571.852µs 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.706513923Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.707090766Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=577.114µs 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.707454007Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-16T10:14:45.752 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.70798909Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=535.294µs 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.710315828Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.710866985Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=550.822µs 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.713084871Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.713709405Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=624.786µs 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.715870851Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.716520942Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=651.393µs 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.71847985Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.718500269Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=21.649µs 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.721064997Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.723718494Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.654157ms 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.724186161Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.724784652Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=598.902µs 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.72631973Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-16T10:14:45.753 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.728839337Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.518414ms 2024-09-16T10:14:45.754 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.729314247Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-16T10:14:45.754 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.72983078Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=516.107µs 2024-09-16T10:14:45.764 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:45.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:45.844 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:45.866 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:45.866 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:45.896 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:45.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:45.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: osdmap e83: 8 total, 8 up, 8 in 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: 3.3 scrub starts 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: 3.3 scrub ok 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: pgmap v32: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: 4.f scrub starts 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: 4.f scrub ok 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:45.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:45.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: Deploying daemon node-exporter.a on smithi063 2024-09-16T10:14:45.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-mon[32696]: osdmap e84: 8 total, 8 up, 8 in 2024-09-16T10:14:45.980 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.735550976Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-16T10:14:45.980 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.736276913Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=725.332µs 2024-09-16T10:14:45.980 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.738365806Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.738969709Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=603.924µs 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.739449884Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.746993937Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=7.544275ms 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.747352897Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.747857176Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=504.153µs 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.750273123Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.750836826Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=563.39µs 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.753076268Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.753267681Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=191.294µs 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.756032696Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.756405569Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=372.817µs 2024-09-16T10:14:45.981 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.758757721Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.758877914Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=121.65µs 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.761785154Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.764277329Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.494231ms 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.764646156Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.766986458Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.339912ms 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.767475578Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.768026889Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=550.913µs 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.770187562Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.770718604Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=530.749µs 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.772952175Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.773094661Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=142.487µs 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.775690848Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-16T10:14:45.982 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.778075845Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.385728ms 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.778455299Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.778996203Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=540.535µs 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.78127453Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.781384195Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=109.688µs 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.784071212Z level=info msg="Executing migration" id="Move region to single row" 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.784393087Z level=info msg="Migration successfully executed" id="Move region to single row" duration=324.547µs 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.786855744Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.787461074Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=607.533µs 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.789455602Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.790018491Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=562.501µs 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.792189103Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-16T10:14:45.983 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.792775703Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=584.784µs 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.794747928Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.795328581Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=580.116µs 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.797552885Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.798131331Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=580.601µs 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.800262664Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.800823985Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=561.14µs 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.802908755Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.803031894Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=107.412µs 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.805461843Z level=info msg="Executing migration" id="create test_data table" 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.806004584Z level=info msg="Migration successfully executed" id="create test_data table" duration=542.396µs 2024-09-16T10:14:45.984 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.808202035Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-16T10:14:45.985 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.808748419Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=546.262µs 2024-09-16T10:14:45.985 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.810721288Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.811315849Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=594.814µs 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.811959824Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.812684229Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=723.084µs 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.813059428Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.813178491Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=119.107µs 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.813923804Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.814160327Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=236.416µs 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.81476726Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.814811544Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=44.456µs 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.81562449Z level=info msg="Executing migration" id="create team table" 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.816104924Z level=info msg="Migration successfully executed" id="create team table" duration=480.175µs 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.81648394Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-16T10:14:45.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.817108309Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=624.136µs 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.817461569Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.818013647Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=551.989µs 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.818355721Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.820929668Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.573663ms 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.821266984Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.821374152Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=107.208µs 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.822048723Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.822637134Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=587.853µs 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.822994402Z level=info msg="Executing migration" id="create team member table" 2024-09-16T10:14:45.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.823477581Z level=info msg="Migration successfully executed" id="create team member table" duration=483.024µs 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.823851697Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.824414118Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=562.432µs 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.824769786Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.825326076Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=554.792µs 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.825680995Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.826235736Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=553.871µs 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.826625113Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.830713533Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=4.086508ms 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.83112552Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.834027472Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.901745ms 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.834377352Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.837050937Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.67315ms 2024-09-16T10:14:45.988 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.837401626Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.838044751Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=642.844µs 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.83843299Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.839010312Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=574.971µs 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.839347437Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.839973424Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=625.497µs 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.84032768Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.840905974Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=576.138µs 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.841360613Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.841933889Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=573.057µs 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.842271954Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.842829469Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=557.277µs 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.843168846Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-16T10:14:45.989 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.843742617Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=573.578µs 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.84410525Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.84469353Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=588.39µs 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.84505877Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.845387Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=328.087µs 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.845868508Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.846006914Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=138.346µs 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.846693681Z level=info msg="Executing migration" id="create tag table" 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.847214954Z level=info msg="Migration successfully executed" id="create tag table" duration=521.016µs 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.847591333Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.848155507Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=563.873µs 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.848505718Z level=info msg="Executing migration" id="create login attempt table" 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.848973285Z level=info msg="Migration successfully executed" id="create login attempt table" duration=467.148µs 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.849310288Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-16T10:14:45.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.849861135Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=550.688µs 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.850206938Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.850752934Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=544.902µs 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.851101334Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.859173577Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=8.0707ms 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.859532058Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.860013845Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=480.506µs 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.860411437Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.86098128Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=571.057µs 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.863344274Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.863557833Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=213.349µs 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.866002035Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.866421412Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=418.736µs 2024-09-16T10:14:45.991 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.868856758Z level=info msg="Executing migration" id="create user auth table" 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.869333734Z level=info msg="Migration successfully executed" id="create user auth table" duration=476.749µs 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.86968136Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.870249974Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=568.262µs 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.872280398Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.872336633Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=56.514µs 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.874921271Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.87792769Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.006556ms 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.878278045Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.881179053Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.90087ms 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.881534668Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.8843Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.764998ms 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.88466532Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.887632403Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.966883ms 2024-09-16T10:14:45.992 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.887996204Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.888565386Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=569.001µs 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.888923976Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.89176387Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=2.83976ms 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.892134531Z level=info msg="Executing migration" id="create server_lock table" 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.892646173Z level=info msg="Migration successfully executed" id="create server_lock table" duration=511.434µs 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.893027083Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.8936101Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=584.437µs 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.893994637Z level=info msg="Executing migration" id="create user auth token table" 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.894547488Z level=info msg="Migration successfully executed" id="create user auth token table" duration=552.755µs 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.894906915Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.895488434Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=581.19µs 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.895845377Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.896425469Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=579.892µs 2024-09-16T10:14:45.993 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.896833881Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.897475872Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=641.804µs 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.897844748Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.900994892Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.149906ms 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.901340821Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.901938149Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=595.716µs 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.902282634Z level=info msg="Executing migration" id="create cache_data table" 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.90284261Z level=info msg="Migration successfully executed" id="create cache_data table" duration=559.726µs 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.903188106Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.903771088Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=583.011µs 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.90412244Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.904685802Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=563.206µs 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.905053207Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.905677399Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=624.002µs 2024-09-16T10:14:45.994 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.907856763Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-16T10:14:45.995 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.907976651Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=122.252µs 2024-09-16T10:14:45.995 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.908440449Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-16T10:14:45.995 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.908499994Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=59.781µs 2024-09-16T10:14:45.995 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.908865924Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-16T10:14:45.995 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.909421184Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=555.627µs 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.909782264Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.910428324Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=645.633µs 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.910814206Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.911428873Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=614.527µs 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.911816925Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.911860328Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=44.218µs 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.912297527Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.912886524Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=587.602µs 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.913219509Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.913807677Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=588.027µs 2024-09-16T10:14:45.996 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.91414904Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.914768198Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=619.434µs 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.915158051Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.915764662Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=606.315µs 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.917844955Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.921230163Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.385036ms 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.921583624Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.922233571Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=649.806µs 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.922593781Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.922650269Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=55.662µs 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.923051909Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.923611472Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=559.368µs 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.923996121Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-16T10:14:45.997 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.924602162Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=604.763µs 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.926747945Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.927368425Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=621.992µs 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.929550126Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.929627809Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=80.489µs 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.930028825Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.930649327Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=620.208µs 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.931024889Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.931615977Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=590.78µs 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.933955972Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.934591712Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=635.472µs 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.934952858Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-16T10:14:45.998 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.935542128Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=589.112µs 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.935895855Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.939371827Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.476044ms 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.939778886Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.940386384Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=607.594µs 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.940726117Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.941299041Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=572.805µs 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.941650698Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.960913756Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=19.259603ms 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.961361135Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.977772275Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=16.411189ms 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.97820789Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.978895951Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=687.721µs 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.979252165Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-16T10:14:45.999 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.979933458Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=680.415µs 2024-09-16T10:14:46.000 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.980300609Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-16T10:14:46.000 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.983605173Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.305993ms 2024-09-16T10:14:46.000 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.983992199Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-16T10:14:46.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:46.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:46.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:46.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:46.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: osdmap e83: 8 total, 8 up, 8 in 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: 3.3 scrub starts 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: 3.3 scrub ok 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: pgmap v32: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: 4.f scrub starts 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: 4.f scrub ok 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: Deploying daemon node-exporter.a on smithi063 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[39470]: osdmap e84: 8 total, 8 up, 8 in 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: osdmap e83: 8 total, 8 up, 8 in 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: 3.3 scrub starts 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: 3.3 scrub ok 2024-09-16T10:14:46.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: pgmap v32: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: 4.f scrub starts 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: 4.f scrub ok 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: Deploying daemon node-exporter.a on smithi063 2024-09-16T10:14:46.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:45 smithi063 ceph-mon[29993]: osdmap e84: 8 total, 8 up, 8 in 2024-09-16T10:14:46.229 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.988191815Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=4.198706ms 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.988606836Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.989405274Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=798.269µs 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.989818826Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.990565085Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=746.258µs 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.990957957Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.991629117Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=670.986µs 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.991981519Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.992656008Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=672.378µs 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.993010314Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-16T10:14:46.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.993049096Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=39.005µs 2024-09-16T10:14:46.231 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.99385286Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-16T10:14:46.231 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.998485538Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=4.632281ms 2024-09-16T10:14:46.231 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:45.998837442Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-16T10:14:46.231 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.002472926Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.634097ms 2024-09-16T10:14:46.231 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.002914091Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-16T10:14:46.231 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.006412709Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.497982ms 2024-09-16T10:14:46.232 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.006840355Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-16T10:14:46.232 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.007445261Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=605.056µs 2024-09-16T10:14:46.232 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.007809293Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-16T10:14:46.232 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.008430413Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=621.355µs 2024-09-16T10:14:46.232 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.008768502Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-16T10:14:46.232 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.012213564Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.444236ms 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.012611571Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.016419099Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.804571ms 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.017044779Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.01772736Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=682.275µs 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.018097272Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.021545878Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.445186ms 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.021951828Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.025373641Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.419664ms 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.025741736Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-16T10:14:46.233 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.025780431Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=39.008µs 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.026654777Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.027394718Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=739.846µs 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.027759759Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.028429021Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=668.986µs 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.028783095Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.029473289Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=689.88µs 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.029847465Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.029888965Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=42.981µs 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.030731513Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.034366735Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.634584ms 2024-09-16T10:14:46.234 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.034730487Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.038307562Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.576611ms 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.038679513Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.042250243Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.570293ms 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.042614911Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.046170708Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.553213ms 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.0465384Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.050053678Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.514402ms 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.050439761Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.05047803Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=38.788µs 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.053305482Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.053851484Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=545.788µs 2024-09-16T10:14:46.235 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.054226642Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.058151837Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.92489ms 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.058514466Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.058556727Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=42.477µs 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.061471429Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.065176228Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.704463ms 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.065548456Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.066180065Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=631.373µs 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.066546611Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.070222044Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.674865ms 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.070604214Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.071142521Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=538.072µs 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.073492617Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-16T10:14:46.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.074163077Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=670.326µs 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.07632471Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.080098244Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.772991ms 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.08047968Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.081005811Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=525.688µs 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.083290027Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.083965362Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=674.957µs 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.086230512Z level=info msg="Executing migration" id="create alert_image table" 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.086755223Z level=info msg="Migration successfully executed" id="create alert_image table" duration=524.496µs 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.089039311Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.089661527Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=622.026µs 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.090053079Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.090092272Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=39.458µs 2024-09-16T10:14:46.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.092861422Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.093626552Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=765.056µs 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.09567586Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.096291304Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=615.469µs 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.098608721Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.09887082Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.099259333Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.099541008Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=281.667µs 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.101914213Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-16T10:14:46.238 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.102532825Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=618.492µs 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.104579721Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.108437305Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.856979ms 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.108841333Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.109584346Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=742.727µs 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.109985554Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.110675652Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=689.978µs 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.11259093Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.113173951Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=582.936µs 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.115349223Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.116030352Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=680.701µs 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.117936698Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.118543411Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=606.487µs 2024-09-16T10:14:46.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.118928986Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.118950941Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=25.154µs 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.121475447Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.121521625Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=47.085µs 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.124198589Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.12439659Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=200.208µs 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.126969374Z level=info msg="Executing migration" id="create data_keys table" 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.127669265Z level=info msg="Migration successfully executed" id="create data_keys table" duration=700.512µs 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.12960569Z level=info msg="Executing migration" id="create secrets table" 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.130263495Z level=info msg="Migration successfully executed" id="create secrets table" duration=657.871µs 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.130629456Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.152330985Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=21.695591ms 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.152808464Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-16T10:14:46.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.156951316Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.143407ms 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.157307566Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.157404757Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=97.37µs 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.157789564Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.179432304Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=21.642017ms 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.179866421Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.201626605Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=21.757596ms 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.202132064Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.202732335Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=602.888µs 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.203109694Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.203808871Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=698.698µs 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.204158057Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-16T10:14:46.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.204300968Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=143.366µs 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.205042319Z level=info msg="Executing migration" id="create permission table" 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.205641117Z level=info msg="Migration successfully executed" id="create permission table" duration=599.026µs 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.206009424Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.206635813Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=626.549µs 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.206982303Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.20763968Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=657.631µs 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.207998439Z level=info msg="Executing migration" id="create role table" 2024-09-16T10:14:46.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.20857096Z level=info msg="Migration successfully executed" id="create role table" duration=572.351µs 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.208916403Z level=info msg="Executing migration" id="add column display_name" 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.213359568Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.442742ms 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.213743967Z level=info msg="Executing migration" id="add column group_name" 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.218029322Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.284985ms 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.218386061Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.219069468Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=683.332µs 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.219438882Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.220138284Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=699.062µs 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.220505363Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.221181248Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=675.641µs 2024-09-16T10:14:46.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.221542578Z level=info msg="Executing migration" id="create team role table" 2024-09-16T10:14:46.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.222110776Z level=info msg="Migration successfully executed" id="create team role table" duration=567.966µs 2024-09-16T10:14:46.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.2224543Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-16T10:14:46.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.223158181Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=703.653µs 2024-09-16T10:14:46.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.223528246Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-16T10:14:46.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.224248219Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=719.686µs 2024-09-16T10:14:46.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.224615316Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-16T10:14:46.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.225261588Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=647.641µs 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.225600925Z level=info msg="Executing migration" id="create user role table" 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.226162891Z level=info msg="Migration successfully executed" id="create user role table" duration=558.417µs 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.226521905Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.227181604Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=659.62µs 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.227517788Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.228172892Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=654.946µs 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.228509632Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.229417702Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=908.003µs 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.229868491Z level=info msg="Executing migration" id="create builtin role table" 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.230536941Z level=info msg="Migration successfully executed" id="create builtin role table" duration=668.615µs 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.230884761Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-16T10:14:46.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.23183505Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=949.366µs 2024-09-16T10:14:46.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.232274203Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-16T10:14:46.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:46.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:46.297 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:14:46.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:46.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:46.414 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:46.439 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:14:46.439 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:14:46.492 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:46.503 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.233248942Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=974.297µs 2024-09-16T10:14:46.503 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.233687505Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.242205306Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=8.51421ms 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.242790211Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.243844416Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=1.054344ms 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.244279597Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.244950928Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=670.968µs 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.245288575Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.245936846Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=644.698µs 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.24803821Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.248684631Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=647.932µs 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.250780388Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-16T10:14:46.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.251312541Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=532.072µs 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.253764382Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.254497634Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=733.295µs 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.254882404Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.259689673Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.805991ms 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.260144657Z level=info msg="Executing migration" id="permission kind migration" 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.264869542Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.724144ms 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.265325072Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.270065467Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.740578ms 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.270522495Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.276885936Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=6.359626ms 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.277620332Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.278718396Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=1.096562ms 2024-09-16T10:14:46.505 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.280405139Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.281486209Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=1.080793ms 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.28329869Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.284292516Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=993.713µs 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.285960598Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.286838476Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=877.603µs 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.28726497Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.288275281Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=1.010013ms 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.289962033Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-16T10:14:46.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.290015557Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=53.636µs 2024-09-16T10:14:46.507 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.292725517Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.292767846Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=43.137µs 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.295663941Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.296057422Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=391.534µs 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.296551395Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.297032971Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=482.2µs 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.299513212Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.300118875Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=605.601µs 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.302365849Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.302542091Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=184.165µs 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.304896188Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.305251827Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=355.577µs 2024-09-16T10:14:46.508 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.305669587Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.306250325Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=580.537µs 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.30851157Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.309243282Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=729.717µs 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.311258557Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.316017931Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.758637ms 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.316388417Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.316435044Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=46.372µs 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.31689715Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.317626856Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=729.19µs 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.31948741Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.320161481Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=673.932µs 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.322150503Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-16T10:14:46.509 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.322828579Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=677.945µs 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.324774596Z level=info msg="Executing migration" id="add correlation config column" 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.329832958Z level=info msg="Migration successfully executed" id="add correlation config column" duration=5.055832ms 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.330304917Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.331010123Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=705.294µs 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.333016374Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.333679858Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=663.501µs 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.335790022Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.349398361Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=13.586419ms 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.349810949Z level=info msg="Executing migration" id="create correlation v2" 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.35056573Z level=info msg="Migration successfully executed" id="create correlation v2" duration=754.588µs 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.350965931Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.35162554Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=659.484µs 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.353824633Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-16T10:14:46.510 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.354672443Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=847.454µs 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.356466535Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.35715571Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=688.952µs 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.357523179Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.357708891Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=185.659µs 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.358122352Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.358671686Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=549.279µs 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.359052011Z level=info msg="Executing migration" id="add provisioning column" 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.363922903Z level=info msg="Migration successfully executed" id="add provisioning column" duration=4.87056ms 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.364292517Z level=info msg="Executing migration" id="create entity_events table" 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.364876739Z level=info msg="Migration successfully executed" id="create entity_events table" duration=583.974µs 2024-09-16T10:14:46.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.365248169Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.365943747Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=695.433µs 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.366420738Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.36669924Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.367250968Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.367526928Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.368071706Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.368637953Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=566.114µs 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.369044383Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.369716918Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=672.161µs 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.370104081Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.370945137Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=840.774µs 2024-09-16T10:14:46.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.371335974Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.372070983Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=734.73µs 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.37244963Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.373144877Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=695.278µs 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.373513064Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.374195656Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=684.727µs 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.37456466Z level=info msg="Executing migration" id="Drop public config table" 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.375142584Z level=info msg="Migration successfully executed" id="Drop public config table" duration=577.731µs 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.375523567Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.376280595Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=756.589µs 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.376656319Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.377354317Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=696.111µs 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.377798375Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-16T10:14:46.513 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.378904147Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=1.110991ms 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.379373731Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.380432497Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=1.05997ms 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.38091894Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.408599887Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=27.678447ms 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.409195376Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.414365545Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.169564ms 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.414751648Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.419609123Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.855598ms 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.420073319Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.420225385Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=152.176µs 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.420894645Z level=info msg="Executing migration" id="add share column" 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.425839298Z level=info msg="Migration successfully executed" id="add share column" duration=4.941363ms 2024-09-16T10:14:46.514 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.426344389Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.426494301Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=149.906µs 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.427184839Z level=info msg="Executing migration" id="create file table" 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.427822264Z level=info msg="Migration successfully executed" id="create file table" duration=637.217µs 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.428179196Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.42884606Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=668.287µs 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.429192361Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.429843251Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=650.719µs 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.430185709Z level=info msg="Executing migration" id="create file_meta table" 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.430676201Z level=info msg="Migration successfully executed" id="create file_meta table" duration=490.131µs 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.43105437Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.431711469Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=656.795µs 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.432089592Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-16T10:14:46.515 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.432128054Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=38.79µs 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.432931654Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.432975828Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=43.999µs 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.43380863Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.434150703Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=341.227µs 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.43464483Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.434778236Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=133.231µs 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.435167547Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.43603998Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=872.327µs 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.436656697Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.441853896Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=5.194542ms 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.442365899Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.442495052Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=129.552µs 2024-09-16T10:14:46.516 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.445230963Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.446061028Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=832.576µs 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.446454816Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.446714953Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=260.225µs 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.449289014Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.449438458Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=149.178µs 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.452084232Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.452385292Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=301.327µs 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.454992133Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.459955719Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=4.963195ms 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.460300459Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.465221626Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=4.920972ms 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.465573236Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-16T10:14:46.517 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.466272527Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=697.591µs 2024-09-16T10:14:46.518 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.46839398Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-16T10:14:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.518 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:46.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:46.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:46.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:46.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:46.759 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.526390412Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=57.992848ms 2024-09-16T10:14:46.759 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.527084395Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.527866224Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=781.584µs 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.529756881Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.530501945Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=744.727µs 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.532545898Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.549457869Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=16.909992ms 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.549968877Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.55515465Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.185412ms 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.555544414Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-16T10:14:46.760 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.555761335Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=216.762µs 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.556179679Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.556308651Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=128.865µs 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.558990104Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.559140736Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=150.778µs 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.561887737Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.562016689Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=128.957µs 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.564842891Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.564982306Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=139.477µs 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.565390533Z level=info msg="Executing migration" id="create folder table" 2024-09-16T10:14:46.761 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.56602218Z level=info msg="Migration successfully executed" id="create folder table" duration=631.272µs 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.568105121Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.568973238Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=865.991µs 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.570767499Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.571528313Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=761.112µs 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.573566007Z level=info msg="Executing migration" id="Update folder title length" 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.573585405Z level=info msg="Migration successfully executed" id="Update folder title length" duration=19.662µs 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.57399802Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.574723965Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=725.428µs 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.576632835Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.57732992Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=696.959µs 2024-09-16T10:14:46.762 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.579392271Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.580160329Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=767.591µs 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.582129924Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.58243509Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=305.075µs 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.582766124Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.582957217Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=190.939µs 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.585561501Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.586282518Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=720.946µs 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.58834453Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.589100215Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=755.935µs 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.59095955Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-16T10:14:46.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.591641763Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=682.1µs 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.592166716Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.592918351Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=751.205µs 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.594700696Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.595456296Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=754.585µs 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.597321023Z level=info msg="Executing migration" id="create anon_device table" 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.597917234Z level=info msg="Migration successfully executed" id="create anon_device table" duration=596.399µs 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.599940197Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.600735478Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=795.55µs 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.601134139Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.601860012Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=725.576µs 2024-09-16T10:14:46.764 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.60223639Z level=info msg="Executing migration" id="create signing_key table" 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.602885109Z level=info msg="Migration successfully executed" id="create signing_key table" duration=648.521µs 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.603248635Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.60395115Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=702.218µs 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.60431234Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.605025372Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=713.104µs 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.605365242Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.605547402Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=184.755µs 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.606258201Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.612099677Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.824036ms 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.61250354Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-16T10:14:46.765 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.613007288Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=505.023µs 2024-09-16T10:14:46.766 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.613405215Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-16T10:14:46.766 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.614134634Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=729.776µs 2024-09-16T10:14:46.766 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.614497831Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-16T10:14:46.766 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.615354006Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=855.939µs 2024-09-16T10:14:46.766 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.615711259Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.616457551Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=748.728µs 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.616821126Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.61755783Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=736.792µs 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.617906003Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.618609106Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=702.869µs 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.618975149Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.61963825Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=662.979µs 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.619997736Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-16T10:14:46.767 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.620500979Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=503.352µs 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.620870505Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.621042586Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=172.025µs 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.621476355Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.621523323Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=47.753µs 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.621973645Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.627230737Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.256827ms 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.627587533Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.632924167Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.336326ms 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.633260454Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.63347199Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=211.514µs 2024-09-16T10:14:46.768 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=migrator t=2024-09-16T10:14:46.633834546Z level=info msg="migrations completed" performed=547 skipped=0 duration=1.279469794s 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore t=2024-09-16T10:14:46.634687105Z level=info msg="Created default organization" 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=secrets t=2024-09-16T10:14:46.635199249Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=plugin.store t=2024-09-16T10:14:46.653718096Z level=info msg="Loading plugins..." 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=local.finder t=2024-09-16T10:14:46.696424781Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=plugin.store t=2024-09-16T10:14:46.696460451Z level=info msg="Plugins loaded" count=55 duration=42.743086ms 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=query_data t=2024-09-16T10:14:46.698639733Z level=info msg="Query Service initialization" 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=live.push_http t=2024-09-16T10:14:46.70133664Z level=info msg="Live Push Gateway initialization" 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.migration t=2024-09-16T10:14:46.702345998Z level=info msg=Starting 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.migration t=2024-09-16T10:14:46.702617952Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-16T10:14:46.769 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.migration orgID=1 t=2024-09-16T10:14:46.702944709Z level=info msg="Migrating alerts for organisation" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.migration orgID=1 t=2024-09-16T10:14:46.703386687Z level=info msg="Alerts found to migrate" alerts=0 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.migration t=2024-09-16T10:14:46.704567291Z level=info msg="Completed alerting migration" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.state.manager t=2024-09-16T10:14:46.715537449Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=infra.usagestats.collector t=2024-09-16T10:14:46.717005472Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=provisioning.datasources t=2024-09-16T10:14:46.718718935Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=provisioning.datasources t=2024-09-16T10:14:46.731289994Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=provisioning.alerting t=2024-09-16T10:14:46.744362482Z level=info msg="starting to provision alerting" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=provisioning.alerting t=2024-09-16T10:14:46.744528614Z level=info msg="finished to provision alerting" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=grafanaStorageLogger t=2024-09-16T10:14:46.744659563Z level=info msg="Storage starting" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.state.manager t=2024-09-16T10:14:46.745075647Z level=info msg="Warming state cache for startup" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.multiorg.alertmanager t=2024-09-16T10:14:46.747850885Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-16T10:14:46.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=http.server t=2024-09-16T10:14:46.748309111Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-16T10:14:46.771 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=http.server t=2024-09-16T10:14:46.748592787Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-16T10:14:46.771 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:14:46.755246799Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-16T10:14:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:46.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:46.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:46.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:46.958 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:14:46.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:47.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:47.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-mon[32696]: 5.10 scrub starts 2024-09-16T10:14:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-mon[32696]: 5.10 scrub ok 2024-09-16T10:14:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-mon[32696]: 6.9 scrub starts 2024-09-16T10:14:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-mon[32696]: 6.9 scrub ok 2024-09-16T10:14:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3584830359' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:14:47.052 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.state.manager t=2024-09-16T10:14:46.76150271Z level=info msg="State cache has been initialized" states=0 duration=16.425426ms 2024-09-16T10:14:47.052 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ngalert.scheduler t=2024-09-16T10:14:46.761544951Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-16T10:14:47.052 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ticker t=2024-09-16T10:14:46.761592562Z level=info msg=starting first_tick=2024-09-16T10:14:50Z 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=provisioning.dashboard t=2024-09-16T10:14:46.762592543Z level=info msg="starting to provision dashboards" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:14:46.769355506Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:14:46.779613541Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:14:46.793214305Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:14:46.803666476Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=plugins.update.checker t=2024-09-16T10:14:46.835461807Z level=info msg="Update check succeeded" duration=87.201125ms 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=grafana.update.checker t=2024-09-16T10:14:46.838634035Z level=info msg="Update check succeeded" duration=91.680708ms 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:14:46.84739335Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=provisioning.dashboard t=2024-09-16T10:14:46.898973243Z level=info msg="finished to provision dashboards" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=grafana-apiserver t=2024-09-16T10:14:46.922626419Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-16T10:14:47.053 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:14:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=grafana-apiserver t=2024-09-16T10:14:46.923046476Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-16T10:14:47.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:47.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:47.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:14:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[29993]: 5.10 scrub starts 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[29993]: 5.10 scrub ok 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[29993]: 6.9 scrub starts 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[29993]: 6.9 scrub ok 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3584830359' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[39470]: 5.10 scrub starts 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[39470]: 5.10 scrub ok 2024-09-16T10:14:47.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[39470]: 6.9 scrub starts 2024-09-16T10:14:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[39470]: 6.9 scrub ok 2024-09-16T10:14:47.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3584830359' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:14:47.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:47.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:47.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:47.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:47.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:47.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:47.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:47.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:47.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:47.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:14:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:47.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:47.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:47.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:47.967 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:14:47.967 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3584830359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:14:47.967 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[29993]: osdmap e85: 8 total, 8 up, 8 in 2024-09-16T10:14:47.967 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[29993]: pgmap v35: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1686769715' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[29993]: from='client.44729 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3584830359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[39470]: osdmap e85: 8 total, 8 up, 8 in 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[39470]: pgmap v35: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1686769715' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[39470]: from='client.44729 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:14:47.968 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:47 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:47.969 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T10:14:47.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:48.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:48.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:48.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:48.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:48.256 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:48.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:48.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:14:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:14:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:47 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:14:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3584830359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:14:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:47 smithi165 ceph-mon[32696]: osdmap e85: 8 total, 8 up, 8 in 2024-09-16T10:14:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:47 smithi165 ceph-mon[32696]: pgmap v35: 148 pgs: 16 creating+peering, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-16T10:14:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1686769715' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:14:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:47 smithi165 ceph-mon[32696]: from='client.44729 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:14:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:47 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:48.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:48.380 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:48.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:48.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:48.606 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:48.629 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:48.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.754 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:48.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:48.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:48.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:48.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:48.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:49.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:49.104 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:49.127 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:49.191 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:49.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:48 smithi063 ceph-mon[29993]: from='client.44729 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:14:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:48 smithi063 ceph-mon[29993]: osdmap e86: 8 total, 8 up, 8 in 2024-09-16T10:14:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:48 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:48 smithi063 ceph-mon[39470]: from='client.44729 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:14:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:48 smithi063 ceph-mon[39470]: osdmap e86: 8 total, 8 up, 8 in 2024-09-16T10:14:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:48 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:49.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.253 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:49.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:49.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:48 smithi165 ceph-mon[32696]: from='client.44729 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:14:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:48 smithi165 ceph-mon[32696]: osdmap e86: 8 total, 8 up, 8 in 2024-09-16T10:14:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:48 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:49.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:49.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:49.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:49.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:49.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:49.683 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:14:49.683 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:14:49.702 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:14:49.709 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:49.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:14:49.726 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:14:49.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:49.743 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:14:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:14:49.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:49.774 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:14:49.774 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:14:49.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:14:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:14:49.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:14:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:14:49.830 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:14:49.830 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:14:49.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:49.881 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:49.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:49.974 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T10:14:49.982 DEBUG:teuthology.orchestra.run.smithi063:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-16T10:14:49.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:49.985 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-16T10:14:49.985 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-16T10:14:50.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:50.079 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:hints = 1 2024-09-16T10:14:50.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-16T10:14:50.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object prefix: benchmark_data_smithi063_73086 2024-09-16T10:14:50.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:14:50.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:14:50.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[39470]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:50.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/560613285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:14:50.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[39470]: from='client.44735 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:14:50.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2785384728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:14:50.099 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:50 smithi063 bash[72954]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-16T10:14:50.099 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[29993]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:50.100 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/560613285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:14:50.100 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[29993]: from='client.44735 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:14:50.100 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:49 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2785384728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:14:50.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:50.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:50.244 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:50.287 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:49 smithi165 ceph-mon[32696]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:49 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/560613285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:14:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:49 smithi165 ceph-mon[32696]: from='client.44735 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:14:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:49 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2785384728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:14:50.389 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:50.428 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:50.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:50.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:50.570 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:50.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:14:50.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:50.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:50.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:50.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:50.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:50.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:50.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:51.040 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:14:51.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:14:51.041 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:14:51.041 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:14:51.041 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:14:51.060 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:51.079 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 1 16 3015 2999 187.427 187.438 0.00440095 0.00531785 2024-09-16T10:14:51.107 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:51.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:51.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:51.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:51.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:50 smithi165 ceph-mon[32696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:14:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:50 smithi165 ceph-mon[32696]: from='client.44735 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:14:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:50 smithi165 ceph-mon[32696]: osdmap e87: 8 total, 8 up, 8 in 2024-09-16T10:14:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:50 smithi165 ceph-mon[32696]: osdmap e88: 8 total, 8 up, 8 in 2024-09-16T10:14:51.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:51.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:51.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[29993]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:14:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[29993]: from='client.44735 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:14:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[29993]: osdmap e87: 8 total, 8 up, 8 in 2024-09-16T10:14:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[29993]: osdmap e88: 8 total, 8 up, 8 in 2024-09-16T10:14:51.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[39470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:14:51.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[39470]: from='client.44735 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:14:51.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[39470]: osdmap e87: 8 total, 8 up, 8 in 2024-09-16T10:14:51.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:50 smithi063 ceph-mon[39470]: osdmap e88: 8 total, 8 up, 8 in 2024-09-16T10:14:51.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:51.514 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:51.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:51.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:51.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:51.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:51.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:51.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:51.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:51.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:51.941 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:51.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:52.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:52.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:52.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:52.081 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 2 15 6134 6119 191.205 195 0.00451339 0.00521742 2024-09-16T10:14:52.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:52.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:51 smithi063 ceph-mon[29993]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:52.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:51 smithi063 ceph-mon[39470]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:52.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:52.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:52.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:51 smithi165 ceph-mon[32696]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:14:52.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:52.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:52.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:52.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:52.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:52.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:52.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:52.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:52.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:52.702 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:52.744 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:52.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:52.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:52.893 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:52.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:52.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:53.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:53.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 3 16 8906 8890 185.194 173.188 0.00436979 0.00539117 2024-09-16T10:14:53.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:53.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:53.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:53.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:53.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:52 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:14:53.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:52 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:14:53.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:52 smithi165 ceph-mon[32696]: osdmap e89: 8 total, 8 up, 8 in 2024-09-16T10:14:53.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:52 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:53.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:52 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:53.344 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:53 smithi063 bash[72954]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[29993]: osdmap e89: 8 total, 8 up, 8 in 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:14:53.345 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[39470]: osdmap e89: 8 total, 8 up, 8 in 2024-09-16T10:14:53.346 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:53.346 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:52 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:14:53.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:53.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:53.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:53.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:53.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:53.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:53.701 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:53 smithi063 bash[72954]: Writing manifest to image destination 2024-09-16T10:14:53.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:53.791 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T10:14:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.828 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:53.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:53.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:53.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:53.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:54.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:54.079 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 4 16 11396 11380 177.799 155.625 0.00662455 0.00561305 2024-09-16T10:14:54.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:53 smithi063 ceph-mon[29993]: pgmap v42: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-16T10:14:54.088 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:53 smithi063 ceph-mon[39470]: pgmap v42: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-16T10:14:54.089 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:53 smithi063 podman[72954]: 2024-09-16 10:14:53.795498858 +0000 UTC m=+4.315993947 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-16T10:14:54.089 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:53 smithi063 podman[72954]: 2024-09-16 10:14:53.950940749 +0000 UTC m=+4.471435818 container create e72b9c1b21673229e09f2c498590216431c0c9968da61f9906ed70b3cdee833c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:54.107 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:54.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:54.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:54.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:54.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:54.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:53 smithi165 ceph-mon[32696]: pgmap v42: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-16T10:14:54.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:54.376 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:54.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:54.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:14:54.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:54.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:54.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:54.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:54.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:54.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:54.660 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 podman[72954]: 2024-09-16 10:14:54.541024804 +0000 UTC m=+5.061519881 container init e72b9c1b21673229e09f2c498590216431c0c9968da61f9906ed70b3cdee833c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:54.660 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 podman[72954]: 2024-09-16 10:14:54.547159156 +0000 UTC m=+5.067654229 container start e72b9c1b21673229e09f2c498590216431c0c9968da61f9906ed70b3cdee833c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T10:14:54.660 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.554Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-16T10:14:54.660 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.554Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-16T10:14:54.660 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.555Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-16T10:14:54.660 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.555Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-16T10:14:54.660 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.555Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.555Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=arp 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-16T10:14:54.661 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-16T10:14:54.662 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-16T10:14:54.662 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-16T10:14:54.662 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=edac 2024-09-16T10:14:54.662 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-16T10:14:54.662 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-16T10:14:54.662 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-16T10:14:54.662 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.556Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-16T10:14:54.663 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=os 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=stat 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=time 2024-09-16T10:14:54.664 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-16T10:14:54.665 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=uname 2024-09-16T10:14:54.665 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-16T10:14:54.665 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-16T10:14:54.665 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-16T10:14:54.665 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-16T10:14:54.665 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a[73460]: ts=2024-09-16T10:14:54.557Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-16T10:14:54.666 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 bash[72954]: e72b9c1b21673229e09f2c498590216431c0c9968da61f9906ed70b3cdee833c 2024-09-16T10:14:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:54.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:54.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:54.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:54.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:54.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:54.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:54.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:54.921 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:14:54 smithi063 systemd[1]: Started Ceph node-exporter.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:14:54.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:55.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:55.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:55.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 5 16 13638 13622 170.261 140.125 0.00453842 0.0058666 2024-09-16T10:14:55.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:55.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:55.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:55.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:55.303 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:55.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:55.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:55.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:55.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:55.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:55.592 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:55.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:55.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:55.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:55.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:55.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:55.871 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:55.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:55.875 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:14:55.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:55.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:55.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:55.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:55.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:55 smithi165 ceph-mon[32696]: Deploying daemon node-exporter.b on smithi165 2024-09-16T10:14:55.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:55 smithi165 ceph-mon[32696]: pgmap v43: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-16T10:14:55.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:55 smithi165 ceph-mon[32696]: osdmap e90: 8 total, 8 up, 8 in 2024-09-16T10:14:56.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:56.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 6 16 15961 15945 166.08 145.188 0.00494567 0.0060142 2024-09-16T10:14:56.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:56.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:56.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:56.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:56.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[39470]: Deploying daemon node-exporter.b on smithi165 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[39470]: pgmap v43: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[39470]: osdmap e90: 8 total, 8 up, 8 in 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[29993]: Deploying daemon node-exporter.b on smithi165 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[29993]: pgmap v43: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-16T10:14:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:55 smithi063 ceph-mon[29993]: osdmap e90: 8 total, 8 up, 8 in 2024-09-16T10:14:56.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:56.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:56.269 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:55 smithi165 systemd[1]: Starting Ceph node-exporter.b for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:14:56.281 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:56.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:56.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:56.502 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:14:56.543 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:14:56.551 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:56 smithi165 bash[60885]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-16T10:14:56.646 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.696 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:56.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:56.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:56.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:56.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:56.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:56.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:57.042 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:56 smithi165 bash[60885]: Getting image source signatures 2024-09-16T10:14:57.042 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:56 smithi165 bash[60885]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-16T10:14:57.043 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:56 smithi165 bash[60885]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-16T10:14:57.043 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:56 smithi165 bash[60885]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-16T10:14:57.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:57.065 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:14:57.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 7 16 18149 18133 161.888 136.75 0.0114663 0.00616718 2024-09-16T10:14:57.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:57.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:57.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:57.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:57.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:57 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1603710805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:14:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:57 smithi165 ceph-mon[32696]: from='client.24772 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:14:57.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:57.368 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:57.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:57.389 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:57 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1603710805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:14:57.389 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:57 smithi063 ceph-mon[39470]: from='client.24772 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:14:57.390 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:14:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:14:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:14:57.390 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:57 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1603710805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:14:57.392 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:57 smithi063 ceph-mon[29993]: from='client.24772 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:14:57.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:57.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:57.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:57.603 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:57.629 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:57.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:57.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:14:57.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:57.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:14:57.862 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:14:57.903 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:57.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:14:58.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:58.044 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:58.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 8 16 20449 20433 159.619 143.75 0.00428945 0.00625881 2024-09-16T10:14:58.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:58.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:58.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:58.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:58 smithi165 ceph-mon[32696]: pgmap v45: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 812 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:14:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:58 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:14:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:58 smithi165 ceph-mon[32696]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:14:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:58 smithi165 ceph-mon[32696]: osdmap e91: 8 total, 8 up, 8 in 2024-09-16T10:14:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:58 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:58.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:14:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:14:58.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:58.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[39470]: pgmap v45: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 812 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:14:58.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:14:58.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[39470]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:14:58.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[39470]: osdmap e91: 8 total, 8 up, 8 in 2024-09-16T10:14:58.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:58.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[29993]: pgmap v45: 148 pgs: 148 active+clean; 545 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 812 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:14:58.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:14:58.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[29993]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:14:58.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[29993]: osdmap e91: 8 total, 8 up, 8 in 2024-09-16T10:14:58.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:58 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:14:58.374 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:58.374 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:14:58.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:58.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:58.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:58.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:58.641 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:58 smithi165 bash[60885]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-16T10:14:58.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:58.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:58.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:58.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:58.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:58.953 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:58 smithi165 bash[60885]: Writing manifest to image destination 2024-09-16T10:14:58.953 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:58 smithi165 podman[60885]: 2024-09-16 10:14:58.907118796 +0000 UTC m=+2.787940692 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-16T10:14:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:58.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:58.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:59.057 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T10:14:59.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:59.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 9 16 22619 22603 156.952 135.625 0.00825128 0.00636398 2024-09-16T10:14:59.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:59.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:59.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:59.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:14:59.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:59.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1377046251' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:14:59.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:14:59 smithi063 ceph-mon[29993]: from='client.24778 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:14:59.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:59.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1377046251' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:14:59.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:14:59 smithi063 ceph-mon[39470]: from='client.24778 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:14:59.231 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:59.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:14:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:14:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1377046251' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:14:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:14:59 smithi165 ceph-mon[32696]: from='client.24778 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:14:59.302 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:14:59 smithi165 podman[60885]: 2024-09-16 10:14:59.177199392 +0000 UTC m=+3.058021273 container create 640aeecfab3246c99db2bff4b960e0e7a66f96fb50580dd969f8c9ea6d5b669d (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T10:14:59.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:14:59.335 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:14:59.335 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:14:59.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:59.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:14:59.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:14:59.507 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:59.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:14:59.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:14:59.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:14:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:59.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:14:59.743 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:59.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:14:59.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:14:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:14:59.930 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:14:59.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:00.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:00.062 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:00.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:00.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 10 16 24617 24601 153.743 124.875 0.0015923 0.00648577 2024-09-16T10:15:00.163 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:00.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[29993]: pgmap v47: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 180 MiB/s wr, 2.88k op/s 2024-09-16T10:15:00.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[29993]: from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:15:00.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[29993]: osdmap e92: 8 total, 8 up, 8 in 2024-09-16T10:15:00.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1855717842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:00.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[39470]: pgmap v47: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 180 MiB/s wr, 2.88k op/s 2024-09-16T10:15:00.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[39470]: from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:15:00.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[39470]: osdmap e92: 8 total, 8 up, 8 in 2024-09-16T10:15:00.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1855717842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:00.203 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:00.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:00.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:15:00.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:00.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:00.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:00.344 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:00 smithi165 ceph-mon[32696]: pgmap v47: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 180 MiB/s wr, 2.88k op/s 2024-09-16T10:15:00.345 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:00 smithi165 ceph-mon[32696]: from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:15:00.345 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:00 smithi165 ceph-mon[32696]: osdmap e92: 8 total, 8 up, 8 in 2024-09-16T10:15:00.345 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1855717842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:00.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:00.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:00.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:00.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:00.586 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:00.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:00.688 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:00.734 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:00.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:00.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:00.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:00.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:00.968 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:01.008 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:01.044 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:15:01.044 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:15:01.044 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:15:01.044 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:15:01.044 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:15:01.045 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:15:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:15:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:15:01.046 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:15:01.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 11 16 26490 26474 150.407 117.062 0.00785313 0.0066422 2024-09-16T10:15:01.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.153 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:01.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:01.255 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:01.295 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:01.397 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.439 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:01.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:01.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:01.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:01.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:01.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:01.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:01.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:01.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-mon[32696]: pgmap v49: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 podman[60885]: 2024-09-16 10:15:01.598903238 +0000 UTC m=+5.479725126 container init 640aeecfab3246c99db2bff4b960e0e7a66f96fb50580dd969f8c9ea6d5b669d (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 podman[60885]: 2024-09-16 10:15:01.603362078 +0000 UTC m=+5.484183968 container start 640aeecfab3246c99db2bff4b960e0e7a66f96fb50580dd969f8c9ea6d5b669d (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.608Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.609Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.610Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.610Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.610Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.610Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-16T10:15:01.881 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.610Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=arp 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-16T10:15:01.882 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=edac 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-16T10:15:01.883 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=os 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-16T10:15:01.884 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=stat 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=time 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=uname 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.611Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.612Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-16T10:15:01.885 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b[61338]: ts=2024-09-16T10:15:01.612Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-16T10:15:01.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:01.964 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:01.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:02.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:02.080 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 12 16 28560 28544 148.653 129.375 0.00696008 0.00672086 2024-09-16T10:15:02.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:02.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:02.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:01 smithi063 ceph-mon[29993]: pgmap v49: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-09-16T10:15:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:01 smithi063 ceph-mon[39470]: pgmap v49: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-09-16T10:15:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:02.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:02.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:02.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T10:15:02.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:02.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:02.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:02.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:02.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:02.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:02.591 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:02.635 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:02.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:02.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:02.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:02.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:02.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:02.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:02.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:02.959 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:02.971 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:03.015 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:03.037 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:03.052 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:02 smithi165 bash[60885]: 640aeecfab3246c99db2bff4b960e0e7a66f96fb50580dd969f8c9ea6d5b669d 2024-09-16T10:15:03.052 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:15:02 smithi165 systemd[1]: Started Ceph node-exporter.b for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:15:03.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 13 16 30324 30308 145.692 110.25 0.0259303 0.00685529 2024-09-16T10:15:03.139 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.169 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:03.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:03.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:03.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:03.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:03.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:03.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:03.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.569 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:03.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:03.670 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.703 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:03.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:03.804 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:03.845 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:03.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[39470]: pgmap v50: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 701 B/s rd, 162 MiB/s wr, 2.60k op/s 2024-09-16T10:15:03.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:03.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:15:03.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:03 smithi063 ceph-mon[29993]: pgmap v50: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 701 B/s rd, 162 MiB/s wr, 2.60k op/s 2024-09-16T10:15:03.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:03.977 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:03.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:04.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:03 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:03 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:03 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:03 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:03 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:03 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:15:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:03 smithi165 ceph-mon[32696]: pgmap v50: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 701 B/s rd, 162 MiB/s wr, 2.60k op/s 2024-09-16T10:15:04.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:04.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 14 16 31892 31876 142.285 98 0.00849084 0.00702042 2024-09-16T10:15:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:04.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:04.219 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:15:04.220 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:15:04.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:04.297 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:04.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:04.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:04.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:04.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:04.531 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:04.685 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:04.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:04.787 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:04.831 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:04.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:04.885 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T10:15:04.885 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:04 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/120239796' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:15:04.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:04 smithi063 ceph-mon[29993]: from='client.24790 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:15:04.886 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:04 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/120239796' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:15:04.886 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:04 smithi063 ceph-mon[39470]: from='client.24790 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:15:04.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:04.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:15:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:04.965 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:04.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:05.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:05.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 15 16 33165 33149 138.103 79.5625 0.00196184 0.0071659 2024-09-16T10:15:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:05.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:05.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:05.129 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:15:05.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:05.263 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:05.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:05.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:04 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/120239796' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:15:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:04 smithi165 ceph-mon[32696]: from='client.24790 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:15:05.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:05.437 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:05.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:05.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:05.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:05.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:05.671 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:05.710 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:05.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:05.812 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:05.847 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:05.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:05.900 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T10:15:05.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:05.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:05.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:05 smithi165 ceph-mon[32696]: from='client.24790 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:15:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:05 smithi165 ceph-mon[32696]: osdmap e93: 8 total, 8 up, 8 in 2024-09-16T10:15:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:05 smithi165 ceph-mon[32696]: pgmap v52: 148 pgs: 148 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 169 MiB/s wr, 2.71k op/s 2024-09-16T10:15:06.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:05 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:15:06.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:05 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:15:06.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:05 smithi165 ceph-mon[32696]: osdmap e94: 8 total, 8 up, 8 in 2024-09-16T10:15:06.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:06.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 16 16 34744 34728 135.639 98.6875 0.00978377 0.0073664 2024-09-16T10:15:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:06.120 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:06.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:06.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[39470]: from='client.24790 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:15:06.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[39470]: osdmap e93: 8 total, 8 up, 8 in 2024-09-16T10:15:06.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[39470]: pgmap v52: 148 pgs: 148 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 169 MiB/s wr, 2.71k op/s 2024-09-16T10:15:06.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:15:06.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:15:06.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[39470]: osdmap e94: 8 total, 8 up, 8 in 2024-09-16T10:15:06.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[29993]: from='client.24790 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:15:06.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[29993]: osdmap e93: 8 total, 8 up, 8 in 2024-09-16T10:15:06.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[29993]: pgmap v52: 148 pgs: 148 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 169 MiB/s wr, 2.71k op/s 2024-09-16T10:15:06.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:15:06.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:15:06.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:05 smithi063 ceph-mon[29993]: osdmap e94: 8 total, 8 up, 8 in 2024-09-16T10:15:06.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:06.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:06.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:06.259 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:06.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:06.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:06.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:06.477 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:06.517 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:06.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:06.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:06.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:06.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:06.663 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:06.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:06.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:06.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:06.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:06.918 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:06.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:07.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:07.032 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:07.066 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:07.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:07.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 17 16 36331 36315 133.494 99.1875 0.0137237 0.00748563 2024-09-16T10:15:07.167 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:07.178 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:15:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:15:07.230 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:07.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:07.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:07.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:07.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:07.370 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:15:07.370 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:15:07.388 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:15:07.388 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:15:07.425 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:15:07.425 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:15:07.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:07.469 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:15:07.469 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:15:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:07.502 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:07.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:07.503 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:15:07.504 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:15:07.540 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:15:07.540 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:15:07.576 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:15:07.577 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:15:07.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:07.614 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:15:07.614 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:15:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:07.647 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:07.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:07.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:07.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:07.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:07.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:07.938 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:07.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:08.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: osdmap e95: 8 total, 8 up, 8 in 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: pgmap v55: 148 pgs: 148 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:15:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:15:08.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4053241685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:08.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:08.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:08.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:08.081 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 18 16 37928 37912 131.623 99.8125 0.00393563 0.00759175 2024-09-16T10:15:08.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:08.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:08.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:15:08.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: osdmap e95: 8 total, 8 up, 8 in 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: pgmap v55: 148 pgs: 148 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:15:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4053241685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: osdmap e95: 8 total, 8 up, 8 in 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: pgmap v55: 148 pgs: 148 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-16T10:15:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:15:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:15:08.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4053241685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:08.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:08.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:08.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:08.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:08.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:08.467 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:08.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:08.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:15:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:15:08.569 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.609 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:08.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:08.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:08.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:08.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:08.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:15:08.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:08.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:08.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:09.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 19 16 39477 39461 129.79 96.8125 0.00214898 0.00768738 2024-09-16T10:15:09.095 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:09.124 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:09.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:09.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[29993]: osdmap e96: 8 total, 8 up, 8 in 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1061565778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[39470]: osdmap e96: 8 total, 8 up, 8 in 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:09.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1061565778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:15:09.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:09.267 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:09.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-16T10:15:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-16T10:15:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T10:15:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:08 smithi165 ceph-mon[32696]: osdmap e96: 8 total, 8 up, 8 in 2024-09-16T10:15:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:08 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1061565778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:15:09.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:09.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:09.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:09.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:09.537 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:09.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:09.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:09.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:09.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:09.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:09.794 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:09.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:09.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:09.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:09.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:10.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:10.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:10.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:10.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:15:10.081329+0000 min lat: 0.00122794 max lat: 0.201222 avg lat: 0.00783591 2024-09-16T10:15:10.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:15:10.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 20 16 40784 40768 127.384 81.6875 0.00975976 0.00783591 2024-09-16T10:15:10.167 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:10.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:10.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1061565778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:15:10.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[29993]: osdmap e97: 8 total, 8 up, 8 in 2024-09-16T10:15:10.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[29993]: pgmap v58: 148 pgs: 3 peering, 145 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-16T10:15:10.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:10.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1061565778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:15:10.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[39470]: osdmap e97: 8 total, 8 up, 8 in 2024-09-16T10:15:10.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:09 smithi063 ceph-mon[39470]: pgmap v58: 148 pgs: 3 peering, 145 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-16T10:15:10.204 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:10.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:10.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:09 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:10.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1061565778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:15:10.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:09 smithi165 ceph-mon[32696]: osdmap e97: 8 total, 8 up, 8 in 2024-09-16T10:15:10.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:09 smithi165 ceph-mon[32696]: pgmap v58: 148 pgs: 3 peering, 145 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-16T10:15:10.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:10.347 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:10.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:10.449 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:10.491 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:10.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:10.593 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:10.633 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:10.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:10.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:10.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:10.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:10.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:10.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:10.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:11.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:11.048 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:11.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 21 15 42527 42512 126.508 109 0.00684518 0.0078994 2024-09-16T10:15:11.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:11.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:11.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:11.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:11.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:11.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:11.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:11.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.558 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:11.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:11.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:11.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:11.742 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T10:15:11.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:11.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:11.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:11.953 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:11.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: 4.19 scrub starts 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: 4.19 scrub ok 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: osdmap e98: 8 total, 8 up, 8 in 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: 5.1d scrub starts 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: 5.1d scrub ok 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: pgmap v60: 148 pgs: 3 peering, 145 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:15:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1895231816' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: 4.19 scrub starts 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: 4.19 scrub ok 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: osdmap e98: 8 total, 8 up, 8 in 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: 5.1d scrub starts 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: 5.1d scrub ok 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: pgmap v60: 148 pgs: 3 peering, 145 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:15:12.053 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1895231816' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: 4.19 scrub starts 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: 4.19 scrub ok 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: osdmap e98: 8 total, 8 up, 8 in 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: 5.1d scrub starts 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: 5.1d scrub ok 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: pgmap v60: 148 pgs: 3 peering, 145 active+clean; 2.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:15:12.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1895231816' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:15:12.054 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:12.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 22 16 43885 43869 124.613 84.8125 0.0101852 0.00801895 2024-09-16T10:15:12.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:12.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:12.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:12.236 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:12.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:12.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:12.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:12.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:12.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:12.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:12.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:12.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:12.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:12.748 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:15:12.750 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:12.791 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:12.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:12.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:12.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:12.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:13.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:13.044 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 3.e scrub starts 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 3.e scrub ok 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 6.f scrub starts 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 6.f scrub ok 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1895231816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: osdmap e99: 8 total, 8 up, 8 in 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 6.2 scrub ok 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 5.1f scrub starts 2024-09-16T10:15:13.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[29993]: 5.1f scrub ok 2024-09-16T10:15:13.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:15:13.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:15:13.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 3.e scrub starts 2024-09-16T10:15:13.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 3.e scrub ok 2024-09-16T10:15:13.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 6.f scrub starts 2024-09-16T10:15:13.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 6.f scrub ok 2024-09-16T10:15:13.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1895231816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:15:13.047 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: osdmap e99: 8 total, 8 up, 8 in 2024-09-16T10:15:13.047 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:13.047 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:13.047 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:15:13.047 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 6.2 scrub ok 2024-09-16T10:15:13.047 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 5.1f scrub starts 2024-09-16T10:15:13.047 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:12 smithi063 ceph-mon[39470]: 5.1f scrub ok 2024-09-16T10:15:13.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 3.e scrub starts 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 3.e scrub ok 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 6.f scrub starts 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 6.f scrub ok 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1895231816' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: osdmap e99: 8 total, 8 up, 8 in 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:15:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 6.2 scrub ok 2024-09-16T10:15:13.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 5.1f scrub starts 2024-09-16T10:15:13.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:12 smithi165 ceph-mon[32696]: 5.1f scrub ok 2024-09-16T10:15:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:13.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:13.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 23 16 45640 45624 123.964 109.688 0.00802432 0.00806097 2024-09-16T10:15:13.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:13.201 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:13.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:13.300 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:13.302 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:13.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:13.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:13.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:13.503 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:13.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:13.606 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:13.645 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:13.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:13.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:13.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:13.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:13.860 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T10:15:13.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:13.944 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:13.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:14.018 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[29993]: pgmap v62: 148 pgs: 2 peering, 146 active+clean; 2.4 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1001 B/s rd, 111 MiB/s wr, 1.77k op/s; 21 B/s, 0 objects/s recovering 2024-09-16T10:15:14.018 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[29993]: 4.1e scrub starts 2024-09-16T10:15:14.018 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[29993]: 4.1e scrub ok 2024-09-16T10:15:14.018 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[29993]: 5.11 scrub starts 2024-09-16T10:15:14.018 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[29993]: 5.11 scrub ok 2024-09-16T10:15:14.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[39470]: pgmap v62: 148 pgs: 2 peering, 146 active+clean; 2.4 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1001 B/s rd, 111 MiB/s wr, 1.77k op/s; 21 B/s, 0 objects/s recovering 2024-09-16T10:15:14.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[39470]: 4.1e scrub starts 2024-09-16T10:15:14.019 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[39470]: 4.1e scrub ok 2024-09-16T10:15:14.019 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[39470]: 5.11 scrub starts 2024-09-16T10:15:14.019 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:13 smithi063 ceph-mon[39470]: 5.11 scrub ok 2024-09-16T10:15:14.046 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:14.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:13 smithi165 ceph-mon[32696]: pgmap v62: 148 pgs: 2 peering, 146 active+clean; 2.4 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1001 B/s rd, 111 MiB/s wr, 1.77k op/s; 21 B/s, 0 objects/s recovering 2024-09-16T10:15:14.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:13 smithi165 ceph-mon[32696]: 4.1e scrub starts 2024-09-16T10:15:14.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:13 smithi165 ceph-mon[32696]: 4.1e scrub ok 2024-09-16T10:15:14.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:13 smithi165 ceph-mon[32696]: 5.11 scrub starts 2024-09-16T10:15:14.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:13 smithi165 ceph-mon[32696]: 5.11 scrub ok 2024-09-16T10:15:14.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 24 16 47293 47277 123.103 103.312 0.0101098 0.00811665 2024-09-16T10:15:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:14.098 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:14.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:14.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:14.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:14.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:14.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:14.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:14.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:14.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:14.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:14.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:14.512 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:15:14.512 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:15:14.512 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:15:14.512 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:15:14.512 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:15:14.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:15:14.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:14.664 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:14.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:14.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:14.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:14.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:14.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:15:14.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:15:14.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[39470]: 2.1 scrub starts 2024-09-16T10:15:14.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[39470]: 2.1 scrub ok 2024-09-16T10:15:14.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[39470]: 5.3 scrub starts 2024-09-16T10:15:14.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[39470]: 5.3 scrub ok 2024-09-16T10:15:14.928 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:14.929 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:15:14.929 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:15:14.929 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[29993]: 2.1 scrub starts 2024-09-16T10:15:14.929 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[29993]: 2.1 scrub ok 2024-09-16T10:15:14.929 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[29993]: 5.3 scrub starts 2024-09-16T10:15:14.929 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:14 smithi063 ceph-mon[29993]: 5.3 scrub ok 2024-09-16T10:15:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:14.975 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:14.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:15.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:15:15.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:14 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:15:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:14 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:15:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:14 smithi165 ceph-mon[32696]: 2.1 scrub starts 2024-09-16T10:15:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:14 smithi165 ceph-mon[32696]: 2.1 scrub ok 2024-09-16T10:15:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:14 smithi165 ceph-mon[32696]: 5.3 scrub starts 2024-09-16T10:15:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:14 smithi165 ceph-mon[32696]: 5.3 scrub ok 2024-09-16T10:15:15.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:15.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 25 15 48776 48761 121.888 92.75 0.00580796 0.00819884 2024-09-16T10:15:15.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:15.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:15.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:15.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:15.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:15.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:15.388 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:15.429 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:15.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:15.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:15:15.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:15.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:15.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:15.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:15.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:15.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:15.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:15.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:15.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:15.917 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:15.968 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:15.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:16.010 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: 4.12 scrub starts 2024-09-16T10:15:16.011 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: 4.12 scrub ok 2024-09-16T10:15:16.011 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: pgmap v63: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 838 B/s rd, 79 MiB/s wr, 1.26k op/s; 54 B/s, 1 objects/s recovering 2024-09-16T10:15:16.011 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: 4.9 scrub starts 2024-09-16T10:15:16.011 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: 4.9 scrub ok 2024-09-16T10:15:16.011 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: 3.1f scrub starts 2024-09-16T10:15:16.011 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: 3.1f scrub ok 2024-09-16T10:15:16.012 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[29993]: osdmap e100: 8 total, 8 up, 8 in 2024-09-16T10:15:16.012 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: 4.12 scrub starts 2024-09-16T10:15:16.012 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: 4.12 scrub ok 2024-09-16T10:15:16.012 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: pgmap v63: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 838 B/s rd, 79 MiB/s wr, 1.26k op/s; 54 B/s, 1 objects/s recovering 2024-09-16T10:15:16.012 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: 4.9 scrub starts 2024-09-16T10:15:16.013 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: 4.9 scrub ok 2024-09-16T10:15:16.013 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: 3.1f scrub starts 2024-09-16T10:15:16.013 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: 3.1f scrub ok 2024-09-16T10:15:16.013 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:15 smithi063 ceph-mon[39470]: osdmap e100: 8 total, 8 up, 8 in 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: 4.12 scrub starts 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: 4.12 scrub ok 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: pgmap v63: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 838 B/s rd, 79 MiB/s wr, 1.26k op/s; 54 B/s, 1 objects/s recovering 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: 4.9 scrub starts 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: 4.9 scrub ok 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: 3.1f scrub starts 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: 3.1f scrub ok 2024-09-16T10:15:16.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:15 smithi165 ceph-mon[32696]: osdmap e100: 8 total, 8 up, 8 in 2024-09-16T10:15:16.070 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:16.083 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 26 16 50110 50094 120.404 83.3125 0.0154614 0.00829765 2024-09-16T10:15:16.094 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:16.115 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:16.174 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:16.196 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:16.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:16.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:16.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:16.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:16.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:16.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:16.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:16.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:16.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:16.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:16.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:16.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:16.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:16.794 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T10:15:16.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:16.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:16.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:16.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:16.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:16.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:17.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:17.076 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:15:17.076 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: 6.0 scrub starts 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: 6.0 scrub ok 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: 4.15 scrub starts 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: 4.15 scrub ok 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/975349596' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[39470]: from='client.24811 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: 6.0 scrub starts 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: 6.0 scrub ok 2024-09-16T10:15:17.077 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: 4.15 scrub starts 2024-09-16T10:15:17.078 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: 4.15 scrub ok 2024-09-16T10:15:17.078 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/975349596' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:15:17.078 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:16 smithi063 ceph-mon[29993]: from='client.24811 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:15:17.082 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 27 16 51455 51439 119.057 84.0625 0.0111231 0.00839288 2024-09-16T10:15:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:17.114 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:17.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:17.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:17.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:17.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: 6.0 scrub starts 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: 6.0 scrub ok 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: 4.15 scrub starts 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: 4.15 scrub ok 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/975349596' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:15:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:16 smithi165 ceph-mon[32696]: from='client.24811 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:15:17.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:17.389 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:17.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:17.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:15:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:15:17.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:17.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:17.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:17.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:17.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:17.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:17.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:17.787 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:17.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:17.826 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:15:17.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:17.931 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:17.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:17.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:18.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:18.074 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:18.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:18.083 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 28 16 52893 52877 118.015 89.875 0.00684337 0.0084679 2024-09-16T10:15:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:15:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:15:18.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: osdmap e101: 8 total, 8 up, 8 in 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: pgmap v66: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.29k op/s; 54 B/s, 1 objects/s recovering 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: 6.1d scrub starts 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: 6.1d scrub ok 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/959363612' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:18.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:17 smithi165 ceph-mon[32696]: from='client.24817 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:18.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:15:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:15:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:15:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: osdmap e101: 8 total, 8 up, 8 in 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: pgmap v66: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.29k op/s; 54 B/s, 1 objects/s recovering 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: 6.1d scrub starts 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: 6.1d scrub ok 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/959363612' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[39470]: from='client.24817 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:15:18.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: osdmap e101: 8 total, 8 up, 8 in 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: pgmap v66: 148 pgs: 148 active+clean; 2.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 80 MiB/s wr, 1.29k op/s; 54 B/s, 1 objects/s recovering 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: 6.1d scrub starts 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: 6.1d scrub ok 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/959363612' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:18.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:17 smithi063 ceph-mon[29993]: from='client.24817 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:18.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:18.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:18.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:18.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:18.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:18.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:18.470 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:18.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:18.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:15:18.572 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:18.613 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:18.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:18.715 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:18.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:18.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:18.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:18.894 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:18.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:18.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:19.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:19.083 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 29 15 54394 54379 117.182 93.875 0.00905454 0.00852829 2024-09-16T10:15:19.127 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:19.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: osdmap e102: 8 total, 8 up, 8 in 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: 3.4 scrub starts 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: 3.4 scrub ok 2024-09-16T10:15:19.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: osdmap e102: 8 total, 8 up, 8 in 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: 3.4 scrub starts 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: 3.4 scrub ok 2024-09-16T10:15:19.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:18 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:19.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:19.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: osdmap e102: 8 total, 8 up, 8 in 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: 3.4 scrub starts 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: 3.4 scrub ok 2024-09-16T10:15:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:18 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:19.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:19.435 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:19.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:19.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:19.665 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:19.705 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:19.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:19.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:19.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:19.936 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:19.978 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:19.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:20.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:20.083 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 30 15 55722 55707 116.042 83 0.00830457 0.00861248 2024-09-16T10:15:20.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:20.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:20.133 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[29993]: 6.b scrub starts 2024-09-16T10:15:20.133 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[29993]: 6.b scrub ok 2024-09-16T10:15:20.133 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[29993]: pgmap v68: 148 pgs: 148 active+clean; 3.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 140 MiB/s wr, 2.25k op/s; 36 B/s, 1 objects/s recovering 2024-09-16T10:15:20.133 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2851239822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:20.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[39470]: 6.b scrub starts 2024-09-16T10:15:20.134 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[39470]: 6.b scrub ok 2024-09-16T10:15:20.134 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[39470]: pgmap v68: 148 pgs: 148 active+clean; 3.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 140 MiB/s wr, 2.25k op/s; 36 B/s, 1 objects/s recovering 2024-09-16T10:15:20.134 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2851239822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:20.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:20.242 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:20.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:20.252 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:15:20.252 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:15:20.252 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:15:20.253 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:15:20.254 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:15:20.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:19 smithi165 ceph-mon[32696]: 6.b scrub starts 2024-09-16T10:15:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:19 smithi165 ceph-mon[32696]: 6.b scrub ok 2024-09-16T10:15:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:19 smithi165 ceph-mon[32696]: pgmap v68: 148 pgs: 148 active+clean; 3.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 140 MiB/s wr, 2.25k op/s; 36 B/s, 1 objects/s recovering 2024-09-16T10:15:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2851239822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:20.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:20.389 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:20.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:20.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:20.533 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:20.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:20.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:20.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:20.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:20.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:20.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:20.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:20.882 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:15:20.882 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:15:20.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:20.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:20.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:21.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:21.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:21.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:21.083 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 31 16 57061 57045 114.996 83.625 0.0118559 0.0086909 2024-09-16T10:15:21.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:21.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:20 smithi063 ceph-mon[29993]: 3.d scrub starts 2024-09-16T10:15:21.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:20 smithi063 ceph-mon[29993]: 3.d scrub ok 2024-09-16T10:15:21.173 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:20 smithi063 ceph-mon[39470]: 3.d scrub starts 2024-09-16T10:15:21.173 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:20 smithi063 ceph-mon[39470]: 3.d scrub ok 2024-09-16T10:15:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:21.204 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:21.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:20 smithi165 ceph-mon[32696]: 3.d scrub starts 2024-09-16T10:15:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:20 smithi165 ceph-mon[32696]: 3.d scrub ok 2024-09-16T10:15:21.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:21.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:21.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:21.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:21.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:21.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:21.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:21.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:21.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:21.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:21.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:21.734 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:21.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:21.837 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:21.880 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T10:15:21.885 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:21.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:21.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:22.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:22.083 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 32 16 58339 58323 113.898 79.875 0.00488371 0.00877377 2024-09-16T10:15:22.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:22.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: 5.0 scrub starts 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: 5.0 scrub ok 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: osdmap e103: 8 total, 8 up, 8 in 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: 5.1c scrub starts 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: pgmap v70: 148 pgs: 148 active+clean; 3.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 966 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-16T10:15:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[29993]: 5.1c scrub ok 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: 5.0 scrub starts 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: 5.0 scrub ok 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: osdmap e103: 8 total, 8 up, 8 in 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: 5.1c scrub starts 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: pgmap v70: 148 pgs: 148 active+clean; 3.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 966 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-16T10:15:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:21 smithi063 ceph-mon[39470]: 5.1c scrub ok 2024-09-16T10:15:22.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:22.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:22.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: 5.0 scrub starts 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: 5.0 scrub ok 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: osdmap e103: 8 total, 8 up, 8 in 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: 5.1c scrub starts 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: pgmap v70: 148 pgs: 148 active+clean; 3.2 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 966 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-16T10:15:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:21 smithi165 ceph-mon[32696]: 5.1c scrub ok 2024-09-16T10:15:22.373 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.424 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:22.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:22.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:22.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:22.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:22.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:22.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.827 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:22.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:22.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:22.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:22.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:23.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:23.083 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 33 16 59590 59574 112.816 78.1875 0.0156871 0.00885887 2024-09-16T10:15:23.095 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:23.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: 3.13 scrub starts 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: 3.13 scrub ok 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: 5.1b scrub starts 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: 5.1b scrub ok 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:23.133 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: 3.13 scrub starts 2024-09-16T10:15:23.134 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: 3.13 scrub ok 2024-09-16T10:15:23.134 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:15:23.134 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:15:23.134 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: 5.1b scrub starts 2024-09-16T10:15:23.134 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: 5.1b scrub ok 2024-09-16T10:15:23.134 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:23.134 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:22 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:23.197 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:23.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:23.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: 3.13 scrub starts 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: 3.13 scrub ok 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: 5.1b scrub starts 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: 5.1b scrub ok 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:22 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:23.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:23.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:23.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:23.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:15:23.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:23.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:23.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:23.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:23.600 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:23.651 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:23.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:23.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:23.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:23.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:23.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:23.952 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:23.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:23.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:15:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:23 smithi165 ceph-mon[32696]: pgmap v71: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 820 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-16T10:15:24.052 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:15:23 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=infra.usagestats t=2024-09-16T10:15:23.752979184Z level=info msg="Usage stats are ready to report" 2024-09-16T10:15:24.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:24.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:24.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:24.084 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 34 16 60903 60887 111.908 82.0625 0.0127623 0.00892879 2024-09-16T10:15:24.093 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:24.108 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:24.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:24.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:23 smithi063 ceph-mon[29993]: pgmap v71: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 820 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-16T10:15:24.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:23 smithi063 ceph-mon[39470]: pgmap v71: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 820 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-16T10:15:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:24.230 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:24.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:24.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:24.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:24.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:24.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:24.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:24.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:24.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:15:24.588 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:24.650 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:24.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:24.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:24.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:24.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:24.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:24.929 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:24.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:25.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:25.075 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:25.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:25.085 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 35 16 62146 62130 110.93 77.6875 0.00906973 0.00900877 2024-09-16T10:15:25.138 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 3.5 scrub starts 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 3.5 scrub ok 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 4.0 scrub starts 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 4.0 scrub ok 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: osdmap e104: 8 total, 8 up, 8 in 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 5.1e scrub starts 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 5.1e scrub ok 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 4.a scrub starts 2024-09-16T10:15:25.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[29993]: 4.a scrub ok 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 3.5 scrub starts 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 3.5 scrub ok 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 4.0 scrub starts 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 4.0 scrub ok 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: osdmap e104: 8 total, 8 up, 8 in 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 5.1e scrub starts 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 5.1e scrub ok 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 4.a scrub starts 2024-09-16T10:15:25.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:24 smithi063 ceph-mon[39470]: 4.a scrub ok 2024-09-16T10:15:25.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:25.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:25.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 3.5 scrub starts 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 3.5 scrub ok 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 4.0 scrub starts 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 4.0 scrub ok 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: osdmap e104: 8 total, 8 up, 8 in 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 5.1e scrub starts 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 5.1e scrub ok 2024-09-16T10:15:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 4.a scrub starts 2024-09-16T10:15:25.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:24 smithi165 ceph-mon[32696]: 4.a scrub ok 2024-09-16T10:15:25.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:25.348 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:25.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:25.449 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:25.493 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:25.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:25.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:25.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:25.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:25.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:25.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:25.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:25.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:25.875 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T10:15:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:25.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:25.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:25.983 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.025 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:26.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:26.085 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 36 15 63267 63252 109.796 70.125 0.0203457 0.00910276 2024-09-16T10:15:26.126 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:26.170 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:26.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[29993]: 3.18 scrub starts 2024-09-16T10:15:26.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[29993]: 3.18 scrub ok 2024-09-16T10:15:26.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[29993]: pgmap v73: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-16T10:15:26.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1706091211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:26.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2428130698' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:15:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[39470]: 3.18 scrub starts 2024-09-16T10:15:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[39470]: 3.18 scrub ok 2024-09-16T10:15:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[39470]: pgmap v73: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-16T10:15:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1706091211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:26.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2428130698' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:15:26.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:26.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:25 smithi165 ceph-mon[32696]: 3.18 scrub starts 2024-09-16T10:15:26.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:25 smithi165 ceph-mon[32696]: 3.18 scrub ok 2024-09-16T10:15:26.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:25 smithi165 ceph-mon[32696]: pgmap v73: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-16T10:15:26.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1706091211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:26.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2428130698' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:15:26.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:26.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:26.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:26.584 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:26.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:26.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:26.809 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:26.860 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:26.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:26.991 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:26.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:27.085 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 37 16 64449 64433 108.824 73.8125 0.00783897 0.0091843 2024-09-16T10:15:27.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:27.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:27.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:15:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: 3.f scrub starts 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: 3.f scrub ok 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: 6.c scrub starts 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: 6.c scrub ok 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2428130698' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: osdmap e105: 8 total, 8 up, 8 in 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: 4.3 scrub starts 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: 4.3 scrub ok 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:27.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: 3.f scrub starts 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: 3.f scrub ok 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: 6.c scrub starts 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: 6.c scrub ok 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2428130698' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: osdmap e105: 8 total, 8 up, 8 in 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: 4.3 scrub starts 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: 4.3 scrub ok 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:27.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:26 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:27.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.248 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:27.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: 3.f scrub starts 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: 3.f scrub ok 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: 6.c scrub starts 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: 6.c scrub ok 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2428130698' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: osdmap e105: 8 total, 8 up, 8 in 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: 4.3 scrub starts 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: 4.3 scrub ok 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:26 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:27.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.387 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:27.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:27.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:27.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:27.618 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:27.640 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:27.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:27.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:27.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.899 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:27.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:28.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:28.044 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:28.085 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 38 15 65603 65588 107.859 72.1875 0.0388766 0.009265 2024-09-16T10:15:28.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:28.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:28.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[29993]: 5.4 scrub starts 2024-09-16T10:15:28.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[29993]: 5.4 scrub ok 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[29993]: pgmap v75: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 824 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[29993]: 4.b scrub starts 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[29993]: 4.b scrub ok 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[39470]: 5.4 scrub starts 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[39470]: 5.4 scrub ok 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[39470]: pgmap v75: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 824 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[39470]: 4.b scrub starts 2024-09-16T10:15:28.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:27 smithi063 ceph-mon[39470]: 4.b scrub ok 2024-09-16T10:15:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:27 smithi165 ceph-mon[32696]: 5.4 scrub starts 2024-09-16T10:15:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:27 smithi165 ceph-mon[32696]: 5.4 scrub ok 2024-09-16T10:15:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:27 smithi165 ceph-mon[32696]: pgmap v75: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 824 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-16T10:15:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:27 smithi165 ceph-mon[32696]: 4.b scrub starts 2024-09-16T10:15:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:27 smithi165 ceph-mon[32696]: 4.b scrub ok 2024-09-16T10:15:28.204 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:15:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:15:28.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:28.318 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:28.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:28.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:28.484 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:15:28.484 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:15:28.484 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:15:28.484 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:15:28.485 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:15:28.486 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:15:28.486 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:15:28.486 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:15:28.486 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:15:28.554 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:28.591 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:28.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:28.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:28.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:28.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:28.959 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:28.996 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:28.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:28 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:15:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:28 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:15:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:28 smithi165 ceph-mon[32696]: osdmap e106: 8 total, 8 up, 8 in 2024-09-16T10:15:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:28 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:29.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 39 16 66950 66934 107.251 84.125 0.0136014 0.00931858 2024-09-16T10:15:29.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:29.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:29.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[39470]: osdmap e106: 8 total, 8 up, 8 in 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[29993]: osdmap e106: 8 total, 8 up, 8 in 2024-09-16T10:15:29.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:28 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:29.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:29.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:29.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:29.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:29.403 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:29.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:29.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:29.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:29.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:29.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:29.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:29.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:29.772 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:29.819 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:29.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:29.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:29.922 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T10:15:29.972 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:29.985 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:30.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:30.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:30.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:15:30.085402+0000 min lat: 0.00122794 max lat: 0.201222 avg lat: 0.00939556 2024-09-16T10:15:30.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:15:30.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 40 16 68103 68087 106.369 72.0625 0.00724887 0.00939556 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: pgmap v77: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: 3.6 scrub starts 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: 3.6 scrub ok 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:30.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-16T10:15:30.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[29993]: Reconfiguring daemon alertmanager.a on smithi063 2024-09-16T10:15:30.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:15:30.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:15:30.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:15:30.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: pgmap v77: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: 3.6 scrub starts 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: 3.6 scrub ok 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-16T10:15:30.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:29 smithi063 ceph-mon[39470]: Reconfiguring daemon alertmanager.a on smithi063 2024-09-16T10:15:30.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:30.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:30.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:30.274 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: pgmap v77: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: 3.6 scrub starts 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: 3.6 scrub ok 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:15:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:15:30.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:30.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-16T10:15:30.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:29 smithi165 ceph-mon[32696]: Reconfiguring daemon alertmanager.a on smithi063 2024-09-16T10:15:30.344 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:30.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:30.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:30.495 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:30.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:30.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:30.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:30.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:30.724 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:30.770 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:30.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:30.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:30.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:30.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:30.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:30 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:15:30.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:30 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:15:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:30 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:15:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:30 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:15:31.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:31.053 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:31.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:31.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 41 16 69335 69319 105.652 77 0.00813431 0.00946011 2024-09-16T10:15:31.158 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:31.202 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:31.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:31.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:15:31.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:30 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:15:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:30 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:15:31.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:31.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:31.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:31.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:31.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:31.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:31.574 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:31.646 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:31.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:31.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:31.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:31.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:31.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:31.930 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:31.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:70154 7.5 e106/81 7:ae67add3:::benchmark_data_smithi063_73086_object70153:head v 106'4400, mlcod=106'4397)", 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.766885+0000", 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0078759339999999994, 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079379810000000002, 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:31.944 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.766885+0000", 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.766885+0000", 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:31.945 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.766886+0000", 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.181e-06 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.766930+0000", 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3770999999999998e-05 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:31.946 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.766940+0000", 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0047999999999999e-05 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.766948+0000", 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6559999999999992e-06 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772606+0000", 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056577420000000003 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.947 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772625+0000", 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8974000000000002e-05 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.774738+0000", 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00211305 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:70158 7.2 e106/81 7:42138869:::benchmark_data_smithi063_73086_object70157:head v 106'4320, mlcod=106'4317)", 2024-09-16T10:15:31.948 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.772390+0000", 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0023714489999999999, 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025149920000000002, 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772390+0000", 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:31.949 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772390+0000", 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772391+0000", 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5880000000000001e-06 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772418+0000", 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.658e-05 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:31.950 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772429+0000", 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0569e-05 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.772435+0000", 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9659999999999999e-06 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.774430+0000", 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019947039999999999 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.774448+0000", 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7419999999999999e-05 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 2 2024-09-16T10:15:31.952 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:15:32.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:32.076 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:32.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:32.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 42 16 70406 70390 104.73 66.9375 0.00712175 0.00954213 2024-09-16T10:15:32.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:15:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:15:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:15:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:15:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[29993]: pgmap v78: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 713 B/s rd, 53 MiB/s wr, 841 op/s 2024-09-16T10:15:32.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:15:32.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:15:32.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:15:32.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:15:32.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:31 smithi063 ceph-mon[39470]: pgmap v78: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 713 B/s rd, 53 MiB/s wr, 841 op/s 2024-09-16T10:15:32.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:32.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:31 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:15:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:31 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:15:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:31 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:15:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:31 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:15:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:31 smithi165 ceph-mon[32696]: pgmap v78: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 713 B/s rd, 53 MiB/s wr, 841 op/s 2024-09-16T10:15:32.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:32.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:32.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:32.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:32.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:32.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:32.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:32.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:32.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:32.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:32.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:32.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:32.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:32.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:32.949 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:15:32.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:33.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:33.055 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2894358196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:15:33.055 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:32 smithi063 ceph-mon[29993]: 6.18 scrub starts 2024-09-16T10:15:33.055 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:32 smithi063 ceph-mon[29993]: 6.18 scrub ok 2024-09-16T10:15:33.055 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2894358196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:15:33.055 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:32 smithi063 ceph-mon[39470]: 6.18 scrub starts 2024-09-16T10:15:33.055 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:32 smithi063 ceph-mon[39470]: 6.18 scrub ok 2024-09-16T10:15:33.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 43 16 71716 71700 104.199 81.875 0.00704082 0.00959193 2024-09-16T10:15:33.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:33.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:33.237 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:33.278 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:33.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2894358196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:15:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:32 smithi165 ceph-mon[32696]: 6.18 scrub starts 2024-09-16T10:15:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:32 smithi165 ceph-mon[32696]: 6.18 scrub ok 2024-09-16T10:15:33.380 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:33.424 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:33.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:33.571 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:33.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:33.712 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:33.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.854 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:33.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:33.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:33.978 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:33.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:34.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:34.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 44 16 72855 72839 103.448 71.1875 0.0138339 0.00966115 2024-09-16T10:15:34.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:34.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:34.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:34.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2894358196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:15:34.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[29993]: osdmap e107: 8 total, 8 up, 8 in 2024-09-16T10:15:34.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[29993]: pgmap v80: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 714 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-16T10:15:34.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:34.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2894358196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:15:34.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[39470]: osdmap e107: 8 total, 8 up, 8 in 2024-09-16T10:15:34.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:33 smithi063 ceph-mon[39470]: pgmap v80: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 714 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-16T10:15:34.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:34.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:34.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:34.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:33 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2894358196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:15:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:33 smithi165 ceph-mon[32696]: osdmap e107: 8 total, 8 up, 8 in 2024-09-16T10:15:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:33 smithi165 ceph-mon[32696]: pgmap v80: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 714 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-16T10:15:34.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:34.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:34.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:34.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:15:34.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:34.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:34.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:34.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:34.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:34.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:34.825 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:34.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:34.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:34.971 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:34.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:35.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:35.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 45 16 74038 74022 102.792 73.9375 0.0151908 0.009723 2024-09-16T10:15:35.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:35.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:35.211 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:35.249 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:35.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:35.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:35.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:35.389 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:35.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:35.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:35.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:35.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:35.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:35.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:35.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:35.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:35.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:35.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:35.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:35.927 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:35.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:36.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:36.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:36.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:36.086 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 46 16 75304 75288 102.278 79.125 0.00609211 0.00977216 2024-09-16T10:15:36.160 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:36 smithi063 ceph-mon[29993]: pgmap v81: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-16T10:15:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:36 smithi063 ceph-mon[29993]: osdmap e108: 8 total, 8 up, 8 in 2024-09-16T10:15:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:36 smithi063 ceph-mon[39470]: pgmap v81: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-16T10:15:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:36 smithi063 ceph-mon[39470]: osdmap e108: 8 total, 8 up, 8 in 2024-09-16T10:15:36.205 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:36.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:36.248 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:36.257 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:36.307 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:36.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:36.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:36.517 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:36.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:36.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:36 smithi165 ceph-mon[32696]: pgmap v81: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-16T10:15:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:36 smithi165 ceph-mon[32696]: osdmap e108: 8 total, 8 up, 8 in 2024-09-16T10:15:36.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:36.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:36.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:36.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:36.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:36.862 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:36.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:36.943 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:36.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:37.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:37.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 47 16 76487 76471 101.675 73.9375 0.0112738 0.00982979 2024-09-16T10:15:37.092 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:37.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:37.152 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:15:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:15:37.153 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:36 smithi063 systemd[1]: Stopping Ceph alertmanager.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:15:37.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:37.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:37.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:37.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:37.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:37.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:37.448 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:37.501 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:37.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:37.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:37.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:37.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:37.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:37.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:37.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:37.839 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:15:37.839 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:15:37.860 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:37.875 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:15:37.875 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:15:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:37.904 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:37.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:37.913 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:15:37.914 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:15:37.948 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:15:37.949 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:15:37.986 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:15:37.987 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:15:38.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:38.021 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:15:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:15:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:38.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:38.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:38.055 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:15:38.055 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:15:38.057 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:15:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:15:38.089 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 48 16 77571 77555 100.968 67.75 0.0115714 0.00989873 2024-09-16T10:15:38.090 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:15:38.091 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:15:38.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:38.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:38.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:38.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:15:38.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:38.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:38.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:38.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:38.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:38.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:15:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:15:38.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:38.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:38.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:38.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:38.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:38.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:38.835 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:38.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[29993]: pgmap v83: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-16T10:15:38.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:38.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:38.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4176669940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:38.858 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[39470]: pgmap v83: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-16T10:15:38.858 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:38.858 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:38.858 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4176669940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:38.889 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:38.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:38.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:39.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:38 smithi165 ceph-mon[32696]: pgmap v83: 148 pgs: 148 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-16T10:15:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4176669940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:39.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 49 16 78742 78726 100.401 73.1875 0.00988137 0.00995513 2024-09-16T10:15:39.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.168 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:39.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:39.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:39.320 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:39.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:39.451 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:39 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[70754]: ts=2024-09-16T10:15:39.120Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-16T10:15:39.452 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:39 smithi063 podman[78745]: 2024-09-16 10:15:39.163093485 +0000 UTC m=+1.111066171 container died 0f853ece66bf260487e9fe535ef7777d55832ff43ac5fb18b6e85c3581cee928 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:39.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:39.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.603 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:39.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:39.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:39.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:39.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:39.833 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.871 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:39.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:39.910 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:39 smithi063 podman[78745]: 2024-09-16 10:15:39.631702556 +0000 UTC m=+1.579675242 container cleanup 0f853ece66bf260487e9fe535ef7777d55832ff43ac5fb18b6e85c3581cee928 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:39.910 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:39 smithi063 bash[78745]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a 2024-09-16T10:15:39.910 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:39.910 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:39.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:39.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:39.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:39.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:40.089 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 50 16 79900 79884 99.8401 72.375 0.0170246 0.0100106 2024-09-16T10:15:40.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:40.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:40.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:40.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:40.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:40.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:40.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:40.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:40.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:40.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:40.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:40.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:40.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:15:40.548 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:40.610 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:40.629 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T10:15:40.666 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:40.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:40.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:40 smithi063 ceph-mon[29993]: pgmap v84: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-16T10:15:40.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/674486817' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:15:40.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:40 smithi063 ceph-mon[29993]: from='client.44822 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:15:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:40 smithi063 ceph-mon[39470]: pgmap v84: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-16T10:15:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/674486817' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:15:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:40 smithi063 ceph-mon[39470]: from='client.44822 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:15:40.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:40.815 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:40.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:40.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:15:40.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:40.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:40 smithi165 ceph-mon[32696]: pgmap v84: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-16T10:15:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/674486817' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:15:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:40 smithi165 ceph-mon[32696]: from='client.44822 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:15:41.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:41.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:41.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:41.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 51 16 80845 80829 99.0405 59.0625 0.00662025 0.0100912 2024-09-16T10:15:41.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:41.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:41.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:41.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:41.361 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:41.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:41.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:41.501 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:41.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:41.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:41.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:41.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:41.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:41.793 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:41.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:41.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:41.934 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:41.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:41.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:41 smithi063 ceph-mon[29993]: from='client.44822 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:15:41.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:41 smithi063 ceph-mon[29993]: osdmap e109: 8 total, 8 up, 8 in 2024-09-16T10:15:41.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:41 smithi063 ceph-mon[29993]: osdmap e110: 8 total, 8 up, 8 in 2024-09-16T10:15:41.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:41 smithi063 ceph-mon[39470]: from='client.44822 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:15:41.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:41 smithi063 ceph-mon[39470]: osdmap e109: 8 total, 8 up, 8 in 2024-09-16T10:15:41.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:41 smithi063 ceph-mon[39470]: osdmap e110: 8 total, 8 up, 8 in 2024-09-16T10:15:42.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:42.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:41 smithi165 ceph-mon[32696]: from='client.44822 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:15:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:41 smithi165 ceph-mon[32696]: osdmap e109: 8 total, 8 up, 8 in 2024-09-16T10:15:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:41 smithi165 ceph-mon[32696]: osdmap e110: 8 total, 8 up, 8 in 2024-09-16T10:15:42.079 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:42.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:42.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 52 16 81985 81969 98.5059 71.25 0.00990268 0.0101465 2024-09-16T10:15:42.181 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:42.222 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:42.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:42.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:42.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:42.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:42.464 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:42.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:42.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:42.610 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:42.649 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:42.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:42.751 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:42.785 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:42 smithi063 ceph-mon[29993]: pgmap v87: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 980 op/s 2024-09-16T10:15:42.786 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:42 smithi063 ceph-mon[39470]: pgmap v87: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 980 op/s 2024-09-16T10:15:42.786 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:42 smithi063 podman[78871]: 2024-09-16 10:15:42.520088368 +0000 UTC m=+3.349221416 container remove 0f853ece66bf260487e9fe535ef7777d55832ff43ac5fb18b6e85c3581cee928 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:42.786 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:42 smithi063 podman[78871]: 2024-09-16 10:15:42.612813515 +0000 UTC m=+3.441946564 volume remove 7d5bb02cd27e3eaecc466c0aead5d1f01864e4929f9fe11d02224b2bb225e2a2 2024-09-16T10:15:42.793 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:42.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:42.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:42.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:42.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:43.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:42 smithi165 ceph-mon[32696]: pgmap v87: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 980 op/s 2024-09-16T10:15:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:43.066 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:43.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:43.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 53 16 83141 83125 98.0104 72.25 0.0125162 0.0101971 2024-09-16T10:15:43.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:43.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:43.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:43.296 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:43.335 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:43.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:43.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:43.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:43.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:43.563 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:43.611 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:43.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:43.643 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:15:43.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:43.753 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:43.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:43.855 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:43.892 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:43.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:43.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/115841570' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:43.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:43 smithi063 ceph-mon[29993]: from='client.24853 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:43.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/115841570' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:43.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:43 smithi063 ceph-mon[39470]: from='client.24853 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:43.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:44.039 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:44.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:44.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/115841570' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:44.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:43 smithi165 ceph-mon[32696]: from='client.24853 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:15:44.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 54 16 84288 84272 97.5229 71.6875 0.0207307 0.0102491 2024-09-16T10:15:44.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:44.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:44.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:44.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:44.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:44.286 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T10:15:44.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:15:44.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:44.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:44.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:44.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:44.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:44.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:44.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:44.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:44.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:44.654 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:44.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:44.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:44.799 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:44.852 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:44.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:44.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[29993]: pgmap v88: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 700 B/s rd, 59 MiB/s wr, 942 op/s 2024-09-16T10:15:44.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:44.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[29993]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:15:44.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[29993]: osdmap e111: 8 total, 8 up, 8 in 2024-09-16T10:15:44.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[39470]: pgmap v88: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 700 B/s rd, 59 MiB/s wr, 942 op/s 2024-09-16T10:15:44.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:44.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[39470]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:15:44.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:44 smithi063 ceph-mon[39470]: osdmap e111: 8 total, 8 up, 8 in 2024-09-16T10:15:44.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:44.995 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:45.018 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:45.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:44 smithi165 ceph-mon[32696]: pgmap v88: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 700 B/s rd, 59 MiB/s wr, 942 op/s 2024-09-16T10:15:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:44 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:44 smithi165 ceph-mon[32696]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:15:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:44 smithi165 ceph-mon[32696]: osdmap e111: 8 total, 8 up, 8 in 2024-09-16T10:15:45.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 55 16 85290 85274 96.888 62.625 0.016143 0.0103157 2024-09-16T10:15:45.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:45.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:45.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:45.157 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:44 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-16T10:15:45.157 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:44 smithi063 systemd[1]: Stopped Ceph alertmanager.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:15:45.157 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:44 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.277s CPU time. 2024-09-16T10:15:45.157 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:44 smithi063 systemd[1]: Starting Ceph alertmanager.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:15:45.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:45.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:45.400 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:45.452 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:45.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:45.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:45.596 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:45.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:45.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:45.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:45.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:45.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:45.860 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:45.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:45.961 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:45.999 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:46.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:46.087 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 56 16 86393 86377 96.3888 68.9375 0.00964723 0.0103697 2024-09-16T10:15:46.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:46.108 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:15:46.109 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:15:46.128 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:46.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:46.148 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:15:46.148 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:15:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:15:46.189 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:15:46.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:15:46.237 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:15:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:15:46.268 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:15:46.276 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:46.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:46.304 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:15:46.305 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:15:46.346 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:15:46.346 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:40100 7.e 7:7326c3b0:::benchmark_data_smithi063_73086_object40099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:15:46.372 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 36.778534942, 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.113082665, 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 40100 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546900+0000", 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2330000000000001e-06 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546927+0000", 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6721999999999999e-05 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546939+0000", 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1793999999999999e-05 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546947+0000", 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.704e-06 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547026+0000", 2024-09-16T10:15:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8479999999999994e-05 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547144+0000", 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011862999999999999 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547278+0000", 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013377800000000001 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.548993+0000", 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001715261 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549008+0000", 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4158000000000001e-05 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659763+0000", 2024-09-16T10:15:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.110755881 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659886+0000", 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012276600000000001 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659980+0000", 2024-09-16T10:15:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4258000000000005e-05 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:43040 7.e 7:7096c4f7:::benchmark_data_smithi063_73086_object43039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.406090+0000", 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 34.91934294, 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.065817186, 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 43040 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406090+0000", 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406090+0000", 2024-09-16T10:15:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406091+0000", 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5009999999999999e-06 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406111+0000", 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0072e-05 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406120+0000", 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4780000000000008e-06 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406126+0000", 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8880000000000002e-06 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406377+0000", 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025136700000000001 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406641+0000", 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026427200000000002 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.406821+0000", 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000179624 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.463994+0000", 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.057172948000000001 2024-09-16T10:15:46.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.464052+0000", 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8103000000000003e-05 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.471789+0000", 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.007737023 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.471863+0000", 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3610999999999997e-05 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.471907+0000", 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4298999999999997e-05 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:43495 7.e 7:7a740846:::benchmark_data_smithi063_73086_object43494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:15:46.386 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 34.572537336000003, 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115699248, 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 43495 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752898+0000", 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1710000000000001e-06 2024-09-16T10:15:46.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752922+0000", 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4593e-05 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752933+0000", 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1083999999999999e-05 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752941+0000", 2024-09-16T10:15:46.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6920000000000002e-06 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753001+0000", 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0109000000000003e-05 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753158+0000", 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000156668 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753289+0000", 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013114999999999999 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.755266+0000", 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00197679 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.755404+0000", 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013795999999999999 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868497+0000", 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11309365 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868554+0000", 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6641999999999997e-05 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868595+0000", 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0738999999999997e-05 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:51054 7.e 7:72b76afe:::benchmark_data_smithi063_73086_object51053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:16.779675+0000", 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 29.545758076999999, 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.076105037, 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51054 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.779675+0000", 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.779675+0000", 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.779676+0000", 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5099999999999998e-07 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.779696+0000", 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0210000000000001e-05 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.779706+0000", 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0581000000000001e-05 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.779714+0000", 2024-09-16T10:15:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.165e-06 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.783850+0000", 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041362630000000003 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.783977+0000", 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012748499999999999 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.784112+0000", 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013459600000000001 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791624+0000", 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0075121650000000003 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.800791+0000", 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0091674600000000005 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.855504+0000", 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.054712080000000003 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.855735+0000", 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00023151199999999999 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.855780+0000", 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4569e-05 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:51068 7.e 7:79cd42ab:::benchmark_data_smithi063_73086_object51067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:16.791001+0000", 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 29.534431956999999, 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.065011533999999996, 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51068 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791001+0000", 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791001+0000", 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791003+0000", 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5689999999999999e-06 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791044+0000", 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0352999999999998e-05 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791062+0000", 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8825e-05 2024-09-16T10:15:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791073+0000", 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0267e-05 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791657+0000", 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058429399999999996 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791782+0000", 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125165 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.791919+0000", 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013670800000000001 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.796611+0000", 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046919079999999998 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.800800+0000", 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041892780000000003 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.855826+0000", 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.055025406999999998 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.855976+0000", 2024-09-16T10:15:46.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015074899999999999 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:16.856012+0000", 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6010999999999998e-05 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:54692 7.e 7:7c312c5d:::benchmark_data_smithi063_73086_object54691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:19.281689+0000", 2024-09-16T10:15:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 27.043744091000001, 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.083823231999999998, 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 54692 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281689+0000", 2024-09-16T10:15:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281689+0000", 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281691+0000", 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2479999999999999e-06 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281715+0000", 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4430000000000002e-05 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281728+0000", 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2527000000000001e-05 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281736+0000", 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0930000000000003e-06 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281785+0000", 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9017000000000002e-05 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281874+0000", 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9064e-05 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.281953+0000", 2024-09-16T10:15:46.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9097999999999994e-05 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.283540+0000", 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001587197 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.283752+0000", 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00021195599999999999 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.365366+0000", 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.081613702999999996 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.365470+0000", 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010439500000000001 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:19.365512+0000", 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1504000000000001e-05 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.412 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:63762 7.e 7:71c9074c:::benchmark_data_smithi063_73086_object63761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:26.480256+0000", 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 19.845176457000001, 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.065064907000000005, 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.413 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 63762 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480256+0000", 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480256+0000", 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480259+0000", 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.033e-06 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480288+0000", 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9085999999999999e-05 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480306+0000", 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7246999999999999e-05 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480315+0000", 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8309999999999996e-06 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480384+0000", 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8417000000000006e-05 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480513+0000", 2024-09-16T10:15:46.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012904999999999999 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.480640+0000", 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012685800000000001 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.483416+0000", 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027765390000000002 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.483809+0000", 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000392276 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.545214+0000", 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.061404959000000002 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.545273+0000", 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8962999999999998e-05 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:26.545321+0000", 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8647999999999999e-05 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65096 7.e 7:7bb27b3e:::benchmark_data_smithi063_73086_object65095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 18.761529081999999, 2024-09-16T10:15:46.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.125924595, 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65096 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563905+0000", 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.606e-06 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563926+0000", 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1009000000000002e-05 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563936+0000", 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6460000000000005e-06 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563942+0000", 2024-09-16T10:15:46.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1589999999999998e-06 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563991+0000", 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9279e-05 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564082+0000", 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0890000000000006e-05 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564170+0000", 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7435000000000006e-05 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.565824+0000", 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001654345 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.566103+0000", 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027876700000000002 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689664+0000", 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123561491 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689778+0000", 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113677 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689828+0000", 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0290999999999997e-05 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.425 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70300 7.8 7:10588a9d:::benchmark_data_smithi063_73086_object70299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 14.422339645999999, 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12214414, 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.426 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.427 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70300 2024-09-16T10:15:46.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.427 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903095+0000", 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9099999999999999e-06 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903119+0000", 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3856999999999999e-05 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903132+0000", 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2873e-05 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903139+0000", 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9739999999999996e-06 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903195+0000", 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6121e-05 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903309+0000", 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113962 2024-09-16T10:15:46.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903448+0000", 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013888799999999999 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905996+0000", 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025485479999999999 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909244+0000", 2024-09-16T10:15:46.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032479420000000002 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.024965+0000", 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115720986 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025185+0000", 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220177 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025237+0000", 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1901999999999999e-05 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70303 7.8 7:176924a2:::benchmark_data_smithi063_73086_object70302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 14.420039191000001, 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120210123, 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.433 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70303 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905396+0000", 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5900000000000002e-06 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905422+0000", 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6364999999999999e-05 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905438+0000", 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5795e-05 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905448+0000", 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8020000000000003e-06 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906033+0000", 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058453800000000001 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906161+0000", 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012883799999999999 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906300+0000", 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013873499999999999 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909282+0000", 2024-09-16T10:15:46.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029820070000000001 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.910327+0000", 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001044332 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025288+0000", 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11496116100000001 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025559+0000", 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027147 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025604+0000", 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4490000000000003e-05 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70328 7.8 7:1d48f836:::benchmark_data_smithi063_73086_object70327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.940350+0000", 2024-09-16T10:15:46.439 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 14.385082204, 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.085849485000000003, 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70328 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.440 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940350+0000", 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940350+0000", 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940353+0000", 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3070000000000001e-06 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940379+0000", 2024-09-16T10:15:46.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5905999999999999e-05 2024-09-16T10:15:46.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940395+0000", 2024-09-16T10:15:46.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6529e-05 2024-09-16T10:15:46.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940404+0000", 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9080000000000001e-06 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940468+0000", 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4240999999999997e-05 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940564+0000", 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5577999999999999e-05 2024-09-16T10:15:46.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.940654+0000", 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0258000000000002e-05 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.942770+0000", 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002115839 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.942898+0000", 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012830099999999999 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025899+0000", 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.083000421000000005 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.026158+0000", 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000258769 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.026200+0000", 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2428000000000001e-05 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72399 7.e 7:78165852:::benchmark_data_smithi063_73086_object72398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.593479+0000", 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.731953939, 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.068155825000000003, 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.448 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72399 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.593479+0000", 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.593479+0000", 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.593481+0000", 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.995e-06 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.593505+0000", 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4666000000000001e-05 2024-09-16T10:15:46.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.593517+0000", 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1719000000000001e-05 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.593525+0000", 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0690000000000002e-06 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.594739+0000", 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012137039999999999 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.594851+0000", 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112015 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.594970+0000", 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011923 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.597729+0000", 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027590560000000002 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.597741+0000", 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1637000000000001e-05 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.661189+0000", 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.063448020999999993 2024-09-16T10:15:46.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.661428+0000", 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000238648 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.661635+0000", 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020706500000000001 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72689 7.e 7:7f5699d3:::benchmark_data_smithi063_73086_object72688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.455945420000001, 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12073505599999999, 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72689 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.455 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869489+0000", 2024-09-16T10:15:46.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.237e-06 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869521+0000", 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2728999999999998e-05 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869531+0000", 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8989999999999996e-06 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869539+0000", 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2209999999999997e-06 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872936+0000", 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003396296 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873090+0000", 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015414000000000001 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873225+0000", 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013536000000000001 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878402+0000", 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0051764419999999998 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878413+0000", 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1810999999999999e-05 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.989923+0000", 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.111509897 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990168+0000", 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00024499400000000001 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990222+0000", 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4030000000000003e-05 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72696 7.e 7:7eb0d49d:::benchmark_data_smithi063_73086_object72695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:15:46.462 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.452646035000001, 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120396989, 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72696 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872789+0000", 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.979e-06 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872820+0000", 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1522000000000001e-05 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872830+0000", 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7550000000000006e-06 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872837+0000", 2024-09-16T10:15:46.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7399999999999998e-06 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875523+0000", 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026860759999999999 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875632+0000", 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000109343 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875757+0000", 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012532799999999999 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878452+0000", 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026941249999999999 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.880841+0000", 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023892340000000001 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993090+0000", 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.112248941 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993145+0000", 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5043999999999997e-05 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993184+0000", 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8902000000000001e-05 2024-09-16T10:15:46.468 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72702 7.e 7:720829b1:::benchmark_data_smithi063_73086_object72701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.447382249, 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119080255, 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.469 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72702 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878052+0000", 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3689999999999999e-06 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878078+0000", 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6140999999999999e-05 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878088+0000", 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0147e-05 2024-09-16T10:15:46.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878096+0000", 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5089999999999996e-06 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878485+0000", 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00038902000000000001 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878652+0000", 2024-09-16T10:15:46.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016732800000000001 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878733+0000", 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0873999999999999e-05 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.883480+0000", 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047467120000000002 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884355+0000", 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087525899999999998 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997027+0000", 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11267184399999999 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997084+0000", 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.766e-05 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997131+0000", 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6391999999999999e-05 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72710 7.e 7:792abf92:::benchmark_data_smithi063_73086_object72709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.441183595, 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11643550799999999, 2024-09-16T10:15:46.475 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72710 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884251+0000", 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8929999999999999e-06 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884276+0000", 2024-09-16T10:15:46.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4837999999999998e-05 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884287+0000", 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1464000000000001e-05 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884295+0000", 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2300000000000002e-06 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884389+0000", 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4073000000000005e-05 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884555+0000", 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000166782 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884698+0000", 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000142917 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.479 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888931+0000", 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042330529999999996 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984052+0000", 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.095120447999999996 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000566+0000", 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016514217000000001 2024-09-16T10:15:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000630+0000", 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4459000000000001e-05 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000685+0000", 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4134e-05 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72712 7.e 7:7d21703e:::benchmark_data_smithi063_73086_object72711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.440862138, 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.118747798, 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72712 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884572+0000", 2024-09-16T10:15:46.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4270000000000001e-06 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884607+0000", 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4535000000000001e-05 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884617+0000", 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0774000000000001e-05 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884625+0000", 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.644e-06 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.886978+0000", 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00235257 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887080+0000", 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010283 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.485 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887167+0000", 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6706999999999995e-05 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984059+0000", 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096891692000000001 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984099+0000", 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0516000000000003e-05 2024-09-16T10:15:46.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003069+0000", 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.018970035999999999 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003200+0000", 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000130961 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003318+0000", 2024-09-16T10:15:46.487 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118106 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72715 7.e 7:70bb1898:::benchmark_data_smithi063_73086_object72714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.437226449000001, 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115258741, 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:15:46.488 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72715 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.490 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:15:46.490 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:15:46.490 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:15:46.490 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888208+0000", 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0990000000000001e-06 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888233+0000", 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4961999999999999e-05 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888243+0000", 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3680000000000008e-06 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888248+0000", 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1479999999999997e-06 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888960+0000", 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00071208700000000003 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889036+0000", 2024-09-16T10:15:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6175999999999997e-05 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981822+0000", 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.092785459000000001 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984260+0000", 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002438127 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.986355+0000", 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020951099999999999 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003378+0000", 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.017022926000000001 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003430+0000", 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1847e-05 2024-09-16T10:15:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003465+0000", 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5432000000000002e-05 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:72718 7.2 e107/81 7:450c0e00:::benchmark_data_smithi063_73086_object72717:head v 107'4483, mlcod=107'4481)", 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.436062296999999, 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.094911379000000004, 2024-09-16T10:15:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889372+0000", 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3349999999999999e-06 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889396+0000", 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3961999999999999e-05 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981880+0000", 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.092484449999999996 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981892+0000", 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1661000000000001e-05 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981956+0000", 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4168e-05 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981973+0000", 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7357999999999999e-05 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984187+0000", 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022133159999999999 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984282+0000", 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5129000000000003e-05 2024-09-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:72712 7.e e107/81 ondisk, result = 0)", 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.889417+0000", 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.436016118, 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.094690055999999995, 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889417+0000", 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889417+0000", 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889418+0000", 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0699999999999999e-06 2024-09-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889422+0000", 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4159999999999997e-06 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981506+0000", 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.092083898999999997 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981527+0000", 2024-09-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1279000000000001e-05 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984086+0000", 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025590029999999998 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984094+0000", 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3819999999999998e-06 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984107+0000", 2024-09-16T10:15:46.503 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3006999999999999e-05 2024-09-16T10:15:46.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.504 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:15:46.504 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:15:46.504 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:15:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:46.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:46.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:46.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:46.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:46.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:46.639 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:46 smithi063 podman[79688]: 2024-09-16 10:15:46.384078996 +0000 UTC m=+1.037155217 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-16T10:15:46.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:46.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:46.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:46.775 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:15:46.778 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:15:46.845 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:15:46.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:46.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:46 smithi063 ceph-mon[29993]: pgmap v90: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-16T10:15:46.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:46 smithi063 ceph-mon[29993]: osdmap e112: 8 total, 8 up, 8 in 2024-09-16T10:15:46.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/850107717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:46.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:46 smithi063 ceph-mon[39470]: pgmap v90: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-16T10:15:46.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:46 smithi063 ceph-mon[39470]: osdmap e112: 8 total, 8 up, 8 in 2024-09-16T10:15:46.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/850107717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:46.950 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:46.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:47.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:47.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:46 smithi165 ceph-mon[32696]: pgmap v90: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-16T10:15:47.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:46 smithi165 ceph-mon[32696]: osdmap e112: 8 total, 8 up, 8 in 2024-09-16T10:15:47.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/850107717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:47.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:47.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:47.095 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 57 16 87353 87337 95.7396 60 0.00712798 0.0104383 2024-09-16T10:15:47.178 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:47.201 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:46 smithi063 podman[79688]: 2024-09-16 10:15:46.898487792 +0000 UTC m=+1.551564003 volume create 3e05a0e8bf3db90169849de1c5861262611af42f6acbd08e64ea67a63b5fc221 2024-09-16T10:15:47.202 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:47 smithi063 podman[79688]: 2024-09-16 10:15:47.074166995 +0000 UTC m=+1.727243204 container create c82a0fcfe301e57df27402a073c295e3a01020bd41cbc363cd671d9c941e7716 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:47.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:15:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:15:47.231 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:47.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:47.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:47.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:47.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:47.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:47.510 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:47.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:47.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:47.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:47.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:47.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:47.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:47.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:47.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:47.912 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:47.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:48.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:48.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:48.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:48.096 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 58 16 88495 88479 95.3195 71.375 0.0140357 0.0104858 2024-09-16T10:15:48.143 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:48.188 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:48.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:48.290 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:48.333 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:48.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:48.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:48.480 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:48.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:48.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:15:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:15:48.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:48.623 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:48.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:48.644 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 podman[79688]: 2024-09-16 10:15:48.518465362 +0000 UTC m=+3.171541578 container init c82a0fcfe301e57df27402a073c295e3a01020bd41cbc363cd671d9c941e7716 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:48.644 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 podman[79688]: 2024-09-16 10:15:48.524451141 +0000 UTC m=+3.177527353 container start c82a0fcfe301e57df27402a073c295e3a01020bd41cbc363cd671d9c941e7716 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:48.644 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.550Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-16T10:15:48.644 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.550Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-16T10:15:48.644 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.551Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.63 port=9094 2024-09-16T10:15:48.644 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.555Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-16T10:15:48.644 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.601Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T10:15:48.645 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.602Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T10:15:48.645 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.605Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-16T10:15:48.645 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:48.605Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-16T10:15:48.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:48.774 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:48.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:48.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:48.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:48.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:48.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-mon[29993]: from='client.24865 v1:172.21.15.63:0/1876476134' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:15:48.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-mon[29993]: pgmap v92: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 812 B/s rd, 55 MiB/s wr, 884 op/s 2024-09-16T10:15:48.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-mon[39470]: from='client.24865 v1:172.21.15.63:0/1876476134' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:15:48.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:48 smithi063 ceph-mon[39470]: pgmap v92: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 812 B/s rd, 55 MiB/s wr, 884 op/s 2024-09-16T10:15:48.952 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 bash[79688]: c82a0fcfe301e57df27402a073c295e3a01020bd41cbc363cd671d9c941e7716 2024-09-16T10:15:48.953 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:48 smithi063 systemd[1]: Started Ceph alertmanager.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:15:49.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:48 smithi165 ceph-mon[32696]: from='client.24865 v1:172.21.15.63:0/1876476134' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:15:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:48 smithi165 ceph-mon[32696]: pgmap v92: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 812 B/s rd, 55 MiB/s wr, 884 op/s 2024-09-16T10:15:49.060 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:49.094 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 59 16 89593 89577 94.867 68.625 0.00801656 0.0105343 2024-09-16T10:15:49.162 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.203 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:49.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:49.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:49.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:49.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:49.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:49.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:49.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:49.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:49.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:49.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:49.857 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:49.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:49.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:49.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:49.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:50.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:50.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:50.094 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:15:50.093800+0000 min lat: 0.00122794 max lat: 0.201222 avg lat: 0.0105998 2024-09-16T10:15:50.094 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:15:50.094 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 60 16 90559 90543 94.2921 60.375 0.0115336 0.0105998 2024-09-16T10:15:50.131 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:50.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:50.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:50.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:50.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:50.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:50.400 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:50.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:50.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:15:50.502 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:50.540 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:50.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:50.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:50.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:50.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:50.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:50.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:50.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:50.853 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:50 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:50.556Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000551039s 2024-09-16T10:15:50.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:50 smithi063 ceph-mon[29993]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-16T10:15:50.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:50 smithi063 ceph-mon[29993]: pgmap v93: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-16T10:15:50.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:50 smithi063 ceph-mon[29993]: Reconfiguring daemon prometheus.a on smithi165 2024-09-16T10:15:50.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:50 smithi063 ceph-mon[39470]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-16T10:15:50.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:50 smithi063 ceph-mon[39470]: pgmap v93: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-16T10:15:50.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:50 smithi063 ceph-mon[39470]: Reconfiguring daemon prometheus.a on smithi165 2024-09-16T10:15:50.902 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:50.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:50 smithi165 ceph-mon[32696]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-16T10:15:50.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:50 smithi165 ceph-mon[32696]: pgmap v93: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-16T10:15:50.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:50 smithi165 ceph-mon[32696]: Reconfiguring daemon prometheus.a on smithi165 2024-09-16T10:15:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:50.941 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:50.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:51.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:51.087 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:51.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:51.094 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 61 16 91546 91530 93.7576 61.6875 0.00431663 0.0106597 2024-09-16T10:15:51.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:51.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:51.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:51.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:51.366 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:51.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:51.467 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:51.507 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:51.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:51.530 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-16T10:15:45.736466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5447096859,"num_objects":83306,"num_object_clones":0,"num_object_copies":249918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83306,"num_whiteouts":0,"num_read":968,"num_read_kb":825,"num_write":83656,"num_write_kb":5320521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81559,"ondisk_log_size":81559,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16356980,"kb_used_data":15874268,"kb_used_omap":12,"kb_used_meta":482547,"kb_avail":733604236,"statfs":{"total":767960285184,"available":751210737664,"internally_reserved":0,"allocated":16255250432,"data_stored":16249993681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":494128689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":365166592,"num_objects":5572,"num_object_clones":0,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5572,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.115549"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762960+0000","last_change":"2024-09-16T10:15:15.058932+0000","last_active":"2024-09-16T10:15:40.762960+0000","last_peered":"2024-09-16T10:15:40.762960+0000","last_clean":"2024-09-16T10:15:40.762960+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:15:40.762960+0000","last_undegraded":"2024-09-16T10:15:40.762960+0000","last_fullsized":"2024-09-16T10:15:40.762960+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:15.058652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.058652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.247209+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":150,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735053+0000","last_change":"2024-09-16T10:15:11.063493+0000","last_active":"2024-09-16T10:15:40.735053+0000","last_peered":"2024-09-16T10:15:40.735053+0000","last_clean":"2024-09-16T10:15:40.735053+0000","last_became_active":"2024-09-16T10:15:08.925269+0000","last_became_peered":"2024-09-16T10:15:08.925269+0000","last_unstale":"2024-09-16T10:15:40.735053+0000","last_undegraded":"2024-09-16T10:15:40.735053+0000","last_fullsized":"2024-09-16T10:15:40.735053+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:11.063426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.063426+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:23.888178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738822+0000","last_change":"2024-09-16T10:13:41.108286+0000","last_active":"2024-09-16T10:15:40.738822+0000","last_peered":"2024-09-16T10:15:40.738822+0000","last_clean":"2024-09-16T10:15:40.738822+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:15:40.738822+0000","last_undegraded":"2024-09-16T10:15:40.738822+0000","last_fullsized":"2024-09-16T10:15:40.738822+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.053767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735155+0000","last_change":"2024-09-16T10:13:43.113148+0000","last_active":"2024-09-16T10:15:40.735155+0000","last_peered":"2024-09-16T10:15:40.735155+0000","last_clean":"2024-09-16T10:15:40.735155+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:15:40.735155+0000","last_undegraded":"2024-09-16T10:15:40.735155+0000","last_fullsized":"2024-09-16T10:15:40.735155+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:01.951377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748961+0000","last_change":"2024-09-16T10:13:43.287323+0000","last_active":"2024-09-16T10:15:40.748961+0000","last_peered":"2024-09-16T10:15:40.748961+0000","last_clean":"2024-09-16T10:15:40.748961+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:15:40.748961+0000","last_undegraded":"2024-09-16T10:15:40.748961+0000","last_fullsized":"2024-09-16T10:15:40.748961+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:37.873818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.749027+0000","last_change":"2024-09-16T10:13:37.092457+0000","last_active":"2024-09-16T10:15:40.749027+0000","last_peered":"2024-09-16T10:15:40.749027+0000","last_clean":"2024-09-16T10:15:40.749027+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:15:40.749027+0000","last_undegraded":"2024-09-16T10:15:40.749027+0000","last_fullsized":"2024-09-16T10:15:40.749027+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:51.455656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":143,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.751320+0000","last_change":"2024-09-16T10:15:10.668770+0000","last_active":"2024-09-16T10:15:40.751320+0000","last_peered":"2024-09-16T10:15:40.751320+0000","last_clean":"2024-09-16T10:15:40.751320+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:15:40.751320+0000","last_undegraded":"2024-09-16T10:15:40.751320+0000","last_fullsized":"2024-09-16T10:15:40.751320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:15:10.668521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:10.668521+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:18.492671+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.734991+0000","last_change":"2024-09-16T10:13:41.121501+0000","last_active":"2024-09-16T10:15:40.734991+0000","last_peered":"2024-09-16T10:15:40.734991+0000","last_clean":"2024-09-16T10:15:40.734991+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:15:40.734991+0000","last_undegraded":"2024-09-16T10:15:40.734991+0000","last_fullsized":"2024-09-16T10:15:40.734991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.954155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754179+0000","last_change":"2024-09-16T10:15:29.351071+0000","last_active":"2024-09-16T10:15:40.754179+0000","last_peered":"2024-09-16T10:15:40.754179+0000","last_clean":"2024-09-16T10:15:40.754179+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:15:40.754179+0000","last_undegraded":"2024-09-16T10:15:40.754179+0000","last_fullsized":"2024-09-16T10:15:40.754179+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:29.351000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.351000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:52.980664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":133,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735304+0000","last_change":"2024-09-16T10:15:16.028344+0000","last_active":"2024-09-16T10:15:40.735304+0000","last_peered":"2024-09-16T10:15:40.735304+0000","last_clean":"2024-09-16T10:15:40.735304+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:15:40.735304+0000","last_undegraded":"2024-09-16T10:15:40.735304+0000","last_fullsized":"2024-09-16T10:15:40.735304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:16.028283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.028283+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.455289+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.753242+0000","last_change":"2024-09-16T10:15:22.271162+0000","last_active":"2024-09-16T10:15:40.753242+0000","last_peered":"2024-09-16T10:15:40.753242+0000","last_clean":"2024-09-16T10:15:40.753242+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:15:40.753242+0000","last_undegraded":"2024-09-16T10:15:40.753242+0000","last_fullsized":"2024-09-16T10:15:40.753242+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:22.271101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:22.271101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.480708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":109,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.761703+0000","last_change":"2024-09-16T10:15:32.193135+0000","last_active":"2024-09-16T10:15:40.761703+0000","last_peered":"2024-09-16T10:15:40.761703+0000","last_clean":"2024-09-16T10:15:40.761703+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:15:40.761703+0000","last_undegraded":"2024-09-16T10:15:40.761703+0000","last_fullsized":"2024-09-16T10:15:40.761703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:32.193067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:32.193067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:59.551498+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.753890+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:15:40.753890+0000","last_peered":"2024-09-16T10:15:40.753890+0000","last_clean":"2024-09-16T10:15:40.753890+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:15:40.753890+0000","last_undegraded":"2024-09-16T10:15:40.753890+0000","last_fullsized":"2024-09-16T10:15:40.753890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.232971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":119,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735150+0000","last_change":"2024-09-16T10:13:39.092696+0000","last_active":"2024-09-16T10:15:40.735150+0000","last_peered":"2024-09-16T10:15:40.735150+0000","last_clean":"2024-09-16T10:15:40.735150+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:15:40.735150+0000","last_undegraded":"2024-09-16T10:15:40.735150+0000","last_fullsized":"2024-09-16T10:15:40.735150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:54.990979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745583+0000","last_change":"2024-09-16T10:15:16.573685+0000","last_active":"2024-09-16T10:15:40.745583+0000","last_peered":"2024-09-16T10:15:40.745583+0000","last_clean":"2024-09-16T10:15:40.745583+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:15:40.745583+0000","last_undegraded":"2024-09-16T10:15:40.745583+0000","last_fullsized":"2024-09-16T10:15:40.745583+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.573614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.573614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.536483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752846+0000","last_change":"2024-09-16T10:15:11.297702+0000","last_active":"2024-09-16T10:15:40.752846+0000","last_peered":"2024-09-16T10:15:40.752846+0000","last_clean":"2024-09-16T10:15:40.752846+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:15:40.752846+0000","last_undegraded":"2024-09-16T10:15:40.752846+0000","last_fullsized":"2024-09-16T10:15:40.752846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.297599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.297599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.846847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735659+0000","last_change":"2024-09-16T10:13:43.284208+0000","last_active":"2024-09-16T10:15:40.735659+0000","last_peered":"2024-09-16T10:15:40.735659+0000","last_clean":"2024-09-16T10:15:40.735659+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:15:40.735659+0000","last_undegraded":"2024-09-16T10:15:40.735659+0000","last_fullsized":"2024-09-16T10:15:40.735659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.935116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.763110+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:15:40.763110+0000","last_peered":"2024-09-16T10:15:40.763110+0000","last_clean":"2024-09-16T10:15:40.763110+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:15:40.763110+0000","last_undegraded":"2024-09-16T10:15:40.763110+0000","last_fullsized":"2024-09-16T10:15:40.763110+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:50.979003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":135,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.739827+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:15:40.739827+0000","last_peered":"2024-09-16T10:15:40.739827+0000","last_clean":"2024-09-16T10:15:40.739827+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:15:40.739827+0000","last_undegraded":"2024-09-16T10:15:40.739827+0000","last_fullsized":"2024-09-16T10:15:40.739827+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:02.543443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752958+0000","last_change":"2024-09-16T10:15:10.987453+0000","last_active":"2024-09-16T10:15:40.752958+0000","last_peered":"2024-09-16T10:15:40.752958+0000","last_clean":"2024-09-16T10:15:40.752958+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:15:40.752958+0000","last_undegraded":"2024-09-16T10:15:40.752958+0000","last_fullsized":"2024-09-16T10:15:40.752958+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:10.987312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:10.987312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.809648+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752047+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:15:40.752047+0000","last_peered":"2024-09-16T10:15:40.752047+0000","last_clean":"2024-09-16T10:15:40.752047+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:15:40.752047+0000","last_undegraded":"2024-09-16T10:15:40.752047+0000","last_fullsized":"2024-09-16T10:15:40.752047+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735454+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:15:40.735454+0000","last_peered":"2024-09-16T10:15:40.735454+0000","last_clean":"2024-09-16T10:15:40.735454+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:15:40.735454+0000","last_undegraded":"2024-09-16T10:15:40.735454+0000","last_fullsized":"2024-09-16T10:15:40.735454+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:46.960702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":160,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.753362+0000","last_change":"2024-09-16T10:15:21.639445+0000","last_active":"2024-09-16T10:15:40.753362+0000","last_peered":"2024-09-16T10:15:40.753362+0000","last_clean":"2024-09-16T10:15:40.753362+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:15:40.753362+0000","last_undegraded":"2024-09-16T10:15:40.753362+0000","last_fullsized":"2024-09-16T10:15:40.753362+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:21.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.639361+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:30.197909+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735560+0000","last_change":"2024-09-16T10:15:21.032515+0000","last_active":"2024-09-16T10:15:40.735560+0000","last_peered":"2024-09-16T10:15:40.735560+0000","last_clean":"2024-09-16T10:15:40.735560+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:15:40.735560+0000","last_undegraded":"2024-09-16T10:15:40.735560+0000","last_fullsized":"2024-09-16T10:15:40.735560+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.032449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.032449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:21.047016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":103,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746108+0000","last_change":"2024-09-16T10:13:43.119664+0000","last_active":"2024-09-16T10:15:40.746108+0000","last_peered":"2024-09-16T10:15:40.746108+0000","last_clean":"2024-09-16T10:15:40.746108+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:15:40.746108+0000","last_undegraded":"2024-09-16T10:15:40.746108+0000","last_fullsized":"2024-09-16T10:15:40.746108+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:34.427312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":121,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738963+0000","last_change":"2024-09-16T10:13:37.082594+0000","last_active":"2024-09-16T10:15:40.738963+0000","last_peered":"2024-09-16T10:15:40.738963+0000","last_clean":"2024-09-16T10:15:40.738963+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:15:40.738963+0000","last_undegraded":"2024-09-16T10:15:40.738963+0000","last_fullsized":"2024-09-16T10:15:40.738963+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.024905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":133,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762520+0000","last_change":"2024-09-16T10:15:13.091323+0000","last_active":"2024-09-16T10:15:40.762520+0000","last_peered":"2024-09-16T10:15:40.762520+0000","last_clean":"2024-09-16T10:15:40.762520+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:15:40.762520+0000","last_undegraded":"2024-09-16T10:15:40.762520+0000","last_fullsized":"2024-09-16T10:15:40.762520+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:15:13.091193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.091193+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:26.457536+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":150,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.749068+0000","last_change":"2024-09-16T10:15:12.451691+0000","last_active":"2024-09-16T10:15:40.749068+0000","last_peered":"2024-09-16T10:15:40.749068+0000","last_clean":"2024-09-16T10:15:40.749068+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:15:40.749068+0000","last_undegraded":"2024-09-16T10:15:40.749068+0000","last_fullsized":"2024-09-16T10:15:40.749068+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:12.451619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:12.451619+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:31.691482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"110'5366","reported_seq":5435,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.005243+0000","last_change":"2024-09-16T10:14:43.957473+0000","last_active":"2024-09-16T10:15:43.005243+0000","last_peered":"2024-09-16T10:15:43.005243+0000","last_clean":"2024-09-16T10:15:43.005243+0000","last_became_active":"2024-09-16T10:14:43.957174+0000","last_became_peered":"2024-09-16T10:14:43.957174+0000","last_unstale":"2024-09-16T10:15:43.005243+0000","last_undegraded":"2024-09-16T10:15:43.005243+0000","last_fullsized":"2024-09-16T10:15:43.005243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:45.621994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351666176,"num_objects":5366,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":343424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":143,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738653+0000","last_change":"2024-09-16T10:14:45.136557+0000","last_active":"2024-09-16T10:15:40.738653+0000","last_peered":"2024-09-16T10:15:40.738653+0000","last_clean":"2024-09-16T10:15:40.738653+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:15:40.738653+0000","last_undegraded":"2024-09-16T10:15:40.738653+0000","last_fullsized":"2024-09-16T10:15:40.738653+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:14:45.136460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.136460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:45.551571+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.750832+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:15:40.750832+0000","last_peered":"2024-09-16T10:15:40.750832+0000","last_clean":"2024-09-16T10:15:40.750832+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:15:40.750832+0000","last_undegraded":"2024-09-16T10:15:40.750832+0000","last_fullsized":"2024-09-16T10:15:40.750832+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.842587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":139,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735780+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:15:40.735780+0000","last_peered":"2024-09-16T10:15:40.735780+0000","last_clean":"2024-09-16T10:15:40.735780+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:15:40.735780+0000","last_undegraded":"2024-09-16T10:15:40.735780+0000","last_fullsized":"2024-09-16T10:15:40.735780+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.097659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.753824+0000","last_change":"2024-09-16T10:13:43.114886+0000","last_active":"2024-09-16T10:15:40.753824+0000","last_peered":"2024-09-16T10:15:40.753824+0000","last_clean":"2024-09-16T10:15:40.753824+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:15:40.753824+0000","last_undegraded":"2024-09-16T10:15:40.753824+0000","last_fullsized":"2024-09-16T10:15:40.753824+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:51.280322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"110'5167","reported_seq":5236,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.062445+0000","last_change":"2024-09-16T10:14:43.953298+0000","last_active":"2024-09-16T10:15:43.062445+0000","last_peered":"2024-09-16T10:15:43.062445+0000","last_clean":"2024-09-16T10:15:43.062445+0000","last_became_active":"2024-09-16T10:14:43.952713+0000","last_became_peered":"2024-09-16T10:14:43.952713+0000","last_unstale":"2024-09-16T10:15:43.062445+0000","last_undegraded":"2024-09-16T10:15:43.062445+0000","last_fullsized":"2024-09-16T10:15:43.062445+0000","mapping_epoch":81,"log_start":"89'500","ondisk_log_start":"89'500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:50.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338624512,"num_objects":5167,"num_object_clones":0,"num_object_copies":15501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":330688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":151,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.750624+0000","last_change":"2024-09-16T10:15:23.703368+0000","last_active":"2024-09-16T10:15:40.750624+0000","last_peered":"2024-09-16T10:15:40.750624+0000","last_clean":"2024-09-16T10:15:40.750624+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:15:40.750624+0000","last_undegraded":"2024-09-16T10:15:40.750624+0000","last_fullsized":"2024-09-16T10:15:40.750624+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:15:23.703278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.703278+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.263361+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752201+0000","last_change":"2024-09-16T10:13:37.093660+0000","last_active":"2024-09-16T10:15:40.752201+0000","last_peered":"2024-09-16T10:15:40.752201+0000","last_clean":"2024-09-16T10:15:40.752201+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:15:40.752201+0000","last_undegraded":"2024-09-16T10:15:40.752201+0000","last_fullsized":"2024-09-16T10:15:40.752201+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.591570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735143+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:15:40.735143+0000","last_peered":"2024-09-16T10:15:40.735143+0000","last_clean":"2024-09-16T10:15:40.735143+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:15:40.735143+0000","last_undegraded":"2024-09-16T10:15:40.735143+0000","last_fullsized":"2024-09-16T10:15:40.735143+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:02.988717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":109,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735286+0000","last_change":"2024-09-16T10:15:12.079425+0000","last_active":"2024-09-16T10:15:40.735286+0000","last_peered":"2024-09-16T10:15:40.735286+0000","last_clean":"2024-09-16T10:15:40.735286+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:15:40.735286+0000","last_undegraded":"2024-09-16T10:15:40.735286+0000","last_fullsized":"2024-09-16T10:15:40.735286+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:12.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:12.079353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.080838+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"110'5161","reported_seq":5230,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.069379+0000","last_change":"2024-09-16T10:14:43.953541+0000","last_active":"2024-09-16T10:15:43.069379+0000","last_peered":"2024-09-16T10:15:43.069379+0000","last_clean":"2024-09-16T10:15:43.069379+0000","last_became_active":"2024-09-16T10:14:43.952908+0000","last_became_peered":"2024-09-16T10:14:43.952908+0000","last_unstale":"2024-09-16T10:15:43.069379+0000","last_undegraded":"2024-09-16T10:15:43.069379+0000","last_fullsized":"2024-09-16T10:15:43.069379+0000","mapping_epoch":81,"log_start":"89'500","ondisk_log_start":"89'500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.479727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338231296,"num_objects":5161,"num_object_clones":0,"num_object_copies":15483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":330304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":139,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735217+0000","last_change":"2024-09-16T10:15:17.025884+0000","last_active":"2024-09-16T10:15:40.735217+0000","last_peered":"2024-09-16T10:15:40.735217+0000","last_clean":"2024-09-16T10:15:40.735217+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:15:40.735217+0000","last_undegraded":"2024-09-16T10:15:40.735217+0000","last_fullsized":"2024-09-16T10:15:40.735217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:15:17.025825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.025825+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.846848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762793+0000","last_change":"2024-09-16T10:15:29.146696+0000","last_active":"2024-09-16T10:15:40.762793+0000","last_peered":"2024-09-16T10:15:40.762793+0000","last_clean":"2024-09-16T10:15:40.762793+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:15:40.762793+0000","last_undegraded":"2024-09-16T10:15:40.762793+0000","last_fullsized":"2024-09-16T10:15:40.762793+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:29.146482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.146482+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:56.251083+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":152,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:42.175789+0000","last_change":"2024-09-16T10:15:20.675679+0000","last_active":"2024-09-16T10:15:42.175789+0000","last_peered":"2024-09-16T10:15:42.175789+0000","last_clean":"2024-09-16T10:15:42.175789+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:15:42.175789+0000","last_undegraded":"2024-09-16T10:15:42.175789+0000","last_fullsized":"2024-09-16T10:15:42.175789+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:20.675600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:20.675600+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.817505+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745724+0000","last_change":"2024-09-16T10:13:43.284631+0000","last_active":"2024-09-16T10:15:40.745724+0000","last_peered":"2024-09-16T10:15:40.745724+0000","last_clean":"2024-09-16T10:15:40.745724+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:15:40.745724+0000","last_undegraded":"2024-09-16T10:15:40.745724+0000","last_fullsized":"2024-09-16T10:15:40.745724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.518019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"110'5299","reported_seq":5368,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.011439+0000","last_change":"2024-09-16T10:14:43.952332+0000","last_active":"2024-09-16T10:15:43.011439+0000","last_peered":"2024-09-16T10:15:43.011439+0000","last_clean":"2024-09-16T10:15:43.011439+0000","last_became_active":"2024-09-16T10:14:43.952043+0000","last_became_peered":"2024-09-16T10:14:43.952043+0000","last_unstale":"2024-09-16T10:15:43.011439+0000","last_undegraded":"2024-09-16T10:15:43.011439+0000","last_fullsized":"2024-09-16T10:15:43.011439+0000","mapping_epoch":81,"log_start":"89'500","ondisk_log_start":"89'500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.379216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347275264,"num_objects":5299,"num_object_clones":0,"num_object_copies":15897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":339136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":133,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.747143+0000","last_change":"2024-09-16T10:14:42.119372+0000","last_active":"2024-09-16T10:15:40.747143+0000","last_peered":"2024-09-16T10:15:40.747143+0000","last_clean":"2024-09-16T10:15:40.747143+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:15:40.747143+0000","last_undegraded":"2024-09-16T10:15:40.747143+0000","last_fullsized":"2024-09-16T10:15:40.747143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:14:42.119310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.119310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.359829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754266+0000","last_change":"2024-09-16T10:15:23.304088+0000","last_active":"2024-09-16T10:15:40.754266+0000","last_peered":"2024-09-16T10:15:40.754266+0000","last_clean":"2024-09-16T10:15:40.754266+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:15:40.754266+0000","last_undegraded":"2024-09-16T10:15:40.754266+0000","last_fullsized":"2024-09-16T10:15:40.754266+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:23.304033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.304033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.813644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":147,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762705+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:15:40.762705+0000","last_peered":"2024-09-16T10:15:40.762705+0000","last_clean":"2024-09-16T10:15:40.762705+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:15:40.762705+0000","last_undegraded":"2024-09-16T10:15:40.762705+0000","last_fullsized":"2024-09-16T10:15:40.762705+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.731855+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":109,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.761933+0000","last_change":"2024-09-16T10:15:16.104168+0000","last_active":"2024-09-16T10:15:40.761933+0000","last_peered":"2024-09-16T10:15:40.761933+0000","last_clean":"2024-09-16T10:15:40.761933+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:15:40.761933+0000","last_undegraded":"2024-09-16T10:15:40.761933+0000","last_fullsized":"2024-09-16T10:15:40.761933+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.104092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.104092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:41.867492+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"110'5089","reported_seq":5158,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:42.966167+0000","last_change":"2024-09-16T10:14:43.956078+0000","last_active":"2024-09-16T10:15:42.966167+0000","last_peered":"2024-09-16T10:15:42.966167+0000","last_clean":"2024-09-16T10:15:42.966167+0000","last_became_active":"2024-09-16T10:14:43.955808+0000","last_became_peered":"2024-09-16T10:14:43.955808+0000","last_unstale":"2024-09-16T10:15:42.966167+0000","last_undegraded":"2024-09-16T10:15:42.966167+0000","last_fullsized":"2024-09-16T10:15:42.966167+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:54.992452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333512704,"num_objects":5089,"num_object_clones":0,"num_object_copies":15267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":325696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":154,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762905+0000","last_change":"2024-09-16T10:15:26.113170+0000","last_active":"2024-09-16T10:15:40.762905+0000","last_peered":"2024-09-16T10:15:40.762905+0000","last_clean":"2024-09-16T10:15:40.762905+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:15:40.762905+0000","last_undegraded":"2024-09-16T10:15:40.762905+0000","last_fullsized":"2024-09-16T10:15:40.762905+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:15:26.110570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:26.110570+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.337507+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":123,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.739299+0000","last_change":"2024-09-16T10:15:18.059218+0000","last_active":"2024-09-16T10:15:40.739299+0000","last_peered":"2024-09-16T10:15:40.739299+0000","last_clean":"2024-09-16T10:15:40.739299+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:15:40.739299+0000","last_undegraded":"2024-09-16T10:15:40.739299+0000","last_fullsized":"2024-09-16T10:15:40.739299+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:18.059145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:18.059145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.142283+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748324+0000","last_change":"2024-09-16T10:13:41.114703+0000","last_active":"2024-09-16T10:15:40.748324+0000","last_peered":"2024-09-16T10:15:40.748324+0000","last_clean":"2024-09-16T10:15:40.748324+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:15:40.748324+0000","last_undegraded":"2024-09-16T10:15:40.748324+0000","last_fullsized":"2024-09-16T10:15:40.748324+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.378226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738729+0000","last_change":"2024-09-16T10:13:43.283556+0000","last_active":"2024-09-16T10:15:40.738729+0000","last_peered":"2024-09-16T10:15:40.738729+0000","last_clean":"2024-09-16T10:15:40.738729+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:15:40.738729+0000","last_undegraded":"2024-09-16T10:15:40.738729+0000","last_fullsized":"2024-09-16T10:15:40.738729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.117062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"110'5236","reported_seq":5305,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.001949+0000","last_change":"2024-09-16T10:14:43.957466+0000","last_active":"2024-09-16T10:15:43.001949+0000","last_peered":"2024-09-16T10:15:43.001949+0000","last_clean":"2024-09-16T10:15:43.001949+0000","last_became_active":"2024-09-16T10:14:43.957195+0000","last_became_peered":"2024-09-16T10:14:43.957195+0000","last_unstale":"2024-09-16T10:15:43.001949+0000","last_undegraded":"2024-09-16T10:15:43.001949+0000","last_fullsized":"2024-09-16T10:15:43.001949+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":335104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":192,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.739225+0000","last_change":"2024-09-16T10:14:41.095857+0000","last_active":"2024-09-16T10:15:40.739225+0000","last_peered":"2024-09-16T10:15:40.739225+0000","last_clean":"2024-09-16T10:15:40.739225+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:15:40.739225+0000","last_undegraded":"2024-09-16T10:15:40.739225+0000","last_fullsized":"2024-09-16T10:15:40.739225+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:14:41.095779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:41.095779+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:53.960257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735426+0000","last_change":"2024-09-16T10:14:44.986853+0000","last_active":"2024-09-16T10:15:40.735426+0000","last_peered":"2024-09-16T10:15:40.735426+0000","last_clean":"2024-09-16T10:15:40.735426+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:15:40.735426+0000","last_undegraded":"2024-09-16T10:15:40.735426+0000","last_fullsized":"2024-09-16T10:15:40.735426+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:44.986805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.986805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.455538+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":125,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.753271+0000","last_change":"2024-09-16T10:13:39.091269+0000","last_active":"2024-09-16T10:15:40.753271+0000","last_peered":"2024-09-16T10:15:40.753271+0000","last_clean":"2024-09-16T10:15:40.753271+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:15:40.753271+0000","last_undegraded":"2024-09-16T10:15:40.753271+0000","last_fullsized":"2024-09-16T10:15:40.753271+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:06.263288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":105,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.764275+0000","last_change":"2024-09-16T10:13:41.118142+0000","last_active":"2024-09-16T10:15:40.764275+0000","last_peered":"2024-09-16T10:15:40.764275+0000","last_clean":"2024-09-16T10:15:40.764275+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:15:40.764275+0000","last_undegraded":"2024-09-16T10:15:40.764275+0000","last_fullsized":"2024-09-16T10:15:40.764275+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.036482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748759+0000","last_change":"2024-09-16T10:13:43.124755+0000","last_active":"2024-09-16T10:15:40.748759+0000","last_peered":"2024-09-16T10:15:40.748759+0000","last_clean":"2024-09-16T10:15:40.748759+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:15:40.748759+0000","last_undegraded":"2024-09-16T10:15:40.748759+0000","last_fullsized":"2024-09-16T10:15:40.748759+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"110'5161","reported_seq":5230,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.012399+0000","last_change":"2024-09-16T10:14:43.956747+0000","last_active":"2024-09-16T10:15:43.012399+0000","last_peered":"2024-09-16T10:15:43.012399+0000","last_clean":"2024-09-16T10:15:43.012399+0000","last_became_active":"2024-09-16T10:14:43.956362+0000","last_became_peered":"2024-09-16T10:14:43.956362+0000","last_unstale":"2024-09-16T10:15:43.012399+0000","last_undegraded":"2024-09-16T10:15:43.012399+0000","last_fullsized":"2024-09-16T10:15:43.012399+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:43.324827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338231296,"num_objects":5161,"num_object_clones":0,"num_object_copies":15483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":330304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":137,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738754+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:15:40.738754+0000","last_peered":"2024-09-16T10:15:40.738754+0000","last_clean":"2024-09-16T10:15:40.738754+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:15:40.738754+0000","last_undegraded":"2024-09-16T10:15:40.738754+0000","last_fullsized":"2024-09-16T10:15:40.738754+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.052025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":115,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738340+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:15:40.738340+0000","last_peered":"2024-09-16T10:15:40.738340+0000","last_clean":"2024-09-16T10:15:40.738340+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:15:40.738340+0000","last_undegraded":"2024-09-16T10:15:40.738340+0000","last_fullsized":"2024-09-16T10:15:40.738340+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.073692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":132,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.740063+0000","last_change":"2024-09-16T10:15:13.676523+0000","last_active":"2024-09-16T10:15:40.740063+0000","last_peered":"2024-09-16T10:15:40.740063+0000","last_clean":"2024-09-16T10:15:40.740063+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:15:40.740063+0000","last_undegraded":"2024-09-16T10:15:40.740063+0000","last_fullsized":"2024-09-16T10:15:40.740063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:13.676437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.676437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.794617+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.739973+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:15:40.739973+0000","last_peered":"2024-09-16T10:15:40.739973+0000","last_clean":"2024-09-16T10:15:40.739973+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:15:40.739973+0000","last_undegraded":"2024-09-16T10:15:40.739973+0000","last_fullsized":"2024-09-16T10:15:40.739973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.448883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746475+0000","last_change":"2024-09-16T10:13:43.283490+0000","last_active":"2024-09-16T10:15:40.746475+0000","last_peered":"2024-09-16T10:15:40.746475+0000","last_clean":"2024-09-16T10:15:40.746475+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:15:40.746475+0000","last_undegraded":"2024-09-16T10:15:40.746475+0000","last_fullsized":"2024-09-16T10:15:40.746475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:06.569452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"110'5239","reported_seq":5308,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.041642+0000","last_change":"2024-09-16T10:14:43.959854+0000","last_active":"2024-09-16T10:15:43.041642+0000","last_peered":"2024-09-16T10:15:43.041642+0000","last_clean":"2024-09-16T10:15:43.041642+0000","last_became_active":"2024-09-16T10:14:43.959612+0000","last_became_peered":"2024-09-16T10:14:43.959612+0000","last_unstale":"2024-09-16T10:15:43.041642+0000","last_undegraded":"2024-09-16T10:15:43.041642+0000","last_fullsized":"2024-09-16T10:15:43.041642+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.134999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343343104,"num_objects":5239,"num_object_clones":0,"num_object_copies":15717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":335296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":135,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762582+0000","last_change":"2024-09-16T10:14:44.905004+0000","last_active":"2024-09-16T10:15:40.762582+0000","last_peered":"2024-09-16T10:15:40.762582+0000","last_clean":"2024-09-16T10:15:40.762582+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:15:40.762582+0000","last_undegraded":"2024-09-16T10:15:40.762582+0000","last_fullsized":"2024-09-16T10:15:40.762582+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:14:44.904949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.904949+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.392628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.761837+0000","last_change":"2024-09-16T10:14:39.888045+0000","last_active":"2024-09-16T10:15:40.761837+0000","last_peered":"2024-09-16T10:15:40.761837+0000","last_clean":"2024-09-16T10:15:40.761837+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:15:40.761837+0000","last_undegraded":"2024-09-16T10:15:40.761837+0000","last_fullsized":"2024-09-16T10:15:40.761837+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:39.887993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:39.887993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.458794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":254,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:41.153206+0000","last_change":"2024-09-16T10:14:36.812774+0000","last_active":"2024-09-16T10:15:41.153206+0000","last_peered":"2024-09-16T10:15:41.153206+0000","last_clean":"2024-09-16T10:15:41.153206+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:15:41.153206+0000","last_undegraded":"2024-09-16T10:15:41.153206+0000","last_fullsized":"2024-09-16T10:15:41.153206+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:14:36.812720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:14:36.812720+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.775101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":115,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754695+0000","last_change":"2024-09-16T10:13:41.112756+0000","last_active":"2024-09-16T10:15:40.754695+0000","last_peered":"2024-09-16T10:15:40.754695+0000","last_clean":"2024-09-16T10:15:40.754695+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:15:40.754695+0000","last_undegraded":"2024-09-16T10:15:40.754695+0000","last_fullsized":"2024-09-16T10:15:40.754695+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.271006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738898+0000","last_change":"2024-09-16T10:13:43.123048+0000","last_active":"2024-09-16T10:15:40.738898+0000","last_peered":"2024-09-16T10:15:40.738898+0000","last_clean":"2024-09-16T10:15:40.738898+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:15:40.738898+0000","last_undegraded":"2024-09-16T10:15:40.738898+0000","last_fullsized":"2024-09-16T10:15:40.738898+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:03.084010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"110'5301","reported_seq":5370,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.029868+0000","last_change":"2024-09-16T10:14:43.957718+0000","last_active":"2024-09-16T10:15:43.029868+0000","last_peered":"2024-09-16T10:15:43.029868+0000","last_clean":"2024-09-16T10:15:43.029868+0000","last_became_active":"2024-09-16T10:14:43.957476+0000","last_became_peered":"2024-09-16T10:14:43.957476+0000","last_unstale":"2024-09-16T10:15:43.029868+0000","last_undegraded":"2024-09-16T10:15:43.029868+0000","last_fullsized":"2024-09-16T10:15:43.029868+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5301,"log_dups_size":0,"ondisk_log_size":5301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:45.068072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347406336,"num_objects":5301,"num_object_clones":0,"num_object_copies":15903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":339264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":157,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748884+0000","last_change":"2024-09-16T10:13:39.106378+0000","last_active":"2024-09-16T10:15:40.748884+0000","last_peered":"2024-09-16T10:15:40.748884+0000","last_clean":"2024-09-16T10:15:40.748884+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:15:40.748884+0000","last_undegraded":"2024-09-16T10:15:40.748884+0000","last_fullsized":"2024-09-16T10:15:40.748884+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.385090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":123,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.757881+0000","last_change":"2024-09-16T10:14:37.299040+0000","last_active":"2024-09-16T10:15:40.757881+0000","last_peered":"2024-09-16T10:15:40.757881+0000","last_clean":"2024-09-16T10:15:40.757881+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:15:40.757881+0000","last_undegraded":"2024-09-16T10:15:40.757881+0000","last_fullsized":"2024-09-16T10:15:40.757881+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.298992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.298992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.526394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":160,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.747262+0000","last_change":"2024-09-16T10:14:14.911115+0000","last_active":"2024-09-16T10:15:40.747262+0000","last_peered":"2024-09-16T10:15:40.747262+0000","last_clean":"2024-09-16T10:15:40.747262+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:15:40.747262+0000","last_undegraded":"2024-09-16T10:15:40.747262+0000","last_fullsized":"2024-09-16T10:15:40.747262+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:14:14.911041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:14:14.911041+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746206+0000","last_change":"2024-09-16T10:15:26.571995+0000","last_active":"2024-09-16T10:15:40.746206+0000","last_peered":"2024-09-16T10:15:40.746206+0000","last_clean":"2024-09-16T10:15:40.746206+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:15:40.746206+0000","last_undegraded":"2024-09-16T10:15:40.746206+0000","last_fullsized":"2024-09-16T10:15:40.746206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:26.571945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:26.571945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.414009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754520+0000","last_change":"2024-09-16T10:13:43.121658+0000","last_active":"2024-09-16T10:15:40.754520+0000","last_peered":"2024-09-16T10:15:40.754520+0000","last_clean":"2024-09-16T10:15:40.754520+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:15:40.754520+0000","last_undegraded":"2024-09-16T10:15:40.754520+0000","last_fullsized":"2024-09-16T10:15:40.754520+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.316999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"110'5252","reported_seq":5320,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.045752+0000","last_change":"2024-09-16T10:14:43.958855+0000","last_active":"2024-09-16T10:15:43.045752+0000","last_peered":"2024-09-16T10:15:43.045752+0000","last_clean":"2024-09-16T10:15:43.045752+0000","last_became_active":"2024-09-16T10:14:43.958616+0000","last_became_peered":"2024-09-16T10:14:43.958616+0000","last_unstale":"2024-09-16T10:15:43.045752+0000","last_undegraded":"2024-09-16T10:15:43.045752+0000","last_fullsized":"2024-09-16T10:15:43.045752+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.711102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344195072,"num_objects":5252,"num_object_clones":0,"num_object_copies":15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5252,"num_write_kb":336128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":136,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735768+0000","last_change":"2024-09-16T10:14:40.987310+0000","last_active":"2024-09-16T10:15:40.735768+0000","last_peered":"2024-09-16T10:15:40.735768+0000","last_clean":"2024-09-16T10:15:40.735768+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:15:40.735768+0000","last_undegraded":"2024-09-16T10:15:40.735768+0000","last_fullsized":"2024-09-16T10:15:40.735768+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:14:40.987224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:40.987224+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.140235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735430+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:15:40.735430+0000","last_peered":"2024-09-16T10:15:40.735430+0000","last_clean":"2024-09-16T10:15:40.735430+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:15:40.735430+0000","last_undegraded":"2024-09-16T10:15:40.735430+0000","last_fullsized":"2024-09-16T10:15:40.735430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.991258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754450+0000","last_change":"2024-09-16T10:15:30.302921+0000","last_active":"2024-09-16T10:15:40.754450+0000","last_peered":"2024-09-16T10:15:40.754450+0000","last_clean":"2024-09-16T10:15:40.754450+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:15:40.754450+0000","last_undegraded":"2024-09-16T10:15:40.754450+0000","last_fullsized":"2024-09-16T10:15:40.754450+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.302859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.302859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.575611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.753725+0000","last_change":"2024-09-16T10:15:25.689754+0000","last_active":"2024-09-16T10:15:40.753725+0000","last_peered":"2024-09-16T10:15:40.753725+0000","last_clean":"2024-09-16T10:15:40.753725+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:15:40.753725+0000","last_undegraded":"2024-09-16T10:15:40.753725+0000","last_fullsized":"2024-09-16T10:15:40.753725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:25.689615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.689615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.500134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"110'5103","reported_seq":5170,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.070355+0000","last_change":"2024-09-16T10:14:43.954148+0000","last_active":"2024-09-16T10:15:43.070355+0000","last_peered":"2024-09-16T10:15:43.070355+0000","last_clean":"2024-09-16T10:15:43.070355+0000","last_became_active":"2024-09-16T10:14:43.953871+0000","last_became_peered":"2024-09-16T10:14:43.953871+0000","last_unstale":"2024-09-16T10:15:43.070355+0000","last_undegraded":"2024-09-16T10:15:43.070355+0000","last_fullsized":"2024-09-16T10:15:43.070355+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5103,"log_dups_size":0,"ondisk_log_size":5103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.930149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334430208,"num_objects":5103,"num_object_clones":0,"num_object_copies":15309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5103,"num_write_kb":326592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":137,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735900+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:15:40.735900+0000","last_peered":"2024-09-16T10:15:40.735900+0000","last_clean":"2024-09-16T10:15:40.735900+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:15:40.735900+0000","last_undegraded":"2024-09-16T10:15:40.735900+0000","last_fullsized":"2024-09-16T10:15:40.735900+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.029277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748534+0000","last_change":"2024-09-16T10:13:37.089527+0000","last_active":"2024-09-16T10:15:40.748534+0000","last_peered":"2024-09-16T10:15:40.748534+0000","last_clean":"2024-09-16T10:15:40.748534+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:15:40.748534+0000","last_undegraded":"2024-09-16T10:15:40.748534+0000","last_fullsized":"2024-09-16T10:15:40.748534+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.738991+0000","last_change":"2024-09-16T10:13:41.121389+0000","last_active":"2024-09-16T10:15:40.738991+0000","last_peered":"2024-09-16T10:15:40.738991+0000","last_clean":"2024-09-16T10:15:40.738991+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:15:40.738991+0000","last_undegraded":"2024-09-16T10:15:40.738991+0000","last_fullsized":"2024-09-16T10:15:40.738991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:53.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.740010+0000","last_change":"2024-09-16T10:15:11.699384+0000","last_active":"2024-09-16T10:15:40.740010+0000","last_peered":"2024-09-16T10:15:40.740010+0000","last_clean":"2024-09-16T10:15:40.740010+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:15:40.740010+0000","last_undegraded":"2024-09-16T10:15:40.740010+0000","last_fullsized":"2024-09-16T10:15:40.740010+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.699308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.699308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.370786+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"110'5041","reported_seq":5109,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.039218+0000","last_change":"2024-09-16T10:14:43.955268+0000","last_active":"2024-09-16T10:15:43.039218+0000","last_peered":"2024-09-16T10:15:43.039218+0000","last_clean":"2024-09-16T10:15:43.039218+0000","last_became_active":"2024-09-16T10:14:43.954996+0000","last_became_peered":"2024-09-16T10:14:43.954996+0000","last_unstale":"2024-09-16T10:15:43.039218+0000","last_undegraded":"2024-09-16T10:15:43.039218+0000","last_fullsized":"2024-09-16T10:15:43.039218+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5041,"log_dups_size":0,"ondisk_log_size":5041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.950717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330366976,"num_objects":5041,"num_object_clones":0,"num_object_copies":15123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5041,"num_write_kb":322624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":124,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735606+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:15:40.735606+0000","last_peered":"2024-09-16T10:15:40.735606+0000","last_clean":"2024-09-16T10:15:40.735606+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:15:40.735606+0000","last_undegraded":"2024-09-16T10:15:40.735606+0000","last_fullsized":"2024-09-16T10:15:40.735606+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:50.001374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.749320+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:15:40.749320+0000","last_peered":"2024-09-16T10:15:40.749320+0000","last_clean":"2024-09-16T10:15:40.749320+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:15:40.749320+0000","last_undegraded":"2024-09-16T10:15:40.749320+0000","last_fullsized":"2024-09-16T10:15:40.749320+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.481976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":140,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.740021+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:15:40.740021+0000","last_peered":"2024-09-16T10:15:40.740021+0000","last_clean":"2024-09-16T10:15:40.740021+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:15:40.740021+0000","last_undegraded":"2024-09-16T10:15:40.740021+0000","last_fullsized":"2024-09-16T10:15:40.740021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.466166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735574+0000","last_change":"2024-09-16T10:13:43.119606+0000","last_active":"2024-09-16T10:15:40.735574+0000","last_peered":"2024-09-16T10:15:40.735574+0000","last_clean":"2024-09-16T10:15:40.735574+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:15:40.735574+0000","last_undegraded":"2024-09-16T10:15:40.735574+0000","last_fullsized":"2024-09-16T10:15:40.735574+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:56.977559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"110'5215","reported_seq":5284,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.049883+0000","last_change":"2024-09-16T10:14:43.952726+0000","last_active":"2024-09-16T10:15:43.049883+0000","last_peered":"2024-09-16T10:15:43.049883+0000","last_clean":"2024-09-16T10:15:43.049883+0000","last_became_active":"2024-09-16T10:14:43.952378+0000","last_became_peered":"2024-09-16T10:14:43.952378+0000","last_unstale":"2024-09-16T10:15:43.049883+0000","last_undegraded":"2024-09-16T10:15:43.049883+0000","last_fullsized":"2024-09-16T10:15:43.049883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:56.742869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341770240,"num_objects":5215,"num_object_clones":0,"num_object_copies":15645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":333760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":133,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762757+0000","last_change":"2024-09-16T10:15:27.103866+0000","last_active":"2024-09-16T10:15:40.762757+0000","last_peered":"2024-09-16T10:15:40.762757+0000","last_clean":"2024-09-16T10:15:40.762757+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:15:40.762757+0000","last_undegraded":"2024-09-16T10:15:40.762757+0000","last_fullsized":"2024-09-16T10:15:40.762757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:27.103781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:27.103781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.276258+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754082+0000","last_change":"2024-09-16T10:15:13.340979+0000","last_active":"2024-09-16T10:15:40.754082+0000","last_peered":"2024-09-16T10:15:40.754082+0000","last_clean":"2024-09-16T10:15:40.754082+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:15:40.754082+0000","last_undegraded":"2024-09-16T10:15:40.754082+0000","last_fullsized":"2024-09-16T10:15:40.754082+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.340929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.340929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:15.076648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746317+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:15:40.746317+0000","last_peered":"2024-09-16T10:15:40.746317+0000","last_clean":"2024-09-16T10:15:40.746317+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:15:40.746317+0000","last_undegraded":"2024-09-16T10:15:40.746317+0000","last_fullsized":"2024-09-16T10:15:40.746317+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:50.536465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.761681+0000","last_change":"2024-09-16T10:14:45.941203+0000","last_active":"2024-09-16T10:15:40.761681+0000","last_peered":"2024-09-16T10:15:40.761681+0000","last_clean":"2024-09-16T10:15:40.761681+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:15:40.761681+0000","last_undegraded":"2024-09-16T10:15:40.761681+0000","last_fullsized":"2024-09-16T10:15:40.761681+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:14:45.941146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.941146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.998588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"110'5094","reported_seq":5163,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.012404+0000","last_change":"2024-09-16T10:14:43.960133+0000","last_active":"2024-09-16T10:15:43.012404+0000","last_peered":"2024-09-16T10:15:43.012404+0000","last_clean":"2024-09-16T10:15:43.012404+0000","last_became_active":"2024-09-16T10:14:43.959635+0000","last_became_peered":"2024-09-16T10:14:43.959635+0000","last_unstale":"2024-09-16T10:15:43.012404+0000","last_undegraded":"2024-09-16T10:15:43.012404+0000","last_fullsized":"2024-09-16T10:15:43.012404+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.705141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333840384,"num_objects":5094,"num_object_clones":0,"num_object_copies":15282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5094,"num_write_kb":326016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":148,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748339+0000","last_change":"2024-09-16T10:15:24.413737+0000","last_active":"2024-09-16T10:15:40.748339+0000","last_peered":"2024-09-16T10:15:40.748339+0000","last_clean":"2024-09-16T10:15:40.748339+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:15:40.748339+0000","last_undegraded":"2024-09-16T10:15:40.748339+0000","last_fullsized":"2024-09-16T10:15:40.748339+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:15:24.413675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.413675+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.734723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746777+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:15:40.746777+0000","last_peered":"2024-09-16T10:15:40.746777+0000","last_clean":"2024-09-16T10:15:40.746777+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:15:40.746777+0000","last_undegraded":"2024-09-16T10:15:40.746777+0000","last_fullsized":"2024-09-16T10:15:40.746777+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.009197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745173+0000","last_change":"2024-09-16T10:14:43.459469+0000","last_active":"2024-09-16T10:15:40.745173+0000","last_peered":"2024-09-16T10:15:40.745173+0000","last_clean":"2024-09-16T10:15:40.745173+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:15:40.745173+0000","last_undegraded":"2024-09-16T10:15:40.745173+0000","last_fullsized":"2024-09-16T10:15:40.745173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:43.459419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.459419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.271593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.747237+0000","last_change":"2024-09-16T10:13:43.122094+0000","last_active":"2024-09-16T10:15:40.747237+0000","last_peered":"2024-09-16T10:15:40.747237+0000","last_clean":"2024-09-16T10:15:40.747237+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:15:40.747237+0000","last_undegraded":"2024-09-16T10:15:40.747237+0000","last_fullsized":"2024-09-16T10:15:40.747237+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.656883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"110'5103","reported_seq":5172,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:43.064925+0000","last_change":"2024-09-16T10:14:43.957583+0000","last_active":"2024-09-16T10:15:43.064925+0000","last_peered":"2024-09-16T10:15:43.064925+0000","last_clean":"2024-09-16T10:15:43.064925+0000","last_became_active":"2024-09-16T10:14:43.957207+0000","last_became_peered":"2024-09-16T10:14:43.957207+0000","last_unstale":"2024-09-16T10:15:43.064925+0000","last_undegraded":"2024-09-16T10:15:43.064925+0000","last_fullsized":"2024-09-16T10:15:43.064925+0000","mapping_epoch":81,"log_start":"89'400","ondisk_log_start":"89'400","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.367126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334430208,"num_objects":5103,"num_object_clones":0,"num_object_copies":15309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5103,"num_write_kb":326592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":140,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735326+0000","last_change":"2024-09-16T10:15:15.038399+0000","last_active":"2024-09-16T10:15:40.735326+0000","last_peered":"2024-09-16T10:15:40.735326+0000","last_clean":"2024-09-16T10:15:40.735326+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:15:40.735326+0000","last_undegraded":"2024-09-16T10:15:40.735326+0000","last_fullsized":"2024-09-16T10:15:40.735326+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:15.038322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.038322+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:16.384094+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746574+0000","last_change":"2024-09-16T10:15:11.631330+0000","last_active":"2024-09-16T10:15:40.746574+0000","last_peered":"2024-09-16T10:15:40.746574+0000","last_clean":"2024-09-16T10:15:40.746574+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:15:40.746574+0000","last_undegraded":"2024-09-16T10:15:40.746574+0000","last_fullsized":"2024-09-16T10:15:40.746574+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.631262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.631262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.886243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.739741+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:15:40.739741+0000","last_peered":"2024-09-16T10:15:40.739741+0000","last_clean":"2024-09-16T10:15:40.739741+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:15:40.739741+0000","last_undegraded":"2024-09-16T10:15:40.739741+0000","last_fullsized":"2024-09-16T10:15:40.739741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.452518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.749046+0000","last_change":"2024-09-16T10:15:18.722246+0000","last_active":"2024-09-16T10:15:40.749046+0000","last_peered":"2024-09-16T10:15:40.749046+0000","last_clean":"2024-09-16T10:15:40.749046+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:15:40.749046+0000","last_undegraded":"2024-09-16T10:15:40.749046+0000","last_fullsized":"2024-09-16T10:15:40.749046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:18.722025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:18.722025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:29.585204+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"110'5280","reported_seq":5349,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:42.997520+0000","last_change":"2024-09-16T10:14:43.954758+0000","last_active":"2024-09-16T10:15:42.997520+0000","last_peered":"2024-09-16T10:15:42.997520+0000","last_clean":"2024-09-16T10:15:42.997520+0000","last_became_active":"2024-09-16T10:14:43.954475+0000","last_became_peered":"2024-09-16T10:14:43.954475+0000","last_unstale":"2024-09-16T10:15:42.997520+0000","last_undegraded":"2024-09-16T10:15:42.997520+0000","last_fullsized":"2024-09-16T10:15:42.997520+0000","mapping_epoch":81,"log_start":"88'200","ondisk_log_start":"88'200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5080,"log_dups_size":0,"ondisk_log_size":5080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:53.333918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346030080,"num_objects":5280,"num_object_clones":0,"num_object_copies":15840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5280,"num_write_kb":337920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":135,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752801+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:15:40.752801+0000","last_peered":"2024-09-16T10:15:40.752801+0000","last_clean":"2024-09-16T10:15:40.752801+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:15:40.752801+0000","last_undegraded":"2024-09-16T10:15:40.752801+0000","last_fullsized":"2024-09-16T10:15:40.752801+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.246286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":134,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746590+0000","last_change":"2024-09-16T10:15:25.529229+0000","last_active":"2024-09-16T10:15:40.746590+0000","last_peered":"2024-09-16T10:15:40.746590+0000","last_clean":"2024-09-16T10:15:40.746590+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:15:40.746590+0000","last_undegraded":"2024-09-16T10:15:40.746590+0000","last_fullsized":"2024-09-16T10:15:40.746590+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:15:25.529178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.529178+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:46.039966+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":140,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745695+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:15:40.745695+0000","last_peered":"2024-09-16T10:15:40.745695+0000","last_clean":"2024-09-16T10:15:40.745695+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:15:40.745695+0000","last_undegraded":"2024-09-16T10:15:40.745695+0000","last_fullsized":"2024-09-16T10:15:40.745695+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.613927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752954+0000","last_change":"2024-09-16T10:15:17.306478+0000","last_active":"2024-09-16T10:15:40.752954+0000","last_peered":"2024-09-16T10:15:40.752954+0000","last_clean":"2024-09-16T10:15:40.752954+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:15:40.752954+0000","last_undegraded":"2024-09-16T10:15:40.752954+0000","last_fullsized":"2024-09-16T10:15:40.752954+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.306392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.306392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.352322+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748232+0000","last_change":"2024-09-16T10:13:37.089575+0000","last_active":"2024-09-16T10:15:40.748232+0000","last_peered":"2024-09-16T10:15:40.748232+0000","last_clean":"2024-09-16T10:15:40.748232+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:15:40.748232+0000","last_undegraded":"2024-09-16T10:15:40.748232+0000","last_fullsized":"2024-09-16T10:15:40.748232+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.480266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":127,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752103+0000","last_change":"2024-09-16T10:14:41.331207+0000","last_active":"2024-09-16T10:15:40.752103+0000","last_peered":"2024-09-16T10:15:40.752103+0000","last_clean":"2024-09-16T10:15:40.752103+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:15:40.752103+0000","last_undegraded":"2024-09-16T10:15:40.752103+0000","last_fullsized":"2024-09-16T10:15:40.752103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:14:41.331145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:41.331145+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:42.419653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754629+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:15:40.754629+0000","last_peered":"2024-09-16T10:15:40.754629+0000","last_clean":"2024-09-16T10:15:40.754629+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:15:40.754629+0000","last_undegraded":"2024-09-16T10:15:40.754629+0000","last_fullsized":"2024-09-16T10:15:40.754629+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.231519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746397+0000","last_change":"2024-09-16T10:13:43.283644+0000","last_active":"2024-09-16T10:15:40.746397+0000","last_peered":"2024-09-16T10:15:40.746397+0000","last_clean":"2024-09-16T10:15:40.746397+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:15:40.746397+0000","last_undegraded":"2024-09-16T10:15:40.746397+0000","last_fullsized":"2024-09-16T10:15:40.746397+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.487936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":133,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762688+0000","last_change":"2024-09-16T10:14:40.919863+0000","last_active":"2024-09-16T10:15:40.762688+0000","last_peered":"2024-09-16T10:15:40.762688+0000","last_clean":"2024-09-16T10:15:40.762688+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:15:40.762688+0000","last_undegraded":"2024-09-16T10:15:40.762688+0000","last_fullsized":"2024-09-16T10:15:40.762688+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:14:40.919755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:40.919755+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.992871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745811+0000","last_change":"2024-09-16T10:15:28.537739+0000","last_active":"2024-09-16T10:15:40.745811+0000","last_peered":"2024-09-16T10:15:40.745811+0000","last_clean":"2024-09-16T10:15:40.745811+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:15:40.745811+0000","last_undegraded":"2024-09-16T10:15:40.745811+0000","last_fullsized":"2024-09-16T10:15:40.745811+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:28.537673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.537673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.723859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.751874+0000","last_change":"2024-09-16T10:13:41.118905+0000","last_active":"2024-09-16T10:15:40.751874+0000","last_peered":"2024-09-16T10:15:40.751874+0000","last_clean":"2024-09-16T10:15:40.751874+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:15:40.751874+0000","last_undegraded":"2024-09-16T10:15:40.751874+0000","last_fullsized":"2024-09-16T10:15:40.751874+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.856494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.739484+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:15:40.739484+0000","last_peered":"2024-09-16T10:15:40.739484+0000","last_clean":"2024-09-16T10:15:40.739484+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:15:40.739484+0000","last_undegraded":"2024-09-16T10:15:40.739484+0000","last_fullsized":"2024-09-16T10:15:40.739484+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.602226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":134,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754163+0000","last_change":"2024-09-16T10:15:16.264104+0000","last_active":"2024-09-16T10:15:40.754163+0000","last_peered":"2024-09-16T10:15:40.754163+0000","last_clean":"2024-09-16T10:15:40.754163+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:15:40.754163+0000","last_undegraded":"2024-09-16T10:15:40.754163+0000","last_fullsized":"2024-09-16T10:15:40.754163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:16.264023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.264023+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.067434+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":121,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762374+0000","last_change":"2024-09-16T10:14:37.844181+0000","last_active":"2024-09-16T10:15:40.762374+0000","last_peered":"2024-09-16T10:15:40.762374+0000","last_clean":"2024-09-16T10:15:40.762374+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:15:40.762374+0000","last_undegraded":"2024-09-16T10:15:40.762374+0000","last_fullsized":"2024-09-16T10:15:40.762374+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.844113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.844113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.542157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":148,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.753864+0000","last_change":"2024-09-16T10:15:28.707612+0000","last_active":"2024-09-16T10:15:40.753864+0000","last_peered":"2024-09-16T10:15:40.753864+0000","last_clean":"2024-09-16T10:15:40.753864+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:15:40.753864+0000","last_undegraded":"2024-09-16T10:15:40.753864+0000","last_fullsized":"2024-09-16T10:15:40.753864+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:28.707502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.707502+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:55.133889+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735982+0000","last_change":"2024-09-16T10:13:43.119772+0000","last_active":"2024-09-16T10:15:40.735982+0000","last_peered":"2024-09-16T10:15:40.735982+0000","last_clean":"2024-09-16T10:15:40.735982+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:15:40.735982+0000","last_undegraded":"2024-09-16T10:15:40.735982+0000","last_fullsized":"2024-09-16T10:15:40.735982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.974833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":125,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.756414+0000","last_change":"2024-09-16T10:13:39.105939+0000","last_active":"2024-09-16T10:15:40.756414+0000","last_peered":"2024-09-16T10:15:40.756414+0000","last_clean":"2024-09-16T10:15:40.756414+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:15:40.756414+0000","last_undegraded":"2024-09-16T10:15:40.756414+0000","last_fullsized":"2024-09-16T10:15:40.756414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.532265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745980+0000","last_change":"2024-09-16T10:15:21.601115+0000","last_active":"2024-09-16T10:15:40.745980+0000","last_peered":"2024-09-16T10:15:40.745980+0000","last_clean":"2024-09-16T10:15:40.745980+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:15:40.745980+0000","last_undegraded":"2024-09-16T10:15:40.745980+0000","last_fullsized":"2024-09-16T10:15:40.745980+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.601066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.601066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:45.291040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":140,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754049+0000","last_change":"2024-09-16T10:13:41.116455+0000","last_active":"2024-09-16T10:15:40.754049+0000","last_peered":"2024-09-16T10:15:40.754049+0000","last_clean":"2024-09-16T10:15:40.754049+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:15:40.754049+0000","last_undegraded":"2024-09-16T10:15:40.754049+0000","last_fullsized":"2024-09-16T10:15:40.754049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.292433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762106+0000","last_change":"2024-09-16T10:13:43.120971+0000","last_active":"2024-09-16T10:15:40.762106+0000","last_peered":"2024-09-16T10:15:40.762106+0000","last_clean":"2024-09-16T10:15:40.762106+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:15:40.762106+0000","last_undegraded":"2024-09-16T10:15:40.762106+0000","last_fullsized":"2024-09-16T10:15:40.762106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":128,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735180+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:15:40.735180+0000","last_peered":"2024-09-16T10:15:40.735180+0000","last_clean":"2024-09-16T10:15:40.735180+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:15:40.735180+0000","last_undegraded":"2024-09-16T10:15:40.735180+0000","last_fullsized":"2024-09-16T10:15:40.735180+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.072263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.735708+0000","last_change":"2024-09-16T10:13:37.090129+0000","last_active":"2024-09-16T10:15:40.735708+0000","last_peered":"2024-09-16T10:15:40.735708+0000","last_clean":"2024-09-16T10:15:40.735708+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:15:40.735708+0000","last_undegraded":"2024-09-16T10:15:40.735708+0000","last_fullsized":"2024-09-16T10:15:40.735708+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:54.001301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746993+0000","last_change":"2024-09-16T10:13:41.112912+0000","last_active":"2024-09-16T10:15:40.746993+0000","last_peered":"2024-09-16T10:15:40.746993+0000","last_clean":"2024-09-16T10:15:40.746993+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:15:40.746993+0000","last_undegraded":"2024-09-16T10:15:40.746993+0000","last_fullsized":"2024-09-16T10:15:40.746993+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.005626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.750928+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:15:40.750928+0000","last_peered":"2024-09-16T10:15:40.750928+0000","last_clean":"2024-09-16T10:15:40.750928+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:15:40.750928+0000","last_undegraded":"2024-09-16T10:15:40.750928+0000","last_fullsized":"2024-09-16T10:15:40.750928+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":134,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.747239+0000","last_change":"2024-09-16T10:15:15.010801+0000","last_active":"2024-09-16T10:15:40.747239+0000","last_peered":"2024-09-16T10:15:40.747239+0000","last_clean":"2024-09-16T10:15:40.747239+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:15:40.747239+0000","last_undegraded":"2024-09-16T10:15:40.747239+0000","last_fullsized":"2024-09-16T10:15:40.747239+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:15.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.010709+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.214304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.746653+0000","last_change":"2024-09-16T10:13:37.091081+0000","last_active":"2024-09-16T10:15:40.746653+0000","last_peered":"2024-09-16T10:15:40.746653+0000","last_clean":"2024-09-16T10:15:40.746653+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:15:40.746653+0000","last_undegraded":"2024-09-16T10:15:40.746653+0000","last_fullsized":"2024-09-16T10:15:40.746653+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.620886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.749122+0000","last_change":"2024-09-16T10:13:41.114260+0000","last_active":"2024-09-16T10:15:40.749122+0000","last_peered":"2024-09-16T10:15:40.749122+0000","last_clean":"2024-09-16T10:15:40.749122+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:15:40.749122+0000","last_undegraded":"2024-09-16T10:15:40.749122+0000","last_fullsized":"2024-09-16T10:15:40.749122+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:56.528615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":101,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.761545+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:15:40.761545+0000","last_peered":"2024-09-16T10:15:40.761545+0000","last_clean":"2024-09-16T10:15:40.761545+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:15:40.761545+0000","last_undegraded":"2024-09-16T10:15:40.761545+0000","last_fullsized":"2024-09-16T10:15:40.761545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":129,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.751066+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:15:40.751066+0000","last_peered":"2024-09-16T10:15:40.751066+0000","last_clean":"2024-09-16T10:15:40.751066+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:15:40.751066+0000","last_undegraded":"2024-09-16T10:15:40.751066+0000","last_fullsized":"2024-09-16T10:15:40.751066+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:47.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.754633+0000","last_change":"2024-09-16T10:13:37.088453+0000","last_active":"2024-09-16T10:15:40.754633+0000","last_peered":"2024-09-16T10:15:40.754633+0000","last_clean":"2024-09-16T10:15:40.754633+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:15:40.754633+0000","last_undegraded":"2024-09-16T10:15:40.754633+0000","last_fullsized":"2024-09-16T10:15:40.754633+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.296404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745459+0000","last_change":"2024-09-16T10:14:45.829620+0000","last_active":"2024-09-16T10:15:40.745459+0000","last_peered":"2024-09-16T10:15:40.745459+0000","last_clean":"2024-09-16T10:15:40.745459+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:15:40.745459+0000","last_undegraded":"2024-09-16T10:15:40.745459+0000","last_fullsized":"2024-09-16T10:15:40.745459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:45.829572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.829572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:46.942231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.749246+0000","last_change":"2024-09-16T10:15:30.762339+0000","last_active":"2024-09-16T10:15:40.749246+0000","last_peered":"2024-09-16T10:15:40.749246+0000","last_clean":"2024-09-16T10:15:40.749246+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:15:40.749246+0000","last_undegraded":"2024-09-16T10:15:40.749246+0000","last_fullsized":"2024-09-16T10:15:40.749246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.762137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.762137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.668208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":137,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.750282+0000","last_change":"2024-09-16T10:15:08.953214+0000","last_active":"2024-09-16T10:15:40.750282+0000","last_peered":"2024-09-16T10:15:40.750282+0000","last_clean":"2024-09-16T10:15:40.750282+0000","last_became_active":"2024-09-16T10:15:08.930505+0000","last_became_peered":"2024-09-16T10:15:08.930505+0000","last_unstale":"2024-09-16T10:15:40.750282+0000","last_undegraded":"2024-09-16T10:15:40.750282+0000","last_fullsized":"2024-09-16T10:15:40.750282+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.615906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":119,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.763012+0000","last_change":"2024-09-16T10:13:37.094533+0000","last_active":"2024-09-16T10:15:40.763012+0000","last_peered":"2024-09-16T10:15:40.763012+0000","last_clean":"2024-09-16T10:15:40.763012+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:15:40.763012+0000","last_undegraded":"2024-09-16T10:15:40.763012+0000","last_fullsized":"2024-09-16T10:15:40.763012+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.358935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.748247+0000","last_change":"2024-09-16T10:15:13.474188+0000","last_active":"2024-09-16T10:15:40.748247+0000","last_peered":"2024-09-16T10:15:40.748247+0000","last_clean":"2024-09-16T10:15:40.748247+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:15:40.748247+0000","last_undegraded":"2024-09-16T10:15:40.748247+0000","last_fullsized":"2024-09-16T10:15:40.748247+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.474125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.474125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:29.683499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.745979+0000","last_change":"2024-09-16T10:13:43.119917+0000","last_active":"2024-09-16T10:15:40.745979+0000","last_peered":"2024-09-16T10:15:40.745979+0000","last_clean":"2024-09-16T10:15:40.745979+0000","last_became_active":"2024-09-16T10:13:43.119612+0000","last_became_peered":"2024-09-16T10:13:43.119612+0000","last_unstale":"2024-09-16T10:15:40.745979+0000","last_undegraded":"2024-09-16T10:15:40.745979+0000","last_fullsized":"2024-09-16T10:15:40.745979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.561593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.752670+0000","last_change":"2024-09-16T10:15:17.082954+0000","last_active":"2024-09-16T10:15:40.752670+0000","last_peered":"2024-09-16T10:15:40.752670+0000","last_clean":"2024-09-16T10:15:40.752670+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:15:40.752670+0000","last_undegraded":"2024-09-16T10:15:40.752670+0000","last_fullsized":"2024-09-16T10:15:40.752670+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.082687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.082687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:26.257770+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.750732+0000","last_change":"2024-09-16T10:15:24.663908+0000","last_active":"2024-09-16T10:15:40.750732+0000","last_peered":"2024-09-16T10:15:40.750732+0000","last_clean":"2024-09-16T10:15:40.750732+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:15:40.750732+0000","last_undegraded":"2024-09-16T10:15:40.750732+0000","last_fullsized":"2024-09-16T10:15:40.750732+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.663832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.663832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:46.507773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":137,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.749070+0000","last_change":"2024-09-16T10:14:44.362078+0000","last_active":"2024-09-16T10:15:40.749070+0000","last_peered":"2024-09-16T10:15:40.749070+0000","last_clean":"2024-09-16T10:15:40.749070+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:15:40.749070+0000","last_undegraded":"2024-09-16T10:15:40.749070+0000","last_fullsized":"2024-09-16T10:15:40.749070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:14:44.362007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.362007+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.313495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T10:15:40.762379+0000","last_change":"2024-09-16T10:15:24.135166+0000","last_active":"2024-09-16T10:15:40.762379+0000","last_peered":"2024-09-16T10:15:40.762379+0000","last_clean":"2024-09-16T10:15:40.762379+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:15:40.762379+0000","last_undegraded":"2024-09-16T10:15:40.762379+0000","last_fullsized":"2024-09-16T10:15:40.762379+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.135091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.135091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.981416+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5446500352,"num_objects":83107,"num_object_clones":0,"num_object_copies":249321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83107,"num_write_kb":5318848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16339697664,"data_stored":16339697664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81007,"ondisk_log_size":81007,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":115,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234013,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712364,"kb_used_data":1663124,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92032788,"statfs":{"total":95995035648,"available":94241574912,"internally_reserved":0,"allocated":1703038976,"data_stored":1702381559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528353,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1726284,"kb_used_data":1677052,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92018868,"statfs":{"total":95995035648,"available":94227320832,"internally_reserved":0,"allocated":1717301248,"data_stored":1716667309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789991,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350752,"kb_used_data":1305672,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92394400,"statfs":{"total":95995035648,"available":94611865600,"internally_reserved":0,"allocated":1337008128,"data_stored":1336362929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1710760,"kb_used_data":1657368,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92034392,"statfs":{"total":95995035648,"available":94243217408,"internally_reserved":0,"allocated":1697144832,"data_stored":1696482718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215153,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3062268,"kb_used_data":2986660,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":90682884,"statfs":{"total":95995035648,"available":92859273216,"internally_reserved":0,"allocated":3058339840,"data_stored":3057666717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77396424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2043212,"kb_used_data":1967612,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91701940,"statfs":{"total":95995035648,"available":93902786560,"internally_reserved":0,"allocated":2014834688,"data_stored":2014200476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2058240,"kb_used_data":1986800,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91686912,"statfs":{"total":95995035648,"available":93887397888,"internally_reserved":0,"allocated":2034483200,"data_stored":2033797360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2693100,"kb_used_data":2629980,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91052052,"statfs":{"total":95995035648,"available":93237301248,"internally_reserved":0,"allocated":2693099520,"data_stored":2692434613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2715025408,"data_stored":2715025408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2033844224,"data_stored":2033844224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2031026176,"data_stored":2031026176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3065643008,"data_stored":3065643008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1699151872,"data_stored":1699151872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1347747840,"data_stored":1347747840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1732706304,"data_stored":1732706304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1714552832,"data_stored":1714552832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:15:51.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:15:51.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:51.690 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:51.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:51.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:51.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:51.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:51.917 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:51.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:51 smithi063 ceph-mon[29993]: pgmap v94: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-16T10:15:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:51 smithi063 ceph-mon[39470]: pgmap v94: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-16T10:15:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:51.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:51.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:52.008 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:51 smithi165 ceph-mon[32696]: pgmap v94: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-16T10:15:52.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:52.095 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 62 16 92423 92407 93.1282 54.8125 0.00978871 0.0107323 2024-09-16T10:15:52.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:52.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:52.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:15:52.162 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:52.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:52.216 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:52.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:52.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:52.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:52.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:52.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:52.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:52.501 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:52.540 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:52.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:52.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:52.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:52.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:52.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:52.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:52.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:52.899 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:52 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:52.900 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:52 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:52.954 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:52.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:53.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:52 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:15:53.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:53.095 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 63 16 93231 93215 92.4516 50.5 0.00859835 0.0108105 2024-09-16T10:15:53.099 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:53.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:53.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:53.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:53.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:53.233 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:15:53.233 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:15:53.283 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:15:53.283 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:15:53.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:53.334 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:15:53.334 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:15:53.379 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:53.379 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:15:53.380 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:15:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:53.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:53.416 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:15:53.416 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:15:53.457 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:15:53.457 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:15:53.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:53.490 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:15:53.491 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:15:53.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:53.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:53.530 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:15:53.530 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:15:53.608 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:53.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:53.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:53.767 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 systemd[1]: Stopping Ceph prometheus.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:15:53.774 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:15:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:53.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:53.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:53.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:53.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:54.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:54.033 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[39470]: pgmap v95: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 544 B/s rd, 39 MiB/s wr, 623 op/s 2024-09-16T10:15:54.034 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2695950266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:54.034 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:15:54.034 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:15:54.034 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2562112422' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:15:54.034 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[29993]: pgmap v95: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 544 B/s rd, 39 MiB/s wr, 623 op/s 2024-09-16T10:15:54.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2695950266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:54.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:15:54.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:15:54.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2562112422' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:15:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-mon[32696]: pgmap v95: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 544 B/s rd, 39 MiB/s wr, 623 op/s 2024-09-16T10:15:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2695950266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:15:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:15:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:15:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2562112422' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:15:54.052 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-16T10:15:54.052 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.966Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.967Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.967Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.967Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[58129]: ts=2024-09-16T10:15:53.967Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-16T10:15:54.053 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:53 smithi165 podman[65107]: 2024-09-16 10:15:53.976725167 +0000 UTC m=+0.142917714 container died 927c0e755445f4680ea9f5561e62ef4cae6d4d2fdde2252cb73feacc44ee09f4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:54.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:54.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:54.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:54.095 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 64 16 94197 94181 91.9504 60.375 0.0153452 0.0108701 2024-09-16T10:15:54.159 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:54.204 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:54.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:54.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:54.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:54.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:54.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:54.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:54.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:54.482 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:15:54.489 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:15:54.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:54.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:54.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:54.653 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:54 smithi165 podman[65107]: 2024-09-16 10:15:54.408400706 +0000 UTC m=+0.574593244 container cleanup 927c0e755445f4680ea9f5561e62ef4cae6d4d2fdde2252cb73feacc44ee09f4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:54.653 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:54 smithi165 bash[65107]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a 2024-09-16T10:15:54.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:54.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:54.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:54.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:54.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2562112422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:15:54.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:54.869 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:54.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2562112422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: osdmap e113: 8 total, 8 up, 8 in 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: 3.1e scrub starts 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: 3.1e scrub ok 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: 5.12 scrub starts 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: 5.12 scrub ok 2024-09-16T10:15:54.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: 4.16 scrub starts 2024-09-16T10:15:54.962 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: 4.16 scrub ok 2024-09-16T10:15:54.962 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: 7.e scrub starts 2024-09-16T10:15:54.962 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:54 smithi165 ceph-mon[32696]: from='client.24874 v1:172.21.15.63:0/2334145576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:15:54.962 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:54 smithi165 podman[65135]: 2024-09-16 10:15:54.652248259 +0000 UTC m=+0.675560417 container remove 927c0e755445f4680ea9f5561e62ef4cae6d4d2fdde2252cb73feacc44ee09f4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:54.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:55.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:55.096 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 65 16 95224 95208 91.5233 64.1875 0.0145928 0.0109206 2024-09-16T10:15:55.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:55.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:55.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: osdmap e113: 8 total, 8 up, 8 in 2024-09-16T10:15:55.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: 3.1e scrub starts 2024-09-16T10:15:55.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: 3.1e scrub ok 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: 5.12 scrub starts 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: 5.12 scrub ok 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: 4.16 scrub starts 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: 4.16 scrub ok 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: 7.e scrub starts 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[29993]: from='client.24874 v1:172.21.15.63:0/2334145576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2562112422' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: osdmap e113: 8 total, 8 up, 8 in 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: 3.1e scrub starts 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: 3.1e scrub ok 2024-09-16T10:15:55.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: 5.12 scrub starts 2024-09-16T10:15:55.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: 5.12 scrub ok 2024-09-16T10:15:55.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: 4.16 scrub starts 2024-09-16T10:15:55.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: 4.16 scrub ok 2024-09-16T10:15:55.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: 7.e scrub starts 2024-09-16T10:15:55.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:54 smithi063 ceph-mon[39470]: from='client.24874 v1:172.21.15.63:0/2334145576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:15:55.241 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:54 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-16T10:15:55.241 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:54 smithi165 systemd[1]: Stopped Ceph prometheus.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:15:55.241 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:54 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.219s CPU time. 2024-09-16T10:15:55.241 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:54 smithi165 systemd[1]: Starting Ceph prometheus.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:15:55.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:55.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:55.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:55.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:55.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:55.525 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 podman[65304]: 2024-09-16 10:15:55.147617571 +0000 UTC m=+0.030017282 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-16T10:15:55.525 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 podman[65304]: 2024-09-16 10:15:55.278857013 +0000 UTC m=+0.161256713 container create 6fecfcdf6c17bb87e9ac226e24f82093b36bcc651117fb3eb67de184751a3869 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:55.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:55.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:55.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:55.761 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 podman[65304]: 2024-09-16 10:15:55.523730279 +0000 UTC m=+0.406129986 container init 6fecfcdf6c17bb87e9ac226e24f82093b36bcc651117fb3eb67de184751a3869 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 podman[65304]: 2024-09-16 10:15:55.528271548 +0000 UTC m=+0.410671256 container start 6fecfcdf6c17bb87e9ac226e24f82093b36bcc651117fb3eb67de184751a3869 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.558Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.558Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.558Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi165 (none))" 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.558Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.558Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-16T10:15:55.796 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.560Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.561Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.563Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.564Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.566Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.566Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.58µs 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.566Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.567Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.567Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.567Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.532µs wal_replay_duration=466.621µs wbl_replay_duration=126ns total_replay_duration=550.321µs 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.569Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.569Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-16T10:15:55.797 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.569Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-16T10:15:55.798 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.590Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.984454ms db_storage=879ns remote_storage=1.205µs web_handler=375ns query_engine=775ns scrape=421.125µs scrape_sd=102.346µs notify=10.551µs notify_sd=10.271µs rules=20.049128ms tracing=8.449µs 2024-09-16T10:15:55.798 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.590Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-16T10:15:55.798 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:15:55.590Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-16T10:15:55.798 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 bash[65304]: 6fecfcdf6c17bb87e9ac226e24f82093b36bcc651117fb3eb67de184751a3869 2024-09-16T10:15:55.798 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:15:55 smithi165 systemd[1]: Started Ceph prometheus.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:15:55.823 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:55.835 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-16T10:15:53.769210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6108420635,"num_objects":93397,"num_object_clones":0,"num_object_copies":280191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93397,"num_whiteouts":0,"num_read":978,"num_read_kb":835,"num_write":93747,"num_write_kb":5966345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87150,"ondisk_log_size":87150,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18301788,"kb_used_data":17809372,"kb_used_omap":12,"kb_used_meta":492275,"kb_avail":731659428,"statfs":{"total":767960285184,"available":749219254272,"internally_reserved":0,"allocated":18236796928,"data_stored":18231452753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":504090161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":661323776,"num_objects":10091,"num_object_clones":0,"num_object_copies":30273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10091,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10091,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.727218"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.740698+0000","last_change":"2024-09-16T10:15:15.058932+0000","last_active":"2024-09-16T10:15:45.740698+0000","last_peered":"2024-09-16T10:15:45.740698+0000","last_clean":"2024-09-16T10:15:45.740698+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:15:45.740698+0000","last_undegraded":"2024-09-16T10:15:45.740698+0000","last_fullsized":"2024-09-16T10:15:45.740698+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:15.058652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.058652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.247209+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":154,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735375+0000","last_change":"2024-09-16T10:15:11.063493+0000","last_active":"2024-09-16T10:15:45.735375+0000","last_peered":"2024-09-16T10:15:45.735375+0000","last_clean":"2024-09-16T10:15:45.735375+0000","last_became_active":"2024-09-16T10:15:08.925269+0000","last_became_peered":"2024-09-16T10:15:08.925269+0000","last_unstale":"2024-09-16T10:15:45.735375+0000","last_undegraded":"2024-09-16T10:15:45.735375+0000","last_fullsized":"2024-09-16T10:15:45.735375+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:11.063426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.063426+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:23.888178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.756274+0000","last_change":"2024-09-16T10:13:41.108286+0000","last_active":"2024-09-16T10:15:45.756274+0000","last_peered":"2024-09-16T10:15:45.756274+0000","last_clean":"2024-09-16T10:15:45.756274+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:15:45.756274+0000","last_undegraded":"2024-09-16T10:15:45.756274+0000","last_fullsized":"2024-09-16T10:15:45.756274+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.053767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735480+0000","last_change":"2024-09-16T10:13:43.113148+0000","last_active":"2024-09-16T10:15:45.735480+0000","last_peered":"2024-09-16T10:15:45.735480+0000","last_clean":"2024-09-16T10:15:45.735480+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:15:45.735480+0000","last_undegraded":"2024-09-16T10:15:45.735480+0000","last_fullsized":"2024-09-16T10:15:45.735480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:01.951377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751264+0000","last_change":"2024-09-16T10:13:43.287323+0000","last_active":"2024-09-16T10:15:45.751264+0000","last_peered":"2024-09-16T10:15:45.751264+0000","last_clean":"2024-09-16T10:15:45.751264+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:15:45.751264+0000","last_undegraded":"2024-09-16T10:15:45.751264+0000","last_fullsized":"2024-09-16T10:15:45.751264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:37.873818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752312+0000","last_change":"2024-09-16T10:13:37.092457+0000","last_active":"2024-09-16T10:15:45.752312+0000","last_peered":"2024-09-16T10:15:45.752312+0000","last_clean":"2024-09-16T10:15:45.752312+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:15:45.752312+0000","last_undegraded":"2024-09-16T10:15:45.752312+0000","last_fullsized":"2024-09-16T10:15:45.752312+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:51.455656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749775+0000","last_change":"2024-09-16T10:15:10.668770+0000","last_active":"2024-09-16T10:15:45.749775+0000","last_peered":"2024-09-16T10:15:45.749775+0000","last_clean":"2024-09-16T10:15:45.749775+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:15:45.749775+0000","last_undegraded":"2024-09-16T10:15:45.749775+0000","last_fullsized":"2024-09-16T10:15:45.749775+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:15:10.668521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:10.668521+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:18.492671+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735288+0000","last_change":"2024-09-16T10:13:41.121501+0000","last_active":"2024-09-16T10:15:45.735288+0000","last_peered":"2024-09-16T10:15:45.735288+0000","last_clean":"2024-09-16T10:15:45.735288+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:15:45.735288+0000","last_undegraded":"2024-09-16T10:15:45.735288+0000","last_fullsized":"2024-09-16T10:15:45.735288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.954155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753190+0000","last_change":"2024-09-16T10:15:29.351071+0000","last_active":"2024-09-16T10:15:45.753190+0000","last_peered":"2024-09-16T10:15:45.753190+0000","last_clean":"2024-09-16T10:15:45.753190+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:15:45.753190+0000","last_undegraded":"2024-09-16T10:15:45.753190+0000","last_fullsized":"2024-09-16T10:15:45.753190+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:29.351000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.351000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:52.980664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735651+0000","last_change":"2024-09-16T10:15:16.028344+0000","last_active":"2024-09-16T10:15:45.735651+0000","last_peered":"2024-09-16T10:15:45.735651+0000","last_clean":"2024-09-16T10:15:45.735651+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:15:45.735651+0000","last_undegraded":"2024-09-16T10:15:45.735651+0000","last_fullsized":"2024-09-16T10:15:45.735651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:16.028283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.028283+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.455289+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.754223+0000","last_change":"2024-09-16T10:15:22.271162+0000","last_active":"2024-09-16T10:15:45.754223+0000","last_peered":"2024-09-16T10:15:45.754223+0000","last_clean":"2024-09-16T10:15:45.754223+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:15:45.754223+0000","last_undegraded":"2024-09-16T10:15:45.754223+0000","last_fullsized":"2024-09-16T10:15:45.754223+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:22.271101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:22.271101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.480708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":113,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.740455+0000","last_change":"2024-09-16T10:15:32.193135+0000","last_active":"2024-09-16T10:15:45.740455+0000","last_peered":"2024-09-16T10:15:45.740455+0000","last_clean":"2024-09-16T10:15:45.740455+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:15:45.740455+0000","last_undegraded":"2024-09-16T10:15:45.740455+0000","last_fullsized":"2024-09-16T10:15:45.740455+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:32.193067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:32.193067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:59.551498+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752747+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:15:45.752747+0000","last_peered":"2024-09-16T10:15:45.752747+0000","last_clean":"2024-09-16T10:15:45.752747+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:15:45.752747+0000","last_undegraded":"2024-09-16T10:15:45.752747+0000","last_fullsized":"2024-09-16T10:15:45.752747+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.232971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735479+0000","last_change":"2024-09-16T10:13:39.092696+0000","last_active":"2024-09-16T10:15:45.735479+0000","last_peered":"2024-09-16T10:15:45.735479+0000","last_clean":"2024-09-16T10:15:45.735479+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:15:45.735479+0000","last_undegraded":"2024-09-16T10:15:45.735479+0000","last_fullsized":"2024-09-16T10:15:45.735479+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:54.990979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747012+0000","last_change":"2024-09-16T10:15:16.573685+0000","last_active":"2024-09-16T10:15:45.747012+0000","last_peered":"2024-09-16T10:15:45.747012+0000","last_clean":"2024-09-16T10:15:45.747012+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:15:45.747012+0000","last_undegraded":"2024-09-16T10:15:45.747012+0000","last_fullsized":"2024-09-16T10:15:45.747012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.573614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.573614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.536483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752515+0000","last_change":"2024-09-16T10:15:11.297702+0000","last_active":"2024-09-16T10:15:45.752515+0000","last_peered":"2024-09-16T10:15:45.752515+0000","last_clean":"2024-09-16T10:15:45.752515+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:15:45.752515+0000","last_undegraded":"2024-09-16T10:15:45.752515+0000","last_fullsized":"2024-09-16T10:15:45.752515+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.297599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.297599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.846847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.736190+0000","last_change":"2024-09-16T10:13:43.284208+0000","last_active":"2024-09-16T10:15:45.736190+0000","last_peered":"2024-09-16T10:15:45.736190+0000","last_clean":"2024-09-16T10:15:45.736190+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:15:45.736190+0000","last_undegraded":"2024-09-16T10:15:45.736190+0000","last_fullsized":"2024-09-16T10:15:45.736190+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.935116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.742721+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:15:45.742721+0000","last_peered":"2024-09-16T10:15:45.742721+0000","last_clean":"2024-09-16T10:15:45.742721+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:15:45.742721+0000","last_undegraded":"2024-09-16T10:15:45.742721+0000","last_fullsized":"2024-09-16T10:15:45.742721+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:50.979003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":139,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.739026+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:15:45.739026+0000","last_peered":"2024-09-16T10:15:45.739026+0000","last_clean":"2024-09-16T10:15:45.739026+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:15:45.739026+0000","last_undegraded":"2024-09-16T10:15:45.739026+0000","last_fullsized":"2024-09-16T10:15:45.739026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:02.543443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.750845+0000","last_change":"2024-09-16T10:15:10.987453+0000","last_active":"2024-09-16T10:15:45.750845+0000","last_peered":"2024-09-16T10:15:45.750845+0000","last_clean":"2024-09-16T10:15:45.750845+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:15:45.750845+0000","last_undegraded":"2024-09-16T10:15:45.750845+0000","last_fullsized":"2024-09-16T10:15:45.750845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:10.987312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:10.987312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.809648+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752445+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:15:45.752445+0000","last_peered":"2024-09-16T10:15:45.752445+0000","last_clean":"2024-09-16T10:15:45.752445+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:15:45.752445+0000","last_undegraded":"2024-09-16T10:15:45.752445+0000","last_fullsized":"2024-09-16T10:15:45.752445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:45.886306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735741+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:15:45.735741+0000","last_peered":"2024-09-16T10:15:45.735741+0000","last_clean":"2024-09-16T10:15:45.735741+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:15:45.735741+0000","last_undegraded":"2024-09-16T10:15:45.735741+0000","last_fullsized":"2024-09-16T10:15:45.735741+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:46.960702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":164,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751624+0000","last_change":"2024-09-16T10:15:21.639445+0000","last_active":"2024-09-16T10:15:45.751624+0000","last_peered":"2024-09-16T10:15:45.751624+0000","last_clean":"2024-09-16T10:15:45.751624+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:15:45.751624+0000","last_undegraded":"2024-09-16T10:15:45.751624+0000","last_fullsized":"2024-09-16T10:15:45.751624+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:21.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.639361+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:30.197909+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.737624+0000","last_change":"2024-09-16T10:15:21.032515+0000","last_active":"2024-09-16T10:15:45.737624+0000","last_peered":"2024-09-16T10:15:45.737624+0000","last_clean":"2024-09-16T10:15:45.737624+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:15:45.737624+0000","last_undegraded":"2024-09-16T10:15:45.737624+0000","last_fullsized":"2024-09-16T10:15:45.737624+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.032449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.032449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:21.047016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747393+0000","last_change":"2024-09-16T10:13:43.119664+0000","last_active":"2024-09-16T10:15:45.747393+0000","last_peered":"2024-09-16T10:15:45.747393+0000","last_clean":"2024-09-16T10:15:45.747393+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:15:45.747393+0000","last_undegraded":"2024-09-16T10:15:45.747393+0000","last_fullsized":"2024-09-16T10:15:45.747393+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:34.427312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.756415+0000","last_change":"2024-09-16T10:13:37.082594+0000","last_active":"2024-09-16T10:15:45.756415+0000","last_peered":"2024-09-16T10:15:45.756415+0000","last_clean":"2024-09-16T10:15:45.756415+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:15:45.756415+0000","last_undegraded":"2024-09-16T10:15:45.756415+0000","last_fullsized":"2024-09-16T10:15:45.756415+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.024905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753405+0000","last_change":"2024-09-16T10:15:13.091323+0000","last_active":"2024-09-16T10:15:45.753405+0000","last_peered":"2024-09-16T10:15:45.753405+0000","last_clean":"2024-09-16T10:15:45.753405+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:15:45.753405+0000","last_undegraded":"2024-09-16T10:15:45.753405+0000","last_fullsized":"2024-09-16T10:15:45.753405+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:15:13.091193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.091193+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:26.457536+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":157,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:48.051002+0000","last_change":"2024-09-16T10:15:12.451691+0000","last_active":"2024-09-16T10:15:48.051002+0000","last_peered":"2024-09-16T10:15:48.051002+0000","last_clean":"2024-09-16T10:15:48.051002+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:15:48.051002+0000","last_undegraded":"2024-09-16T10:15:48.051002+0000","last_fullsized":"2024-09-16T10:15:48.051002+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:12.451619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:12.451619+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:31.691482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"112'6035","reported_seq":6108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.021413+0000","last_change":"2024-09-16T10:14:43.957473+0000","last_active":"2024-09-16T10:15:53.021413+0000","last_peered":"2024-09-16T10:15:53.021413+0000","last_clean":"2024-09-16T10:15:53.021413+0000","last_became_active":"2024-09-16T10:14:43.957174+0000","last_became_peered":"2024-09-16T10:14:43.957174+0000","last_unstale":"2024-09-16T10:15:53.021413+0000","last_undegraded":"2024-09-16T10:15:53.021413+0000","last_fullsized":"2024-09-16T10:15:53.021413+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6035,"log_dups_size":0,"ondisk_log_size":6035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:47.305264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395509760,"num_objects":6035,"num_object_clones":0,"num_object_copies":18105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6035,"num_write_kb":386240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.750067+0000","last_change":"2024-09-16T10:14:45.136557+0000","last_active":"2024-09-16T10:15:45.750067+0000","last_peered":"2024-09-16T10:15:45.750067+0000","last_clean":"2024-09-16T10:15:45.750067+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:15:45.750067+0000","last_undegraded":"2024-09-16T10:15:45.750067+0000","last_fullsized":"2024-09-16T10:15:45.750067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:14:45.136460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.136460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749501+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:15:45.749501+0000","last_peered":"2024-09-16T10:15:45.749501+0000","last_clean":"2024-09-16T10:15:45.749501+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:15:45.749501+0000","last_undegraded":"2024-09-16T10:15:45.749501+0000","last_fullsized":"2024-09-16T10:15:45.749501+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.842587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":145,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:47.175478+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:15:47.175478+0000","last_peered":"2024-09-16T10:15:47.175478+0000","last_clean":"2024-09-16T10:15:47.175478+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:15:47.175478+0000","last_undegraded":"2024-09-16T10:15:47.175478+0000","last_fullsized":"2024-09-16T10:15:47.175478+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.097659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753402+0000","last_change":"2024-09-16T10:13:43.114886+0000","last_active":"2024-09-16T10:15:45.753402+0000","last_peered":"2024-09-16T10:15:45.753402+0000","last_clean":"2024-09-16T10:15:45.753402+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:15:45.753402+0000","last_undegraded":"2024-09-16T10:15:45.753402+0000","last_fullsized":"2024-09-16T10:15:45.753402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:51.280322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"112'5779","reported_seq":5852,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.037241+0000","last_change":"2024-09-16T10:14:43.953298+0000","last_active":"2024-09-16T10:15:53.037241+0000","last_peered":"2024-09-16T10:15:53.037241+0000","last_clean":"2024-09-16T10:15:53.037241+0000","last_became_active":"2024-09-16T10:14:43.952713+0000","last_became_peered":"2024-09-16T10:14:43.952713+0000","last_unstale":"2024-09-16T10:15:53.037241+0000","last_undegraded":"2024-09-16T10:15:53.037241+0000","last_fullsized":"2024-09-16T10:15:53.037241+0000","mapping_epoch":81,"log_start":"90'1100","ondisk_log_start":"90'1100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378732544,"num_objects":5779,"num_object_clones":0,"num_object_copies":17337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5779,"num_write_kb":369856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":155,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749293+0000","last_change":"2024-09-16T10:15:23.703368+0000","last_active":"2024-09-16T10:15:45.749293+0000","last_peered":"2024-09-16T10:15:45.749293+0000","last_clean":"2024-09-16T10:15:45.749293+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:15:45.749293+0000","last_undegraded":"2024-09-16T10:15:45.749293+0000","last_fullsized":"2024-09-16T10:15:45.749293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:15:23.703278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.703278+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.263361+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752881+0000","last_change":"2024-09-16T10:13:37.093660+0000","last_active":"2024-09-16T10:15:45.752881+0000","last_peered":"2024-09-16T10:15:45.752881+0000","last_clean":"2024-09-16T10:15:45.752881+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:15:45.752881+0000","last_undegraded":"2024-09-16T10:15:45.752881+0000","last_fullsized":"2024-09-16T10:15:45.752881+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.591570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735489+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:15:45.735489+0000","last_peered":"2024-09-16T10:15:45.735489+0000","last_clean":"2024-09-16T10:15:45.735489+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:15:45.735489+0000","last_undegraded":"2024-09-16T10:15:45.735489+0000","last_fullsized":"2024-09-16T10:15:45.735489+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:02.988717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735645+0000","last_change":"2024-09-16T10:15:12.079425+0000","last_active":"2024-09-16T10:15:45.735645+0000","last_peered":"2024-09-16T10:15:45.735645+0000","last_clean":"2024-09-16T10:15:45.735645+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:15:45.735645+0000","last_undegraded":"2024-09-16T10:15:45.735645+0000","last_fullsized":"2024-09-16T10:15:45.735645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:12.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:12.079353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.080838+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"112'5799","reported_seq":5872,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.009510+0000","last_change":"2024-09-16T10:14:43.953541+0000","last_active":"2024-09-16T10:15:53.009510+0000","last_peered":"2024-09-16T10:15:53.009510+0000","last_clean":"2024-09-16T10:15:53.009510+0000","last_became_active":"2024-09-16T10:14:43.952908+0000","last_became_peered":"2024-09-16T10:14:43.952908+0000","last_unstale":"2024-09-16T10:15:53.009510+0000","last_undegraded":"2024-09-16T10:15:53.009510+0000","last_fullsized":"2024-09-16T10:15:53.009510+0000","mapping_epoch":81,"log_start":"90'1100","ondisk_log_start":"90'1100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.479727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380043264,"num_objects":5799,"num_object_clones":0,"num_object_copies":17397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":371136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":143,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735536+0000","last_change":"2024-09-16T10:15:17.025884+0000","last_active":"2024-09-16T10:15:45.735536+0000","last_peered":"2024-09-16T10:15:45.735536+0000","last_clean":"2024-09-16T10:15:45.735536+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:15:45.735536+0000","last_undegraded":"2024-09-16T10:15:45.735536+0000","last_fullsized":"2024-09-16T10:15:45.735536+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:15:17.025825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.025825+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.846848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752795+0000","last_change":"2024-09-16T10:15:29.146696+0000","last_active":"2024-09-16T10:15:45.752795+0000","last_peered":"2024-09-16T10:15:45.752795+0000","last_clean":"2024-09-16T10:15:45.752795+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:15:45.752795+0000","last_undegraded":"2024-09-16T10:15:45.752795+0000","last_fullsized":"2024-09-16T10:15:45.752795+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:29.146482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.146482+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:56.251083+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":158,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:47.176001+0000","last_change":"2024-09-16T10:15:20.675679+0000","last_active":"2024-09-16T10:15:47.176001+0000","last_peered":"2024-09-16T10:15:47.176001+0000","last_clean":"2024-09-16T10:15:47.176001+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:15:47.176001+0000","last_undegraded":"2024-09-16T10:15:47.176001+0000","last_fullsized":"2024-09-16T10:15:47.176001+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:20.675600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:20.675600+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.817505+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747004+0000","last_change":"2024-09-16T10:13:43.284631+0000","last_active":"2024-09-16T10:15:45.747004+0000","last_peered":"2024-09-16T10:15:45.747004+0000","last_clean":"2024-09-16T10:15:45.747004+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:15:45.747004+0000","last_undegraded":"2024-09-16T10:15:45.747004+0000","last_fullsized":"2024-09-16T10:15:45.747004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.518019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"112'5896","reported_seq":5969,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.017410+0000","last_change":"2024-09-16T10:14:43.952332+0000","last_active":"2024-09-16T10:15:53.017410+0000","last_peered":"2024-09-16T10:15:53.017410+0000","last_clean":"2024-09-16T10:15:53.017410+0000","last_became_active":"2024-09-16T10:14:43.952043+0000","last_became_peered":"2024-09-16T10:14:43.952043+0000","last_unstale":"2024-09-16T10:15:53.017410+0000","last_undegraded":"2024-09-16T10:15:53.017410+0000","last_fullsized":"2024-09-16T10:15:53.017410+0000","mapping_epoch":81,"log_start":"90'1100","ondisk_log_start":"90'1100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.379216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386400256,"num_objects":5896,"num_object_clones":0,"num_object_copies":17688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5896,"num_write_kb":377344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749791+0000","last_change":"2024-09-16T10:14:42.119372+0000","last_active":"2024-09-16T10:15:45.749791+0000","last_peered":"2024-09-16T10:15:45.749791+0000","last_clean":"2024-09-16T10:15:45.749791+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:15:45.749791+0000","last_undegraded":"2024-09-16T10:15:45.749791+0000","last_fullsized":"2024-09-16T10:15:45.749791+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:14:42.119310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.119310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.359829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753524+0000","last_change":"2024-09-16T10:15:23.304088+0000","last_active":"2024-09-16T10:15:45.753524+0000","last_peered":"2024-09-16T10:15:45.753524+0000","last_clean":"2024-09-16T10:15:45.753524+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:15:45.753524+0000","last_undegraded":"2024-09-16T10:15:45.753524+0000","last_fullsized":"2024-09-16T10:15:45.753524+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:23.304033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.304033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.813644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":154,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:48.079403+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:15:48.079403+0000","last_peered":"2024-09-16T10:15:48.079403+0000","last_clean":"2024-09-16T10:15:48.079403+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:15:48.079403+0000","last_undegraded":"2024-09-16T10:15:48.079403+0000","last_fullsized":"2024-09-16T10:15:48.079403+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.731855+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.740716+0000","last_change":"2024-09-16T10:15:16.104168+0000","last_active":"2024-09-16T10:15:45.740716+0000","last_peered":"2024-09-16T10:15:45.740716+0000","last_clean":"2024-09-16T10:15:45.740716+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:15:45.740716+0000","last_undegraded":"2024-09-16T10:15:45.740716+0000","last_fullsized":"2024-09-16T10:15:45.740716+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.104092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.104092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:41.867492+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"112'5670","reported_seq":5743,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.001090+0000","last_change":"2024-09-16T10:14:43.956078+0000","last_active":"2024-09-16T10:15:53.001090+0000","last_peered":"2024-09-16T10:15:53.001090+0000","last_clean":"2024-09-16T10:15:53.001090+0000","last_became_active":"2024-09-16T10:14:43.955808+0000","last_became_peered":"2024-09-16T10:14:43.955808+0000","last_unstale":"2024-09-16T10:15:53.001090+0000","last_undegraded":"2024-09-16T10:15:53.001090+0000","last_fullsized":"2024-09-16T10:15:53.001090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5670,"log_dups_size":0,"ondisk_log_size":5670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:54.992452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371589120,"num_objects":5670,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":362880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":158,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.742497+0000","last_change":"2024-09-16T10:15:26.113170+0000","last_active":"2024-09-16T10:15:45.742497+0000","last_peered":"2024-09-16T10:15:45.742497+0000","last_clean":"2024-09-16T10:15:45.742497+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:15:45.742497+0000","last_undegraded":"2024-09-16T10:15:45.742497+0000","last_fullsized":"2024-09-16T10:15:45.742497+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:15:26.110570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:26.110570+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.337507+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752348+0000","last_change":"2024-09-16T10:15:18.059218+0000","last_active":"2024-09-16T10:15:45.752348+0000","last_peered":"2024-09-16T10:15:45.752348+0000","last_clean":"2024-09-16T10:15:45.752348+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:15:45.752348+0000","last_undegraded":"2024-09-16T10:15:45.752348+0000","last_fullsized":"2024-09-16T10:15:45.752348+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:18.059145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:18.059145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.142283+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749064+0000","last_change":"2024-09-16T10:13:41.114703+0000","last_active":"2024-09-16T10:15:45.749064+0000","last_peered":"2024-09-16T10:15:45.749064+0000","last_clean":"2024-09-16T10:15:45.749064+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:15:45.749064+0000","last_undegraded":"2024-09-16T10:15:45.749064+0000","last_fullsized":"2024-09-16T10:15:45.749064+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.378226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.756231+0000","last_change":"2024-09-16T10:13:43.283556+0000","last_active":"2024-09-16T10:15:45.756231+0000","last_peered":"2024-09-16T10:15:45.756231+0000","last_clean":"2024-09-16T10:15:45.756231+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:15:45.756231+0000","last_undegraded":"2024-09-16T10:15:45.756231+0000","last_fullsized":"2024-09-16T10:15:45.756231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.117062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"112'5870","reported_seq":5943,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.036518+0000","last_change":"2024-09-16T10:14:43.957466+0000","last_active":"2024-09-16T10:15:53.036518+0000","last_peered":"2024-09-16T10:15:53.036518+0000","last_clean":"2024-09-16T10:15:53.036518+0000","last_became_active":"2024-09-16T10:14:43.957195+0000","last_became_peered":"2024-09-16T10:14:43.957195+0000","last_unstale":"2024-09-16T10:15:53.036518+0000","last_undegraded":"2024-09-16T10:15:53.036518+0000","last_fullsized":"2024-09-16T10:15:53.036518+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5870,"log_dups_size":0,"ondisk_log_size":5870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384696320,"num_objects":5870,"num_object_clones":0,"num_object_copies":17610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5870,"num_write_kb":375680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":199,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:48.054707+0000","last_change":"2024-09-16T10:14:41.095857+0000","last_active":"2024-09-16T10:15:48.054707+0000","last_peered":"2024-09-16T10:15:48.054707+0000","last_clean":"2024-09-16T10:15:48.054707+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:15:48.054707+0000","last_undegraded":"2024-09-16T10:15:48.054707+0000","last_fullsized":"2024-09-16T10:15:48.054707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:14:41.095779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:41.095779+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:53.960257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.736016+0000","last_change":"2024-09-16T10:14:44.986853+0000","last_active":"2024-09-16T10:15:45.736016+0000","last_peered":"2024-09-16T10:15:45.736016+0000","last_clean":"2024-09-16T10:15:45.736016+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:15:45.736016+0000","last_undegraded":"2024-09-16T10:15:45.736016+0000","last_fullsized":"2024-09-16T10:15:45.736016+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:44.986805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.986805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.455538+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752858+0000","last_change":"2024-09-16T10:13:39.091269+0000","last_active":"2024-09-16T10:15:45.752858+0000","last_peered":"2024-09-16T10:15:45.752858+0000","last_clean":"2024-09-16T10:15:45.752858+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:15:45.752858+0000","last_undegraded":"2024-09-16T10:15:45.752858+0000","last_fullsized":"2024-09-16T10:15:45.752858+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:06.263288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.748378+0000","last_change":"2024-09-16T10:13:41.118142+0000","last_active":"2024-09-16T10:15:45.748378+0000","last_peered":"2024-09-16T10:15:45.748378+0000","last_clean":"2024-09-16T10:15:45.748378+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:15:45.748378+0000","last_undegraded":"2024-09-16T10:15:45.748378+0000","last_fullsized":"2024-09-16T10:15:45.748378+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.036482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752187+0000","last_change":"2024-09-16T10:13:43.124755+0000","last_active":"2024-09-16T10:15:45.752187+0000","last_peered":"2024-09-16T10:15:45.752187+0000","last_clean":"2024-09-16T10:15:45.752187+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:15:45.752187+0000","last_undegraded":"2024-09-16T10:15:45.752187+0000","last_fullsized":"2024-09-16T10:15:45.752187+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"112'5791","reported_seq":5864,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.044210+0000","last_change":"2024-09-16T10:14:43.956747+0000","last_active":"2024-09-16T10:15:53.044210+0000","last_peered":"2024-09-16T10:15:53.044210+0000","last_clean":"2024-09-16T10:15:53.044210+0000","last_became_active":"2024-09-16T10:14:43.956362+0000","last_became_peered":"2024-09-16T10:14:43.956362+0000","last_unstale":"2024-09-16T10:15:53.044210+0000","last_undegraded":"2024-09-16T10:15:53.044210+0000","last_fullsized":"2024-09-16T10:15:53.044210+0000","mapping_epoch":81,"log_start":"88'200","ondisk_log_start":"88'200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5591,"log_dups_size":0,"ondisk_log_size":5591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:45.338229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379518976,"num_objects":5791,"num_object_clones":0,"num_object_copies":17373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5791,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":142,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.750208+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:15:45.750208+0000","last_peered":"2024-09-16T10:15:45.750208+0000","last_clean":"2024-09-16T10:15:45.750208+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:15:45.750208+0000","last_undegraded":"2024-09-16T10:15:45.750208+0000","last_fullsized":"2024-09-16T10:15:45.750208+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:48.075358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.750392+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:15:45.750392+0000","last_peered":"2024-09-16T10:15:45.750392+0000","last_clean":"2024-09-16T10:15:45.750392+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:15:45.750392+0000","last_undegraded":"2024-09-16T10:15:45.750392+0000","last_fullsized":"2024-09-16T10:15:45.750392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.073692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":136,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.742025+0000","last_change":"2024-09-16T10:15:13.676523+0000","last_active":"2024-09-16T10:15:45.742025+0000","last_peered":"2024-09-16T10:15:45.742025+0000","last_clean":"2024-09-16T10:15:45.742025+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:15:45.742025+0000","last_undegraded":"2024-09-16T10:15:45.742025+0000","last_fullsized":"2024-09-16T10:15:45.742025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:13.676437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.676437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.794617+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.742270+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:15:45.742270+0000","last_peered":"2024-09-16T10:15:45.742270+0000","last_clean":"2024-09-16T10:15:45.742270+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:15:45.742270+0000","last_undegraded":"2024-09-16T10:15:45.742270+0000","last_fullsized":"2024-09-16T10:15:45.742270+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.448883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747420+0000","last_change":"2024-09-16T10:13:43.283490+0000","last_active":"2024-09-16T10:15:45.747420+0000","last_peered":"2024-09-16T10:15:45.747420+0000","last_clean":"2024-09-16T10:15:45.747420+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:15:45.747420+0000","last_undegraded":"2024-09-16T10:15:45.747420+0000","last_fullsized":"2024-09-16T10:15:45.747420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:06.569452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"112'5903","reported_seq":5976,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:52.979786+0000","last_change":"2024-09-16T10:14:43.959854+0000","last_active":"2024-09-16T10:15:52.979786+0000","last_peered":"2024-09-16T10:15:52.979786+0000","last_clean":"2024-09-16T10:15:52.979786+0000","last_became_active":"2024-09-16T10:14:43.959612+0000","last_became_peered":"2024-09-16T10:14:43.959612+0000","last_unstale":"2024-09-16T10:15:52.979786+0000","last_undegraded":"2024-09-16T10:15:52.979786+0000","last_fullsized":"2024-09-16T10:15:52.979786+0000","mapping_epoch":81,"log_start":"89'700","ondisk_log_start":"89'700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.433783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386859008,"num_objects":5903,"num_object_clones":0,"num_object_copies":17709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":377792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":140,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752610+0000","last_change":"2024-09-16T10:14:44.905004+0000","last_active":"2024-09-16T10:15:45.752610+0000","last_peered":"2024-09-16T10:15:45.752610+0000","last_clean":"2024-09-16T10:15:45.752610+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:15:45.752610+0000","last_undegraded":"2024-09-16T10:15:45.752610+0000","last_fullsized":"2024-09-16T10:15:45.752610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:14:44.904949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.904949+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.392628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.743389+0000","last_change":"2024-09-16T10:14:39.888045+0000","last_active":"2024-09-16T10:15:45.743389+0000","last_peered":"2024-09-16T10:15:45.743389+0000","last_clean":"2024-09-16T10:15:45.743389+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:15:45.743389+0000","last_undegraded":"2024-09-16T10:15:45.743389+0000","last_fullsized":"2024-09-16T10:15:45.743389+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:39.887993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:39.887993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.458794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":268,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:51.166123+0000","last_change":"2024-09-16T10:14:36.812774+0000","last_active":"2024-09-16T10:15:51.166123+0000","last_peered":"2024-09-16T10:15:51.166123+0000","last_clean":"2024-09-16T10:15:51.166123+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:15:51.166123+0000","last_undegraded":"2024-09-16T10:15:51.166123+0000","last_fullsized":"2024-09-16T10:15:51.166123+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:14:36.812720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:14:36.812720+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.775101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":125,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753441+0000","last_change":"2024-09-16T10:13:41.112756+0000","last_active":"2024-09-16T10:15:45.753441+0000","last_peered":"2024-09-16T10:15:45.753441+0000","last_clean":"2024-09-16T10:15:45.753441+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:15:45.753441+0000","last_undegraded":"2024-09-16T10:15:45.753441+0000","last_fullsized":"2024-09-16T10:15:45.753441+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.271006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752087+0000","last_change":"2024-09-16T10:13:43.123048+0000","last_active":"2024-09-16T10:15:45.752087+0000","last_peered":"2024-09-16T10:15:45.752087+0000","last_clean":"2024-09-16T10:15:45.752087+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:15:45.752087+0000","last_undegraded":"2024-09-16T10:15:45.752087+0000","last_fullsized":"2024-09-16T10:15:45.752087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:03.084010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"112'5971","reported_seq":6043,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.062949+0000","last_change":"2024-09-16T10:14:43.957718+0000","last_active":"2024-09-16T10:15:53.062949+0000","last_peered":"2024-09-16T10:15:53.062949+0000","last_clean":"2024-09-16T10:15:53.062949+0000","last_became_active":"2024-09-16T10:14:43.957476+0000","last_became_peered":"2024-09-16T10:14:43.957476+0000","last_unstale":"2024-09-16T10:15:53.062949+0000","last_undegraded":"2024-09-16T10:15:53.062949+0000","last_fullsized":"2024-09-16T10:15:53.062949+0000","mapping_epoch":81,"log_start":"89'400","ondisk_log_start":"89'400","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:48.330630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391315456,"num_objects":5971,"num_object_clones":0,"num_object_copies":17913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5971,"num_write_kb":382144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":164,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:48.050986+0000","last_change":"2024-09-16T10:13:39.106378+0000","last_active":"2024-09-16T10:15:48.050986+0000","last_peered":"2024-09-16T10:15:48.050986+0000","last_clean":"2024-09-16T10:15:48.050986+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:15:48.050986+0000","last_undegraded":"2024-09-16T10:15:48.050986+0000","last_fullsized":"2024-09-16T10:15:48.050986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.385090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.754692+0000","last_change":"2024-09-16T10:14:37.299040+0000","last_active":"2024-09-16T10:15:45.754692+0000","last_peered":"2024-09-16T10:15:45.754692+0000","last_clean":"2024-09-16T10:15:45.754692+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:15:45.754692+0000","last_undegraded":"2024-09-16T10:15:45.754692+0000","last_fullsized":"2024-09-16T10:15:45.754692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.298992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.298992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.526394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":164,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.748642+0000","last_change":"2024-09-16T10:14:14.911115+0000","last_active":"2024-09-16T10:15:45.748642+0000","last_peered":"2024-09-16T10:15:45.748642+0000","last_clean":"2024-09-16T10:15:45.748642+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:15:45.748642+0000","last_undegraded":"2024-09-16T10:15:45.748642+0000","last_fullsized":"2024-09-16T10:15:45.748642+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:14:14.911041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:14:14.911041+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:42.469843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747582+0000","last_change":"2024-09-16T10:15:26.571995+0000","last_active":"2024-09-16T10:15:45.747582+0000","last_peered":"2024-09-16T10:15:45.747582+0000","last_clean":"2024-09-16T10:15:45.747582+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:15:45.747582+0000","last_undegraded":"2024-09-16T10:15:45.747582+0000","last_fullsized":"2024-09-16T10:15:45.747582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:26.571945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:26.571945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.414009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753342+0000","last_change":"2024-09-16T10:13:43.121658+0000","last_active":"2024-09-16T10:15:45.753342+0000","last_peered":"2024-09-16T10:15:45.753342+0000","last_clean":"2024-09-16T10:15:45.753342+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:15:45.753342+0000","last_undegraded":"2024-09-16T10:15:45.753342+0000","last_fullsized":"2024-09-16T10:15:45.753342+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.316999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"112'5930","reported_seq":6001,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.026620+0000","last_change":"2024-09-16T10:14:43.958855+0000","last_active":"2024-09-16T10:15:53.026620+0000","last_peered":"2024-09-16T10:15:53.026620+0000","last_clean":"2024-09-16T10:15:53.026620+0000","last_became_active":"2024-09-16T10:14:43.958616+0000","last_became_peered":"2024-09-16T10:14:43.958616+0000","last_unstale":"2024-09-16T10:15:53.026620+0000","last_undegraded":"2024-09-16T10:15:53.026620+0000","last_fullsized":"2024-09-16T10:15:53.026620+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5930,"log_dups_size":0,"ondisk_log_size":5930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.220931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388628480,"num_objects":5930,"num_object_clones":0,"num_object_copies":17790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":379520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":140,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.736318+0000","last_change":"2024-09-16T10:14:40.987310+0000","last_active":"2024-09-16T10:15:45.736318+0000","last_peered":"2024-09-16T10:15:45.736318+0000","last_clean":"2024-09-16T10:15:45.736318+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:15:45.736318+0000","last_undegraded":"2024-09-16T10:15:45.736318+0000","last_fullsized":"2024-09-16T10:15:45.736318+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:14:40.987224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:40.987224+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.140235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735929+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:15:45.735929+0000","last_peered":"2024-09-16T10:15:45.735929+0000","last_clean":"2024-09-16T10:15:45.735929+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:15:45.735929+0000","last_undegraded":"2024-09-16T10:15:45.735929+0000","last_fullsized":"2024-09-16T10:15:45.735929+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.814101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753265+0000","last_change":"2024-09-16T10:15:30.302921+0000","last_active":"2024-09-16T10:15:45.753265+0000","last_peered":"2024-09-16T10:15:45.753265+0000","last_clean":"2024-09-16T10:15:45.753265+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:15:45.753265+0000","last_undegraded":"2024-09-16T10:15:45.753265+0000","last_fullsized":"2024-09-16T10:15:45.753265+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.302859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.302859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.575611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751111+0000","last_change":"2024-09-16T10:15:25.689754+0000","last_active":"2024-09-16T10:15:45.751111+0000","last_peered":"2024-09-16T10:15:45.751111+0000","last_clean":"2024-09-16T10:15:45.751111+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:15:45.751111+0000","last_undegraded":"2024-09-16T10:15:45.751111+0000","last_fullsized":"2024-09-16T10:15:45.751111+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:25.689615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.689615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.500134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"112'5721","reported_seq":5794,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.064160+0000","last_change":"2024-09-16T10:14:43.954148+0000","last_active":"2024-09-16T10:15:53.064160+0000","last_peered":"2024-09-16T10:15:53.064160+0000","last_clean":"2024-09-16T10:15:53.064160+0000","last_became_active":"2024-09-16T10:14:43.953871+0000","last_became_peered":"2024-09-16T10:14:43.953871+0000","last_unstale":"2024-09-16T10:15:53.064160+0000","last_undegraded":"2024-09-16T10:15:53.064160+0000","last_fullsized":"2024-09-16T10:15:53.064160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374931456,"num_objects":5721,"num_object_clones":0,"num_object_copies":17163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5721,"num_write_kb":366144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":142,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.736333+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:15:45.736333+0000","last_peered":"2024-09-16T10:15:45.736333+0000","last_clean":"2024-09-16T10:15:45.736333+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:15:45.736333+0000","last_undegraded":"2024-09-16T10:15:45.736333+0000","last_fullsized":"2024-09-16T10:15:45.736333+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.029277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749518+0000","last_change":"2024-09-16T10:13:37.089527+0000","last_active":"2024-09-16T10:15:45.749518+0000","last_peered":"2024-09-16T10:15:45.749518+0000","last_clean":"2024-09-16T10:15:45.749518+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:15:45.749518+0000","last_undegraded":"2024-09-16T10:15:45.749518+0000","last_fullsized":"2024-09-16T10:15:45.749518+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751859+0000","last_change":"2024-09-16T10:13:41.121389+0000","last_active":"2024-09-16T10:15:45.751859+0000","last_peered":"2024-09-16T10:15:45.751859+0000","last_clean":"2024-09-16T10:15:45.751859+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:15:45.751859+0000","last_undegraded":"2024-09-16T10:15:45.751859+0000","last_fullsized":"2024-09-16T10:15:45.751859+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:53.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.739054+0000","last_change":"2024-09-16T10:15:11.699384+0000","last_active":"2024-09-16T10:15:45.739054+0000","last_peered":"2024-09-16T10:15:45.739054+0000","last_clean":"2024-09-16T10:15:45.739054+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:15:45.739054+0000","last_undegraded":"2024-09-16T10:15:45.739054+0000","last_fullsized":"2024-09-16T10:15:45.739054+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.699308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.699308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.370786+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"112'5651","reported_seq":5723,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.081447+0000","last_change":"2024-09-16T10:14:43.955268+0000","last_active":"2024-09-16T10:15:53.081447+0000","last_peered":"2024-09-16T10:15:53.081447+0000","last_clean":"2024-09-16T10:15:53.081447+0000","last_became_active":"2024-09-16T10:14:43.954996+0000","last_became_peered":"2024-09-16T10:14:43.954996+0000","last_unstale":"2024-09-16T10:15:53.081447+0000","last_undegraded":"2024-09-16T10:15:53.081447+0000","last_fullsized":"2024-09-16T10:15:53.081447+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.950717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370343936,"num_objects":5651,"num_object_clones":0,"num_object_copies":16953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":361664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.737667+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:15:45.737667+0000","last_peered":"2024-09-16T10:15:45.737667+0000","last_clean":"2024-09-16T10:15:45.737667+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:15:45.737667+0000","last_undegraded":"2024-09-16T10:15:45.737667+0000","last_fullsized":"2024-09-16T10:15:45.737667+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:51.830974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751738+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:15:45.751738+0000","last_peered":"2024-09-16T10:15:45.751738+0000","last_clean":"2024-09-16T10:15:45.751738+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:15:45.751738+0000","last_undegraded":"2024-09-16T10:15:45.751738+0000","last_fullsized":"2024-09-16T10:15:45.751738+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.481976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:47.175611+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:15:47.175611+0000","last_peered":"2024-09-16T10:15:47.175611+0000","last_clean":"2024-09-16T10:15:47.175611+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:15:47.175611+0000","last_undegraded":"2024-09-16T10:15:47.175611+0000","last_fullsized":"2024-09-16T10:15:47.175611+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.466166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.736073+0000","last_change":"2024-09-16T10:13:43.119606+0000","last_active":"2024-09-16T10:15:45.736073+0000","last_peered":"2024-09-16T10:15:45.736073+0000","last_clean":"2024-09-16T10:15:45.736073+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:15:45.736073+0000","last_undegraded":"2024-09-16T10:15:45.736073+0000","last_fullsized":"2024-09-16T10:15:45.736073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:56.977559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"112'5867","reported_seq":5937,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.033962+0000","last_change":"2024-09-16T10:14:43.952726+0000","last_active":"2024-09-16T10:15:53.033962+0000","last_peered":"2024-09-16T10:15:53.033962+0000","last_clean":"2024-09-16T10:15:53.033962+0000","last_became_active":"2024-09-16T10:14:43.952378+0000","last_became_peered":"2024-09-16T10:14:43.952378+0000","last_unstale":"2024-09-16T10:15:53.033962+0000","last_undegraded":"2024-09-16T10:15:53.033962+0000","last_fullsized":"2024-09-16T10:15:53.033962+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:56.742869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384499712,"num_objects":5867,"num_object_clones":0,"num_object_copies":17601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5867,"num_write_kb":375488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.742447+0000","last_change":"2024-09-16T10:15:27.103866+0000","last_active":"2024-09-16T10:15:45.742447+0000","last_peered":"2024-09-16T10:15:45.742447+0000","last_clean":"2024-09-16T10:15:45.742447+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:15:45.742447+0000","last_undegraded":"2024-09-16T10:15:45.742447+0000","last_fullsized":"2024-09-16T10:15:45.742447+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:27.103781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:27.103781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.276258+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752879+0000","last_change":"2024-09-16T10:15:13.340979+0000","last_active":"2024-09-16T10:15:45.752879+0000","last_peered":"2024-09-16T10:15:45.752879+0000","last_clean":"2024-09-16T10:15:45.752879+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:15:45.752879+0000","last_undegraded":"2024-09-16T10:15:45.752879+0000","last_fullsized":"2024-09-16T10:15:45.752879+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.340929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.340929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:15.076648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747049+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:15:45.747049+0000","last_peered":"2024-09-16T10:15:45.747049+0000","last_clean":"2024-09-16T10:15:45.747049+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:15:45.747049+0000","last_undegraded":"2024-09-16T10:15:45.747049+0000","last_fullsized":"2024-09-16T10:15:45.747049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:51.338754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.743274+0000","last_change":"2024-09-16T10:14:45.941203+0000","last_active":"2024-09-16T10:15:45.743274+0000","last_peered":"2024-09-16T10:15:45.743274+0000","last_clean":"2024-09-16T10:15:45.743274+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:15:45.743274+0000","last_undegraded":"2024-09-16T10:15:45.743274+0000","last_fullsized":"2024-09-16T10:15:45.743274+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:14:45.941146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.941146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.998588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"112'5691","reported_seq":5764,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.042686+0000","last_change":"2024-09-16T10:14:43.960133+0000","last_active":"2024-09-16T10:15:53.042686+0000","last_peered":"2024-09-16T10:15:53.042686+0000","last_clean":"2024-09-16T10:15:53.042686+0000","last_became_active":"2024-09-16T10:14:43.959635+0000","last_became_peered":"2024-09-16T10:14:43.959635+0000","last_unstale":"2024-09-16T10:15:53.042686+0000","last_undegraded":"2024-09-16T10:15:53.042686+0000","last_fullsized":"2024-09-16T10:15:53.042686+0000","mapping_epoch":81,"log_start":"87'100","ondisk_log_start":"87'100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5591,"log_dups_size":0,"ondisk_log_size":5591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.705141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372965376,"num_objects":5691,"num_object_clones":0,"num_object_copies":17073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5691,"num_write_kb":364224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":153,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749304+0000","last_change":"2024-09-16T10:15:24.413737+0000","last_active":"2024-09-16T10:15:45.749304+0000","last_peered":"2024-09-16T10:15:45.749304+0000","last_clean":"2024-09-16T10:15:45.749304+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:15:45.749304+0000","last_undegraded":"2024-09-16T10:15:45.749304+0000","last_fullsized":"2024-09-16T10:15:45.749304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:15:24.413675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.413675+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.734723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747607+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:15:45.747607+0000","last_peered":"2024-09-16T10:15:45.747607+0000","last_clean":"2024-09-16T10:15:45.747607+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:15:45.747607+0000","last_undegraded":"2024-09-16T10:15:45.747607+0000","last_fullsized":"2024-09-16T10:15:45.747607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.009197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.739468+0000","last_change":"2024-09-16T10:14:43.459469+0000","last_active":"2024-09-16T10:15:45.739468+0000","last_peered":"2024-09-16T10:15:45.739468+0000","last_clean":"2024-09-16T10:15:45.739468+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:15:45.739468+0000","last_undegraded":"2024-09-16T10:15:45.739468+0000","last_fullsized":"2024-09-16T10:15:45.739468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:43.459419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.459419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.271593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.748637+0000","last_change":"2024-09-16T10:13:43.122094+0000","last_active":"2024-09-16T10:15:45.748637+0000","last_peered":"2024-09-16T10:15:45.748637+0000","last_clean":"2024-09-16T10:15:45.748637+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:15:45.748637+0000","last_undegraded":"2024-09-16T10:15:45.748637+0000","last_fullsized":"2024-09-16T10:15:45.748637+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.656883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"112'5684","reported_seq":5757,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.060535+0000","last_change":"2024-09-16T10:14:43.957583+0000","last_active":"2024-09-16T10:15:53.060535+0000","last_peered":"2024-09-16T10:15:53.060535+0000","last_clean":"2024-09-16T10:15:53.060535+0000","last_became_active":"2024-09-16T10:14:43.957207+0000","last_became_peered":"2024-09-16T10:14:43.957207+0000","last_unstale":"2024-09-16T10:15:53.060535+0000","last_undegraded":"2024-09-16T10:15:53.060535+0000","last_fullsized":"2024-09-16T10:15:53.060535+0000","mapping_epoch":81,"log_start":"90'1000","ondisk_log_start":"90'1000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.367126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372506624,"num_objects":5684,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":363776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":145,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735755+0000","last_change":"2024-09-16T10:15:15.038399+0000","last_active":"2024-09-16T10:15:45.735755+0000","last_peered":"2024-09-16T10:15:45.735755+0000","last_clean":"2024-09-16T10:15:45.735755+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:15:45.735755+0000","last_undegraded":"2024-09-16T10:15:45.735755+0000","last_fullsized":"2024-09-16T10:15:45.735755+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:15.038322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.038322+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:16.384094+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747759+0000","last_change":"2024-09-16T10:15:11.631330+0000","last_active":"2024-09-16T10:15:45.747759+0000","last_peered":"2024-09-16T10:15:45.747759+0000","last_clean":"2024-09-16T10:15:45.747759+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:15:45.747759+0000","last_undegraded":"2024-09-16T10:15:45.747759+0000","last_fullsized":"2024-09-16T10:15:45.747759+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.631262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.631262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.886243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.743423+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:15:45.743423+0000","last_peered":"2024-09-16T10:15:45.743423+0000","last_clean":"2024-09-16T10:15:45.743423+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:15:45.743423+0000","last_undegraded":"2024-09-16T10:15:45.743423+0000","last_fullsized":"2024-09-16T10:15:45.743423+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.452518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751361+0000","last_change":"2024-09-16T10:15:18.722246+0000","last_active":"2024-09-16T10:15:45.751361+0000","last_peered":"2024-09-16T10:15:45.751361+0000","last_clean":"2024-09-16T10:15:45.751361+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:15:45.751361+0000","last_undegraded":"2024-09-16T10:15:45.751361+0000","last_fullsized":"2024-09-16T10:15:45.751361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:18.722025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:18.722025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:29.585204+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"112'5940","reported_seq":6013,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:53.023864+0000","last_change":"2024-09-16T10:14:43.954758+0000","last_active":"2024-09-16T10:15:53.023864+0000","last_peered":"2024-09-16T10:15:53.023864+0000","last_clean":"2024-09-16T10:15:53.023864+0000","last_became_active":"2024-09-16T10:14:43.954475+0000","last_became_peered":"2024-09-16T10:14:43.954475+0000","last_unstale":"2024-09-16T10:15:53.023864+0000","last_undegraded":"2024-09-16T10:15:53.023864+0000","last_fullsized":"2024-09-16T10:15:53.023864+0000","mapping_epoch":81,"log_start":"89'900","ondisk_log_start":"89'900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:53.333918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389283840,"num_objects":5940,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5940,"num_write_kb":380160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":140,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752500+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:15:45.752500+0000","last_peered":"2024-09-16T10:15:45.752500+0000","last_clean":"2024-09-16T10:15:45.752500+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:15:45.752500+0000","last_undegraded":"2024-09-16T10:15:45.752500+0000","last_fullsized":"2024-09-16T10:15:45.752500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.246286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":139,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.748033+0000","last_change":"2024-09-16T10:15:25.529229+0000","last_active":"2024-09-16T10:15:45.748033+0000","last_peered":"2024-09-16T10:15:45.748033+0000","last_clean":"2024-09-16T10:15:45.748033+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:15:45.748033+0000","last_undegraded":"2024-09-16T10:15:45.748033+0000","last_fullsized":"2024-09-16T10:15:45.748033+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:15:25.529178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.529178+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:46.039966+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:52.181181+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:15:52.181181+0000","last_peered":"2024-09-16T10:15:52.181181+0000","last_clean":"2024-09-16T10:15:52.181181+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:15:52.181181+0000","last_undegraded":"2024-09-16T10:15:52.181181+0000","last_fullsized":"2024-09-16T10:15:52.181181+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.613927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752517+0000","last_change":"2024-09-16T10:15:17.306478+0000","last_active":"2024-09-16T10:15:45.752517+0000","last_peered":"2024-09-16T10:15:45.752517+0000","last_clean":"2024-09-16T10:15:45.752517+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:15:45.752517+0000","last_undegraded":"2024-09-16T10:15:45.752517+0000","last_fullsized":"2024-09-16T10:15:45.752517+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.306392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.306392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.352322+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.748871+0000","last_change":"2024-09-16T10:13:37.089575+0000","last_active":"2024-09-16T10:15:45.748871+0000","last_peered":"2024-09-16T10:15:45.748871+0000","last_clean":"2024-09-16T10:15:45.748871+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:15:45.748871+0000","last_undegraded":"2024-09-16T10:15:45.748871+0000","last_fullsized":"2024-09-16T10:15:45.748871+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.480266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752641+0000","last_change":"2024-09-16T10:14:41.331207+0000","last_active":"2024-09-16T10:15:45.752641+0000","last_peered":"2024-09-16T10:15:45.752641+0000","last_clean":"2024-09-16T10:15:45.752641+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:15:45.752641+0000","last_undegraded":"2024-09-16T10:15:45.752641+0000","last_fullsized":"2024-09-16T10:15:45.752641+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:14:41.331145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:41.331145+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753073+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:15:45.753073+0000","last_peered":"2024-09-16T10:15:45.753073+0000","last_clean":"2024-09-16T10:15:45.753073+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:15:45.753073+0000","last_undegraded":"2024-09-16T10:15:45.753073+0000","last_fullsized":"2024-09-16T10:15:45.753073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.231519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747336+0000","last_change":"2024-09-16T10:13:43.283644+0000","last_active":"2024-09-16T10:15:45.747336+0000","last_peered":"2024-09-16T10:15:45.747336+0000","last_clean":"2024-09-16T10:15:45.747336+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:15:45.747336+0000","last_undegraded":"2024-09-16T10:15:45.747336+0000","last_fullsized":"2024-09-16T10:15:45.747336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.487936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752664+0000","last_change":"2024-09-16T10:14:40.919863+0000","last_active":"2024-09-16T10:15:45.752664+0000","last_peered":"2024-09-16T10:15:45.752664+0000","last_clean":"2024-09-16T10:15:45.752664+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:15:45.752664+0000","last_undegraded":"2024-09-16T10:15:45.752664+0000","last_fullsized":"2024-09-16T10:15:45.752664+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:14:40.919755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:40.919755+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:49.992871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747059+0000","last_change":"2024-09-16T10:15:28.537739+0000","last_active":"2024-09-16T10:15:45.747059+0000","last_peered":"2024-09-16T10:15:45.747059+0000","last_clean":"2024-09-16T10:15:45.747059+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:15:45.747059+0000","last_undegraded":"2024-09-16T10:15:45.747059+0000","last_fullsized":"2024-09-16T10:15:45.747059+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:28.537673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.537673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.723859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752493+0000","last_change":"2024-09-16T10:13:41.118905+0000","last_active":"2024-09-16T10:15:45.752493+0000","last_peered":"2024-09-16T10:15:45.752493+0000","last_clean":"2024-09-16T10:15:45.752493+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:15:45.752493+0000","last_undegraded":"2024-09-16T10:15:45.752493+0000","last_fullsized":"2024-09-16T10:15:45.752493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.856494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.738659+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:15:45.738659+0000","last_peered":"2024-09-16T10:15:45.738659+0000","last_clean":"2024-09-16T10:15:45.738659+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:15:45.738659+0000","last_undegraded":"2024-09-16T10:15:45.738659+0000","last_fullsized":"2024-09-16T10:15:45.738659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.602226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753106+0000","last_change":"2024-09-16T10:15:16.264104+0000","last_active":"2024-09-16T10:15:45.753106+0000","last_peered":"2024-09-16T10:15:45.753106+0000","last_clean":"2024-09-16T10:15:45.753106+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:15:45.753106+0000","last_undegraded":"2024-09-16T10:15:45.753106+0000","last_fullsized":"2024-09-16T10:15:45.753106+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:16.264023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.264023+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.067434+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.740539+0000","last_change":"2024-09-16T10:14:37.844181+0000","last_active":"2024-09-16T10:15:45.740539+0000","last_peered":"2024-09-16T10:15:45.740539+0000","last_clean":"2024-09-16T10:15:45.740539+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:15:45.740539+0000","last_undegraded":"2024-09-16T10:15:45.740539+0000","last_fullsized":"2024-09-16T10:15:45.740539+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.844113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.844113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:48.542157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":154,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:47.176055+0000","last_change":"2024-09-16T10:15:28.707612+0000","last_active":"2024-09-16T10:15:47.176055+0000","last_peered":"2024-09-16T10:15:47.176055+0000","last_clean":"2024-09-16T10:15:47.176055+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:15:47.176055+0000","last_undegraded":"2024-09-16T10:15:47.176055+0000","last_fullsized":"2024-09-16T10:15:47.176055+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:28.707502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.707502+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:55.133889+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.736613+0000","last_change":"2024-09-16T10:13:43.119772+0000","last_active":"2024-09-16T10:15:45.736613+0000","last_peered":"2024-09-16T10:15:45.736613+0000","last_clean":"2024-09-16T10:15:45.736613+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:15:45.736613+0000","last_undegraded":"2024-09-16T10:15:45.736613+0000","last_fullsized":"2024-09-16T10:15:45.736613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.974833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751543+0000","last_change":"2024-09-16T10:13:39.105939+0000","last_active":"2024-09-16T10:15:45.751543+0000","last_peered":"2024-09-16T10:15:45.751543+0000","last_clean":"2024-09-16T10:15:45.751543+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:15:45.751543+0000","last_undegraded":"2024-09-16T10:15:45.751543+0000","last_fullsized":"2024-09-16T10:15:45.751543+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.532265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747252+0000","last_change":"2024-09-16T10:15:21.601115+0000","last_active":"2024-09-16T10:15:45.747252+0000","last_peered":"2024-09-16T10:15:45.747252+0000","last_clean":"2024-09-16T10:15:45.747252+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:15:45.747252+0000","last_undegraded":"2024-09-16T10:15:45.747252+0000","last_fullsized":"2024-09-16T10:15:45.747252+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.601066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.601066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:45.291040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:47.175462+0000","last_change":"2024-09-16T10:13:41.116455+0000","last_active":"2024-09-16T10:15:47.175462+0000","last_peered":"2024-09-16T10:15:47.175462+0000","last_clean":"2024-09-16T10:15:47.175462+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:15:47.175462+0000","last_undegraded":"2024-09-16T10:15:47.175462+0000","last_fullsized":"2024-09-16T10:15:47.175462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.292433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.740936+0000","last_change":"2024-09-16T10:13:43.120971+0000","last_active":"2024-09-16T10:15:45.740936+0000","last_peered":"2024-09-16T10:15:45.740936+0000","last_clean":"2024-09-16T10:15:45.740936+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:15:45.740936+0000","last_undegraded":"2024-09-16T10:15:45.740936+0000","last_fullsized":"2024-09-16T10:15:45.740936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:41.426789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":132,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.735577+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:15:45.735577+0000","last_peered":"2024-09-16T10:15:45.735577+0000","last_clean":"2024-09-16T10:15:45.735577+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:15:45.735577+0000","last_undegraded":"2024-09-16T10:15:45.735577+0000","last_fullsized":"2024-09-16T10:15:45.735577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.072263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.737790+0000","last_change":"2024-09-16T10:13:37.090129+0000","last_active":"2024-09-16T10:15:45.737790+0000","last_peered":"2024-09-16T10:15:45.737790+0000","last_clean":"2024-09-16T10:15:45.737790+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:15:45.737790+0000","last_undegraded":"2024-09-16T10:15:45.737790+0000","last_fullsized":"2024-09-16T10:15:45.737790+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:54.001301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749595+0000","last_change":"2024-09-16T10:13:41.112912+0000","last_active":"2024-09-16T10:15:45.749595+0000","last_peered":"2024-09-16T10:15:45.749595+0000","last_clean":"2024-09-16T10:15:45.749595+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:15:45.749595+0000","last_undegraded":"2024-09-16T10:15:45.749595+0000","last_fullsized":"2024-09-16T10:15:45.749595+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.005626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749362+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:15:45.749362+0000","last_peered":"2024-09-16T10:15:45.749362+0000","last_clean":"2024-09-16T10:15:45.749362+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:15:45.749362+0000","last_undegraded":"2024-09-16T10:15:45.749362+0000","last_fullsized":"2024-09-16T10:15:45.749362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.859008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749851+0000","last_change":"2024-09-16T10:15:15.010801+0000","last_active":"2024-09-16T10:15:45.749851+0000","last_peered":"2024-09-16T10:15:45.749851+0000","last_clean":"2024-09-16T10:15:45.749851+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:15:45.749851+0000","last_undegraded":"2024-09-16T10:15:45.749851+0000","last_fullsized":"2024-09-16T10:15:45.749851+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:15.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.010709+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.214304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.747496+0000","last_change":"2024-09-16T10:13:37.091081+0000","last_active":"2024-09-16T10:15:45.747496+0000","last_peered":"2024-09-16T10:15:45.747496+0000","last_clean":"2024-09-16T10:15:45.747496+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:15:45.747496+0000","last_undegraded":"2024-09-16T10:15:45.747496+0000","last_fullsized":"2024-09-16T10:15:45.747496+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.620886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751454+0000","last_change":"2024-09-16T10:13:41.114260+0000","last_active":"2024-09-16T10:15:45.751454+0000","last_peered":"2024-09-16T10:15:45.751454+0000","last_clean":"2024-09-16T10:15:45.751454+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:15:45.751454+0000","last_undegraded":"2024-09-16T10:15:45.751454+0000","last_fullsized":"2024-09-16T10:15:45.751454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:56.528615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.740267+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:15:45.740267+0000","last_peered":"2024-09-16T10:15:45.740267+0000","last_clean":"2024-09-16T10:15:45.740267+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:15:45.740267+0000","last_undegraded":"2024-09-16T10:15:45.740267+0000","last_fullsized":"2024-09-16T10:15:45.740267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.406640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749518+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:15:45.749518+0000","last_peered":"2024-09-16T10:15:45.749518+0000","last_clean":"2024-09-16T10:15:45.749518+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:15:45.749518+0000","last_undegraded":"2024-09-16T10:15:45.749518+0000","last_fullsized":"2024-09-16T10:15:45.749518+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:47.456731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.753078+0000","last_change":"2024-09-16T10:13:37.088453+0000","last_active":"2024-09-16T10:15:45.753078+0000","last_peered":"2024-09-16T10:15:45.753078+0000","last_clean":"2024-09-16T10:15:45.753078+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:15:45.753078+0000","last_undegraded":"2024-09-16T10:15:45.753078+0000","last_fullsized":"2024-09-16T10:15:45.753078+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.296404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.746763+0000","last_change":"2024-09-16T10:14:45.829620+0000","last_active":"2024-09-16T10:15:45.746763+0000","last_peered":"2024-09-16T10:15:45.746763+0000","last_clean":"2024-09-16T10:15:45.746763+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:15:45.746763+0000","last_undegraded":"2024-09-16T10:15:45.746763+0000","last_fullsized":"2024-09-16T10:15:45.746763+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:45.829572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.829572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:46.942231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.751604+0000","last_change":"2024-09-16T10:15:30.762339+0000","last_active":"2024-09-16T10:15:45.751604+0000","last_peered":"2024-09-16T10:15:45.751604+0000","last_clean":"2024-09-16T10:15:45.751604+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:15:45.751604+0000","last_undegraded":"2024-09-16T10:15:45.751604+0000","last_fullsized":"2024-09-16T10:15:45.751604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.762137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.762137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.668208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":141,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749199+0000","last_change":"2024-09-16T10:15:08.953214+0000","last_active":"2024-09-16T10:15:45.749199+0000","last_peered":"2024-09-16T10:15:45.749199+0000","last_clean":"2024-09-16T10:15:45.749199+0000","last_became_active":"2024-09-16T10:15:08.930505+0000","last_became_peered":"2024-09-16T10:15:08.930505+0000","last_unstale":"2024-09-16T10:15:45.749199+0000","last_undegraded":"2024-09-16T10:15:45.749199+0000","last_fullsized":"2024-09-16T10:15:45.749199+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:52.615906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.740780+0000","last_change":"2024-09-16T10:13:37.094533+0000","last_active":"2024-09-16T10:15:45.740780+0000","last_peered":"2024-09-16T10:15:45.740780+0000","last_clean":"2024-09-16T10:15:45.740780+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:15:45.740780+0000","last_undegraded":"2024-09-16T10:15:45.740780+0000","last_fullsized":"2024-09-16T10:15:45.740780+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.358935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.748983+0000","last_change":"2024-09-16T10:15:13.474188+0000","last_active":"2024-09-16T10:15:45.748983+0000","last_peered":"2024-09-16T10:15:45.748983+0000","last_clean":"2024-09-16T10:15:45.748983+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:15:45.748983+0000","last_undegraded":"2024-09-16T10:15:45.748983+0000","last_fullsized":"2024-09-16T10:15:45.748983+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.474125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.474125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:29.683499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.746913+0000","last_change":"2024-09-16T10:13:43.119917+0000","last_active":"2024-09-16T10:15:45.746913+0000","last_peered":"2024-09-16T10:15:45.746913+0000","last_clean":"2024-09-16T10:15:45.746913+0000","last_became_active":"2024-09-16T10:13:43.119612+0000","last_became_peered":"2024-09-16T10:13:43.119612+0000","last_unstale":"2024-09-16T10:15:45.746913+0000","last_undegraded":"2024-09-16T10:15:45.746913+0000","last_fullsized":"2024-09-16T10:15:45.746913+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.561593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.750583+0000","last_change":"2024-09-16T10:15:17.082954+0000","last_active":"2024-09-16T10:15:45.750583+0000","last_peered":"2024-09-16T10:15:45.750583+0000","last_clean":"2024-09-16T10:15:45.750583+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:15:45.750583+0000","last_undegraded":"2024-09-16T10:15:45.750583+0000","last_fullsized":"2024-09-16T10:15:45.750583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.082687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.082687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:26.257770+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749410+0000","last_change":"2024-09-16T10:15:24.663908+0000","last_active":"2024-09-16T10:15:45.749410+0000","last_peered":"2024-09-16T10:15:45.749410+0000","last_clean":"2024-09-16T10:15:45.749410+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:15:45.749410+0000","last_undegraded":"2024-09-16T10:15:45.749410+0000","last_fullsized":"2024-09-16T10:15:45.749410+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.663832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.663832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:46.507773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":141,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.749343+0000","last_change":"2024-09-16T10:14:44.362078+0000","last_active":"2024-09-16T10:15:45.749343+0000","last_peered":"2024-09-16T10:15:45.749343+0000","last_clean":"2024-09-16T10:15:45.749343+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:15:45.749343+0000","last_undegraded":"2024-09-16T10:15:45.749343+0000","last_fullsized":"2024-09-16T10:15:45.749343+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:14:44.362007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.362007+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.313495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T10:15:45.752473+0000","last_change":"2024-09-16T10:15:24.135166+0000","last_active":"2024-09-16T10:15:45.752473+0000","last_peered":"2024-09-16T10:15:45.752473+0000","last_clean":"2024-09-16T10:15:45.752473+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:15:45.752473+0000","last_undegraded":"2024-09-16T10:15:45.752473+0000","last_fullsized":"2024-09-16T10:15:45.752473+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.135091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.135091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.981416+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6107824128,"num_objects":93198,"num_object_clones":0,"num_object_copies":279594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93198,"num_write_kb":5964672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18324062208,"data_stored":18324062208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86598,"ondisk_log_size":86598,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":130,"num_read_kb":125,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234015,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1918716,"kb_used_data":1873644,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":91826436,"statfs":{"total":95995035648,"available":94030270464,"internally_reserved":0,"allocated":1918611456,"data_stored":1917943111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528355,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1934756,"kb_used_data":1885524,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":91810396,"statfs":{"total":95995035648,"available":94013845504,"internally_reserved":0,"allocated":1930776576,"data_stored":1930131709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1518768,"kb_used_data":1465376,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92226384,"statfs":{"total":95995035648,"available":94439817216,"internally_reserved":0,"allocated":1500545024,"data_stored":1499888897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986222,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1904256,"kb_used_data":1855024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":91840896,"statfs":{"total":95995035648,"available":94045077504,"internally_reserved":0,"allocated":1899544576,"data_stored":1898871534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215155,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3443596,"kb_used_data":3349948,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":90301556,"statfs":{"total":95995035648,"available":92468793344,"internally_reserved":0,"allocated":3430346752,"data_stored":3429662701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":95877576},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2282788,"kb_used_data":2211348,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91462364,"statfs":{"total":95995035648,"available":93657460736,"internally_reserved":0,"allocated":2264420352,"data_stored":2263775212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2277720,"kb_used_data":2210440,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91467432,"statfs":{"total":95995035648,"available":93662650368,"internally_reserved":0,"allocated":2263490560,"data_stored":2262793792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3021188,"kb_used_data":2958068,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":90723964,"statfs":{"total":95995035648,"available":92901339136,"internally_reserved":0,"allocated":3029061632,"data_stored":3028385797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3049586688,"data_stored":3049586688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2282356736,"data_stored":2282356736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2277834752,"data_stored":2277834752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3432120320,"data_stored":3432120320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1901264896,"data_stored":1901264896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1507852288,"data_stored":1507852288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1944977408,"data_stored":1944977408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1928069120,"data_stored":1928069120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:15:55.841 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T10:15:55.841 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:15:55.841 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:15:55.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:15:55.925 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:55.988 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:55.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: 6.d scrub starts 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: 6.d scrub ok 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: pgmap v97: 148 pgs: 148 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: 3.10 scrub starts 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: 3.10 scrub ok 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-16T10:15:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"} : dispatch 2024-09-16T10:15:56.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: osdmap e114: 8 total, 8 up, 8 in 2024-09-16T10:15:56.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-16T10:15:56.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:55 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"} : dispatch 2024-09-16T10:15:56.089 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:56.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: 6.d scrub starts 2024-09-16T10:15:56.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: 6.d scrub ok 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: pgmap v97: 148 pgs: 148 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: 3.10 scrub starts 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: 3.10 scrub ok 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"} : dispatch 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: osdmap e114: 8 total, 8 up, 8 in 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-16T10:15:56.096 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"} : dispatch 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:55] ENGINE Bus STOPPING 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:55] ENGINE Bus STOPPED 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:55] ENGINE Bus STARTING 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Serving on http://:::9283 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: 6.d scrub starts 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: 6.d scrub ok 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: pgmap v97: 148 pgs: 148 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-16T10:15:56.097 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: 3.10 scrub starts 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: 3.10 scrub ok 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"} : dispatch 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: osdmap e114: 8 total, 8 up, 8 in 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-16T10:15:56.098 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:55 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"} : dispatch 2024-09-16T10:15:56.098 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 66 16 96127 96111 90.9917 56.4375 0.0303925 0.0109841 2024-09-16T10:15:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:56.148 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:56.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:56.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:56.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:56.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:56.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:56.402 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:56.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:56.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:56.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Bus STARTED 2024-09-16T10:15:56.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Bus STOPPING 2024-09-16T10:15:56.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:56.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:56.573 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:15:56.574 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:56.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:56.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:56.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:56.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:56.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:56.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"} : dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-16T10:15:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: 5.c scrub starts 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: 5.c scrub ok 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:56 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:56.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-16T10:15:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-09-16T10:15:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"} : dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: 5.c scrub starts 2024-09-16T10:15:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: 5.c scrub ok 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Bus STOPPED 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Bus STARTING 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Serving on http://:::9283 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Bus STARTED 2024-09-16T10:15:56.954 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:56] ENGINE Bus STOPPING 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"} : dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:15:56.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:56.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:15:56.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:56.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: 5.c scrub starts 2024-09-16T10:15:56.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: 5.c scrub ok 2024-09-16T10:15:56.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:15:56.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:56.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:56 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:56.982 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:56.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:57.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:57.096 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 67 16 96959 96943 90.4091 52 0.00688062 0.0110555 2024-09-16T10:15:57.124 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:57.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:57.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:57.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:57.266 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:57.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:57.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:57.418 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:57.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:57.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T10:15:57.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:57] ENGINE Bus STOPPED 2024-09-16T10:15:57.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:57] ENGINE Bus STARTING 2024-09-16T10:15:57.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:57] ENGINE Serving on http://:::9283 2024-09-16T10:15:57.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:15:57] ENGINE Bus STARTED 2024-09-16T10:15:57.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:57.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:57.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:57.651 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:57.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:57.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:57.822 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:57.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:57.924 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:15:57.963 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:57.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:15:58.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:15:58.089 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:15:58.089 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: 7.e scrub ok 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: pgmap v99: 148 pgs: 148 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 593 op/s 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: 6.1 scrub starts 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: 6.1 scrub ok 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:15:58.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:15:58.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: 7.e scrub ok 2024-09-16T10:15:58.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: pgmap v99: 148 pgs: 148 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 593 op/s 2024-09-16T10:15:58.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: 6.1 scrub starts 2024-09-16T10:15:58.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: 6.1 scrub ok 2024-09-16T10:15:58.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:58.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:58.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:15:58.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:57 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:15:58.098 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 68 16 97985 97969 90.0216 64.125 0.0235943 0.0111039 2024-09-16T10:15:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:58.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:58.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:15:58.144 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: 7.e scrub ok 2024-09-16T10:15:58.145 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: pgmap v99: 148 pgs: 148 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 593 op/s 2024-09-16T10:15:58.145 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: 6.1 scrub starts 2024-09-16T10:15:58.145 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: 6.1 scrub ok 2024-09-16T10:15:58.145 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:15:58.145 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: from='mon.? v1:172.21.15.63:0/4020829401' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T10:15:58.145 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:15:58.145 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:57 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:15:58.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:58.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:58.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:58.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:15:58.370 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:58.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:15:58.473 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:15:58.532 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:58.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:15:58.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:15:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:15:58.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:58.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:58.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:15:58.701 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:15:58 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:15:58.560Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004446437s 2024-09-16T10:15:58.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:58.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:58.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:58.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:58.900 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:58.900 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:15:58.906 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:15:58.973 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:58.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:59.066 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:58 smithi063 ceph-mon[39470]: 5.19 scrub starts 2024-09-16T10:15:59.066 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:58 smithi063 ceph-mon[39470]: 5.19 scrub ok 2024-09-16T10:15:59.066 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:58 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:59.066 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:58 smithi063 ceph-mon[29993]: 5.19 scrub starts 2024-09-16T10:15:59.066 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:58 smithi063 ceph-mon[29993]: 5.19 scrub ok 2024-09-16T10:15:59.066 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:58 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:59.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:59.098 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 69 16 99048 99032 89.6798 66.4375 0.00722441 0.0111463 2024-09-16T10:15:59.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:59.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:59.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:15:59.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:15:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:59.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:59.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:15:59.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:58 smithi165 ceph-mon[32696]: 5.19 scrub starts 2024-09-16T10:15:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:58 smithi165 ceph-mon[32696]: 5.19 scrub ok 2024-09-16T10:15:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:58 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:15:59.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:59.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:59.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:59.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:59.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:59.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:59.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:15:59.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:59.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:15:59.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:15:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:59.777 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:15:59.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:15:59.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:15:59.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:15:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:59.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:00.009 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:00.013 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[29993]: 3.16 scrub starts 2024-09-16T10:16:00.013 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[29993]: 3.16 scrub ok 2024-09-16T10:16:00.013 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[29993]: 6.8 scrub starts 2024-09-16T10:16:00.013 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[29993]: 6.8 scrub ok 2024-09-16T10:16:00.013 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[29993]: pgmap v100: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-16T10:16:00.013 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[39470]: 3.16 scrub starts 2024-09-16T10:16:00.014 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[39470]: 3.16 scrub ok 2024-09-16T10:16:00.014 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[39470]: 6.8 scrub starts 2024-09-16T10:16:00.014 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[39470]: 6.8 scrub ok 2024-09-16T10:16:00.014 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:15:59 smithi063 ceph-mon[39470]: pgmap v100: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-16T10:16:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:59 smithi165 ceph-mon[32696]: 3.16 scrub starts 2024-09-16T10:16:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:59 smithi165 ceph-mon[32696]: 3.16 scrub ok 2024-09-16T10:16:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:59 smithi165 ceph-mon[32696]: 6.8 scrub starts 2024-09-16T10:16:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:59 smithi165 ceph-mon[32696]: 6.8 scrub ok 2024-09-16T10:16:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:15:59 smithi165 ceph-mon[32696]: pgmap v100: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-16T10:16:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:00.053 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:00.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:00.097 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 70 16 99996 99980 89.245 59.25 0.00605349 0.0111991 2024-09-16T10:16:00.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:00.206 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:00.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:00.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:00.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:00.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:00.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:00.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:00.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:00.464 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:16:00.465 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:16:00.465 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 114, 2024-09-16T10:16:00.465 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:16:00.465 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:16:00.465 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:15:55.740198+0000", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:13:06.309218+0000", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:16:00.466 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 7, 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:16:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:16:00.469 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:16:00.469 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:16:00.469 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 1, 2024-09-16T10:16:00.469 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 1, 2024-09-16T10:16:00.469 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 1, 2024-09-16T10:16:00.469 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 1, 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 1, 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "21", 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:16:00.470 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:16:00.471 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:16:00.472 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:16:00.473 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 7.8899998664855957, 2024-09-16T10:16:00.474 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 7.8899998664855957, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.37999999523162842, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 3, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 3, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:16:00.475 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:16:00.476 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.477 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:16:00.478 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:16:00.479 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:16:00.480 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:16:00.481 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:16:00.482 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:16:00.484 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:16:00.485 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:16:00.486 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:16:00.487 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:16:00.488 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.489 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:16:00.490 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:16:00.491 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:16:00.492 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:16:00.493 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:16:00.494 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.495 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.25, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.25, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.25, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.25, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.496 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:16:00.497 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:16:00.498 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:16:00.499 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:16:00.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:16:00.501 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.25, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.25, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.25, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.25, 2024-09-16T10:16:00.502 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:16:00.503 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:16:00.503 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:16:00.503 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:16:00.504 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:16:00.505 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:16:00.506 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:16:00.507 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:16:00.508 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:16:00.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 7, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_0", 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:14:42.862958+0000", 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:16:00.511 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:16:00.512 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "86", 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:16:00.513 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:16:00.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:16:00.515 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:16:00.516 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:16:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.520 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.521 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.522 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.523 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.524 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:16:00.525 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.527 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.530 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 96, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:16:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 96, 2024-09-16T10:16:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:16:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.541 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:16:00.542 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:16:00.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.542 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.542 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.542 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:16:00.543 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.544 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.545 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.546 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:16:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:16:00.548 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.549 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.552 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:15:20.495316+0000", 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:15:53.414258+0000", 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:16:00.554 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:15:23.680264+0000", 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.555 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:15:05.053001+0000", 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:15:27.811822+0000", 2024-09-16T10:16:00.556 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:15:06.433255+0000", 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:15:20.733362+0000", 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:16:00.558 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:14:51.559165+0000", 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [], 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.10", 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:16:00.559 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.18", 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:16:00.561 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:16:00.562 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:16:00.563 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:00.564 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:16:00.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:00.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:00.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:00.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:00.727 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:00.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:00.829 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:00.876 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:00.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:00.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:01.015 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:01.097 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 71 15 100913 100898 88.7962 57.375 0.0237496 0.0112566 2024-09-16T10:16:01.111 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[39470]: 5.13 scrub starts 2024-09-16T10:16:01.111 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[39470]: 5.13 scrub ok 2024-09-16T10:16:01.111 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[39470]: 7.b scrub starts 2024-09-16T10:16:01.111 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2491371123' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:16:01.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[29993]: 5.13 scrub starts 2024-09-16T10:16:01.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[29993]: 5.13 scrub ok 2024-09-16T10:16:01.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[29993]: 7.b scrub starts 2024-09-16T10:16:01.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2491371123' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:16:01.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:01.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:01.246 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:00 smithi165 ceph-mon[32696]: 5.13 scrub starts 2024-09-16T10:16:01.246 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:00 smithi165 ceph-mon[32696]: 5.13 scrub ok 2024-09-16T10:16:01.247 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:00 smithi165 ceph-mon[32696]: 7.b scrub starts 2024-09-16T10:16:01.247 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2491371123' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:16:01.248 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.296 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:01.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:01.397 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:01.442 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:01.544 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:01.589 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:01.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:01.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:01.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:01.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.865 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:01.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:01.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:01.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:01.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:02.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:02.097 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 72 15 101858 101843 88.3832 59.0625 0.00896479 0.0113094 2024-09-16T10:16:02.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:02.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: 5.15 scrub starts 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: 5.15 scrub ok 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: pgmap v101: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/497679963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:02.192 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: 5.15 scrub starts 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: 5.15 scrub ok 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: pgmap v101: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/497679963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:02.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:01 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:02.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:02.244 INFO:tasks.thrashosds.thrasher:j is {'epoch': 114, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:15:55.740198+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:13:06.309218+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T10:14:42.862958+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:15:20.495316+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:15:53.414258+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:15:23.680264+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:15:05.053001+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:15:27.811822+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:15:06.433255+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:15:20.733362+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:14:51.559165+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:16:02.245 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:16:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:02.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:02.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: 5.15 scrub starts 2024-09-16T10:16:02.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: 5.15 scrub ok 2024-09-16T10:16:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:16:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:16:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: pgmap v101: 148 pgs: 148 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-16T10:16:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/497679963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:01 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:02.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:02.424 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:02.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:02.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:02.526 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:02.526 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:02.526 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:02.526 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:02.526 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:02.526 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:02.526 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:02.527 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:02.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:02.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:02.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:02.702 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:02.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:02.732 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:02.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:02.864 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:02.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:02.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:02.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:02.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:03.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:03.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 73 15 102821 102806 87.997 60.1875 0.0137421 0.0113592 2024-09-16T10:16:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:03.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:03.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:03.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[39470]: 7.0 scrub starts 2024-09-16T10:16:03.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:16:03.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:16:03.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[39470]: 3.14 scrub starts 2024-09-16T10:16:03.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[39470]: 3.14 scrub ok 2024-09-16T10:16:03.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[39470]: 7.b scrub ok 2024-09-16T10:16:03.142 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[29993]: 7.0 scrub starts 2024-09-16T10:16:03.142 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:16:03.142 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:16:03.142 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[29993]: 3.14 scrub starts 2024-09-16T10:16:03.142 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[29993]: 3.14 scrub ok 2024-09-16T10:16:03.142 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:02 smithi063 ceph-mon[29993]: 7.b scrub ok 2024-09-16T10:16:03.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:03.286 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:03.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:03.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:02 smithi165 ceph-mon[32696]: 7.0 scrub starts 2024-09-16T10:16:03.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:02 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:16:03.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:02 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:16:03.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:02 smithi165 ceph-mon[32696]: 3.14 scrub starts 2024-09-16T10:16:03.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:02 smithi165 ceph-mon[32696]: 3.14 scrub ok 2024-09-16T10:16:03.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:02 smithi165 ceph-mon[32696]: 7.b scrub ok 2024-09-16T10:16:03.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:03.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:03.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:03.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:03.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:03.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:03.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:03.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:03.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:03.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:03.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:03.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:03.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:03.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:04.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:04.081 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:04.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:04.097 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 74 16 103872 103856 87.6946 65.625 0.0140353 0.0113979 2024-09-16T10:16:04.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:04.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:03 smithi063 ceph-mon[39470]: 7.0 scrub ok 2024-09-16T10:16:04.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:03 smithi063 ceph-mon[39470]: pgmap v102: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 551 B/s rd, 36 MiB/s wr, 581 op/s 2024-09-16T10:16:04.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:03 smithi063 ceph-mon[29993]: 7.0 scrub ok 2024-09-16T10:16:04.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:03 smithi063 ceph-mon[29993]: pgmap v102: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 551 B/s rd, 36 MiB/s wr, 581 op/s 2024-09-16T10:16:04.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:04.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:04.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:03 smithi165 ceph-mon[32696]: 7.0 scrub ok 2024-09-16T10:16:04.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:03 smithi165 ceph-mon[32696]: pgmap v102: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 551 B/s rd, 36 MiB/s wr, 581 op/s 2024-09-16T10:16:04.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:04.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:04.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:04.437 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:04.468 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:04.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:04.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:04.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:04.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:04.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:04.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:04.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:04.779 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:04.786 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:04.822 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:16:04.822 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:16:04.822 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:16:04.822 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:16:04.822 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:16:04.826 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:04.893 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:04.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:04.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[29993]: 6.3 scrub starts 2024-09-16T10:16:04.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[29993]: 6.3 scrub ok 2024-09-16T10:16:04.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[29993]: 4.1b scrub starts 2024-09-16T10:16:04.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[29993]: 4.1b scrub ok 2024-09-16T10:16:04.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[29993]: 5.5 scrub starts 2024-09-16T10:16:04.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[29993]: 5.5 scrub ok 2024-09-16T10:16:04.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[39470]: 6.3 scrub starts 2024-09-16T10:16:04.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[39470]: 6.3 scrub ok 2024-09-16T10:16:04.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[39470]: 4.1b scrub starts 2024-09-16T10:16:04.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[39470]: 4.1b scrub ok 2024-09-16T10:16:04.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[39470]: 5.5 scrub starts 2024-09-16T10:16:04.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:04 smithi063 ceph-mon[39470]: 5.5 scrub ok 2024-09-16T10:16:04.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:05.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:05.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:05.098 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 75 16 104816 104800 87.312 59 0.00839484 0.0114482 2024-09-16T10:16:05.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:05.186 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:05.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:05.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:16:05.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:05.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:04 smithi165 ceph-mon[32696]: 6.3 scrub starts 2024-09-16T10:16:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:04 smithi165 ceph-mon[32696]: 6.3 scrub ok 2024-09-16T10:16:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:04 smithi165 ceph-mon[32696]: 4.1b scrub starts 2024-09-16T10:16:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:04 smithi165 ceph-mon[32696]: 4.1b scrub ok 2024-09-16T10:16:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:04 smithi165 ceph-mon[32696]: 5.5 scrub starts 2024-09-16T10:16:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:04 smithi165 ceph-mon[32696]: 5.5 scrub ok 2024-09-16T10:16:05.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:05.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:05.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:05.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:05.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:05.544 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:05.591 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:05.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:05.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:05.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:05.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:05.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:05.872 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:05.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:05.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:06.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:06.098 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 76 15 105799 105784 86.9724 61.5 0.0110003 0.0114929 2024-09-16T10:16:06.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:06.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:06.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[29993]: 4.14 scrub starts 2024-09-16T10:16:06.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[29993]: 4.14 scrub ok 2024-09-16T10:16:06.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[29993]: pgmap v103: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-16T10:16:06.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[29993]: 3.a scrub starts 2024-09-16T10:16:06.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[29993]: 3.a scrub ok 2024-09-16T10:16:06.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[39470]: 4.14 scrub starts 2024-09-16T10:16:06.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[39470]: 4.14 scrub ok 2024-09-16T10:16:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[39470]: pgmap v103: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-16T10:16:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[39470]: 3.a scrub starts 2024-09-16T10:16:06.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:05 smithi063 ceph-mon[39470]: 3.a scrub ok 2024-09-16T10:16:06.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:06.283 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:06.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:05 smithi165 ceph-mon[32696]: 4.14 scrub starts 2024-09-16T10:16:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:05 smithi165 ceph-mon[32696]: 4.14 scrub ok 2024-09-16T10:16:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:05 smithi165 ceph-mon[32696]: pgmap v103: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-16T10:16:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:05 smithi165 ceph-mon[32696]: 3.a scrub starts 2024-09-16T10:16:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:05 smithi165 ceph-mon[32696]: 3.a scrub ok 2024-09-16T10:16:06.336 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:16:06.338 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:16:06.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:06.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:06.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:06.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:06.636 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:06.681 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:06.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:06.820 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:06.903 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:16:06.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:06.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:06.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:07.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:07.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:07.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:07.078 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: from='client.14991 v1:172.21.15.63:0/2203680140' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1767464696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[39470]: from='client.24889 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:16:07.079 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:16:07.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:16:07.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:16:07.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:16:07.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:16:07.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: from='client.14991 v1:172.21.15.63:0/2203680140' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:16:07.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1767464696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:07.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:06 smithi063 ceph-mon[29993]: from='client.24889 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:07.098 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 77 16 106669 106653 86.5482 54.3125 0.015669 0.0115491 2024-09-16T10:16:07.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:07.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:07.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:07.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:16:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:16:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:16:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:16:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:16:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:16:07.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: from='client.14991 v1:172.21.15.63:0/2203680140' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:16:07.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1767464696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:07.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:06 smithi165 ceph-mon[32696]: from='client.24889 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:07.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:07.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:07.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:07.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:16:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:16:07.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:07.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:07.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:07.600 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:07.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:07.671 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:16:07.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:07.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:07.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:07.829 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:07.877 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:07.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:07.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:08.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:08.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:08.098 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 78 16 107689 107673 86.2559 63.75 0.037804 0.0115881 2024-09-16T10:16:08.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: osdmap e115: 8 total, 8 up, 8 in 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: 4.17 scrub starts 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: 4.17 scrub ok 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: pgmap v105: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: 3.19 scrub starts 2024-09-16T10:16:08.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: 3.19 scrub ok 2024-09-16T10:16:08.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: 7.9 scrub starts 2024-09-16T10:16:08.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:08.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:08.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:08.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:08.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: osdmap e115: 8 total, 8 up, 8 in 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: 4.17 scrub starts 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: 4.17 scrub ok 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: pgmap v105: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: 3.19 scrub starts 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: 3.19 scrub ok 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: 7.9 scrub starts 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: osdmap e115: 8 total, 8 up, 8 in 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: 4.17 scrub starts 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: 4.17 scrub ok 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: pgmap v105: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:16:08.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: 3.19 scrub starts 2024-09-16T10:16:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: 3.19 scrub ok 2024-09-16T10:16:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: 7.9 scrub starts 2024-09-16T10:16:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:08.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:08.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:08.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:08.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:08.421 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:08.466 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:08.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:08.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:16:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:16:08.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:08.616 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:08.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:08.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:08.672 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-16T10:16:05.050765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6736845339,"num_objects":102986,"num_object_clones":0,"num_object_copies":308958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102986,"num_whiteouts":0,"num_read":988,"num_read_kb":845,"num_write":103336,"num_write_kb":6580041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89339,"ondisk_log_size":89339,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20222876,"kb_used_data":19629148,"kb_used_omap":12,"kb_used_meta":593587,"kb_avail":729738340,"statfs":{"total":767960285184,"available":747252060160,"internally_reserved":0,"allocated":20100247552,"data_stored":20094816353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":607833649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":628424704,"num_objects":9589,"num_object_clones":0,"num_object_copies":28767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9589,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9589,"num_write_kb":613696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002651"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757349+0000","last_change":"2024-09-16T10:15:15.058932+0000","last_active":"2024-09-16T10:15:55.757349+0000","last_peered":"2024-09-16T10:15:55.757349+0000","last_clean":"2024-09-16T10:15:55.757349+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:15:55.757349+0000","last_undegraded":"2024-09-16T10:15:55.757349+0000","last_fullsized":"2024-09-16T10:15:55.757349+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:15.058652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.058652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.247209+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":158,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745501+0000","last_change":"2024-09-16T10:15:11.063493+0000","last_active":"2024-09-16T10:15:55.745501+0000","last_peered":"2024-09-16T10:15:55.745501+0000","last_clean":"2024-09-16T10:15:55.745501+0000","last_became_active":"2024-09-16T10:15:08.925269+0000","last_became_peered":"2024-09-16T10:15:08.925269+0000","last_unstale":"2024-09-16T10:15:55.745501+0000","last_undegraded":"2024-09-16T10:15:55.745501+0000","last_fullsized":"2024-09-16T10:15:55.745501+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:11.063426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.063426+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:23.888178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:58.122926+0000","last_change":"2024-09-16T10:15:58.122926+0000","last_active":"2024-09-16T10:15:58.122926+0000","last_peered":"2024-09-16T10:15:58.122926+0000","last_clean":"2024-09-16T10:15:58.122926+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:15:58.122926+0000","last_undegraded":"2024-09-16T10:15:58.122926+0000","last_fullsized":"2024-09-16T10:15:58.122926+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:58.122856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:58.122856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:00.777958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":109,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745454+0000","last_change":"2024-09-16T10:13:43.113148+0000","last_active":"2024-09-16T10:15:55.745454+0000","last_peered":"2024-09-16T10:15:55.745454+0000","last_clean":"2024-09-16T10:15:55.745454+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:15:55.745454+0000","last_undegraded":"2024-09-16T10:15:55.745454+0000","last_fullsized":"2024-09-16T10:15:55.745454+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:01.951377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752745+0000","last_change":"2024-09-16T10:13:43.287323+0000","last_active":"2024-09-16T10:15:55.752745+0000","last_peered":"2024-09-16T10:15:55.752745+0000","last_clean":"2024-09-16T10:15:55.752745+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:15:55.752745+0000","last_undegraded":"2024-09-16T10:15:55.752745+0000","last_fullsized":"2024-09-16T10:15:55.752745+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:37.873818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753690+0000","last_change":"2024-09-16T10:15:54.003592+0000","last_active":"2024-09-16T10:15:55.753690+0000","last_peered":"2024-09-16T10:15:55.753690+0000","last_clean":"2024-09-16T10:15:55.753690+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:15:55.753690+0000","last_undegraded":"2024-09-16T10:15:55.753690+0000","last_fullsized":"2024-09-16T10:15:55.753690+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:54.003517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.003517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:01.835320+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":151,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.754265+0000","last_change":"2024-09-16T10:15:10.668770+0000","last_active":"2024-09-16T10:15:55.754265+0000","last_peered":"2024-09-16T10:15:55.754265+0000","last_clean":"2024-09-16T10:15:55.754265+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:15:55.754265+0000","last_undegraded":"2024-09-16T10:15:55.754265+0000","last_fullsized":"2024-09-16T10:15:55.754265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:15:10.668521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:10.668521+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:18.492671+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:00.858395+0000","last_change":"2024-09-16T10:16:00.858395+0000","last_active":"2024-09-16T10:16:00.858395+0000","last_peered":"2024-09-16T10:16:00.858395+0000","last_clean":"2024-09-16T10:16:00.858395+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:16:00.858395+0000","last_undegraded":"2024-09-16T10:16:00.858395+0000","last_fullsized":"2024-09-16T10:16:00.858395+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:00.858345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:00.858345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:02.855313+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.761019+0000","last_change":"2024-09-16T10:15:29.351071+0000","last_active":"2024-09-16T10:15:55.761019+0000","last_peered":"2024-09-16T10:15:55.761019+0000","last_clean":"2024-09-16T10:15:55.761019+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:15:55.761019+0000","last_undegraded":"2024-09-16T10:15:55.761019+0000","last_fullsized":"2024-09-16T10:15:55.761019+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:29.351000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.351000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:52.980664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":141,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745681+0000","last_change":"2024-09-16T10:15:16.028344+0000","last_active":"2024-09-16T10:15:55.745681+0000","last_peered":"2024-09-16T10:15:55.745681+0000","last_clean":"2024-09-16T10:15:55.745681+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:15:55.745681+0000","last_undegraded":"2024-09-16T10:15:55.745681+0000","last_fullsized":"2024-09-16T10:15:55.745681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:16.028283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.028283+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.455289+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760730+0000","last_change":"2024-09-16T10:15:22.271162+0000","last_active":"2024-09-16T10:15:55.760730+0000","last_peered":"2024-09-16T10:15:55.760730+0000","last_clean":"2024-09-16T10:15:55.760730+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:15:55.760730+0000","last_undegraded":"2024-09-16T10:15:55.760730+0000","last_fullsized":"2024-09-16T10:15:55.760730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:22.271101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:22.271101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.480708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":117,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.751396+0000","last_change":"2024-09-16T10:15:32.193135+0000","last_active":"2024-09-16T10:15:55.751396+0000","last_peered":"2024-09-16T10:15:55.751396+0000","last_clean":"2024-09-16T10:15:55.751396+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:15:55.751396+0000","last_undegraded":"2024-09-16T10:15:55.751396+0000","last_fullsized":"2024-09-16T10:15:55.751396+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:32.193067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:32.193067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:59.551498+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760785+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:15:55.760785+0000","last_peered":"2024-09-16T10:15:55.760785+0000","last_clean":"2024-09-16T10:15:55.760785+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:15:55.760785+0000","last_undegraded":"2024-09-16T10:15:55.760785+0000","last_fullsized":"2024-09-16T10:15:55.760785+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.232971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":127,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745608+0000","last_change":"2024-09-16T10:13:39.092696+0000","last_active":"2024-09-16T10:15:55.745608+0000","last_peered":"2024-09-16T10:15:55.745608+0000","last_clean":"2024-09-16T10:15:55.745608+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:15:55.745608+0000","last_undegraded":"2024-09-16T10:15:55.745608+0000","last_fullsized":"2024-09-16T10:15:55.745608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757314+0000","last_change":"2024-09-16T10:15:16.573685+0000","last_active":"2024-09-16T10:15:55.757314+0000","last_peered":"2024-09-16T10:15:55.757314+0000","last_clean":"2024-09-16T10:15:55.757314+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:15:55.757314+0000","last_undegraded":"2024-09-16T10:15:55.757314+0000","last_fullsized":"2024-09-16T10:15:55.757314+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.573614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.573614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.536483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760441+0000","last_change":"2024-09-16T10:15:11.297702+0000","last_active":"2024-09-16T10:15:55.760441+0000","last_peered":"2024-09-16T10:15:55.760441+0000","last_clean":"2024-09-16T10:15:55.760441+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:15:55.760441+0000","last_undegraded":"2024-09-16T10:15:55.760441+0000","last_fullsized":"2024-09-16T10:15:55.760441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.297599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.297599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.846847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746028+0000","last_change":"2024-09-16T10:13:43.284208+0000","last_active":"2024-09-16T10:15:55.746028+0000","last_peered":"2024-09-16T10:15:55.746028+0000","last_clean":"2024-09-16T10:15:55.746028+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:15:55.746028+0000","last_undegraded":"2024-09-16T10:15:55.746028+0000","last_fullsized":"2024-09-16T10:15:55.746028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.935116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753179+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:15:55.753179+0000","last_peered":"2024-09-16T10:15:55.753179+0000","last_clean":"2024-09-16T10:15:55.753179+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:15:55.753179+0000","last_undegraded":"2024-09-16T10:15:55.753179+0000","last_fullsized":"2024-09-16T10:15:55.753179+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:51.324103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759824+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:15:55.759824+0000","last_peered":"2024-09-16T10:15:55.759824+0000","last_clean":"2024-09-16T10:15:55.759824+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:15:55.759824+0000","last_undegraded":"2024-09-16T10:15:55.759824+0000","last_fullsized":"2024-09-16T10:15:55.759824+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:02.543443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756398+0000","last_change":"2024-09-16T10:15:10.987453+0000","last_active":"2024-09-16T10:15:55.756398+0000","last_peered":"2024-09-16T10:15:55.756398+0000","last_clean":"2024-09-16T10:15:55.756398+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:15:55.756398+0000","last_undegraded":"2024-09-16T10:15:55.756398+0000","last_fullsized":"2024-09-16T10:15:55.756398+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:10.987312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:10.987312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.809648+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.755569+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:15:55.755569+0000","last_peered":"2024-09-16T10:15:55.755569+0000","last_clean":"2024-09-16T10:15:55.755569+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:15:55.755569+0000","last_undegraded":"2024-09-16T10:15:55.755569+0000","last_fullsized":"2024-09-16T10:15:55.755569+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:45.886306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745754+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:15:55.745754+0000","last_peered":"2024-09-16T10:15:55.745754+0000","last_clean":"2024-09-16T10:15:55.745754+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:15:55.745754+0000","last_undegraded":"2024-09-16T10:15:55.745754+0000","last_fullsized":"2024-09-16T10:15:55.745754+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:48.778978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":168,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752901+0000","last_change":"2024-09-16T10:15:21.639445+0000","last_active":"2024-09-16T10:15:55.752901+0000","last_peered":"2024-09-16T10:15:55.752901+0000","last_clean":"2024-09-16T10:15:55.752901+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:15:55.752901+0000","last_undegraded":"2024-09-16T10:15:55.752901+0000","last_fullsized":"2024-09-16T10:15:55.752901+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:21.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.639361+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:30.197909+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.747767+0000","last_change":"2024-09-16T10:15:21.032515+0000","last_active":"2024-09-16T10:15:55.747767+0000","last_peered":"2024-09-16T10:15:55.747767+0000","last_clean":"2024-09-16T10:15:55.747767+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:15:55.747767+0000","last_undegraded":"2024-09-16T10:15:55.747767+0000","last_fullsized":"2024-09-16T10:15:55.747767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.032449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.032449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:21.047016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759022+0000","last_change":"2024-09-16T10:13:43.119664+0000","last_active":"2024-09-16T10:15:55.759022+0000","last_peered":"2024-09-16T10:15:55.759022+0000","last_clean":"2024-09-16T10:15:55.759022+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:15:55.759022+0000","last_undegraded":"2024-09-16T10:15:55.759022+0000","last_fullsized":"2024-09-16T10:15:55.759022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:34.427312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757730+0000","last_change":"2024-09-16T10:13:37.082594+0000","last_active":"2024-09-16T10:15:55.757730+0000","last_peered":"2024-09-16T10:15:55.757730+0000","last_clean":"2024-09-16T10:15:55.757730+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:15:55.757730+0000","last_undegraded":"2024-09-16T10:15:55.757730+0000","last_fullsized":"2024-09-16T10:15:55.757730+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.024905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":141,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753856+0000","last_change":"2024-09-16T10:15:13.091323+0000","last_active":"2024-09-16T10:15:55.753856+0000","last_peered":"2024-09-16T10:15:55.753856+0000","last_clean":"2024-09-16T10:15:55.753856+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:15:55.753856+0000","last_undegraded":"2024-09-16T10:15:55.753856+0000","last_fullsized":"2024-09-16T10:15:55.753856+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:15:13.091193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.091193+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:26.457536+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":163,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.181690+0000","last_change":"2024-09-16T10:15:12.451691+0000","last_active":"2024-09-16T10:15:57.181690+0000","last_peered":"2024-09-16T10:15:57.181690+0000","last_clean":"2024-09-16T10:15:57.181690+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:15:57.181690+0000","last_undegraded":"2024-09-16T10:15:57.181690+0000","last_fullsized":"2024-09-16T10:15:57.181690+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:12.451619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:12.451619+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:31.691482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"114'6675","reported_seq":6751,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.033166+0000","last_change":"2024-09-16T10:14:43.957473+0000","last_active":"2024-09-16T10:16:03.033166+0000","last_peered":"2024-09-16T10:16:03.033166+0000","last_clean":"2024-09-16T10:16:03.033166+0000","last_became_active":"2024-09-16T10:14:43.957174+0000","last_became_peered":"2024-09-16T10:14:43.957174+0000","last_unstale":"2024-09-16T10:16:03.033166+0000","last_undegraded":"2024-09-16T10:16:03.033166+0000","last_fullsized":"2024-09-16T10:16:03.033166+0000","mapping_epoch":81,"log_start":"89'400","ondisk_log_start":"89'400","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:47.305264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437452800,"num_objects":6675,"num_object_clones":0,"num_object_copies":20025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":427200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":152,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746723+0000","last_change":"2024-09-16T10:14:45.136557+0000","last_active":"2024-09-16T10:15:55.746723+0000","last_peered":"2024-09-16T10:15:55.746723+0000","last_clean":"2024-09-16T10:15:55.746723+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:15:55.746723+0000","last_undegraded":"2024-09-16T10:15:55.746723+0000","last_fullsized":"2024-09-16T10:15:55.746723+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:14:45.136460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.136460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:46.055139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753176+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:15:55.753176+0000","last_peered":"2024-09-16T10:15:55.753176+0000","last_clean":"2024-09-16T10:15:55.753176+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:15:55.753176+0000","last_undegraded":"2024-09-16T10:15:55.753176+0000","last_fullsized":"2024-09-16T10:15:55.753176+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.842587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":152,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:02.180753+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:16:02.180753+0000","last_peered":"2024-09-16T10:16:02.180753+0000","last_clean":"2024-09-16T10:16:02.180753+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:16:02.180753+0000","last_undegraded":"2024-09-16T10:16:02.180753+0000","last_fullsized":"2024-09-16T10:16:02.180753+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.097659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760734+0000","last_change":"2024-09-16T10:15:54.314741+0000","last_active":"2024-09-16T10:15:55.760734+0000","last_peered":"2024-09-16T10:15:55.760734+0000","last_clean":"2024-09-16T10:15:55.760734+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:15:55.760734+0000","last_undegraded":"2024-09-16T10:15:55.760734+0000","last_fullsized":"2024-09-16T10:15:55.760734+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:54.314680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.314680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:01.378112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"114'6366","reported_seq":6443,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.103443+0000","last_change":"2024-09-16T10:14:43.953298+0000","last_active":"2024-09-16T10:16:03.103443+0000","last_peered":"2024-09-16T10:16:03.103443+0000","last_clean":"2024-09-16T10:16:03.103443+0000","last_became_active":"2024-09-16T10:14:43.952713+0000","last_became_peered":"2024-09-16T10:14:43.952713+0000","last_unstale":"2024-09-16T10:16:03.103443+0000","last_undegraded":"2024-09-16T10:16:03.103443+0000","last_fullsized":"2024-09-16T10:16:03.103443+0000","mapping_epoch":81,"log_start":"92'1700","ondisk_log_start":"92'1700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417202176,"num_objects":6366,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6366,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":159,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752978+0000","last_change":"2024-09-16T10:15:23.703368+0000","last_active":"2024-09-16T10:15:55.752978+0000","last_peered":"2024-09-16T10:15:55.752978+0000","last_clean":"2024-09-16T10:15:55.752978+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:15:55.752978+0000","last_undegraded":"2024-09-16T10:15:55.752978+0000","last_fullsized":"2024-09-16T10:15:55.752978+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:15:23.703278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.703278+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.263361+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757137+0000","last_change":"2024-09-16T10:13:37.093660+0000","last_active":"2024-09-16T10:15:55.757137+0000","last_peered":"2024-09-16T10:15:55.757137+0000","last_clean":"2024-09-16T10:15:55.757137+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:15:55.757137+0000","last_undegraded":"2024-09-16T10:15:55.757137+0000","last_fullsized":"2024-09-16T10:15:55.757137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.591570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745579+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:15:55.745579+0000","last_peered":"2024-09-16T10:15:55.745579+0000","last_clean":"2024-09-16T10:15:55.745579+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:15:55.745579+0000","last_undegraded":"2024-09-16T10:15:55.745579+0000","last_fullsized":"2024-09-16T10:15:55.745579+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:02.988717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745625+0000","last_change":"2024-09-16T10:15:12.079425+0000","last_active":"2024-09-16T10:15:55.745625+0000","last_peered":"2024-09-16T10:15:55.745625+0000","last_clean":"2024-09-16T10:15:55.745625+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:15:55.745625+0000","last_undegraded":"2024-09-16T10:15:55.745625+0000","last_fullsized":"2024-09-16T10:15:55.745625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:12.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:12.079353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.080838+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"114'6401","reported_seq":6478,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.099137+0000","last_change":"2024-09-16T10:14:43.953541+0000","last_active":"2024-09-16T10:16:03.099137+0000","last_peered":"2024-09-16T10:16:03.099137+0000","last_clean":"2024-09-16T10:16:03.099137+0000","last_became_active":"2024-09-16T10:14:43.952908+0000","last_became_peered":"2024-09-16T10:14:43.952908+0000","last_unstale":"2024-09-16T10:16:03.099137+0000","last_undegraded":"2024-09-16T10:16:03.099137+0000","last_fullsized":"2024-09-16T10:16:03.099137+0000","mapping_epoch":81,"log_start":"92'1700","ondisk_log_start":"92'1700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419495936,"num_objects":6401,"num_object_clones":0,"num_object_copies":19203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6401,"num_write_kb":409664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":147,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745666+0000","last_change":"2024-09-16T10:15:17.025884+0000","last_active":"2024-09-16T10:15:55.745666+0000","last_peered":"2024-09-16T10:15:55.745666+0000","last_clean":"2024-09-16T10:15:55.745666+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:15:55.745666+0000","last_undegraded":"2024-09-16T10:15:55.745666+0000","last_fullsized":"2024-09-16T10:15:55.745666+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:15:17.025825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.025825+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.846848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.754135+0000","last_change":"2024-09-16T10:15:29.146696+0000","last_active":"2024-09-16T10:15:55.754135+0000","last_peered":"2024-09-16T10:15:55.754135+0000","last_clean":"2024-09-16T10:15:55.754135+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:15:55.754135+0000","last_undegraded":"2024-09-16T10:15:55.754135+0000","last_fullsized":"2024-09-16T10:15:55.754135+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:29.146482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.146482+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:56.251083+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":164,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.190078+0000","last_change":"2024-09-16T10:15:20.675679+0000","last_active":"2024-09-16T10:15:57.190078+0000","last_peered":"2024-09-16T10:15:57.190078+0000","last_clean":"2024-09-16T10:15:57.190078+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:15:57.190078+0000","last_undegraded":"2024-09-16T10:15:57.190078+0000","last_fullsized":"2024-09-16T10:15:57.190078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:20.675600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:20.675600+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.817505+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.758149+0000","last_change":"2024-09-16T10:13:43.284631+0000","last_active":"2024-09-16T10:15:55.758149+0000","last_peered":"2024-09-16T10:15:55.758149+0000","last_clean":"2024-09-16T10:15:55.758149+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:15:55.758149+0000","last_undegraded":"2024-09-16T10:15:55.758149+0000","last_fullsized":"2024-09-16T10:15:55.758149+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:58.518019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"114'6499","reported_seq":6576,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.015019+0000","last_change":"2024-09-16T10:14:43.952332+0000","last_active":"2024-09-16T10:16:03.015019+0000","last_peered":"2024-09-16T10:16:03.015019+0000","last_clean":"2024-09-16T10:16:03.015019+0000","last_became_active":"2024-09-16T10:14:43.952043+0000","last_became_peered":"2024-09-16T10:14:43.952043+0000","last_unstale":"2024-09-16T10:16:03.015019+0000","last_undegraded":"2024-09-16T10:16:03.015019+0000","last_fullsized":"2024-09-16T10:16:03.015019+0000","mapping_epoch":81,"log_start":"92'1700","ondisk_log_start":"92'1700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425918464,"num_objects":6499,"num_object_clones":0,"num_object_copies":19497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6499,"num_write_kb":415936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":142,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.748624+0000","last_change":"2024-09-16T10:14:42.119372+0000","last_active":"2024-09-16T10:15:55.748624+0000","last_peered":"2024-09-16T10:15:55.748624+0000","last_clean":"2024-09-16T10:15:55.748624+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:15:55.748624+0000","last_undegraded":"2024-09-16T10:15:55.748624+0000","last_fullsized":"2024-09-16T10:15:55.748624+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:14:42.119310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.119310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.359829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.761107+0000","last_change":"2024-09-16T10:15:23.304088+0000","last_active":"2024-09-16T10:15:55.761107+0000","last_peered":"2024-09-16T10:15:55.761107+0000","last_clean":"2024-09-16T10:15:55.761107+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:15:55.761107+0000","last_undegraded":"2024-09-16T10:15:55.761107+0000","last_fullsized":"2024-09-16T10:15:55.761107+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:23.304033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.304033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.813644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":160,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.181735+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:15:57.181735+0000","last_peered":"2024-09-16T10:15:57.181735+0000","last_clean":"2024-09-16T10:15:57.181735+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:15:57.181735+0000","last_undegraded":"2024-09-16T10:15:57.181735+0000","last_fullsized":"2024-09-16T10:15:57.181735+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.731855+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.751631+0000","last_change":"2024-09-16T10:15:16.104168+0000","last_active":"2024-09-16T10:15:55.751631+0000","last_peered":"2024-09-16T10:15:55.751631+0000","last_clean":"2024-09-16T10:15:55.751631+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:15:55.751631+0000","last_undegraded":"2024-09-16T10:15:55.751631+0000","last_fullsized":"2024-09-16T10:15:55.751631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.104092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.104092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:41.867492+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"114'6321","reported_seq":6470,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:02.982401+0000","last_change":"2024-09-16T10:16:02.977482+0000","last_active":"2024-09-16T10:16:02.982401+0000","last_peered":"2024-09-16T10:16:02.982401+0000","last_clean":"2024-09-16T10:16:02.982401+0000","last_became_active":"2024-09-16T10:14:43.955808+0000","last_became_peered":"2024-09-16T10:14:43.955808+0000","last_unstale":"2024-09-16T10:16:02.982401+0000","last_undegraded":"2024-09-16T10:16:02.982401+0000","last_fullsized":"2024-09-16T10:16:02.982401+0000","mapping_epoch":81,"log_start":"89'500","ondisk_log_start":"89'500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6320","last_scrub_stamp":"2024-09-16T10:16:02.977318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.977318+0000","objects_scrubbed":6273,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:26.363900+0000","scrub_duration":1630,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414253056,"num_objects":6321,"num_object_clones":0,"num_object_copies":18963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":404544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":162,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752939+0000","last_change":"2024-09-16T10:15:26.113170+0000","last_active":"2024-09-16T10:15:55.752939+0000","last_peered":"2024-09-16T10:15:55.752939+0000","last_clean":"2024-09-16T10:15:55.752939+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:15:55.752939+0000","last_undegraded":"2024-09-16T10:15:55.752939+0000","last_fullsized":"2024-09-16T10:15:55.752939+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:15:26.110570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:26.110570+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.337507+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.750802+0000","last_change":"2024-09-16T10:15:18.059218+0000","last_active":"2024-09-16T10:15:55.750802+0000","last_peered":"2024-09-16T10:15:55.750802+0000","last_clean":"2024-09-16T10:15:55.750802+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:15:55.750802+0000","last_undegraded":"2024-09-16T10:15:55.750802+0000","last_fullsized":"2024-09-16T10:15:55.750802+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:18.059145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:18.059145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.142283+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757076+0000","last_change":"2024-09-16T10:13:41.114703+0000","last_active":"2024-09-16T10:15:55.757076+0000","last_peered":"2024-09-16T10:15:55.757076+0000","last_clean":"2024-09-16T10:15:55.757076+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:15:55.757076+0000","last_undegraded":"2024-09-16T10:15:55.757076+0000","last_fullsized":"2024-09-16T10:15:55.757076+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.378226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.100199+0000","last_change":"2024-09-16T10:15:57.100199+0000","last_active":"2024-09-16T10:15:57.100199+0000","last_peered":"2024-09-16T10:15:57.100199+0000","last_clean":"2024-09-16T10:15:57.100199+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:15:57.100199+0000","last_undegraded":"2024-09-16T10:15:57.100199+0000","last_fullsized":"2024-09-16T10:15:57.100199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:57.099989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:57.099989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:00.664000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"114'6467","reported_seq":6544,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.008518+0000","last_change":"2024-09-16T10:14:43.957466+0000","last_active":"2024-09-16T10:16:03.008518+0000","last_peered":"2024-09-16T10:16:03.008518+0000","last_clean":"2024-09-16T10:16:03.008518+0000","last_became_active":"2024-09-16T10:14:43.957195+0000","last_became_peered":"2024-09-16T10:14:43.957195+0000","last_unstale":"2024-09-16T10:16:03.008518+0000","last_undegraded":"2024-09-16T10:16:03.008518+0000","last_fullsized":"2024-09-16T10:16:03.008518+0000","mapping_epoch":81,"log_start":"88'200","ondisk_log_start":"88'200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.581902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423821312,"num_objects":6467,"num_object_clones":0,"num_object_copies":19401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":205,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.180089+0000","last_change":"2024-09-16T10:14:41.095857+0000","last_active":"2024-09-16T10:15:57.180089+0000","last_peered":"2024-09-16T10:15:57.180089+0000","last_clean":"2024-09-16T10:15:57.180089+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:15:57.180089+0000","last_undegraded":"2024-09-16T10:15:57.180089+0000","last_fullsized":"2024-09-16T10:15:57.180089+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:14:41.095779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:41.095779+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746017+0000","last_change":"2024-09-16T10:14:44.986853+0000","last_active":"2024-09-16T10:15:55.746017+0000","last_peered":"2024-09-16T10:15:55.746017+0000","last_clean":"2024-09-16T10:15:55.746017+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:15:55.746017+0000","last_undegraded":"2024-09-16T10:15:55.746017+0000","last_fullsized":"2024-09-16T10:15:55.746017+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:44.986805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.986805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":133,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760825+0000","last_change":"2024-09-16T10:13:39.091269+0000","last_active":"2024-09-16T10:15:55.760825+0000","last_peered":"2024-09-16T10:15:55.760825+0000","last_clean":"2024-09-16T10:15:55.760825+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:15:55.760825+0000","last_undegraded":"2024-09-16T10:15:55.760825+0000","last_fullsized":"2024-09-16T10:15:55.760825+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:13:35.065568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:06.263288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753036+0000","last_change":"2024-09-16T10:13:41.118142+0000","last_active":"2024-09-16T10:15:55.753036+0000","last_peered":"2024-09-16T10:15:55.753036+0000","last_clean":"2024-09-16T10:15:55.753036+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:15:55.753036+0000","last_undegraded":"2024-09-16T10:15:55.753036+0000","last_fullsized":"2024-09-16T10:15:55.753036+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.036482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.980513+0000","last_change":"2024-09-16T10:15:55.980513+0000","last_active":"2024-09-16T10:15:55.980513+0000","last_peered":"2024-09-16T10:15:55.980513+0000","last_clean":"2024-09-16T10:15:55.980513+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:15:55.980513+0000","last_undegraded":"2024-09-16T10:15:55.980513+0000","last_fullsized":"2024-09-16T10:15:55.980513+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:55.980419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:55.980419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:07.856554+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"114'6373","reported_seq":6450,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.082768+0000","last_change":"2024-09-16T10:14:43.956747+0000","last_active":"2024-09-16T10:16:03.082768+0000","last_peered":"2024-09-16T10:16:03.082768+0000","last_clean":"2024-09-16T10:16:03.082768+0000","last_became_active":"2024-09-16T10:14:43.956362+0000","last_became_peered":"2024-09-16T10:14:43.956362+0000","last_unstale":"2024-09-16T10:16:03.082768+0000","last_undegraded":"2024-09-16T10:16:03.082768+0000","last_fullsized":"2024-09-16T10:16:03.082768+0000","mapping_epoch":81,"log_start":"89'800","ondisk_log_start":"89'800","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5573,"log_dups_size":0,"ondisk_log_size":5573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:45.338229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417660928,"num_objects":6373,"num_object_clones":0,"num_object_copies":19119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":407872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":146,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746769+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:15:55.746769+0000","last_peered":"2024-09-16T10:15:55.746769+0000","last_clean":"2024-09-16T10:15:55.746769+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:15:55.746769+0000","last_undegraded":"2024-09-16T10:15:55.746769+0000","last_fullsized":"2024-09-16T10:15:55.746769+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:48.075358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746333+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:15:55.746333+0000","last_peered":"2024-09-16T10:15:55.746333+0000","last_clean":"2024-09-16T10:15:55.746333+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:15:55.746333+0000","last_undegraded":"2024-09-16T10:15:55.746333+0000","last_fullsized":"2024-09-16T10:15:55.746333+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:51.111950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":140,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759288+0000","last_change":"2024-09-16T10:15:13.676523+0000","last_active":"2024-09-16T10:15:55.759288+0000","last_peered":"2024-09-16T10:15:55.759288+0000","last_clean":"2024-09-16T10:15:55.759288+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:15:55.759288+0000","last_undegraded":"2024-09-16T10:15:55.759288+0000","last_fullsized":"2024-09-16T10:15:55.759288+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:13.676437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.676437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.794617+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.773704+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:15:55.773704+0000","last_peered":"2024-09-16T10:15:55.773704+0000","last_clean":"2024-09-16T10:15:55.773704+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:15:55.773704+0000","last_undegraded":"2024-09-16T10:15:55.773704+0000","last_fullsized":"2024-09-16T10:15:55.773704+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:57.448883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.758823+0000","last_change":"2024-09-16T10:13:43.283490+0000","last_active":"2024-09-16T10:15:55.758823+0000","last_peered":"2024-09-16T10:15:55.758823+0000","last_clean":"2024-09-16T10:15:55.758823+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:15:55.758823+0000","last_undegraded":"2024-09-16T10:15:55.758823+0000","last_fullsized":"2024-09-16T10:15:55.758823+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:06.569452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"114'6493","reported_seq":6570,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.015046+0000","last_change":"2024-09-16T10:14:43.959854+0000","last_active":"2024-09-16T10:16:03.015046+0000","last_peered":"2024-09-16T10:16:03.015046+0000","last_clean":"2024-09-16T10:16:03.015046+0000","last_became_active":"2024-09-16T10:14:43.959612+0000","last_became_peered":"2024-09-16T10:14:43.959612+0000","last_unstale":"2024-09-16T10:16:03.015046+0000","last_undegraded":"2024-09-16T10:16:03.015046+0000","last_fullsized":"2024-09-16T10:16:03.015046+0000","mapping_epoch":81,"log_start":"91'1300","ondisk_log_start":"91'1300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5193,"log_dups_size":0,"ondisk_log_size":5193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.433783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425525248,"num_objects":6493,"num_object_clones":0,"num_object_copies":19479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6493,"num_write_kb":415552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":144,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753948+0000","last_change":"2024-09-16T10:14:44.905004+0000","last_active":"2024-09-16T10:15:55.753948+0000","last_peered":"2024-09-16T10:15:55.753948+0000","last_clean":"2024-09-16T10:15:55.753948+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:15:55.753948+0000","last_undegraded":"2024-09-16T10:15:55.753948+0000","last_fullsized":"2024-09-16T10:15:55.753948+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:14:44.904949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.904949+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.392628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":138,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.380280+0000","last_change":"2024-09-16T10:15:57.380280+0000","last_active":"2024-09-16T10:15:57.380280+0000","last_peered":"2024-09-16T10:15:57.380280+0000","last_clean":"2024-09-16T10:15:57.380280+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:15:57.380280+0000","last_undegraded":"2024-09-16T10:15:57.380280+0000","last_fullsized":"2024-09-16T10:15:57.380280+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:57.380105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:57.380105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:06.347624+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":282,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:01.176126+0000","last_change":"2024-09-16T10:14:36.812774+0000","last_active":"2024-09-16T10:16:01.176126+0000","last_peered":"2024-09-16T10:16:01.176126+0000","last_clean":"2024-09-16T10:16:01.176126+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:16:01.176126+0000","last_undegraded":"2024-09-16T10:16:01.176126+0000","last_fullsized":"2024-09-16T10:16:01.176126+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:14:36.812720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:14:36.812720+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":140,"num_read_kb":135,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.761398+0000","last_change":"2024-09-16T10:13:41.112756+0000","last_active":"2024-09-16T10:15:55.761398+0000","last_peered":"2024-09-16T10:15:55.761398+0000","last_clean":"2024-09-16T10:15:55.761398+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:15:55.761398+0000","last_undegraded":"2024-09-16T10:15:55.761398+0000","last_fullsized":"2024-09-16T10:15:55.761398+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.271006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.748198+0000","last_change":"2024-09-16T10:13:43.123048+0000","last_active":"2024-09-16T10:15:55.748198+0000","last_peered":"2024-09-16T10:15:55.748198+0000","last_clean":"2024-09-16T10:15:55.748198+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:15:55.748198+0000","last_undegraded":"2024-09-16T10:15:55.748198+0000","last_fullsized":"2024-09-16T10:15:55.748198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:03.084010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"114'6600","reported_seq":6673,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.084183+0000","last_change":"2024-09-16T10:14:43.957718+0000","last_active":"2024-09-16T10:16:03.084183+0000","last_peered":"2024-09-16T10:16:03.084183+0000","last_clean":"2024-09-16T10:16:03.084183+0000","last_became_active":"2024-09-16T10:14:43.957476+0000","last_became_peered":"2024-09-16T10:14:43.957476+0000","last_unstale":"2024-09-16T10:16:03.084183+0000","last_undegraded":"2024-09-16T10:16:03.084183+0000","last_fullsized":"2024-09-16T10:16:03.084183+0000","mapping_epoch":81,"log_start":"90'1000","ondisk_log_start":"90'1000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:48.330630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537600,"num_objects":6600,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6600,"num_write_kb":422400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":170,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.181754+0000","last_change":"2024-09-16T10:13:39.106378+0000","last_active":"2024-09-16T10:15:57.181754+0000","last_peered":"2024-09-16T10:15:57.181754+0000","last_clean":"2024-09-16T10:15:57.181754+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:15:57.181754+0000","last_undegraded":"2024-09-16T10:15:57.181754+0000","last_fullsized":"2024-09-16T10:15:57.181754+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.385090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756067+0000","last_change":"2024-09-16T10:14:37.299040+0000","last_active":"2024-09-16T10:15:55.756067+0000","last_peered":"2024-09-16T10:15:55.756067+0000","last_clean":"2024-09-16T10:15:55.756067+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:15:55.756067+0000","last_undegraded":"2024-09-16T10:15:55.756067+0000","last_fullsized":"2024-09-16T10:15:55.756067+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.298992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.298992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":168,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756967+0000","last_change":"2024-09-16T10:14:14.911115+0000","last_active":"2024-09-16T10:15:55.756967+0000","last_peered":"2024-09-16T10:15:55.756967+0000","last_clean":"2024-09-16T10:15:55.756967+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:15:55.756967+0000","last_undegraded":"2024-09-16T10:15:55.756967+0000","last_fullsized":"2024-09-16T10:15:55.756967+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:14:14.911041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:14:14.911041+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:42.469843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759141+0000","last_change":"2024-09-16T10:15:26.571995+0000","last_active":"2024-09-16T10:15:55.759141+0000","last_peered":"2024-09-16T10:15:55.759141+0000","last_clean":"2024-09-16T10:15:55.759141+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:15:55.759141+0000","last_undegraded":"2024-09-16T10:15:55.759141+0000","last_fullsized":"2024-09-16T10:15:55.759141+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:26.571945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:26.571945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.414009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.761231+0000","last_change":"2024-09-16T10:13:43.121658+0000","last_active":"2024-09-16T10:15:55.761231+0000","last_peered":"2024-09-16T10:15:55.761231+0000","last_clean":"2024-09-16T10:15:55.761231+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:15:55.761231+0000","last_undegraded":"2024-09-16T10:15:55.761231+0000","last_fullsized":"2024-09-16T10:15:55.761231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.316999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"114'6519","reported_seq":6594,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.045512+0000","last_change":"2024-09-16T10:14:43.958855+0000","last_active":"2024-09-16T10:16:03.045512+0000","last_peered":"2024-09-16T10:16:03.045512+0000","last_clean":"2024-09-16T10:16:03.045512+0000","last_became_active":"2024-09-16T10:14:43.958616+0000","last_became_peered":"2024-09-16T10:14:43.958616+0000","last_unstale":"2024-09-16T10:16:03.045512+0000","last_undegraded":"2024-09-16T10:16:03.045512+0000","last_fullsized":"2024-09-16T10:16:03.045512+0000","mapping_epoch":81,"log_start":"88'200","ondisk_log_start":"88'200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.220931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427229184,"num_objects":6519,"num_object_clones":0,"num_object_copies":19557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":417216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":145,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746174+0000","last_change":"2024-09-16T10:14:40.987310+0000","last_active":"2024-09-16T10:15:55.746174+0000","last_peered":"2024-09-16T10:15:55.746174+0000","last_clean":"2024-09-16T10:15:55.746174+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:15:55.746174+0000","last_undegraded":"2024-09-16T10:15:55.746174+0000","last_fullsized":"2024-09-16T10:15:55.746174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:14:40.987224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:40.987224+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:52.878254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746019+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:15:55.746019+0000","last_peered":"2024-09-16T10:15:55.746019+0000","last_clean":"2024-09-16T10:15:55.746019+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:15:55.746019+0000","last_undegraded":"2024-09-16T10:15:55.746019+0000","last_fullsized":"2024-09-16T10:15:55.746019+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.814101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.761193+0000","last_change":"2024-09-16T10:15:30.302921+0000","last_active":"2024-09-16T10:15:55.761193+0000","last_peered":"2024-09-16T10:15:55.761193+0000","last_clean":"2024-09-16T10:15:55.761193+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:15:55.761193+0000","last_undegraded":"2024-09-16T10:15:55.761193+0000","last_fullsized":"2024-09-16T10:15:55.761193+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.302859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.302859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.575611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759105+0000","last_change":"2024-09-16T10:15:25.689754+0000","last_active":"2024-09-16T10:15:55.759105+0000","last_peered":"2024-09-16T10:15:55.759105+0000","last_clean":"2024-09-16T10:15:55.759105+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:15:55.759105+0000","last_undegraded":"2024-09-16T10:15:55.759105+0000","last_fullsized":"2024-09-16T10:15:55.759105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:25.689615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.689615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.500134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"114'6264","reported_seq":6410,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:02.999941+0000","last_change":"2024-09-16T10:15:56.898348+0000","last_active":"2024-09-16T10:16:02.999941+0000","last_peered":"2024-09-16T10:16:02.999941+0000","last_clean":"2024-09-16T10:16:02.999941+0000","last_became_active":"2024-09-16T10:14:43.953871+0000","last_became_peered":"2024-09-16T10:14:43.953871+0000","last_unstale":"2024-09-16T10:16:02.999941+0000","last_undegraded":"2024-09-16T10:16:02.999941+0000","last_fullsized":"2024-09-16T10:16:02.999941+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'5944","last_scrub_stamp":"2024-09-16T10:15:56.897937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.897937+0000","objects_scrubbed":5880,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:25.773998+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410517504,"num_objects":6264,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6264,"num_write_kb":400896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":146,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746176+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:15:55.746176+0000","last_peered":"2024-09-16T10:15:55.746176+0000","last_clean":"2024-09-16T10:15:55.746176+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:15:55.746176+0000","last_undegraded":"2024-09-16T10:15:55.746176+0000","last_fullsized":"2024-09-16T10:15:55.746176+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.029277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757413+0000","last_change":"2024-09-16T10:13:37.089527+0000","last_active":"2024-09-16T10:15:55.757413+0000","last_peered":"2024-09-16T10:15:55.757413+0000","last_clean":"2024-09-16T10:15:55.757413+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:15:55.757413+0000","last_undegraded":"2024-09-16T10:15:55.757413+0000","last_fullsized":"2024-09-16T10:15:55.757413+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:56.092378+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:15:56.092378+0000","last_peered":"2024-09-16T10:15:56.092378+0000","last_clean":"2024-09-16T10:15:56.092378+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:15:56.092378+0000","last_undegraded":"2024-09-16T10:15:56.092378+0000","last_fullsized":"2024-09-16T10:15:56.092378+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:19.875908+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.767024+0000","last_change":"2024-09-16T10:15:11.699384+0000","last_active":"2024-09-16T10:15:55.767024+0000","last_peered":"2024-09-16T10:15:55.767024+0000","last_clean":"2024-09-16T10:15:55.767024+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:15:55.767024+0000","last_undegraded":"2024-09-16T10:15:55.767024+0000","last_fullsized":"2024-09-16T10:15:55.767024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.699308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.699308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.370786+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"114'6277","reported_seq":6354,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.073450+0000","last_change":"2024-09-16T10:14:43.955268+0000","last_active":"2024-09-16T10:16:03.073450+0000","last_peered":"2024-09-16T10:16:03.073450+0000","last_clean":"2024-09-16T10:16:03.073450+0000","last_became_active":"2024-09-16T10:14:43.954996+0000","last_became_peered":"2024-09-16T10:14:43.954996+0000","last_unstale":"2024-09-16T10:16:03.073450+0000","last_undegraded":"2024-09-16T10:16:03.073450+0000","last_fullsized":"2024-09-16T10:16:03.073450+0000","mapping_epoch":81,"log_start":"89'700","ondisk_log_start":"89'700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.950717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411369472,"num_objects":6277,"num_object_clones":0,"num_object_copies":18831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":401728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":133,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.747808+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:15:55.747808+0000","last_peered":"2024-09-16T10:15:55.747808+0000","last_clean":"2024-09-16T10:15:55.747808+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:15:55.747808+0000","last_undegraded":"2024-09-16T10:15:55.747808+0000","last_fullsized":"2024-09-16T10:15:55.747808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:51.830974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753360+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:15:55.753360+0000","last_peered":"2024-09-16T10:15:55.753360+0000","last_clean":"2024-09-16T10:15:55.753360+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:15:55.753360+0000","last_undegraded":"2024-09-16T10:15:55.753360+0000","last_fullsized":"2024-09-16T10:15:55.753360+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:53.003172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":153,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:58.104476+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:15:58.104476+0000","last_peered":"2024-09-16T10:15:58.104476+0000","last_clean":"2024-09-16T10:15:58.104476+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:15:58.104476+0000","last_undegraded":"2024-09-16T10:15:58.104476+0000","last_fullsized":"2024-09-16T10:15:58.104476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746017+0000","last_change":"2024-09-16T10:13:43.119606+0000","last_active":"2024-09-16T10:15:55.746017+0000","last_peered":"2024-09-16T10:15:55.746017+0000","last_clean":"2024-09-16T10:15:55.746017+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:15:55.746017+0000","last_undegraded":"2024-09-16T10:15:55.746017+0000","last_fullsized":"2024-09-16T10:15:55.746017+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"114'6435","reported_seq":6512,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.088009+0000","last_change":"2024-09-16T10:14:43.952726+0000","last_active":"2024-09-16T10:16:03.088009+0000","last_peered":"2024-09-16T10:16:03.088009+0000","last_clean":"2024-09-16T10:16:03.088009+0000","last_became_active":"2024-09-16T10:14:43.952378+0000","last_became_peered":"2024-09-16T10:14:43.952378+0000","last_unstale":"2024-09-16T10:16:03.088009+0000","last_undegraded":"2024-09-16T10:16:03.088009+0000","last_fullsized":"2024-09-16T10:16:03.088009+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421724160,"num_objects":6435,"num_object_clones":0,"num_object_copies":19305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":142,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752852+0000","last_change":"2024-09-16T10:15:27.103866+0000","last_active":"2024-09-16T10:15:55.752852+0000","last_peered":"2024-09-16T10:15:55.752852+0000","last_clean":"2024-09-16T10:15:55.752852+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:15:55.752852+0000","last_undegraded":"2024-09-16T10:15:55.752852+0000","last_fullsized":"2024-09-16T10:15:55.752852+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:27.103781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:27.103781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.276258+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760930+0000","last_change":"2024-09-16T10:15:13.340979+0000","last_active":"2024-09-16T10:15:55.760930+0000","last_peered":"2024-09-16T10:15:55.760930+0000","last_clean":"2024-09-16T10:15:55.760930+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:15:55.760930+0000","last_undegraded":"2024-09-16T10:15:55.760930+0000","last_fullsized":"2024-09-16T10:15:55.760930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.340929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.340929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:15.076648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760086+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:15:55.760086+0000","last_peered":"2024-09-16T10:15:55.760086+0000","last_clean":"2024-09-16T10:15:55.760086+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:15:55.760086+0000","last_undegraded":"2024-09-16T10:15:55.760086+0000","last_fullsized":"2024-09-16T10:15:55.760086+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:51.338754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.751176+0000","last_change":"2024-09-16T10:14:45.941203+0000","last_active":"2024-09-16T10:15:55.751176+0000","last_peered":"2024-09-16T10:15:55.751176+0000","last_clean":"2024-09-16T10:15:55.751176+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:15:55.751176+0000","last_undegraded":"2024-09-16T10:15:55.751176+0000","last_fullsized":"2024-09-16T10:15:55.751176+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:14:45.941146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.941146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:53.339798+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"114'6274","reported_seq":6351,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.071928+0000","last_change":"2024-09-16T10:14:43.960133+0000","last_active":"2024-09-16T10:16:03.071928+0000","last_peered":"2024-09-16T10:16:03.071928+0000","last_clean":"2024-09-16T10:16:03.071928+0000","last_became_active":"2024-09-16T10:14:43.959635+0000","last_became_peered":"2024-09-16T10:14:43.959635+0000","last_unstale":"2024-09-16T10:16:03.071928+0000","last_undegraded":"2024-09-16T10:16:03.071928+0000","last_fullsized":"2024-09-16T10:16:03.071928+0000","mapping_epoch":81,"log_start":"89'700","ondisk_log_start":"89'700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411172864,"num_objects":6274,"num_object_clones":0,"num_object_copies":18822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":157,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757186+0000","last_change":"2024-09-16T10:15:24.413737+0000","last_active":"2024-09-16T10:15:55.757186+0000","last_peered":"2024-09-16T10:15:55.757186+0000","last_clean":"2024-09-16T10:15:55.757186+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:15:55.757186+0000","last_undegraded":"2024-09-16T10:15:55.757186+0000","last_fullsized":"2024-09-16T10:15:55.757186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:15:24.413675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.413675+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.734723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759087+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:15:55.759087+0000","last_peered":"2024-09-16T10:15:55.759087+0000","last_clean":"2024-09-16T10:15:55.759087+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:15:55.759087+0000","last_undegraded":"2024-09-16T10:15:55.759087+0000","last_fullsized":"2024-09-16T10:15:55.759087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.009197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.763158+0000","last_change":"2024-09-16T10:14:43.459469+0000","last_active":"2024-09-16T10:15:55.763158+0000","last_peered":"2024-09-16T10:15:55.763158+0000","last_clean":"2024-09-16T10:15:55.763158+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:15:55.763158+0000","last_undegraded":"2024-09-16T10:15:55.763158+0000","last_fullsized":"2024-09-16T10:15:55.763158+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:43.459419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.459419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:05.271593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:58.491618+0000","last_change":"2024-09-16T10:15:58.491618+0000","last_active":"2024-09-16T10:15:58.491618+0000","last_peered":"2024-09-16T10:15:58.491618+0000","last_clean":"2024-09-16T10:15:58.491618+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:15:58.491618+0000","last_undegraded":"2024-09-16T10:15:58.491618+0000","last_fullsized":"2024-09-16T10:15:58.491618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:58.491555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:58.491555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:07.790688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"114'6283","reported_seq":6359,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.068417+0000","last_change":"2024-09-16T10:14:43.957583+0000","last_active":"2024-09-16T10:16:03.068417+0000","last_peered":"2024-09-16T10:16:03.068417+0000","last_clean":"2024-09-16T10:16:03.068417+0000","last_became_active":"2024-09-16T10:14:43.957207+0000","last_became_peered":"2024-09-16T10:14:43.957207+0000","last_unstale":"2024-09-16T10:16:03.068417+0000","last_undegraded":"2024-09-16T10:16:03.068417+0000","last_fullsized":"2024-09-16T10:16:03.068417+0000","mapping_epoch":81,"log_start":"92'1600","ondisk_log_start":"92'1600","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:11.367126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411762688,"num_objects":6283,"num_object_clones":0,"num_object_copies":18849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":149,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745865+0000","last_change":"2024-09-16T10:15:15.038399+0000","last_active":"2024-09-16T10:15:55.745865+0000","last_peered":"2024-09-16T10:15:55.745865+0000","last_clean":"2024-09-16T10:15:55.745865+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:15:55.745865+0000","last_undegraded":"2024-09-16T10:15:55.745865+0000","last_fullsized":"2024-09-16T10:15:55.745865+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:15.038322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.038322+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:16.384094+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759656+0000","last_change":"2024-09-16T10:15:11.631330+0000","last_active":"2024-09-16T10:15:55.759656+0000","last_peered":"2024-09-16T10:15:55.759656+0000","last_clean":"2024-09-16T10:15:55.759656+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:15:55.759656+0000","last_undegraded":"2024-09-16T10:15:55.759656+0000","last_fullsized":"2024-09-16T10:15:55.759656+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.631262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.631262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:35.886243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759362+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:15:55.759362+0000","last_peered":"2024-09-16T10:15:55.759362+0000","last_clean":"2024-09-16T10:15:55.759362+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:15:55.759362+0000","last_undegraded":"2024-09-16T10:15:55.759362+0000","last_fullsized":"2024-09-16T10:15:55.759362+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752904+0000","last_change":"2024-09-16T10:15:18.722246+0000","last_active":"2024-09-16T10:15:55.752904+0000","last_peered":"2024-09-16T10:15:55.752904+0000","last_clean":"2024-09-16T10:15:55.752904+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:15:55.752904+0000","last_undegraded":"2024-09-16T10:15:55.752904+0000","last_fullsized":"2024-09-16T10:15:55.752904+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:18.722025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:18.722025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:29.585204+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"114'6540","reported_seq":6688,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:03.055963+0000","last_change":"2024-09-16T10:16:02.172955+0000","last_active":"2024-09-16T10:16:03.055963+0000","last_peered":"2024-09-16T10:16:03.055963+0000","last_clean":"2024-09-16T10:16:03.055963+0000","last_became_active":"2024-09-16T10:14:43.954475+0000","last_became_peered":"2024-09-16T10:14:43.954475+0000","last_unstale":"2024-09-16T10:16:03.055963+0000","last_undegraded":"2024-09-16T10:16:03.055963+0000","last_fullsized":"2024-09-16T10:16:03.055963+0000","mapping_epoch":81,"log_start":"92'1500","ondisk_log_start":"92'1500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6489","last_scrub_stamp":"2024-09-16T10:16:02.172829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.172829+0000","objects_scrubbed":6427,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.789342+0000","scrub_duration":2056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428605440,"num_objects":6540,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6540,"num_write_kb":418560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":144,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.758913+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:15:55.758913+0000","last_peered":"2024-09-16T10:15:55.758913+0000","last_clean":"2024-09-16T10:15:55.758913+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:15:55.758913+0000","last_undegraded":"2024-09-16T10:15:55.758913+0000","last_fullsized":"2024-09-16T10:15:55.758913+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.246286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760095+0000","last_change":"2024-09-16T10:15:25.529229+0000","last_active":"2024-09-16T10:15:55.760095+0000","last_peered":"2024-09-16T10:15:55.760095+0000","last_clean":"2024-09-16T10:15:55.760095+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:15:55.760095+0000","last_undegraded":"2024-09-16T10:15:55.760095+0000","last_fullsized":"2024-09-16T10:15:55.760095+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:15:25.529178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.529178+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:46.039966+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":153,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:57.181666+0000","last_change":"2024-09-16T10:13:41.112968+0000","last_active":"2024-09-16T10:15:57.181666+0000","last_peered":"2024-09-16T10:15:57.181666+0000","last_clean":"2024-09-16T10:15:57.181666+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:15:57.181666+0000","last_undegraded":"2024-09-16T10:15:57.181666+0000","last_fullsized":"2024-09-16T10:15:57.181666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:07.613927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760313+0000","last_change":"2024-09-16T10:15:17.306478+0000","last_active":"2024-09-16T10:15:55.760313+0000","last_peered":"2024-09-16T10:15:55.760313+0000","last_clean":"2024-09-16T10:15:55.760313+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:15:55.760313+0000","last_undegraded":"2024-09-16T10:15:55.760313+0000","last_fullsized":"2024-09-16T10:15:55.760313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.306392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.306392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.352322+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756922+0000","last_change":"2024-09-16T10:15:55.157938+0000","last_active":"2024-09-16T10:15:55.756922+0000","last_peered":"2024-09-16T10:15:55.756922+0000","last_clean":"2024-09-16T10:15:55.756922+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:15:55.756922+0000","last_undegraded":"2024-09-16T10:15:55.756922+0000","last_fullsized":"2024-09-16T10:15:55.756922+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:55.157860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:55.157860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:00.050522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":135,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756637+0000","last_change":"2024-09-16T10:14:41.331207+0000","last_active":"2024-09-16T10:15:55.756637+0000","last_peered":"2024-09-16T10:15:55.756637+0000","last_clean":"2024-09-16T10:15:55.756637+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:15:55.756637+0000","last_undegraded":"2024-09-16T10:15:55.756637+0000","last_fullsized":"2024-09-16T10:15:55.756637+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:14:41.331145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:41.331145+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":115,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.760895+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:15:55.760895+0000","last_peered":"2024-09-16T10:15:55.760895+0000","last_clean":"2024-09-16T10:15:55.760895+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:15:55.760895+0000","last_undegraded":"2024-09-16T10:15:55.760895+0000","last_fullsized":"2024-09-16T10:15:55.760895+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:50.354301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.758742+0000","last_change":"2024-09-16T10:13:43.283644+0000","last_active":"2024-09-16T10:15:55.758742+0000","last_peered":"2024-09-16T10:15:55.758742+0000","last_clean":"2024-09-16T10:15:55.758742+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:15:55.758742+0000","last_undegraded":"2024-09-16T10:15:55.758742+0000","last_fullsized":"2024-09-16T10:15:55.758742+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.487936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":150,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.754076+0000","last_change":"2024-09-16T10:15:54.369062+0000","last_active":"2024-09-16T10:15:55.754076+0000","last_peered":"2024-09-16T10:15:55.754076+0000","last_clean":"2024-09-16T10:15:55.754076+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:15:55.754076+0000","last_undegraded":"2024-09-16T10:15:55.754076+0000","last_fullsized":"2024-09-16T10:15:55.754076+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:54.368987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.368987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:15.287775+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.758158+0000","last_change":"2024-09-16T10:15:28.537739+0000","last_active":"2024-09-16T10:15:55.758158+0000","last_peered":"2024-09-16T10:15:55.758158+0000","last_clean":"2024-09-16T10:15:55.758158+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:15:55.758158+0000","last_undegraded":"2024-09-16T10:15:55.758158+0000","last_fullsized":"2024-09-16T10:15:55.758158+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:28.537673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.537673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.723859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756445+0000","last_change":"2024-09-16T10:13:41.118905+0000","last_active":"2024-09-16T10:15:55.756445+0000","last_peered":"2024-09-16T10:15:55.756445+0000","last_clean":"2024-09-16T10:15:55.756445+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:15:55.756445+0000","last_undegraded":"2024-09-16T10:15:55.756445+0000","last_fullsized":"2024-09-16T10:15:55.756445+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:36.856494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":111,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759470+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:15:55.759470+0000","last_peered":"2024-09-16T10:15:55.759470+0000","last_clean":"2024-09-16T10:15:55.759470+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:15:55.759470+0000","last_undegraded":"2024-09-16T10:15:55.759470+0000","last_fullsized":"2024-09-16T10:15:55.759470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:04.602226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.761058+0000","last_change":"2024-09-16T10:15:16.264104+0000","last_active":"2024-09-16T10:15:55.761058+0000","last_peered":"2024-09-16T10:15:55.761058+0000","last_clean":"2024-09-16T10:15:55.761058+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:15:55.761058+0000","last_undegraded":"2024-09-16T10:15:55.761058+0000","last_fullsized":"2024-09-16T10:15:55.761058+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:16.264023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.264023+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:32.067434+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752548+0000","last_change":"2024-09-16T10:14:37.844181+0000","last_active":"2024-09-16T10:15:55.752548+0000","last_peered":"2024-09-16T10:15:55.752548+0000","last_clean":"2024-09-16T10:15:55.752548+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:15:55.752548+0000","last_undegraded":"2024-09-16T10:15:55.752548+0000","last_fullsized":"2024-09-16T10:15:55.752548+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.844113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.844113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.294298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":161,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:02.181049+0000","last_change":"2024-09-16T10:15:28.707612+0000","last_active":"2024-09-16T10:16:02.181049+0000","last_peered":"2024-09-16T10:16:02.181049+0000","last_clean":"2024-09-16T10:16:02.181049+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:16:02.181049+0000","last_undegraded":"2024-09-16T10:16:02.181049+0000","last_fullsized":"2024-09-16T10:16:02.181049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:28.707502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.707502+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:55.133889+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":109,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.746184+0000","last_change":"2024-09-16T10:13:43.119772+0000","last_active":"2024-09-16T10:15:55.746184+0000","last_peered":"2024-09-16T10:15:55.746184+0000","last_clean":"2024-09-16T10:15:55.746184+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:15:55.746184+0000","last_undegraded":"2024-09-16T10:15:55.746184+0000","last_fullsized":"2024-09-16T10:15:55.746184+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:15:55.974833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":134,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.755652+0000","last_change":"2024-09-16T10:13:39.105939+0000","last_active":"2024-09-16T10:15:55.755652+0000","last_peered":"2024-09-16T10:15:55.755652+0000","last_clean":"2024-09-16T10:15:55.755652+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:15:55.755652+0000","last_undegraded":"2024-09-16T10:15:55.755652+0000","last_fullsized":"2024-09-16T10:15:55.755652+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.758382+0000","last_change":"2024-09-16T10:15:21.601115+0000","last_active":"2024-09-16T10:15:55.758382+0000","last_peered":"2024-09-16T10:15:55.758382+0000","last_clean":"2024-09-16T10:15:55.758382+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:15:55.758382+0000","last_undegraded":"2024-09-16T10:15:55.758382+0000","last_fullsized":"2024-09-16T10:15:55.758382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.601066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.601066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:45.291040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":161,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:02.180277+0000","last_change":"2024-09-16T10:16:00.378802+0000","last_active":"2024-09-16T10:16:02.180277+0000","last_peered":"2024-09-16T10:16:02.180277+0000","last_clean":"2024-09-16T10:16:02.180277+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:16:02.180277+0000","last_undegraded":"2024-09-16T10:16:02.180277+0000","last_fullsized":"2024-09-16T10:16:02.180277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:16:00.378709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:00.378709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:16.958987+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":109,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753518+0000","last_change":"2024-09-16T10:13:43.120971+0000","last_active":"2024-09-16T10:15:55.753518+0000","last_peered":"2024-09-16T10:15:55.753518+0000","last_clean":"2024-09-16T10:15:55.753518+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:15:55.753518+0000","last_undegraded":"2024-09-16T10:15:55.753518+0000","last_fullsized":"2024-09-16T10:15:55.753518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:41.426789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":137,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.745752+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:15:55.745752+0000","last_peered":"2024-09-16T10:15:55.745752+0000","last_clean":"2024-09-16T10:15:55.745752+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:15:55.745752+0000","last_undegraded":"2024-09-16T10:15:55.745752+0000","last_fullsized":"2024-09-16T10:15:55.745752+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.072263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":130,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:01.807968+0000","last_change":"2024-09-16T10:16:01.807900+0000","last_active":"2024-09-16T10:16:01.807968+0000","last_peered":"2024-09-16T10:16:01.807968+0000","last_clean":"2024-09-16T10:16:01.807968+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:16:01.807968+0000","last_undegraded":"2024-09-16T10:16:01.807968+0000","last_fullsized":"2024-09-16T10:16:01.807968+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:01.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:01.807849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:13.237151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.748533+0000","last_change":"2024-09-16T10:15:54.054120+0000","last_active":"2024-09-16T10:15:55.748533+0000","last_peered":"2024-09-16T10:15:55.748533+0000","last_clean":"2024-09-16T10:15:55.748533+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:15:55.748533+0000","last_undegraded":"2024-09-16T10:15:55.748533+0000","last_fullsized":"2024-09-16T10:15:55.748533+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:54.053761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.053761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:12.885813+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753864+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:15:55.753864+0000","last_peered":"2024-09-16T10:15:55.753864+0000","last_clean":"2024-09-16T10:15:55.753864+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:15:55.753864+0000","last_undegraded":"2024-09-16T10:15:55.753864+0000","last_fullsized":"2024-09-16T10:15:55.753864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:44.859008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.748733+0000","last_change":"2024-09-16T10:15:15.010801+0000","last_active":"2024-09-16T10:15:55.748733+0000","last_peered":"2024-09-16T10:15:55.748733+0000","last_clean":"2024-09-16T10:15:55.748733+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:15:55.748733+0000","last_undegraded":"2024-09-16T10:15:55.748733+0000","last_fullsized":"2024-09-16T10:15:55.748733+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:15.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.010709+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:22.214304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759023+0000","last_change":"2024-09-16T10:13:37.091081+0000","last_active":"2024-09-16T10:15:55.759023+0000","last_peered":"2024-09-16T10:15:55.759023+0000","last_clean":"2024-09-16T10:15:55.759023+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:15:55.759023+0000","last_undegraded":"2024-09-16T10:15:55.759023+0000","last_fullsized":"2024-09-16T10:15:55.759023+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:08.620886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:59.987816+0000","last_change":"2024-09-16T10:15:59.987731+0000","last_active":"2024-09-16T10:15:59.987816+0000","last_peered":"2024-09-16T10:15:59.987816+0000","last_clean":"2024-09-16T10:15:59.987816+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:15:59.987816+0000","last_undegraded":"2024-09-16T10:15:59.987816+0000","last_fullsized":"2024-09-16T10:15:59.987816+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:59.987642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:59.987642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:20.901671+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.751279+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:15:55.751279+0000","last_peered":"2024-09-16T10:15:55.751279+0000","last_clean":"2024-09-16T10:15:55.751279+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:15:55.751279+0000","last_undegraded":"2024-09-16T10:15:55.751279+0000","last_fullsized":"2024-09-16T10:15:55.751279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:43.406640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":138,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756788+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:15:55.756788+0000","last_peered":"2024-09-16T10:15:55.756788+0000","last_clean":"2024-09-16T10:15:55.756788+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:15:55.756788+0000","last_undegraded":"2024-09-16T10:15:55.756788+0000","last_fullsized":"2024-09-16T10:15:55.756788+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:51.957282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:58.378717+0000","last_change":"2024-09-16T10:15:58.378648+0000","last_active":"2024-09-16T10:15:58.378717+0000","last_peered":"2024-09-16T10:15:58.378717+0000","last_clean":"2024-09-16T10:15:58.378717+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:15:58.378717+0000","last_undegraded":"2024-09-16T10:15:58.378717+0000","last_fullsized":"2024-09-16T10:15:58.378717+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:58.378585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:58.378585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.749256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756690+0000","last_change":"2024-09-16T10:14:45.829620+0000","last_active":"2024-09-16T10:15:55.756690+0000","last_peered":"2024-09-16T10:15:55.756690+0000","last_clean":"2024-09-16T10:15:55.756690+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:15:55.756690+0000","last_undegraded":"2024-09-16T10:15:55.756690+0000","last_fullsized":"2024-09-16T10:15:55.756690+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:45.829572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.829572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:48.398932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753280+0000","last_change":"2024-09-16T10:15:30.762339+0000","last_active":"2024-09-16T10:15:55.753280+0000","last_peered":"2024-09-16T10:15:55.753280+0000","last_clean":"2024-09-16T10:15:55.753280+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:15:55.753280+0000","last_undegraded":"2024-09-16T10:15:55.753280+0000","last_fullsized":"2024-09-16T10:15:55.753280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.762137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.762137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:49.668208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":146,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.752932+0000","last_change":"2024-09-16T10:15:08.953214+0000","last_active":"2024-09-16T10:15:55.752932+0000","last_peered":"2024-09-16T10:15:55.752932+0000","last_clean":"2024-09-16T10:15:55.752932+0000","last_became_active":"2024-09-16T10:15:08.930505+0000","last_became_peered":"2024-09-16T10:15:08.930505+0000","last_unstale":"2024-09-16T10:15:55.752932+0000","last_undegraded":"2024-09-16T10:15:55.752932+0000","last_fullsized":"2024-09-16T10:15:55.752932+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":128,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.757610+0000","last_change":"2024-09-16T10:13:37.094533+0000","last_active":"2024-09-16T10:15:55.757610+0000","last_peered":"2024-09-16T10:15:55.757610+0000","last_clean":"2024-09-16T10:15:55.757610+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:15:55.757610+0000","last_undegraded":"2024-09-16T10:15:55.757610+0000","last_fullsized":"2024-09-16T10:15:55.757610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:38.358935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756915+0000","last_change":"2024-09-16T10:15:13.474188+0000","last_active":"2024-09-16T10:15:55.756915+0000","last_peered":"2024-09-16T10:15:55.756915+0000","last_clean":"2024-09-16T10:15:55.756915+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:15:55.756915+0000","last_undegraded":"2024-09-16T10:15:55.756915+0000","last_fullsized":"2024-09-16T10:15:55.756915+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.474125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.474125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:29.683499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":119,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:16:01.542246+0000","last_change":"2024-09-16T10:16:01.542162+0000","last_active":"2024-09-16T10:16:01.542246+0000","last_peered":"2024-09-16T10:16:01.542246+0000","last_clean":"2024-09-16T10:16:01.542246+0000","last_became_active":"2024-09-16T10:13:43.119612+0000","last_became_peered":"2024-09-16T10:13:43.119612+0000","last_unstale":"2024-09-16T10:16:01.542246+0000","last_undegraded":"2024-09-16T10:16:01.542246+0000","last_fullsized":"2024-09-16T10:16:01.542246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:01.542093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:01.542093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:04.108899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.756008+0000","last_change":"2024-09-16T10:15:17.082954+0000","last_active":"2024-09-16T10:15:55.756008+0000","last_peered":"2024-09-16T10:15:55.756008+0000","last_clean":"2024-09-16T10:15:55.756008+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:15:55.756008+0000","last_undegraded":"2024-09-16T10:15:55.756008+0000","last_fullsized":"2024-09-16T10:15:55.756008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.082687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.082687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:26.257770+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":131,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753137+0000","last_change":"2024-09-16T10:15:24.663908+0000","last_active":"2024-09-16T10:15:55.753137+0000","last_peered":"2024-09-16T10:15:55.753137+0000","last_clean":"2024-09-16T10:15:55.753137+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:15:55.753137+0000","last_undegraded":"2024-09-16T10:15:55.753137+0000","last_fullsized":"2024-09-16T10:15:55.753137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.663832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.663832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:46.507773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":145,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.759231+0000","last_change":"2024-09-16T10:14:44.362078+0000","last_active":"2024-09-16T10:15:55.759231+0000","last_peered":"2024-09-16T10:15:55.759231+0000","last_clean":"2024-09-16T10:15:55.759231+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:15:55.759231+0000","last_undegraded":"2024-09-16T10:15:55.759231+0000","last_fullsized":"2024-09-16T10:15:55.759231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:14:44.362007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.362007+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:00.313495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T10:15:55.753762+0000","last_change":"2024-09-16T10:15:24.135166+0000","last_active":"2024-09-16T10:15:55.753762+0000","last_peered":"2024-09-16T10:15:55.753762+0000","last_clean":"2024-09-16T10:15:55.753762+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:15:55.753762+0000","last_undegraded":"2024-09-16T10:15:55.753762+0000","last_fullsized":"2024-09-16T10:15:55.753762+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.135091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.135091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:16:33.981416+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6736248832,"num_objects":102787,"num_object_clones":0,"num_object_copies":308361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102787,"num_write_kb":6578368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20209205248,"data_stored":20209205248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88787,"ondisk_log_size":88787,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":135,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234017,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2144852,"kb_used_data":2069252,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91600300,"statfs":{"total":95995035648,"available":93798707200,"internally_reserved":0,"allocated":2118914048,"data_stored":2118234825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2153148,"kb_used_data":2077548,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91592004,"statfs":{"total":95995035648,"available":93790212096,"internally_reserved":0,"allocated":2127409152,"data_stored":2126753407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789995,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1666120,"kb_used_data":1616888,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92079032,"statfs":{"total":95995035648,"available":94288928768,"internally_reserved":0,"allocated":1655693312,"data_stored":1655026307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2113752,"kb_used_data":2038152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91631400,"statfs":{"total":95995035648,"available":93830553600,"internally_reserved":0,"allocated":2087067648,"data_stored":2086383728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215157,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3792292,"kb_used_data":3698644,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":89952860,"statfs":{"total":95995035648,"available":92111728640,"internally_reserved":0,"allocated":3787411456,"data_stored":3786716527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":95877576},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2498108,"kb_used_data":2430828,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91247044,"statfs":{"total":95995035648,"available":93436973056,"internally_reserved":0,"allocated":2489167872,"data_stored":2488511854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2500016,"kb_used_data":2436896,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91245136,"statfs":{"total":95995035648,"available":93435019264,"internally_reserved":0,"allocated":2495381504,"data_stored":2494673858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3354588,"kb_used_data":3260940,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":90390564,"statfs":{"total":95995035648,"available":92559937536,"internally_reserved":0,"allocated":3339202560,"data_stored":3338515847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3360292864,"data_stored":3360292864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2518286336,"data_stored":2518286336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2506555392,"data_stored":2506555392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3795976192,"data_stored":3795976192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2092695552,"data_stored":2092695552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1664221184,"data_stored":1664221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2143223808,"data_stored":2143223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2127953920,"data_stored":2127953920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:16:08.679 INFO:tasks.thrashosds.thrasher:Setting 6.12 to [6, 2, 7, 3, 1, 5] 2024-09-16T10:16:08.679 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.12', '6', '2', '7', '3', '1', '5'] 2024-09-16T10:16:08.679 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 6.12 6 2 7 3 1 5 2024-09-16T10:16:08.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:08.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:08.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:08.845 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:08.916 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:16:08.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:16:08.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:08.918 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:08.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:09.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:09.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:09.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:09.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 79 16 108635 108619 85.912 59.125 0.0103976 0.0116348 2024-09-16T10:16:09.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:09.155 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:09.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:16:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:16:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:16:09.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:16:09.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:16:09.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:08 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:16:09.216 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:09.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:09.245 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:09.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:08 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:16:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:08 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:16:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:08 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:08 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:16:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:08 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:16:09.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:09.365 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:09.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:09.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:09.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:09.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:09.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:09.647 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:09.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:09.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:09.801 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:09.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:09.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:09.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:09.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:09.999 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:10.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:10.081 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:10.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:16:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:16:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:16:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:16:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[29993]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-09-16T10:16:10.095 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:16:10.095 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:16:10.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:16:10.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:16:10.096 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:09 smithi063 ceph-mon[39470]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-09-16T10:16:10.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:16:10.098466+0000 min lat: 0.00122794 max lat: 0.201222 avg lat: 0.0116855 2024-09-16T10:16:10.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:16:10.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 80 16 109524 109508 85.5322 55.5625 0.0157367 0.0116855 2024-09-16T10:16:10.128 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:10.128 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:10.171 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:10.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:10.224 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:10.225 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:10.251 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:10.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:10.268 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:10.269 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:10.281 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:10.282 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:10.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:09 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:16:10.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:09 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:16:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:09 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:16:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:09 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:16:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:09 smithi165 ceph-mon[32696]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-09-16T10:16:10.304 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:10.304 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:10.340 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:10.340 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:10.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:10.374 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:10.374 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:10.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:10.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:10.410 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:10.410 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:10.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:10.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:10.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:10.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:10.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:10.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:10.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:10.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:10.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:10.924 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:10.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:11.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:11.064 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:11.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:11.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 81 15 110527 110512 85.2509 62.75 0.00711766 0.0117248 2024-09-16T10:16:11.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:11.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[29993]: 7.9 scrub ok 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[29993]: osdmap e116: 8 total, 8 up, 8 in 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3243796064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[29993]: 4.5 scrub starts 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[29993]: 4.5 scrub ok 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:11.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[39470]: 7.9 scrub ok 2024-09-16T10:16:11.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[39470]: osdmap e116: 8 total, 8 up, 8 in 2024-09-16T10:16:11.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3243796064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:11.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[39470]: 4.5 scrub starts 2024-09-16T10:16:11.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:10 smithi063 ceph-mon[39470]: 4.5 scrub ok 2024-09-16T10:16:11.209 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:11.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:11.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:16:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:10 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:10 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:10 smithi165 ceph-mon[32696]: 7.9 scrub ok 2024-09-16T10:16:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:10 smithi165 ceph-mon[32696]: osdmap e116: 8 total, 8 up, 8 in 2024-09-16T10:16:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3243796064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:10 smithi165 ceph-mon[32696]: 4.5 scrub starts 2024-09-16T10:16:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:10 smithi165 ceph-mon[32696]: 4.5 scrub ok 2024-09-16T10:16:11.312 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:11.363 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:11.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:11.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:16:11.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:11.509 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:11.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:11.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:11.649 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:11.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:11.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:11.764 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:11.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:11.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:11.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:11.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:11.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:12.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:12.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:12.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:12.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 82 15 111567 111552 85.0039 65 0.00538111 0.0117586 2024-09-16T10:16:12.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:12.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:12.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:12.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[29993]: 5.9 scrub starts 2024-09-16T10:16:12.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[29993]: 5.9 scrub ok 2024-09-16T10:16:12.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[29993]: pgmap v108: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[39470]: 5.9 scrub starts 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[39470]: 5.9 scrub ok 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[39470]: pgmap v108: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:16:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:11 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:16:12.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:11 smithi165 ceph-mon[32696]: 5.9 scrub starts 2024-09-16T10:16:12.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:11 smithi165 ceph-mon[32696]: 5.9 scrub ok 2024-09-16T10:16:12.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:11 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:16:12.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:11 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:16:12.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:11 smithi165 ceph-mon[32696]: pgmap v108: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:16:12.228 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:11 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:16:12.228 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:11 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:16:12.274 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:12.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:12.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:12.368 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:12.368 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:12.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:12.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:12.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:12.571 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:12.609 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:12.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:12.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:12.752 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:12.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:12.854 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:12.875 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:12.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:12.958 INFO:teuthology.orchestra.run.smithi063.stderr:set 6.12 pg_upmap_items mapping to [6->2,7->3,1->5] 2024-09-16T10:16:12.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:13.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:13.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:13.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 83 16 112457 112441 84.6491 55.5625 0.00887574 0.0118084 2024-09-16T10:16:13.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:13.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:16:13.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:13.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:13.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:16:13.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:16:13.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[29993]: 7.7 scrub starts 2024-09-16T10:16:13.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2294731649' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [6, 2, 7, 3, 1, 5]} : dispatch 2024-09-16T10:16:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:16:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:16:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[39470]: 7.7 scrub starts 2024-09-16T10:16:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:12 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2294731649' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [6, 2, 7, 3, 1, 5]} : dispatch 2024-09-16T10:16:13.246 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:13.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:12 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:16:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:12 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:16:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:12 smithi165 ceph-mon[32696]: 7.7 scrub starts 2024-09-16T10:16:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:12 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2294731649' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [6, 2, 7, 3, 1, 5]} : dispatch 2024-09-16T10:16:13.304 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:13.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:13.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:13.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:13.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:13.511 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:13.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:13.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:13.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:13.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:13.734 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:13.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:13.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:13.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:13.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:14.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:14.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:14.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 84 16 113470 113454 84.3951 63.3125 0.019889 0.0118437 2024-09-16T10:16:14.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:14.144 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:14.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:14.245 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:14.293 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:14.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:14.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2294731649' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [6, 2, 7, 3, 1, 5]}]': finished 2024-09-16T10:16:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:13 smithi165 ceph-mon[32696]: osdmap e117: 8 total, 8 up, 8 in 2024-09-16T10:16:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:13 smithi165 ceph-mon[32696]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 832 B/s rd, 56 MiB/s wr, 894 op/s 2024-09-16T10:16:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:13 smithi165 ceph-mon[32696]: 7.f scrub starts 2024-09-16T10:16:14.348 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:16:14.393 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2294731649' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [6, 2, 7, 3, 1, 5]}]': finished 2024-09-16T10:16:14.393 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[39470]: osdmap e117: 8 total, 8 up, 8 in 2024-09-16T10:16:14.394 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[39470]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 832 B/s rd, 56 MiB/s wr, 894 op/s 2024-09-16T10:16:14.394 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[39470]: 7.f scrub starts 2024-09-16T10:16:14.394 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:14.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2294731649' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [6, 2, 7, 3, 1, 5]}]': finished 2024-09-16T10:16:14.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[29993]: osdmap e117: 8 total, 8 up, 8 in 2024-09-16T10:16:14.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[29993]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 832 B/s rd, 56 MiB/s wr, 894 op/s 2024-09-16T10:16:14.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:13 smithi063 ceph-mon[29993]: 7.f scrub starts 2024-09-16T10:16:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:14.454 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:14.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:14.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:14.603 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:14.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:14.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:14.718 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:14.765 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:14.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:14.866 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:14.927 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:14.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:15.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:15.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:15.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:15.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 85 16 114272 114256 83.992 50.125 0.0188865 0.0119006 2024-09-16T10:16:15.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:16:15.123 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[29993]: osdmap e118: 8 total, 8 up, 8 in 2024-09-16T10:16:15.123 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[29993]: 7.3 scrub starts 2024-09-16T10:16:15.123 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[29993]: 4.2 scrub starts 2024-09-16T10:16:15.123 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[29993]: 4.2 scrub ok 2024-09-16T10:16:15.124 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[39470]: osdmap e118: 8 total, 8 up, 8 in 2024-09-16T10:16:15.124 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[39470]: 7.3 scrub starts 2024-09-16T10:16:15.124 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[39470]: 4.2 scrub starts 2024-09-16T10:16:15.124 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:14 smithi063 ceph-mon[39470]: 4.2 scrub ok 2024-09-16T10:16:15.156 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:15.226 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:15.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:15.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:14 smithi165 ceph-mon[32696]: osdmap e118: 8 total, 8 up, 8 in 2024-09-16T10:16:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:14 smithi165 ceph-mon[32696]: 7.3 scrub starts 2024-09-16T10:16:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:14 smithi165 ceph-mon[32696]: 4.2 scrub starts 2024-09-16T10:16:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:14 smithi165 ceph-mon[32696]: 4.2 scrub ok 2024-09-16T10:16:15.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:15.370 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:15.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:15.437 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:15.472 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:15.516 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:15.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:15.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:15.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:15.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:15.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:15.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:15.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:15.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:15.918 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:15.945 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:15.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:16.012 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:16.012 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:16.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:16.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:16.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:16.099 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 86 15 115081 115066 83.6039 50.625 0.0166617 0.0119551 2024-09-16T10:16:16.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:16.226 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:16.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:16.242 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:16:16.242 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:16:16.242 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: 7.7 scrub ok 2024-09-16T10:16:16.242 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: pgmap v112: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 795 op/s 2024-09-16T10:16:16.242 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.242 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3725310088' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: from='client.24907 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: 7.7 scrub ok 2024-09-16T10:16:16.243 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: pgmap v112: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 795 op/s 2024-09-16T10:16:16.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3725310088' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:16.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: from='client.24907 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:16.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:16:16.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:16:16.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:15 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: 7.7 scrub ok 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: pgmap v112: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 795 op/s 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3725310088' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: from='client.24907 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:16:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:16:16.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:15 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:16.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:16.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:16.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:16.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:16.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:16.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:16.592 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:16.592 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:16.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:16.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:16.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:16.634 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:16.634 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:16.669 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:16.669 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:16.706 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:16.706 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:16.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:16.741 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:16.741 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:16.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:16.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:16.779 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:16.779 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:16.808 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:16.808 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:16.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:16.870 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:16.871 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:16.898 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:16.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:16.925 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:16:17.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:17.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:17.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 87 16 115944 115928 83.2621 53.875 0.00552688 0.0120056 2024-09-16T10:16:17.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:17.157 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:17.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:16:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: 6.1e scrub starts 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: 6.1e scrub ok 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: 5.1d scrub starts 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: 5.1d scrub ok 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: 7.f scrub ok 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3691070429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:17.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: from='client.24907 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[29993]: osdmap e119: 8 total, 8 up, 8 in 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: 6.1e scrub starts 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: 6.1e scrub ok 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: 5.1d scrub starts 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: 5.1d scrub ok 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:16:17.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: 7.f scrub ok 2024-09-16T10:16:17.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3691070429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:17.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:17.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: from='client.24907 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:16:17.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:16 smithi063 ceph-mon[39470]: osdmap e119: 8 total, 8 up, 8 in 2024-09-16T10:16:17.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:17.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:17.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: 6.1e scrub starts 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: 6.1e scrub ok 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: 5.1d scrub starts 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: 5.1d scrub ok 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: 7.f scrub ok 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3691070429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: from='client.24907 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:16:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:16 smithi165 ceph-mon[32696]: osdmap e119: 8 total, 8 up, 8 in 2024-09-16T10:16:17.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:17.444 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:17.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:17.586 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:17.688 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:17.721 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:17.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:17.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:17.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:16:17.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:17.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:17.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:18.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:18.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 88 16 117054 117038 83.1043 69.375 0.00688854 0.0120283 2024-09-16T10:16:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:18.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:18.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:18.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:17 smithi165 ceph-mon[32696]: 2.2 scrub starts 2024-09-16T10:16:18.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:17 smithi165 ceph-mon[32696]: 2.2 scrub ok 2024-09-16T10:16:18.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:17 smithi165 ceph-mon[32696]: 7.3 scrub ok 2024-09-16T10:16:18.169 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:17 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:16:18.169 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:17 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:16:18.169 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:17 smithi165 ceph-mon[32696]: pgmap v114: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 795 op/s 2024-09-16T10:16:18.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[29993]: 2.2 scrub starts 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[29993]: 2.2 scrub ok 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[29993]: 7.3 scrub ok 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[29993]: pgmap v114: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 795 op/s 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[39470]: 2.2 scrub starts 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[39470]: 2.2 scrub ok 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[39470]: 7.3 scrub ok 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:16:18.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:17 smithi063 ceph-mon[39470]: pgmap v114: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 795 op/s 2024-09-16T10:16:18.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:18.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:18.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:18.296 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:18.375 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:18.410 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:18.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:18.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:16:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:16:18.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:18.554 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:18.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:18.657 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:18.694 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:18.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:18.796 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:18.836 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:18.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:18.881 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:16:18.881 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:16:18.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.881 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:117902 7.0 7:0b6b3fe3:::benchmark_data_smithi063_73086_object117901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:16:18.867608+0000", 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.011307434, 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011369542999999999, 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117902 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.882 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867608+0000", 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867608+0000", 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867610+0000", 2024-09-16T10:16:18.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1780000000000002e-06 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867651+0000", 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0689000000000003e-05 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867664+0000", 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2502e-05 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867671+0000", 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6709999999999999e-06 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867732+0000", 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0998000000000001e-05 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867855+0000", 2024-09-16T10:16:18.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122726 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.867949+0000", 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3671999999999994e-05 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.870696+0000", 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027472999999999998 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:18.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:117914 7.0 7:0fd722a8:::benchmark_data_smithi063_73086_object117913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:16:18.875496+0000", 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0034198829999999999, 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00355457, 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117914 2024-09-16T10:16:18.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875496+0000", 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875496+0000", 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:18.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875498+0000", 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.74e-06 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875530+0000", 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1826000000000003e-05 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875549+0000", 2024-09-16T10:16:18.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360999999999999e-05 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875557+0000", 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6829999999999997e-06 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875614+0000", 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6910000000000002e-05 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:18.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875730+0000", 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011540399999999999 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.875875+0000", 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145573 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:18.878994+0000", 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031188420000000001 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:18.891 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:18.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:18.892 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:16:18.892 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-09-16T10:16:18.892 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:16:18.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:18.985 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:18.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:19.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:19.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 89 16 118170 118154 82.9542 69.75 0.00536599 0.0120501 2024-09-16T10:16:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:19.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:19.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:19.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:19.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:19.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:19.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:18 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:19.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:19.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:19.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:19.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:18 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:19.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:19.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:19.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:19.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:19.571 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:16:19.571 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:16:19.571 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T10:16:19.571 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T10:16:19.571 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T10:16:19.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-16T10:16:19.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:19.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:16:19.663 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:19.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:19.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:19.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:19.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:19.910 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:19.950 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:16:19.952 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:19.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:20.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:20.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:20.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:20.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total time run: 90.0117 2024-09-16T10:16:20.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total writes made: 119276 2024-09-16T10:16:20.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Write size: 65536 2024-09-16T10:16:20.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object size: 65536 2024-09-16T10:16:20.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Bandwidth (MB/sec): 82.8198 2024-09-16T10:16:20.100 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Bandwidth: 30.8662 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max bandwidth (MB/sec): 195 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min bandwidth (MB/sec): 50.125 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average IOPS: 1325 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev IOPS: 493.86 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max IOPS: 3120 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min IOPS: 802 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average Latency(s): 0.01207 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Latency(s): 0.00940622 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max latency(s): 0.201222 2024-09-16T10:16:20.101 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min latency(s): 0.00122794 2024-09-16T10:16:20.107 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Cleaning up (deleting benchmark objects) 2024-09-16T10:16:20.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:20.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:20.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:20.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:20 smithi165 ceph-mon[32696]: pgmap v115: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-16T10:16:20.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:20.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:20.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:20.353 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:20 smithi063 ceph-mon[29993]: pgmap v115: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-16T10:16:20.353 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:20 smithi063 ceph-mon[39470]: pgmap v115: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-16T10:16:20.420 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:20.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:20.496 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:20.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:20.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:20.623 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:20.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:20.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:16:20.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:20.740 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:16:20.740 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T10:16:20.740 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:16:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:20.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:20.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:20.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:20.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:20.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:20.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:20.967 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:20.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:21.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:21.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:21.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:21.119 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:21.119 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:21.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:21.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:21.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:21.157 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:21.158 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[29993]: osdmap e120: 8 total, 8 up, 8 in 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T10:16:21.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:21 smithi063 ceph-mon[39470]: osdmap e120: 8 total, 8 up, 8 in 2024-09-16T10:16:21.164 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:21.201 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:21.201 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:21.240 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:21.241 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:21.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:21.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:21.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:21.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:21.273 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:21 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:21.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:21 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:21.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:21 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T10:16:21.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:21 smithi165 ceph-mon[32696]: osdmap e120: 8 total, 8 up, 8 in 2024-09-16T10:16:21.309 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:21.309 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:21.341 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:21.341 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:21.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:21.378 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:21.378 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:21.419 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:21.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:21.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:21.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:21.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:21.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:21.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:21.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:21.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:21.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:21.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:21.963 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:22.011 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:22.043 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:22.044 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:22.113 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:22.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:22.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:22.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:22 smithi165 ceph-mon[32696]: pgmap v117: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 722 B/s rd, 41 MiB/s wr, 651 op/s 2024-09-16T10:16:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:22 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3052517195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.304 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:22.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:22.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:22.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:22 smithi063 ceph-mon[29993]: pgmap v117: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 722 B/s rd, 41 MiB/s wr, 651 op/s 2024-09-16T10:16:22.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:22 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3052517195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:22.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:22 smithi063 ceph-mon[39470]: pgmap v117: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 722 B/s rd, 41 MiB/s wr, 651 op/s 2024-09-16T10:16:22.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:22 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3052517195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:22.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:22.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:22.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:22.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:22.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:22.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:16:22.791 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:22.825 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:22.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:22.965 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:22.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:23.045 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:16:23.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:23.078 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:23.115 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:23.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:23.217 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:23.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:23.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:23.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:23 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3705452382' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:23 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:23.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:23.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:23.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:23.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:23 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3705452382' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:23 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:23 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3705452382' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:23 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:23.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:23.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:23.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:23.649 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:23.649 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:23.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:23.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:23.755 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:23.791 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:23.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:23.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:23.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:23.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:24.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:24.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:24.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:24.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:24.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:24.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:24.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:24.285 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:24.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:24 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3705452382' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:24 smithi165 ceph-mon[32696]: osdmap e121: 8 total, 8 up, 8 in 2024-09-16T10:16:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:24 smithi165 ceph-mon[32696]: pgmap v119: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 834 B/s rd, 50 MiB/s wr, 864 op/s 2024-09-16T10:16:24.325 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:24.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:24.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:16:24.427 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:24.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3705452382' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:24.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[29993]: osdmap e121: 8 total, 8 up, 8 in 2024-09-16T10:16:24.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[29993]: pgmap v119: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 834 B/s rd, 50 MiB/s wr, 864 op/s 2024-09-16T10:16:24.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:24.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3705452382' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:24.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[39470]: osdmap e121: 8 total, 8 up, 8 in 2024-09-16T10:16:24.457 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:24 smithi063 ceph-mon[39470]: pgmap v119: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 834 B/s rd, 50 MiB/s wr, 864 op/s 2024-09-16T10:16:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:24.466 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:24.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:24.471 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:16:24.567 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:24.607 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:24.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:24.709 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:24.726 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:24.732 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:24.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:24.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:24.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:24.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:24.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:25.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:25.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:25.140 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:25.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:25.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:25 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:25.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:25 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:25.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:25 smithi165 ceph-mon[32696]: osdmap e122: 8 total, 8 up, 8 in 2024-09-16T10:16:25.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:25.289 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:25.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:25.313 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:25.314 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:25.388 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:25.390 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:25.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:25 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:25.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:25 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:25.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:25 smithi063 ceph-mon[39470]: osdmap e122: 8 total, 8 up, 8 in 2024-09-16T10:16:25.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:25 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:25.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:25 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:25.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:25 smithi063 ceph-mon[29993]: osdmap e122: 8 total, 8 up, 8 in 2024-09-16T10:16:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:25.454 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:25.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:25.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:25.594 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:25.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:25.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:25.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:25.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:25.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:25.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:25.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:26.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:26.118 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.141 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:26.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:26.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:16:26.242 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:26.280 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:26.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:26.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:26.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:26.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:26 smithi063 ceph-mon[29993]: pgmap v121: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 23 MiB/s wr, 850 op/s 2024-09-16T10:16:26.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:26 smithi063 ceph-mon[39470]: pgmap v121: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 23 MiB/s wr, 850 op/s 2024-09-16T10:16:26.514 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:16:26.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T10:16:26.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:26 smithi165 ceph-mon[32696]: pgmap v121: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 23 MiB/s wr, 850 op/s 2024-09-16T10:16:26.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:26.591 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:26.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:26.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:26.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:26.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:26.987 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:26.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:27.088 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:27.129 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:27.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:27.175 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:16:27.175 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:16:27.175 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:16:27.175 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:16:27.175 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:16:27.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:27.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:27.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:27.271 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:16:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:27 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:27 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:27 smithi165 ceph-mon[32696]: osdmap e123: 8 total, 8 up, 8 in 2024-09-16T10:16:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3809928904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:27.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:27.407 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:27.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:27.428 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:27.428 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:27.428 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:27.428 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:27.428 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:27.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:27.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:27.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:27.429 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:27.430 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:27.430 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:27.430 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:27.430 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:27.430 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:27.431 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:27.431 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:27.431 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:27.432 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:27.432 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[29993]: osdmap e123: 8 total, 8 up, 8 in 2024-09-16T10:16:27.432 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3809928904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:27.432 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:16:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:16:27.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:27.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:27.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[39470]: osdmap e123: 8 total, 8 up, 8 in 2024-09-16T10:16:27.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3809928904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:27.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:27.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:27.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:27.635 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:27.670 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:27.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:27.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:27.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:27.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:27.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:16:27.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:27.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:27.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:28.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:28.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:28.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:28.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:28.247 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:28.247 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:28.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:28.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:28.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:28 smithi165 ceph-mon[32696]: pgmap v123: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 23 MiB/s wr, 850 op/s 2024-09-16T10:16:28.375 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:16:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:16:28.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:28.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:28.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:28 smithi063 ceph-mon[39470]: pgmap v123: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 23 MiB/s wr, 850 op/s 2024-09-16T10:16:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:28 smithi063 ceph-mon[29993]: pgmap v123: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 23 MiB/s wr, 850 op/s 2024-09-16T10:16:28.478 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:28.531 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:28.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:28.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:28.672 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:28.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:28.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:28.822 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:28.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39765 7.c 7:361d2464:::benchmark_data_smithi063_73086_object39764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.588323587999994, 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13511668299999999, 2024-09-16T10:16:28.840 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39765 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:16:28.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247952+0000", 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.889e-06 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248012+0000", 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0418000000000003e-05 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248023+0000", 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0852e-05 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248030+0000", 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9489999999999999e-06 2024-09-16T10:16:28.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248067+0000", 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7120999999999999e-05 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248138+0000", 2024-09-16T10:16:28.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0885999999999999e-05 2024-09-16T10:16:28.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248217+0000", 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9058000000000001e-05 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250443+0000", 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022264720000000002 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250487+0000", 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3823999999999999e-05 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.382955+0000", 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13246761100000001 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383013+0000", 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7670000000000002e-05 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383066+0000", 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3933e-05 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39767 7.c 7:3eeafd38:::benchmark_data_smithi063_73086_object39766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.588228322000006, 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13517669099999999, 2024-09-16T10:16:28.849 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39767 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248046+0000", 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5199999999999996e-07 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248153+0000", 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106803 2024-09-16T10:16:28.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248170+0000", 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.721e-05 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248175+0000", 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5609999999999998e-06 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249333+0000", 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011573200000000001 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249455+0000", 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012275400000000001 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249590+0000", 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013500099999999999 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250643+0000", 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001052966 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252177+0000", 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015336340000000001 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383126+0000", 2024-09-16T10:16:28.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13094891 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383178+0000", 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2141000000000001e-05 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383222+0000", 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3739000000000003e-05 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39768 7.c 7:3b0be1c9:::benchmark_data_smithi063_73086_object39767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.585974613999994, 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13302192400000001, 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.857 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39768 2024-09-16T10:16:28.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.857 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250301+0000", 2024-09-16T10:16:28.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2529999999999999e-06 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250409+0000", 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108145 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250635+0000", 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225582 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250641+0000", 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.63e-06 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250684+0000", 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3006000000000001e-05 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250790+0000", 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105831 2024-09-16T10:16:28.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250923+0000", 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013304199999999999 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252242+0000", 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013184189999999999 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252758+0000", 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051628000000000004 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383257+0000", 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13049871900000001 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383300+0000", 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3223999999999998e-05 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383321+0000", 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0792999999999999e-05 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39770 7.7 7:ec0b9b23:::benchmark_data_smithi063_73086_object39769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.585619055999999, 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13275636499999999, 2024-09-16T10:16:28.863 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39770 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250655+0000", 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7999999999999993e-07 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250688+0000", 2024-09-16T10:16:28.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2687000000000001e-05 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250695+0000", 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0110000000000003e-06 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250700+0000", 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250739+0000", 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8665999999999999e-05 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250820+0000", 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1252000000000001e-05 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250936+0000", 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116036 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.253100+0000", 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002163674 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255541+0000", 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024417549999999999 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383318+0000", 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.127776905 2024-09-16T10:16:28.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383372+0000", 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3826000000000003e-05 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383411+0000", 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8621000000000002e-05 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39786 7.c 7:3b648db5:::benchmark_data_smithi063_73086_object39785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.574761100000003, 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124519465, 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39786 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261514+0000", 2024-09-16T10:16:28.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3430000000000001e-06 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261577+0000", 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3646999999999996e-05 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261587+0000", 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9499999999999996e-06 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261593+0000", 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6810000000000003e-06 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261633+0000", 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0262999999999997e-05 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261709+0000", 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.551e-05 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261799+0000", 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0437000000000002e-05 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.262947+0000", 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001148224 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.263126+0000", 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000178529 2024-09-16T10:16:28.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.385937+0000", 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122810896 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.385988+0000", 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0961000000000002e-05 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.386032+0000", 2024-09-16T10:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4023999999999997e-05 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40098 7.7 7:e51c64ce:::benchmark_data_smithi063_73086_object40097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.290909651000007, 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118499387, 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40098 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545366+0000", 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0839999999999999e-06 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545435+0000", 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9424999999999993e-05 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545449+0000", 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3283e-05 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545456+0000", 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0650000000000001e-06 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545522+0000", 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.614e-05 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545636+0000", 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011427800000000001 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545759+0000", 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012285799999999999 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.547169+0000", 2024-09-16T10:16:28.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001410251 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.550261+0000", 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030920029999999999 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663775+0000", 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113513832 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663832+0000", 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6771999999999998e-05 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663863+0000", 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1396000000000002e-05 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40106 7.d 7:bffd718f:::benchmark_data_smithi063_73086_object40105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:16:28.882 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.286662930000006, 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112057936, 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40106 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:16:28.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549612+0000", 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3289999999999999e-06 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549663+0000", 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1014999999999999e-05 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549673+0000", 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9229999999999997e-06 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549678+0000", 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1120000000000004e-06 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549728+0000", 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9700999999999997e-05 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549857+0000", 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012977400000000001 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549995+0000", 2024-09-16T10:16:28.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137498 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551274+0000", 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012796140000000001 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551903+0000", 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062848199999999998 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661566+0000", 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109663317 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661615+0000", 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9193000000000001e-05 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661668+0000", 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2978e-05 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.888 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40108 7.7 7:e3eeca68:::benchmark_data_smithi063_73086_object40107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.284802049000007, 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12682985199999999, 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40108 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551473+0000", 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3090000000000001e-06 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551526+0000", 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999999e-05 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551536+0000", 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6120000000000005e-06 2024-09-16T10:16:28.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551541+0000", 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8790000000000002e-06 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551580+0000", 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9727000000000001e-05 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551660+0000", 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9480999999999993e-05 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551754+0000", 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4548000000000003e-05 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.552878+0000", 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011237459999999999 2024-09-16T10:16:28.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663896+0000", 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11101776200000001 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678241+0000", 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014344894 2024-09-16T10:16:28.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678275+0000", 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4551000000000002e-05 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678301+0000", 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5843e-05 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:43489 7.d 7:bb6aa884:::benchmark_data_smithi063_73086_object43488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:11.749872+0000", 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.086401844999997, 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111586563, 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 43489 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.896 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749872+0000", 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749872+0000", 2024-09-16T10:16:28.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749873+0000", 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4500000000000001e-06 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749922+0000", 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9478999999999998e-05 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749933+0000", 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0097e-05 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749938+0000", 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2410000000000003e-06 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749987+0000", 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8896000000000001e-05 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.750067+0000", 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0623000000000004e-05 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.750150+0000", 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2922000000000003e-05 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.751459+0000", 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013091610000000001 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754957+0000", 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034977670000000002 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.861381+0000", 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10642346 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.861416+0000", 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5337999999999999e-05 2024-09-16T10:16:28.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.861458+0000", 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2129000000000003e-05 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.906 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:43498 7.c 7:3d8938c3:::benchmark_data_smithi063_73086_object43497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:11.754659+0000", 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.081614789, 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10244017699999999, 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 43498 2024-09-16T10:16:28.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754659+0000", 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754659+0000", 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754661+0000", 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2299999999999998e-06 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.755024+0000", 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036321999999999998 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.755050+0000", 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6264000000000001e-05 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.755059+0000", 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2060000000000007e-06 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.755119+0000", 2024-09-16T10:16:28.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0406999999999999e-05 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.755245+0000", 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012645399999999999 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.755382+0000", 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013694799999999999 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.758154+0000", 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027718830000000002 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.758163+0000", 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5169999999999999e-06 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.857033+0000", 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098870390000000002 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.857069+0000", 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5521000000000001e-05 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.857099+0000", 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0137e-05 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:43504 7.7 7:eb5ac715:::benchmark_data_smithi063_73086_object43503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:11.756623+0000", 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.079649961000001, 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111441769, 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.914 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 43504 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756623+0000", 2024-09-16T10:16:28.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756623+0000", 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756625+0000", 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.742e-06 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756655+0000", 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9899e-05 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756665+0000", 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9769999999999995e-06 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756672+0000", 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7309999999999996e-06 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.757092+0000", 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042053700000000002 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.757230+0000", 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013731600000000001 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.757359+0000", 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012892200000000001 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.758660+0000", 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013009510000000001 2024-09-16T10:16:28.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.865343+0000", 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10668319499999999 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.867941+0000", 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025982100000000001 2024-09-16T10:16:28.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.868000+0000", 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8640000000000001e-05 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.868065+0000", 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5648999999999994e-05 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.923 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:54672 7.d 7:ba0f846c:::benchmark_data_smithi063_73086_object54671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:19.268629+0000", 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.567644064000007, 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089206905000000003, 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.924 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 54672 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268629+0000", 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268629+0000", 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268631+0000", 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.759e-06 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268691+0000", 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9605000000000003e-05 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268703+0000", 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2585000000000001e-05 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268711+0000", 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6650000000000003e-06 2024-09-16T10:16:28.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268758+0000", 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7376999999999997e-05 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.268907+0000", 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000148906 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.269108+0000", 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020033099999999999 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.270849+0000", 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001741484 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.271701+0000", 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000851449 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.357725+0000", 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.086024236000000004 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.357798+0000", 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3708999999999995e-05 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.357836+0000", 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7799000000000002e-05 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65089 7.d 7:bff27887:::benchmark_data_smithi063_73086_object65088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 61.279712760999999, 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12756952299999999, 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.932 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65089 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556562+0000", 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.686e-06 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556698+0000", 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013535199999999999 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556711+0000", 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3067e-05 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556719+0000", 2024-09-16T10:16:28.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7919999999999993e-06 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556773+0000", 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4647e-05 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556924+0000", 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015124 2024-09-16T10:16:28.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.557015+0000", 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0681999999999997e-05 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.558770+0000", 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001754395 2024-09-16T10:16:28.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.562719+0000", 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039499050000000001 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684014+0000", 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.1212949 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684082+0000", 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7417999999999997e-05 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684130+0000", 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8439000000000002e-05 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65092 7.d 7:b5d3643d:::benchmark_data_smithi063_73086_object65091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 61.276622957000001, 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12473208600000001, 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65092 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559652+0000", 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4109999999999999e-06 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559730+0000", 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8343000000000005e-05 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559741+0000", 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-05 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559747+0000", 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.417e-06 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559788+0000", 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1597999999999997e-05 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559877+0000", 2024-09-16T10:16:28.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9018999999999996e-05 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559960+0000", 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2101000000000006e-05 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.561615+0000", 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00165525 2024-09-16T10:16:28.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.562754+0000", 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00113956 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684280+0000", 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121525444 2024-09-16T10:16:28.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684345+0000", 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684383+0000", 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7639999999999999e-05 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.947 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72694 7.d 7:b02443fe:::benchmark_data_smithi063_73086_object72693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.966084512000002, 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11432945999999999, 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72694 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870190+0000", 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0499999999999999e-06 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870260+0000", 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9900000000000005e-05 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870269+0000", 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9959999999999999e-06 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870273+0000", 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6210000000000003e-06 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872430+0000", 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002156748 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872556+0000", 2024-09-16T10:16:28.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125859 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872660+0000", 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103859 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.874597+0000", 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373229999999999 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.877583+0000", 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002986247 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984443+0000", 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10685934800000001 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984481+0000", 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7718e-05 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984518+0000", 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7790999999999998e-05 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72705 7.7 7:eddfad96:::benchmark_data_smithi063_73086_object72704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.878700+0000", 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.957573517999997, 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110056815, 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72705 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878700+0000", 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878700+0000", 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878702+0000", 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9039999999999999e-06 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878874+0000", 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017175799999999999 2024-09-16T10:16:28.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878885+0000", 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1523e-05 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878892+0000", 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0729999999999999e-06 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878939+0000", 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6609e-05 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879055+0000", 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116026 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:16:28.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879181+0000", 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000126305 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.881701+0000", 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025194390000000001 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.884850+0000", 2024-09-16T10:16:28.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031490649999999999 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988668+0000", 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103817933 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988720+0000", 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2361999999999998e-05 2024-09-16T10:16:28.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988757+0000", 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6817999999999999e-05 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72716 7.7 7:e8fcaee4:::benchmark_data_smithi063_73086_object72715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:16:28.963 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.888824+0000", 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.947449120000002, 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104953095, 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72716 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.964 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888824+0000", 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888824+0000", 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888826+0000", 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1600000000000001e-06 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888868+0000", 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1128000000000003e-05 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888882+0000", 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4279e-05 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888890+0000", 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1540000000000002e-06 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888947+0000", 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7034999999999999e-05 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.889048+0000", 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010074 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.889144+0000", 2024-09-16T10:16:28.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6020000000000006e-05 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.891104+0000", 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019601969999999999 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.891457+0000", 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035333599999999999 2024-09-16T10:16:28.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.993715+0000", 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102257213 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.993751+0000", 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6837e-05 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.993777+0000", 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5996e-05 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72717 7.d 7:b23e40d3:::benchmark_data_smithi063_73086_object72716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.888916+0000", 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.947357439999998, 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099611210000000006, 2024-09-16T10:16:28.971 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72717 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888916+0000", 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888916+0000", 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888917+0000", 2024-09-16T10:16:28.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1299999999999999e-07 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888966+0000", 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9264999999999997e-05 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888972+0000", 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1789999999999996e-06 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.888980+0000", 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1529999999999997e-06 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.889029+0000", 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860999999999999e-05 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.889113+0000", 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3805000000000001e-05 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.889200+0000", 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6917999999999997e-05 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.890996+0000", 2024-09-16T10:16:28.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017964789999999999 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.891571+0000", 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000574849 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988424+0000", 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096852524999999995 2024-09-16T10:16:28.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988478+0000", 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4070000000000002e-05 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988527+0000", 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9292999999999997e-05 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.978 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90054 7.c 7:3cae452b:::benchmark_data_smithi063_73086_object90053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.322836199000001, 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12751799899999999, 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90054 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513440+0000", 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.452e-06 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513492+0000", 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1950999999999998e-05 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513503+0000", 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1232e-05 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513509+0000", 2024-09-16T10:16:28.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0680000000000002e-06 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513571+0000", 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2434999999999994e-05 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513694+0000", 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012256499999999999 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513865+0000", 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017080599999999999 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.517203+0000", 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033382860000000002 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.518058+0000", 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085528199999999996 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640875+0000", 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12281645400000001 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640919+0000", 2024-09-16T10:16:28.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.401e-05 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640955+0000", 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6458000000000002e-05 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.986 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90065 7.d 7:bffa18df:::benchmark_data_smithi063_73086_object90064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.314402772000001, 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124785905, 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90065 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.987 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521873+0000", 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.497e-06 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521903+0000", 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9407000000000002e-05 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:28.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521916+0000", 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3597999999999999e-05 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521924+0000", 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521991+0000", 2024-09-16T10:16:28.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7318999999999997e-05 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522105+0000", 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113467 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522205+0000", 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9829999999999998e-05 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525304+0000", 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030995089999999999 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525567+0000", 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026322999999999999 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646542+0000", 2024-09-16T10:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120974217 2024-09-16T10:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646604+0000", 2024-09-16T10:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2543000000000001e-05 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646657+0000", 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2438000000000002e-05 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:28.995 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:28.995 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:16:28.995 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:28.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:28.996 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:29.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:29.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:29.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:29.168 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:29.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:29.270 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:29.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:29.313 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:29.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:29.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:29.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:29.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:29.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:29.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:29.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:29.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:29.682 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:29.717 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:29.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:29.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:29.866 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:29.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:29.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:16:29.968 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:30.015 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:30.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:30.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:30.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:30.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:30.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:30.293 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:30.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:30 smithi165 ceph-mon[32696]: pgmap v124: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.8 KiB/s rd, 20 MiB/s wr, 1.59k op/s 2024-09-16T10:16:30.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:30.423 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:30.445 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:30 smithi063 ceph-mon[29993]: pgmap v124: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.8 KiB/s rd, 20 MiB/s wr, 1.59k op/s 2024-09-16T10:16:30.445 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:30 smithi063 ceph-mon[39470]: pgmap v124: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.8 KiB/s rd, 20 MiB/s wr, 1.59k op/s 2024-09-16T10:16:30.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:30.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:30.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:30.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:30.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:30.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:30.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:30.952 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:30.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:30.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:16:31.001 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:16:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:16:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:130410 7.5 7:acea9e0b:::benchmark_data_smithi063_73086_object11131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-16T10:16:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:16:30.995861+0000", 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0035990330000000002, 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036280029999999999, 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 130410 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.995861+0000", 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.995861+0000", 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.995863+0000", 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0980000000000001e-06 2024-09-16T10:16:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.995870+0000", 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8839999999999997e-06 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.995887+0000", 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7195000000000001e-05 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.995895+0000", 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6639999999999998e-06 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.995960+0000", 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5167999999999997e-05 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.996098+0000", 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013746400000000001 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.996234+0000", 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013617300000000001 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.999314+0000", 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030805440000000002 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:130413 7.c e123/81 7:3f24eee5:::benchmark_data_smithi063_73086_object11134:head v 123'8476, mlcod=123'8474)", 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:16:30.998197+0000", 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0012636640000000001, 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013726820000000001, 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:16:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998197+0000", 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998197+0000", 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998198+0000", 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4580000000000001e-06 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998205+0000", 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2370000000000003e-06 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998220+0000", 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4532000000000001e-05 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998228+0000", 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9289999999999998e-06 2024-09-16T10:16:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998278+0000", 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0519000000000002e-05 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:30.998298+0000", 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9553000000000001e-05 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:16:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:16:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-09-16T10:16:31.011 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:16:31.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:31.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:31.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:31.119 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:16:31.124 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T10:16:31.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:31.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:31.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2774768237' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:16:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:31 smithi165 ceph-mon[32696]: from='client.44897 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:16:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1850109355' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:16:31.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:31.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:31.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:31.398 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2774768237' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:16:31.399 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:31 smithi063 ceph-mon[29993]: from='client.44897 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:16:31.399 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1850109355' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:16:31.399 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2774768237' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:16:31.399 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:31 smithi063 ceph-mon[39470]: from='client.44897 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:16:31.399 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1850109355' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:16:31.439 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:31.479 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:31.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:31.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:31.622 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:31.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:31.724 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:31.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:16:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:31.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:31.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:31.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:32.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:32.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:32.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:32.093 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:32.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:32.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T10:16:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:32.199 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:32.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:32.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:32.352 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:32.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:32.443 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:32 smithi165 ceph-mon[32696]: pgmap v125: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-09-16T10:16:32.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:32 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:32.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:32 smithi165 ceph-mon[32696]: from='client.44897 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:16:32.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1850109355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:16:32.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:32 smithi165 ceph-mon[32696]: osdmap e124: 8 total, 8 up, 8 in 2024-09-16T10:16:32.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:32 smithi165 ceph-mon[32696]: 4.19 scrub starts 2024-09-16T10:16:32.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:32 smithi165 ceph-mon[32696]: 4.19 scrub ok 2024-09-16T10:16:32.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[39470]: pgmap v125: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-09-16T10:16:32.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[39470]: from='client.44897 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1850109355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[39470]: osdmap e124: 8 total, 8 up, 8 in 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[39470]: 4.19 scrub starts 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[39470]: 4.19 scrub ok 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[29993]: pgmap v125: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[29993]: from='client.44897 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1850109355' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[29993]: osdmap e124: 8 total, 8 up, 8 in 2024-09-16T10:16:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[29993]: 4.19 scrub starts 2024-09-16T10:16:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:32 smithi063 ceph-mon[29993]: 4.19 scrub ok 2024-09-16T10:16:32.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:32.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:32.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:32.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:32.630 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:32.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:32.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:32.751 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:32.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:32.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:32.808 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:32.853 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:16:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:16:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:16:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:16:32.853 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:16:32.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:32.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:32.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:33.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:33.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:33.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[29993]: 5.1c scrub starts 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[29993]: 5.1c scrub ok 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[29993]: 6.b scrub starts 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[29993]: 6.b scrub ok 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[39470]: 5.1c scrub starts 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[39470]: 5.1c scrub ok 2024-09-16T10:16:33.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[39470]: 6.b scrub starts 2024-09-16T10:16:33.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[39470]: 6.b scrub ok 2024-09-16T10:16:33.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:33.181 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:33 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:33.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:33.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:33.400 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:33.442 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:33.447 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:33 smithi165 ceph-mon[32696]: 5.1c scrub starts 2024-09-16T10:16:33.447 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:33 smithi165 ceph-mon[32696]: 5.1c scrub ok 2024-09-16T10:16:33.447 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:33 smithi165 ceph-mon[32696]: 6.b scrub starts 2024-09-16T10:16:33.447 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:33 smithi165 ceph-mon[32696]: 6.b scrub ok 2024-09-16T10:16:33.447 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:33 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:33.447 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:33 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:33.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:33.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:16:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:33.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:33.673 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:33.715 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:33.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:33.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.866 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:33.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:33.910 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:16:33.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:33.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:33.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:34.059 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:34.059 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:34.059 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:34.059 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:34.060 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:34.060 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:34.060 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:34.060 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:34.060 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:34.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:34.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:34.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:34.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:34.061 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:34.061 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:34.061 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:34.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:34.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:34.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:34.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:34.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:34 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:16:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:34 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:16:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:34 smithi165 ceph-mon[32696]: 6.1d scrub starts 2024-09-16T10:16:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:34 smithi165 ceph-mon[32696]: 6.1d scrub ok 2024-09-16T10:16:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:34 smithi165 ceph-mon[32696]: pgmap v127: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 723 op/s 2024-09-16T10:16:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:34 smithi165 ceph-mon[32696]: osdmap e125: 8 total, 8 up, 8 in 2024-09-16T10:16:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2513265458' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:34.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:34.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:34.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:16:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:16:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[39470]: 6.1d scrub starts 2024-09-16T10:16:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[39470]: 6.1d scrub ok 2024-09-16T10:16:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[39470]: pgmap v127: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 723 op/s 2024-09-16T10:16:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[39470]: osdmap e125: 8 total, 8 up, 8 in 2024-09-16T10:16:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2513265458' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:16:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:16:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[29993]: 6.1d scrub starts 2024-09-16T10:16:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[29993]: 6.1d scrub ok 2024-09-16T10:16:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[29993]: pgmap v127: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 723 op/s 2024-09-16T10:16:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[29993]: osdmap e125: 8 total, 8 up, 8 in 2024-09-16T10:16:34.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2513265458' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:34.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:34.534 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:34.534 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:34.541 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:34.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:34.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:34.682 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:16:34.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:34.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:34.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:34.851 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:34.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:34.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:34.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:34.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:35.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:35.127 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:35.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:35.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[39470]: 4.15 scrub starts 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[39470]: 4.15 scrub ok 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[39470]: 3.4 scrub starts 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[39470]: 3.4 scrub ok 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[39470]: 5.11 scrub starts 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[39470]: 5.11 scrub ok 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[29993]: 4.15 scrub starts 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[29993]: 4.15 scrub ok 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[29993]: 3.4 scrub starts 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[29993]: 3.4 scrub ok 2024-09-16T10:16:35.205 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[29993]: 5.11 scrub starts 2024-09-16T10:16:35.206 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:35 smithi063 ceph-mon[29993]: 5.11 scrub ok 2024-09-16T10:16:35.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:35.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:16:35.269 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:35.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:35.371 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:35.425 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:35.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:35.485 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:35.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:35.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:35 smithi165 ceph-mon[32696]: 4.15 scrub starts 2024-09-16T10:16:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:35 smithi165 ceph-mon[32696]: 4.15 scrub ok 2024-09-16T10:16:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:35 smithi165 ceph-mon[32696]: 3.4 scrub starts 2024-09-16T10:16:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:35 smithi165 ceph-mon[32696]: 3.4 scrub ok 2024-09-16T10:16:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:35 smithi165 ceph-mon[32696]: 5.11 scrub starts 2024-09-16T10:16:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:35 smithi165 ceph-mon[32696]: 5.11 scrub ok 2024-09-16T10:16:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:35.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:35.654 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:35.705 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:35.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:35.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:35.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:35.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:35.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:35.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:35.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:36.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:36.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:36.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:36.127 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:36.155 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: 6.7 scrub starts 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: 6.7 scrub ok 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: pgmap v129: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2682295037' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: 3.1f scrub starts 2024-09-16T10:16:36.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[29993]: 3.1f scrub ok 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: 6.7 scrub starts 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: 6.7 scrub ok 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: pgmap v129: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2682295037' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: 3.1f scrub starts 2024-09-16T10:16:36.182 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:36 smithi063 ceph-mon[39470]: 3.1f scrub ok 2024-09-16T10:16:36.187 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:36.245 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:36.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:36.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:36.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:36.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:36.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:36.515 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:36.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: 6.7 scrub starts 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: 6.7 scrub ok 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: pgmap v129: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2682295037' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: 3.1f scrub starts 2024-09-16T10:16:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:36 smithi165 ceph-mon[32696]: 3.1f scrub ok 2024-09-16T10:16:36.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:36.661 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:36.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:36.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:36.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:36.900 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:36.939 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:36.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:37.016 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T10:16:37.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:37.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:37.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:37.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:37.174 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:16:37.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:37.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:37.220 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:16:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2682295037' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: osdmap e126: 8 total, 8 up, 8 in 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: 5.1e scrub starts 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: 5.1e scrub ok 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1109228776' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:37.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[29993]: from='client.44918 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2682295037' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: osdmap e126: 8 total, 8 up, 8 in 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: 5.1e scrub starts 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: 5.1e scrub ok 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1109228776' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:37.222 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:37 smithi063 ceph-mon[39470]: from='client.44918 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:37.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:37.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:37.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:37.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:16:37.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:37.490 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:37.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2682295037' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: osdmap e126: 8 total, 8 up, 8 in 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: 5.1e scrub starts 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: 5.1e scrub ok 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1109228776' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:37 smithi165 ceph-mon[32696]: from='client.44918 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:37.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:37.667 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:37.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:37.696 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:37.696 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:37.769 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:37.816 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:37.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:37.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:37.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:37.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:38.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:38.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:38.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:38.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:38.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:38.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:38.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:38.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:38.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:38.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: pgmap v131: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 899 op/s 2024-09-16T10:16:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:16:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:16:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: from='client.44918 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:38.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: osdmap e127: 8 total, 8 up, 8 in 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: 5.17 scrub starts 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: 5.17 scrub ok 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: 5.1f scrub starts 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: 5.1f scrub ok 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: 6.1c scrub starts 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: 6.1c scrub ok 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: pgmap v131: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 899 op/s 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:16:38.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: from='client.44918 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: osdmap e127: 8 total, 8 up, 8 in 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: 5.17 scrub starts 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: 5.17 scrub ok 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: 5.1f scrub starts 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: 5.1f scrub ok 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: 6.1c scrub starts 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: 6.1c scrub ok 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:38.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:38.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:38.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:38.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:38.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:16:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: pgmap v131: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 899 op/s 2024-09-16T10:16:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:16:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: from='client.44918 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: osdmap e127: 8 total, 8 up, 8 in 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: 5.17 scrub starts 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: 5.17 scrub ok 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: 5.1f scrub starts 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: 5.1f scrub ok 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: 6.1c scrub starts 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: 6.1c scrub ok 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:38.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:38.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:38.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:38.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:38.739 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:38.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:38.841 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:38.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:38.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:38.942 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:16:38.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:39.007 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:39.011 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:39.074 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:39.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:39.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:39.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:39.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:39.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:39.397 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:39.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:39.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:39 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:16:39.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:39 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:16:39.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:39.405 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:39 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:16:39.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:39 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:16:39.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:39.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:39.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:39.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:39.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:39 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:16:39.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:39 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:16:39.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:39.627 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:39.679 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:39.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:39.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:39.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:39.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:39.972 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:39.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:40.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:40.074 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:40.075 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:40.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:40.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:40.112 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:40.113 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:40.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:16:40.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:40.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:40.215 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:40.239 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[29993]: pgmap v133: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-16T10:16:40.239 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[29993]: 3.17 scrub starts 2024-09-16T10:16:40.239 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[29993]: 3.17 scrub ok 2024-09-16T10:16:40.239 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[29993]: 6.15 scrub starts 2024-09-16T10:16:40.239 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[29993]: 6.15 scrub ok 2024-09-16T10:16:40.239 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/395275249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:40.240 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[39470]: pgmap v133: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-16T10:16:40.240 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[39470]: 3.17 scrub starts 2024-09-16T10:16:40.240 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[39470]: 3.17 scrub ok 2024-09-16T10:16:40.240 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[39470]: 6.15 scrub starts 2024-09-16T10:16:40.240 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[39470]: 6.15 scrub ok 2024-09-16T10:16:40.240 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/395275249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:40.261 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:40.262 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:40.280 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:40.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:40.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:40.289 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:40.313 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:40.351 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:40.351 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:40.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:40.394 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:40.394 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:40.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:40.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:40.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:40 smithi165 ceph-mon[32696]: pgmap v133: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-16T10:16:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:40 smithi165 ceph-mon[32696]: 3.17 scrub starts 2024-09-16T10:16:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:40 smithi165 ceph-mon[32696]: 3.17 scrub ok 2024-09-16T10:16:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:40 smithi165 ceph-mon[32696]: 6.15 scrub starts 2024-09-16T10:16:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:40 smithi165 ceph-mon[32696]: 6.15 scrub ok 2024-09-16T10:16:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/395275249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:40.561 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:40.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:40.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:40.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:40.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:40.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:40.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:40.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:40.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:40.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:40.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:41.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:41.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:41.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:41.171 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:16:41.171 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:16:41.171 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:16:41.171 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:16:41.171 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:16:41.171 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:41.200 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T10:16:41.218 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:41.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:41.320 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:41.366 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:41.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:41.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3892699118' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:16:41.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3892699118' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:16:41.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:41.509 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:41.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:41.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3892699118' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:16:41.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:41.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:41.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:41.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:41.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:41.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:41.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:41.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:41.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:41.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:42.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:42.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:42.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:42.165 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:42.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:42.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:42.311 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:42.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:42.400 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:42 smithi063 ceph-mon[29993]: pgmap v134: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 643 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-16T10:16:42.400 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3892699118' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:16:42.400 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:42 smithi063 ceph-mon[29993]: osdmap e128: 8 total, 8 up, 8 in 2024-09-16T10:16:42.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:42 smithi063 ceph-mon[39470]: pgmap v134: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 643 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-16T10:16:42.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3892699118' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:16:42.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:42 smithi063 ceph-mon[39470]: osdmap e128: 8 total, 8 up, 8 in 2024-09-16T10:16:42.413 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:42.451 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T10:16:42.466 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:42.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:42 smithi165 ceph-mon[32696]: pgmap v134: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 643 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-16T10:16:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3892699118' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:16:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:42 smithi165 ceph-mon[32696]: osdmap e128: 8 total, 8 up, 8 in 2024-09-16T10:16:42.570 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:42.627 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:42.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:42.712 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:42.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:42.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:42.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:42.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:42.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:42.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:42.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:42.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:43.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:43.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:43.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:43.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:16:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:43.170 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:43.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:43.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:43.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:43.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:43.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[29993]: 6.1b scrub starts 2024-09-16T10:16:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[29993]: 6.1b scrub ok 2024-09-16T10:16:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[29993]: 4.a scrub starts 2024-09-16T10:16:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[29993]: 4.a scrub ok 2024-09-16T10:16:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:16:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:16:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[39470]: 6.1b scrub starts 2024-09-16T10:16:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[39470]: 6.1b scrub ok 2024-09-16T10:16:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[39470]: 4.a scrub starts 2024-09-16T10:16:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[39470]: 4.a scrub ok 2024-09-16T10:16:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:16:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:43 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:16:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:43.456 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:43.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:43 smithi165 ceph-mon[32696]: 6.1b scrub starts 2024-09-16T10:16:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:43 smithi165 ceph-mon[32696]: 6.1b scrub ok 2024-09-16T10:16:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:43 smithi165 ceph-mon[32696]: 4.a scrub starts 2024-09-16T10:16:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:43 smithi165 ceph-mon[32696]: 4.a scrub ok 2024-09-16T10:16:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:43 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:16:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:43 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:16:43.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:43.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:43.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:43.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:43.714 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:16:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:43.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:43.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:43.787 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:43.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:43.876 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:43.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:43.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.035 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:44.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:44.136 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.194 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:44.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:44.229 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:16:44.231 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: pgmap v136: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 846 op/s 2024-09-16T10:16:44.231 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: 6.16 scrub starts 2024-09-16T10:16:44.231 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: 6.16 scrub ok 2024-09-16T10:16:44.231 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:16:44.231 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:16:44.231 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: 5.4 scrub starts 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: 5.4 scrub ok 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2955643395' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[29993]: from='client.24967 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: pgmap v136: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 846 op/s 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: 6.16 scrub starts 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: 6.16 scrub ok 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:16:44.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:44.233 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:44.233 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: 5.4 scrub starts 2024-09-16T10:16:44.233 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: 5.4 scrub ok 2024-09-16T10:16:44.233 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2955643395' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:44.233 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:44 smithi063 ceph-mon[39470]: from='client.24967 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:44.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:44.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:44.425 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:44.466 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: pgmap v136: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 846 op/s 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: 6.16 scrub starts 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: 6.16 scrub ok 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: 5.4 scrub starts 2024-09-16T10:16:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: 5.4 scrub ok 2024-09-16T10:16:44.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2955643395' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:44.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:44 smithi165 ceph-mon[32696]: from='client.24967 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:16:44.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:44.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:44.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:44.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:44.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:44.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:44.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:44.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:44.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:45.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:45.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:45.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:45.220 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:45.244 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T10:16:45.265 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:45.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:45.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:45.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:45.411 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:45.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:45.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:45 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:16:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:45 smithi165 ceph-mon[32696]: 6.2 scrub ok 2024-09-16T10:16:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:45 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:45 smithi165 ceph-mon[32696]: from='client.24967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:16:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:45 smithi165 ceph-mon[32696]: osdmap e129: 8 total, 8 up, 8 in 2024-09-16T10:16:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:45 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4081253752' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:16:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:45 smithi165 ceph-mon[32696]: from='client.44936 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:16:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:45.554 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:45.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:45.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:45.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:45.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:45.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[29993]: 6.2 scrub ok 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[29993]: from='client.24967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[29993]: osdmap e129: 8 total, 8 up, 8 in 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4081253752' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[29993]: from='client.44936 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[39470]: 6.2 scrub ok 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[39470]: from='client.24967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:16:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[39470]: osdmap e129: 8 total, 8 up, 8 in 2024-09-16T10:16:45.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4081253752' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:16:45.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:45 smithi063 ceph-mon[39470]: from='client.44936 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:16:45.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:45.820 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:45.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:45.922 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:45.962 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:45.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:46.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:46.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:46.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:46.146 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:46.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:46.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:46.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:46.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:46.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:46.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:46.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:46.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:46.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:46.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:46 smithi165 ceph-mon[32696]: pgmap v138: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-16T10:16:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:46 smithi165 ceph-mon[32696]: from='client.44936 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:16:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:46 smithi165 ceph-mon[32696]: osdmap e130: 8 total, 8 up, 8 in 2024-09-16T10:16:46.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:46.618 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:46.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:46.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:46 smithi063 ceph-mon[29993]: pgmap v138: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-16T10:16:46.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:46 smithi063 ceph-mon[29993]: from='client.44936 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:16:46.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:46 smithi063 ceph-mon[29993]: osdmap e130: 8 total, 8 up, 8 in 2024-09-16T10:16:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:46 smithi063 ceph-mon[39470]: pgmap v138: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-16T10:16:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:46 smithi063 ceph-mon[39470]: from='client.44936 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:16:46.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:46 smithi063 ceph-mon[39470]: osdmap e130: 8 total, 8 up, 8 in 2024-09-16T10:16:46.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:46.762 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:46.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:46.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T10:16:46.854 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:46.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:46.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:46.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:46.992 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:47.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:47.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:47.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:47.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:47.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:47.318 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:47.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:47.337 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:16:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:16:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:16:47.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:47.474 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:47.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:47.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:47.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:47.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:47.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39764 7.6 7:680231a8:::benchmark_data_smithi063_73086_object39763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 98.389192475000002, 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13555234199999999, 2024-09-16T10:16:47.708 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39764 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246057+0000", 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3999999999999999e-06 2024-09-16T10:16:47.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246080+0000", 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2629999999999998e-05 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246091+0000", 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0978999999999999e-05 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246097+0000", 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6579999999999996e-06 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246213+0000", 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115435 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246353+0000", 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014010199999999999 2024-09-16T10:16:47.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246471+0000", 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117877 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249103+0000", 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026325020000000001 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249476+0000", 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037260100000000002 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381493+0000", 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13201689999999999 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381556+0000", 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3516000000000006e-05 2024-09-16T10:16:47.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381608+0000", 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1742e-05 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39772 7.6 7:6310c551:::benchmark_data_smithi063_73086_object39771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 98.384743607000004, 2024-09-16T10:16:47.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13122466799999999, 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39772 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250506+0000", 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.897e-06 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250528+0000", 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1883999999999999e-05 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250537+0000", 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.285e-06 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250545+0000", 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.08e-06 2024-09-16T10:16:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250610+0000", 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5287e-05 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250734+0000", 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012402299999999999 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252617+0000", 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018828919999999999 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254471+0000", 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001854138 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.255185+0000", 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00071378999999999995 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381651+0000", 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12646639400000001 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381696+0000", 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5364999999999997e-05 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381729+0000", 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2632999999999997e-05 2024-09-16T10:16:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39783 7.f 7:fb2e33c9:::benchmark_data_smithi063_73086_object39782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 98.377645271000006, 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13116349199999999, 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39783 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:16:47.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257604+0000", 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7710000000000001e-06 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257627+0000", 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2589999999999999e-05 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257636+0000", 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.302e-06 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257641+0000", 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257700+0000", 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8791e-05 2024-09-16T10:16:47.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257856+0000", 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015637100000000001 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.258009+0000", 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015277300000000001 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383671+0000", 2024-09-16T10:16:47.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12566208600000001 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383705+0000", 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3278999999999998e-05 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388613+0000", 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049080310000000002 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388687+0000", 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4381000000000005e-05 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388766+0000", 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9198999999999998e-05 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.728 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39785 7.9 7:9b09a424:::benchmark_data_smithi063_73086_object39784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.259335+0000", 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 98.375912994000004, 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12677764899999999, 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39785 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259335+0000", 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259335+0000", 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259336+0000", 2024-09-16T10:16:47.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4869999999999999e-06 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259358+0000", 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1840000000000001e-05 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259366+0000", 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1200000000000002e-06 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259371+0000", 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8300000000000003e-06 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381412+0000", 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.122040865 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381543+0000", 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000130808 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:16:47.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381686+0000", 2024-09-16T10:16:47.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014293199999999999 2024-09-16T10:16:47.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383620+0000", 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001934594 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383728+0000", 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000107037 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.385971+0000", 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022436790000000002 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.386043+0000", 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2208999999999999e-05 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.386113+0000", 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9247999999999998e-05 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.736 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:39783 7.f e97/81 ondisk, result = 0)", 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.259447+0000", 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 98.375801422999999, 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.124265217, 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259447+0000", 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259447+0000", 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259448+0000", 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.08e-06 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.259454+0000", 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3740000000000003e-06 2024-09-16T10:16:47.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381394+0000", 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12194039600000001 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381403+0000", 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9309999999999997e-06 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383694+0000", 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002290238 2024-09-16T10:16:47.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383700+0000", 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9819999999999996e-06 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383712+0000", 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2216e-05 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:40096 7.f 7:f95968e1:::benchmark_data_smithi063_73086_object40095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.544748+0000", 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 98.090499511999994, 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12583075699999999, 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.741 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.743 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.743 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.743 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 40096 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.544748+0000", 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.544748+0000", 2024-09-16T10:16:47.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.544749+0000", 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8100000000000002e-07 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.544776+0000", 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6485000000000001e-05 2024-09-16T10:16:47.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.544782+0000", 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6630000000000004e-06 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.544790+0000", 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3320000000000003e-06 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545160+0000", 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000370152 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545261+0000", 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000100822 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545516+0000", 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025524500000000003 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659429+0000", 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.113913607 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.663512+0000", 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040826530000000003 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.670468+0000", 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0069560020000000002 2024-09-16T10:16:47.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.670532+0000", 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4305000000000002e-05 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.670579+0000", 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6810000000000001e-05 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:40097 7.f 7:f223a0ce:::benchmark_data_smithi063_73086_object40096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 98.089936405000003, 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.133293199, 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 40097 2024-09-16T10:16:47.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545313+0000", 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0470000000000001e-06 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545333+0000", 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0778e-05 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545341+0000", 2024-09-16T10:16:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0739999999999993e-06 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545348+0000", 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3049999999999998e-06 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549229+0000", 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038809500000000002 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549377+0000", 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014856200000000001 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549537+0000", 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015973699999999999 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659510+0000", 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.109972575 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.663519+0000", 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040092649999999997 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678483+0000", 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0149646 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678559+0000", 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5891999999999998e-05 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678605+0000", 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5414000000000003e-05 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65087 7.6 7:66a6545e:::benchmark_data_smithi063_73086_object65086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.555793+0000", 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 80.079455370999995, 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12526327000000001, 2024-09-16T10:16:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65087 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555793+0000", 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555793+0000", 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555795+0000", 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2460000000000002e-06 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555825+0000", 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0094e-05 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555836+0000", 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1124999999999999e-05 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555843+0000", 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7819999999999996e-06 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.556602+0000", 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00075938199999999996 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.556729+0000", 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012689 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.556872+0000", 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014303100000000001 2024-09-16T10:16:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.561698+0000", 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0048260730000000002 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.561709+0000", 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0564000000000001e-05 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680887+0000", 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119178635 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681010+0000", 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012277999999999999 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681056+0000", 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5667999999999997e-05 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65088 7.f 7:f37840e1:::benchmark_data_smithi063_73086_object65087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.555854+0000", 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 80.079394391999998, 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12525853000000001, 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.763 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65088 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555854+0000", 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555854+0000", 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555854+0000", 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4399999999999999e-07 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555874+0000", 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9320000000000001e-05 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555880+0000", 2024-09-16T10:16:47.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0050000000000001e-06 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555885+0000", 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0719999999999999e-06 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.555930+0000", 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5147e-05 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.556046+0000", 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116461 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.556190+0000", 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014320199999999999 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559234+0000", 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003044485 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559444+0000", 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000209502 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680910+0000", 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.121466378 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681060+0000", 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014985899999999999 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681112+0000", 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2355000000000003e-05 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:65089 7.d e105/81 7:bff27887:::benchmark_data_smithi063_73086_object65088:head v 105'4113, mlcod=105'4111)", 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.557085+0000", 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 80.078163458999995, 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12435737199999999, 2024-09-16T10:16:47.769 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557085+0000", 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557085+0000", 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557086+0000", 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.44e-06 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557880+0000", 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00079379699999999995 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557890+0000", 2024-09-16T10:16:47.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9879999999999999e-06 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557896+0000", 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7179999999999999e-06 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.558302+0000", 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00040561699999999999 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.558422+0000", 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120171 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681400+0000", 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12297741199999999 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681442+0000", 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2228999999999998e-05 2024-09-16T10:16:47.773 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65090 7.4 7:2059d57d:::benchmark_data_smithi063_73086_object65089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.557946+0000", 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 80.077302223999993, 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.122939374, 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.774 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65090 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557946+0000", 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557946+0000", 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557947+0000", 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6530000000000001e-06 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557972+0000", 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4108999999999999e-05 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557980+0000", 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8619999999999992e-06 2024-09-16T10:16:47.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.557987+0000", 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1500000000000004e-06 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.558038+0000", 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1069000000000002e-05 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.558167+0000", 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012913900000000001 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.558294+0000", 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012773999999999999 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560632+0000", 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023380179999999999 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.561870+0000", 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012371249999999999 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680778+0000", 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11890798499999999 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680842+0000", 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4331999999999993e-05 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680885+0000", 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3192000000000003e-05 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65091 7.6 7:62394c27:::benchmark_data_smithi063_73086_object65090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.559030+0000", 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 80.076217834000005, 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12217602399999999, 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65091 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559030+0000", 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559030+0000", 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559031+0000", 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1990000000000001e-06 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559057+0000", 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6021999999999999e-05 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559065+0000", 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9680000000000006e-06 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559072+0000", 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6800000000000004e-06 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559156+0000", 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4402999999999998e-05 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559273+0000", 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116674 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559440+0000", 2024-09-16T10:16:47.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016691199999999999 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.561926+0000", 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024856589999999999 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681072+0000", 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119146618 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681125+0000", 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2901000000000001e-05 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681171+0000", 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5587000000000002e-05 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681206+0000", 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5401000000000002e-05 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.786 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:65092 7.d e105/81 7:b5d3643d:::benchmark_data_smithi063_73086_object65091:head v 105'4114, mlcod=105'4111)", 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.560132+0000", 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 80.075116379999997, 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.121337789, 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560132+0000", 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560132+0000", 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560133+0000", 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0419999999999999e-06 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560164+0000", 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1643000000000002e-05 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560174+0000", 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7829999999999993e-06 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560180+0000", 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2469999999999997e-06 2024-09-16T10:16:47.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560976+0000", 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00079542199999999995 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.560993+0000", 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7232999999999999e-05 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681445+0000", 2024-09-16T10:16:47.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120451693 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681469+0000", 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4726e-05 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90038 7.6 7:605b4ba6:::benchmark_data_smithi063_73086_object90037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 58.137211379, 2024-09-16T10:16:47.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.151200375, 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90038 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498038+0000", 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3200000000000001e-06 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498061+0000", 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2646e-05 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498070+0000", 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.217e-06 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498076+0000", 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1759999999999998e-06 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498121+0000", 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4870000000000002e-05 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498227+0000", 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106239 2024-09-16T10:16:47.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498358+0000", 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013121899999999999 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501482+0000", 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031239000000000002 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.503423+0000", 2024-09-16T10:16:47.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019405819999999999 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648604+0000", 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14518165099999999 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648977+0000", 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000372641 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649237+0000", 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025991399999999998 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90048 7.6 7:6327a166:::benchmark_data_smithi063_73086_object90047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 58.130996226000001, 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14568709799999999, 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.798 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90048 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504254+0000", 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1409999999999999e-06 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504278+0000", 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3711999999999999e-05 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504289+0000", 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1331e-05 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504296+0000", 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4599999999999997e-06 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504413+0000", 2024-09-16T10:16:47.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011668299999999999 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504522+0000", 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010877 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504652+0000", 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013002399999999999 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509094+0000", 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044417069999999996 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514152+0000", 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0050583470000000004 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649421+0000", 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13526861600000001 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649711+0000", 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00029075100000000001 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649939+0000", 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00022755599999999999 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90050 7.6 7:60112b96:::benchmark_data_smithi063_73086_object90049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 58.126041725, 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14140651900000001, 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90050 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.805 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509208+0000", 2024-09-16T10:16:47.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.407e-06 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509230+0000", 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1861e-05 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509237+0000", 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3159999999999999e-06 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509242+0000", 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5820000000000001e-06 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509424+0000", 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018168599999999999 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509616+0000", 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019154600000000001 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509757+0000", 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000141485 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514172+0000", 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044148629999999998 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514209+0000", 2024-09-16T10:16:47.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6706999999999999e-05 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650127+0000", 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.135917818 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650433+0000", 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030668399999999999 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650613+0000", 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000179564 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90064 7.6 7:67c07ba3:::benchmark_data_smithi063_73086_object90063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 58.114692374000001, 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.130695903, 2024-09-16T10:16:47.811 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.812 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90064 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520558+0000", 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.984e-06 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520581+0000", 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3700999999999999e-05 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520593+0000", 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.128e-05 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520600+0000", 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520651+0000", 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1133e-05 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520775+0000", 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012390099999999999 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520940+0000", 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016520200000000001 2024-09-16T10:16:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523864+0000", 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029241139999999998 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524288+0000", 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00042372299999999998 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650781+0000", 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12649292200000001 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.651082+0000", 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030106300000000001 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.651252+0000", 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016956999999999999 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90066 7.f 7:fafdc57c:::benchmark_data_smithi063_73086_object90065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 58.113396070999997, 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.132960727, 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90066 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521853+0000", 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4419999999999999e-06 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521876+0000", 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2430999999999999e-05 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521885+0000", 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6989999999999998e-06 2024-09-16T10:16:47.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521892+0000", 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7909999999999999e-06 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521943+0000", 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0321999999999997e-05 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.522046+0000", 2024-09-16T10:16:47.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010361600000000001 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.522161+0000", 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000114833 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.641264+0000", 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11910251299999999 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.641301+0000", 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7638000000000001e-05 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.652401+0000", 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011099367000000001 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.654749+0000", 2024-09-16T10:16:47.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023483110000000001 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.654813+0000", 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3763999999999995e-05 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90070 7.6 7:6e0aafb6:::benchmark_data_smithi063_73086_object90069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:16:47.825 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 58.110963271000003, 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13177412199999999, 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90070 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:16:47.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524286+0000", 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3489999999999999e-06 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524308+0000", 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.228e-05 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524317+0000", 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5310000000000001e-06 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524323+0000", 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3010000000000003e-06 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524360+0000", 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6721000000000003e-05 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524466+0000", 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106162 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640787+0000", 2024-09-16T10:16:47.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116321087 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.645632+0000", 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0048448600000000003 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.647100+0000", 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001468254 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.655959+0000", 2024-09-16T10:16:47.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088585079999999993 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.656015+0000", 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6378000000000001e-05 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.656059+0000", 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3690999999999999e-05 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.831 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90069 7.e e112/81 7:7df0bbc7:::benchmark_data_smithi063_73086_object90068:head v 112'5541, mlcod=112'5539)", 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.524343+0000", 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 58.110905178000003, 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119432106, 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:16:47.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524343+0000", 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524343+0000", 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524343+0000", 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8999999999999996e-07 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524363+0000", 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9568e-05 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524369+0000", 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.178e-06 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:16:47.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524374+0000", 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7530000000000001e-06 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524429+0000", 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5479000000000002e-05 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524449+0000", 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9190000000000001e-05 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.643737+0000", 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119288135 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.643775+0000", 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8112999999999999e-05 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:16:47.836 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:16:47.837 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:47.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:47.838 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:47.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:47.845 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:47.845 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:47.887 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:47.887 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:47.923 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:47.923 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:47.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:47.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:16:47.953 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:47.953 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:47.975 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:47.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:47.991 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:47.991 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:48.031 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:48.071 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:48.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:48.162 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:48.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:48.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:48.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:48.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:48.394 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:48.439 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:48.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:48.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:48.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:48 smithi165 ceph-mon[32696]: pgmap v140: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 882 op/s 2024-09-16T10:16:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1354363941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:48.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:16:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:16:48.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:48.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:48.668 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:48.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:48 smithi063 ceph-mon[29993]: pgmap v140: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 882 op/s 2024-09-16T10:16:48.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1354363941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:48.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:48 smithi063 ceph-mon[39470]: pgmap v140: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 882 op/s 2024-09-16T10:16:48.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1354363941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:48.715 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:48.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:48.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:48.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:48.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:48.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:48.983 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:48.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:49.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:16:49.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:49.157 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:49.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:49.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:49.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:49.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:49.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:49.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:49.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:49.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:49.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:49.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:49.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:49.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:49.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:49.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:49.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:49.834 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:49.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:49.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:49.948 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.012 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:50.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:50.018 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:50.023 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:50.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:50.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:50.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:50.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:50.366 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:50 smithi063 ceph-mon[29993]: pgmap v141: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-16T10:16:50.366 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:50 smithi063 ceph-mon[39470]: pgmap v141: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-16T10:16:50.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:50.415 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:50.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:50.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:50 smithi165 ceph-mon[32696]: pgmap v141: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-16T10:16:50.555 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:50.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:50.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:50.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:50.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:50.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:50.954 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:50.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:51.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:51.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:51.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:51.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:51.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:51.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:51.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:51.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:51.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:51.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:51.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:51.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:51.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:51.623 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:51.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:51.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:51.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:51.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:51.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:51.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:51.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:51.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:16:51.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:52.040 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:52.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.185 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:52.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:52.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.321 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:52.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:52.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:52.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:52.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:52 smithi063 ceph-mon[29993]: pgmap v142: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:16:52.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:52 smithi063 ceph-mon[39470]: pgmap v142: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:16:52.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:52.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:52 smithi165 ceph-mon[32696]: pgmap v142: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:16:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:52.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:52.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:52.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:52.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:52.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:52.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:52.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:52.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:53.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:53.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:53.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:53.167 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:16:53.167 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:16:53.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:53.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:53.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:53.291 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T10:16:53.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:53.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:53.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:53.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2552721676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:16:53.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:53 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:53.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:53.442 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2552721676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:16:53.442 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:53 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:53.442 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:53.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:53.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:53.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2552721676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:16:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:53 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:16:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:16:53.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:53.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:53.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:53.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:53.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:53.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:53.849 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:53.889 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:53.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:53.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:54.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:54.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:54.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:54.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:54.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:54.316 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:16:54.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:54.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:54.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:54.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:54 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:54 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:54 smithi165 ceph-mon[32696]: pgmap v143: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 578 B/s rd, 0 B/s wr, 652 op/s 2024-09-16T10:16:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2552721676' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:16:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:54 smithi165 ceph-mon[32696]: osdmap e131: 8 total, 8 up, 8 in 2024-09-16T10:16:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/443842298' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:54 smithi165 ceph-mon[32696]: from='client.44948 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:54.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:54.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:54.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[29993]: pgmap v143: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 578 B/s rd, 0 B/s wr, 652 op/s 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2552721676' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[29993]: osdmap e131: 8 total, 8 up, 8 in 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/443842298' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[29993]: from='client.44948 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[39470]: pgmap v143: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 578 B/s rd, 0 B/s wr, 652 op/s 2024-09-16T10:16:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2552721676' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:16:54.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[39470]: osdmap e131: 8 total, 8 up, 8 in 2024-09-16T10:16:54.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/443842298' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:54.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:54 smithi063 ceph-mon[39470]: from='client.44948 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:16:54.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:54.819 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:54.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:54.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:54.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:54.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:16:55.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:55.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:55.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:55.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:55.220 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:55.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:55.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:55.377 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:55.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:55.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:55.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:55 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:55 smithi165 ceph-mon[32696]: from='client.44948 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:55 smithi165 ceph-mon[32696]: osdmap e132: 8 total, 8 up, 8 in 2024-09-16T10:16:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:16:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:16:55.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:55.653 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:55.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[29993]: from='client.44948 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[29993]: osdmap e132: 8 total, 8 up, 8 in 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[39470]: from='client.44948 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[39470]: osdmap e132: 8 total, 8 up, 8 in 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:16:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:16:55.754 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:55.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:55.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:55.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:55.900 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:55.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:56.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:56.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:56.144 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:56.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:56.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:56.331 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:56.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:56.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:56.496 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:56.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:56.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:56.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:56 smithi063 ceph-mon[29993]: pgmap v146: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-16T10:16:56.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:56 smithi063 ceph-mon[29993]: osdmap e133: 8 total, 8 up, 8 in 2024-09-16T10:16:56.630 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:56 smithi063 ceph-mon[39470]: pgmap v146: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-16T10:16:56.630 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:56 smithi063 ceph-mon[39470]: osdmap e133: 8 total, 8 up, 8 in 2024-09-16T10:16:56.685 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:56.729 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:56.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:56 smithi165 ceph-mon[32696]: pgmap v146: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-16T10:16:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:56 smithi165 ceph-mon[32696]: osdmap e133: 8 total, 8 up, 8 in 2024-09-16T10:16:56.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:56.868 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:56.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:56.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:56.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:57.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:57.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:57.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:16:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:16:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:16:57.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:57.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:57.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:57.363 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:57.411 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:57.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:57.452 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T10:16:57.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:57.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:57.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:57.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:57.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:57.802 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:57.864 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:57.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:57.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:57.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:57.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:58.007 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:58.090 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:16:58.091 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:16:58.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:58.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:58.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:58.130 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:16:58.130 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:16:58.160 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:16:58.161 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:16:58.207 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:16:58.207 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:16:58.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:58.238 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:16:58.238 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:16:58.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:58.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:58.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:16:58.273 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:16:58.309 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:16:58.309 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:16:58.343 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:16:58.344 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:16:58.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:58.383 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:58 smithi063 ceph-mon[29993]: pgmap v148: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s 2024-09-16T10:16:58.384 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:58 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/404620639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:58.384 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:58 smithi063 ceph-mon[39470]: pgmap v148: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s 2024-09-16T10:16:58.384 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:58 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/404620639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:58.413 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:58.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:58.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:16:58.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:58.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:16:58.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:58 smithi165 ceph-mon[32696]: pgmap v148: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s 2024-09-16T10:16:58.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:58 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/404620639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:16:58.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:16:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:16:58.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:16:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:58.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:16:58.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:58.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:58.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:58.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:58.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:58.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:59.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:16:59.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:59.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:16:59.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:16:59.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:16:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:59.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:16:59.249 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:16:59.296 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:16:59.296 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:16:59.297 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:16:59.297 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:16:59.297 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:16:59.297 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:16:59.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:16:59.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:59.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:59.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:16:59.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:16:59.541 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:59.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:16:59.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:16:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:59.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:16:59.646 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:16:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:59.647 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:16:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:16:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:59.699 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:59.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:16:59.800 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:59.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:59.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:16:59.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:16:59.984 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:16:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:16:59.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:00.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:00.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:00.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:00.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:00.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:00.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:00.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:17:00.352 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:00.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:00.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:00.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:00.677 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:00.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:00.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:00 smithi063 ceph-mon[29993]: pgmap v149: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-16T10:17:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:00 smithi063 ceph-mon[39470]: pgmap v149: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-16T10:17:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:00.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:00 smithi165 ceph-mon[32696]: pgmap v149: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-09-16T10:17:00.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:00.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:00.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:00.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:01.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:01.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:17:01.112 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.149 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:01.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:01.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:01.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:01.376 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.435 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:01.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:01.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:01.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:01.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.701 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:01.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:01.736 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:01.781 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T10:17:01.803 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:01.849 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:01.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:01.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:01.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:02.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/208458461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:17:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:01 smithi165 ceph-mon[32696]: pgmap v150: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 663 op/s 2024-09-16T10:17:02.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:02.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:02.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/208458461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:17:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:01 smithi063 ceph-mon[29993]: pgmap v150: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 663 op/s 2024-09-16T10:17:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/208458461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:17:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:01 smithi063 ceph-mon[39470]: pgmap v150: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 663 op/s 2024-09-16T10:17:02.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:02.265 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:02.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:02.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:02.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:02.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:02.407 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:02.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:02.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:02.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:02.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:02.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:02.665 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:02.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:02.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:02.793 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:17:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:02.810 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:02.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:02.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T10:17:02.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:02.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:02.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:03.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/208458461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:17:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:02 smithi165 ceph-mon[32696]: osdmap e134: 8 total, 8 up, 8 in 2024-09-16T10:17:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1190777903' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:02 smithi165 ceph-mon[32696]: from='client.44963 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:03.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:03.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:03.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:03.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/208458461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:17:03.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[29993]: osdmap e134: 8 total, 8 up, 8 in 2024-09-16T10:17:03.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1190777903' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:03.179 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[29993]: from='client.44963 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:03.179 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/208458461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:17:03.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[39470]: osdmap e134: 8 total, 8 up, 8 in 2024-09-16T10:17:03.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1190777903' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:03.180 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:02 smithi063 ceph-mon[39470]: from='client.44963 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:03.225 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:03.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:03.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:03.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:03.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:03.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:11:24.455842+0000", 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 338.90418282899998, 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.459140253, 2024-09-16T10:17:03.366 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14156", 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/1121825432", 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455842+0000", 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455842+0000", 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455844+0000", 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.429e-06 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455850+0000", 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7109999999999998e-06 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455860+0000", 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4720000000000001e-06 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455870+0000", 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0324e-05 2024-09-16T10:17:03.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455918+0000", 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8244000000000001e-05 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.455924+0000", 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9569999999999999e-06 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.913769+0000", 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.45784443200000002 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.913926+0000", 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015720799999999999 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.914060+0000", 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000133638 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.914664+0000", 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00060403200000000001 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.914851+0000", 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018710800000000001 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.914925+0000", 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3938999999999998e-05 2024-09-16T10:17:03.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.914957+0000", 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2415000000000001e-05 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:11:24.914983+0000", 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5344e-05 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:19431 7.b 7:d9ef71b6:::benchmark_data_smithi063_73086_object19430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.599452+0000", 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.760573409, 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.061062587000000002, 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 19431 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.374 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599452+0000", 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599452+0000", 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599453+0000", 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4e-07 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599477+0000", 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4111e-05 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599485+0000", 2024-09-16T10:17:03.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2050000000000002e-06 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599491+0000", 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8320000000000002e-06 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.602552+0000", 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030617470000000001 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.602662+0000", 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010959 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.602790+0000", 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000127844 2024-09-16T10:17:03.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.657998+0000", 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.055208022000000002 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658062+0000", 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4276999999999995e-05 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660321+0000", 2024-09-16T10:17:03.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022591310000000002 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660449+0000", 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000127868 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660514+0000", 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.512e-05 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.380 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:19435 7.b 7:dc66f280:::benchmark_data_smithi063_73086_object19434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.599774+0000", 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.760251113, 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.06095772, 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.381 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 19435 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599774+0000", 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599774+0000", 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599775+0000", 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.26e-06 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599835+0000", 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9938000000000001e-05 2024-09-16T10:17:03.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599844+0000", 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0399999999999998e-06 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.599851+0000", 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9800000000000001e-06 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603547+0000", 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0036960019999999999 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603667+0000", 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120129 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603867+0000", 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019946800000000001 2024-09-16T10:17:03.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658032+0000", 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.054165014999999997 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658135+0000", 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102959 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660630+0000", 2024-09-16T10:17:03.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00249482 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660694+0000", 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4789999999999995e-05 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660732+0000", 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7319e-05 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:19437 7.b 7:d026da8c:::benchmark_data_smithi063_73086_object19436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.601592+0000", 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.758432728, 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.059237660999999997, 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.388 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 19437 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.601592+0000", 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.601592+0000", 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.601594+0000", 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8309999999999999e-06 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.601621+0000", 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6502000000000001e-05 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.601632+0000", 2024-09-16T10:17:03.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0896e-05 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.601639+0000", 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4390000000000003e-06 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.604688+0000", 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030491170000000001 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.604778+0000", 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0254999999999996e-05 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.604906+0000", 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012744300000000001 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658098+0000", 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.053192340999999997 2024-09-16T10:17:03.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660608+0000", 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002509458 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660742+0000", 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013469299999999999 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660784+0000", 2024-09-16T10:17:03.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2024999999999998e-05 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660830+0000", 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5661000000000002e-05 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:19444 7.b 7:de110380:::benchmark_data_smithi063_73086_object19443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.603012+0000", 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.757012773, 2024-09-16T10:17:03.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.058125527000000003, 2024-09-16T10:17:03.395 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.395 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.395 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 19444 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603012+0000", 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603012+0000", 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603014+0000", 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.716e-06 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603037+0000", 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2464e-05 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603048+0000", 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0983e-05 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603054+0000", 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6800000000000004e-06 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605707+0000", 2024-09-16T10:17:03.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026533070000000001 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605953+0000", 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00024598000000000003 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606087+0000", 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000133139 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658173+0000", 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052086773000000003 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660843+0000", 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026694819999999999 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660886+0000", 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2814e-05 2024-09-16T10:17:03.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.661102+0000", 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000216233 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.661138+0000", 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5955999999999999e-05 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:19446 7.b 7:dbbb25ed:::benchmark_data_smithi063_73086_object19445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.603796+0000", 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.756229128, 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.057886131, 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 19446 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.402 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603796+0000", 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603796+0000", 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603797+0000", 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.359e-06 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603857+0000", 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9558000000000001e-05 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603866+0000", 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5589999999999994e-06 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.603889+0000", 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2481999999999999e-05 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658206+0000", 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.054317012999999997 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658718+0000", 2024-09-16T10:17:03.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00051235699999999996 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658972+0000", 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025383900000000001 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660847+0000", 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001875212 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.661179+0000", 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00033168800000000001 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.661554+0000", 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037482900000000003 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.661644+0000", 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9778999999999995e-05 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.661682+0000", 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8455999999999999e-05 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:19431 7.b e91/81 ondisk, result = 0)", 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.605608+0000", 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.754417491, 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052396595999999997, 2024-09-16T10:17:03.408 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605608+0000", 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605608+0000", 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605609+0000", 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.268e-06 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605615+0000", 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5300000000000002e-06 2024-09-16T10:17:03.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605634+0000", 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8428999999999999e-05 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605649+0000", 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5160000000000001e-05 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.657985+0000", 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052336470000000003 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.657992+0000", 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0140000000000001e-06 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658004+0000", 2024-09-16T10:17:03.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1725e-05 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:19445 7.0 e91/81 7:0e5490bb:::benchmark_data_smithi063_73086_object19444:head v 91'1190, mlcod=91'1188)", 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.605797+0000", 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.754228587, 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052107767999999999, 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605797+0000", 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605797+0000", 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605797+0000", 2024-09-16T10:17:03.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4900000000000004e-07 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605865+0000", 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7743999999999994e-05 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605873+0000", 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1389999999999995e-06 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605880+0000", 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3709999999999996e-06 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606074+0000", 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019403399999999999 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606102+0000", 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7772000000000001e-05 2024-09-16T10:17:03.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.657860+0000", 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051758775 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.657904+0000", 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3983999999999998e-05 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:19431 7.b e91/81 ondisk, result = 0)", 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.605898+0000", 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.754127195, 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052169330999999999, 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605898+0000", 2024-09-16T10:17:03.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605898+0000", 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605899+0000", 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7899999999999998e-07 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605902+0000", 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4529999999999998e-06 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605908+0000", 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7980000000000002e-06 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605913+0000", 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1869999999999996e-06 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658054+0000", 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052140652000000003 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658059+0000", 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.977e-06 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658067+0000", 2024-09-16T10:17:03.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6850000000000007e-06 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:19435 7.b e91/81 ondisk, result = 0)", 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.605962+0000", 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.754063094, 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052075069000000002, 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:17:03.422 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605962+0000", 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605962+0000", 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605963+0000", 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3399999999999999e-07 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605966+0000", 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.743e-06 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605973+0000", 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2330000000000003e-06 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.605977+0000", 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8389999999999997e-06 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658022+0000", 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052044695000000002 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658028+0000", 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3480000000000003e-06 2024-09-16T10:17:03.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658037+0000", 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.577e-06 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:19437 7.b e91/81 ondisk, result = 0)", 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.606221+0000", 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.753804409, 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051883814, 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.427 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606221+0000", 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606221+0000", 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606222+0000", 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5899999999999995e-07 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606225+0000", 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6049999999999998e-06 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606233+0000", 2024-09-16T10:17:03.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4590000000000001e-06 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606242+0000", 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.465e-06 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658087+0000", 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051845371000000001 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658093+0000", 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8080000000000001e-06 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658105+0000", 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1246999999999999e-05 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:19435 7.b e91/81 ondisk, result = 0)", 2024-09-16T10:17:03.431 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.606857+0000", 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.753167797, 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051283545999999999, 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606857+0000", 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606857+0000", 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606858+0000", 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7300000000000005e-07 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606862+0000", 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.357e-06 2024-09-16T10:17:03.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606870+0000", 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9640000000000003e-06 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606876+0000", 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9959999999999999e-06 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658125+0000", 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051248192999999997 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658130+0000", 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6509999999999998e-06 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658141+0000", 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0611999999999999e-05 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.435 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:19449 7.0 e91/81 7:0dffb227:::benchmark_data_smithi063_73086_object19448:head v 91'1191, mlcod=91'1188)", 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.606894+0000", 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.753130631, 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.053210378000000003, 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606894+0000", 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606894+0000", 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606895+0000", 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2600000000000002e-07 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606931+0000", 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5917000000000002e-05 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606938+0000", 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9260000000000003e-06 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.606943+0000", 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2549999999999997e-06 2024-09-16T10:17:03.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.607047+0000", 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000104032 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.607066+0000", 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8376000000000002e-05 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660043+0000", 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052977877999999999 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660105+0000", 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1468000000000002e-05 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:19437 7.b e91/81 ondisk, result = 0)", 2024-09-16T10:17:03.440 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:14:57.608385+0000", 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 125.751640471, 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052463466, 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.608385+0000", 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.608385+0000", 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.608386+0000", 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0160000000000001e-06 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.608389+0000", 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5300000000000001e-06 2024-09-16T10:17:03.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658558+0000", 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.050168327999999998 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.658567+0000", 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6399999999999992e-06 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660580+0000", 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002012921 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660595+0000", 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4829e-05 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:14:57.660848+0000", 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025320200000000001 2024-09-16T10:17:03.444 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:43099 7.b 7:d023f2e8:::benchmark_data_smithi063_73086_object43098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.491188+0000", 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.868837412, 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.055426305000000002, 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.445 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 43099 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491188+0000", 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491188+0000", 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491189+0000", 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6390000000000001e-06 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491213+0000", 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4043e-05 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491225+0000", 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1838000000000001e-05 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491232+0000", 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9410000000000001e-06 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491320+0000", 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7768000000000004e-05 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491462+0000", 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014169000000000001 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.491590+0000", 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012858100000000001 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495298+0000", 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037081179999999998 2024-09-16T10:17:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495338+0000", 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9767000000000001e-05 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546532+0000", 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051193663 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546576+0000", 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4546999999999999e-05 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546614+0000", 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7710000000000003e-05 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:43101 7.b 7:d7aba04e:::benchmark_data_smithi063_73086_object43100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:17:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.492943+0000", 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.867082258, 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.053750943000000002, 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 43101 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.492943+0000", 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.492943+0000", 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.492945+0000", 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9099999999999999e-06 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.492967+0000", 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2619000000000001e-05 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.492977+0000", 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9320000000000008e-06 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.492983+0000", 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.22e-06 2024-09-16T10:17:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.493187+0000", 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020398500000000001 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.493284+0000", 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7724000000000002e-05 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.493411+0000", 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012696499999999999 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495423+0000", 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020115419999999998 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546521+0000", 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051098602999999999 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546625+0000", 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103801 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546661+0000", 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6242000000000003e-05 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546694+0000", 2024-09-16T10:17:03.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2400000000000001e-05 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:43105 7.b 7:d07db7ab:::benchmark_data_smithi063_73086_object43104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.494425+0000", 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.86560032600001, 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.060177617000000003, 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.458 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 43105 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.494425+0000", 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.494425+0000", 2024-09-16T10:17:03.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.494426+0000", 2024-09-16T10:17:03.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5260000000000001e-06 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.494451+0000", 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5119000000000001e-05 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.494462+0000", 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1042e-05 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.494470+0000", 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4390000000000003e-06 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495561+0000", 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0010910679999999999 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.544946+0000", 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.049385489999999997 2024-09-16T10:17:03.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.545095+0000", 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014835099999999999 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.549317+0000", 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00422244 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.552233+0000", 2024-09-16T10:17:03.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029154020000000001 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.553931+0000", 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016987320000000001 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.554307+0000", 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037581399999999998 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.554602+0000", 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00029519399999999998 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:43098 7.2 e98/81 7:44d2079e:::benchmark_data_smithi063_73086_object43097:head v 98'2650, mlcod=98'2646)", 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.495252+0000", 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.864773554, 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.051086288000000001, 2024-09-16T10:17:03.465 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495252+0000", 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495252+0000", 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495253+0000", 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.049e-06 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495275+0000", 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2677e-05 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495285+0000", 2024-09-16T10:17:03.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.363e-06 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495291+0000", 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4489999999999996e-06 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495334+0000", 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2519999999999999e-05 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.495351+0000", 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7452000000000002e-05 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546269+0000", 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.050918357999999997 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.546338+0000", 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8419999999999999e-05 2024-09-16T10:17:03.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:43497 7.0 e98/81 7:0cd41ee2:::benchmark_data_smithi063_73086_object43496:head v 98'2636, mlcod=98'2634)", 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.754977+0000", 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.605047782, 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.081692437000000007, 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.754977+0000", 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.754977+0000", 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:03.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.754979+0000", 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4130000000000001e-06 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.755014+0000", 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4761999999999997e-05 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.833970+0000", 2024-09-16T10:17:03.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.078956858000000005 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.833979+0000", 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7909999999999993e-06 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.834925+0000", 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00094625199999999997 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.834955+0000", 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9246000000000001e-05 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.836632+0000", 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001677273 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.836670+0000", 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7842000000000001e-05 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:119930 7.b 7:d26ad1d1:::benchmark_data_smithi063_73086_object651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:16:20.778662+0000", 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 42.581363179999997, 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.052801156000000002, 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:17:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 119930 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.778662+0000", 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.778662+0000", 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.778664+0000", 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0609999999999998e-06 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.778671+0000", 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8850000000000001e-06 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.778684+0000", 2024-09-16T10:17:03.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3142999999999999e-05 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.778693+0000", 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9080000000000001e-06 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.784228+0000", 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0055355070000000003 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.479 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.784573+0000", 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00034413100000000002 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.785582+0000", 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001008973 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.818691+0000", 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.033109472000000001 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.818749+0000", 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7655000000000003e-05 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.830750+0000", 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012001667000000001 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.831411+0000", 2024-09-16T10:17:03.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00066055900000000002 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:20.831463+0000", 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2194999999999997e-05 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:03.482 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:17:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:03.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:03.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:03.600 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:03.648 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:03.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:03.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:03.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:03.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:03.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:03.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:03.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:04.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.054 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:04.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:04.126 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:04.126 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: from='client.44963 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:17:04.126 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: osdmap e135: 8 total, 8 up, 8 in 2024-09-16T10:17:04.126 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: 6.1 scrub starts 2024-09-16T10:17:04.126 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: 4.3 scrub starts 2024-09-16T10:17:04.126 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: 4.3 scrub ok 2024-09-16T10:17:04.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: 6.1 scrub ok 2024-09-16T10:17:04.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[29993]: pgmap v153: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 751 op/s 2024-09-16T10:17:04.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:04.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: from='client.44963 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:17:04.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: osdmap e135: 8 total, 8 up, 8 in 2024-09-16T10:17:04.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: 6.1 scrub starts 2024-09-16T10:17:04.127 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: 4.3 scrub starts 2024-09-16T10:17:04.128 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: 4.3 scrub ok 2024-09-16T10:17:04.128 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: 6.1 scrub ok 2024-09-16T10:17:04.128 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:03 smithi063 ceph-mon[39470]: pgmap v153: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 751 op/s 2024-09-16T10:17:04.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:04.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:04.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:04.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:04.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: from='client.44963 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:17:04.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: osdmap e135: 8 total, 8 up, 8 in 2024-09-16T10:17:04.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: 6.1 scrub starts 2024-09-16T10:17:04.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: 4.3 scrub starts 2024-09-16T10:17:04.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: 4.3 scrub ok 2024-09-16T10:17:04.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: 6.1 scrub ok 2024-09-16T10:17:04.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:03 smithi165 ceph-mon[32696]: pgmap v153: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 660 B/s rd, 0 B/s wr, 751 op/s 2024-09-16T10:17:04.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:04.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:04.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:04.543 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:04.585 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:04.688 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:04.729 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:04.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:04.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:04.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[29993]: 5.19 scrub starts 2024-09-16T10:17:04.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[29993]: 5.19 scrub ok 2024-09-16T10:17:04.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[29993]: 4.b scrub starts 2024-09-16T10:17:04.887 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[29993]: 4.b scrub ok 2024-09-16T10:17:04.887 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[39470]: 5.19 scrub starts 2024-09-16T10:17:04.887 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[39470]: 5.19 scrub ok 2024-09-16T10:17:04.887 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[39470]: 4.b scrub starts 2024-09-16T10:17:04.887 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:04 smithi063 ceph-mon[39470]: 4.b scrub ok 2024-09-16T10:17:04.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:04.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:04.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:04.981 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:04.982 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:05.025 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:05.061 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:05.061 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:05.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:05.093 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:05.093 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:05.118 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:05.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:05.133 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:05.133 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:05.167 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:05.167 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:05.204 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:05.205 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:05.219 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:05.237 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:05.237 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:05.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:05.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:04 smithi165 ceph-mon[32696]: 5.19 scrub starts 2024-09-16T10:17:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:04 smithi165 ceph-mon[32696]: 5.19 scrub ok 2024-09-16T10:17:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:04 smithi165 ceph-mon[32696]: 4.b scrub starts 2024-09-16T10:17:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:04 smithi165 ceph-mon[32696]: 4.b scrub ok 2024-09-16T10:17:05.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:05.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:05.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:05.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:05.537 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:05.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:05.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:05.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:05.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:05.681 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:17:05.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:05.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:05.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:05.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:05.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:05.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:05.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:05 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3525110876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:05.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:05 smithi063 ceph-mon[29993]: pgmap v154: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-16T10:17:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:05 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3525110876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:05 smithi063 ceph-mon[39470]: pgmap v154: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-16T10:17:06.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:06.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:06.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:06.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:06.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:06.239 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:06.257 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:06.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:06.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:06.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:05 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3525110876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:05 smithi165 ceph-mon[32696]: pgmap v154: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-16T10:17:06.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:06.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:06.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:06.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:06.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:06.668 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:17:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:06.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:06.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:06.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:06.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:06.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:06.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:07.016 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:06 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:17:07.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:06 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:17:07.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:06 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:17:07.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:06 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:17:07.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:07.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:07.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:07.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:07.297 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:17:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:17:07.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:06 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:17:07.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:06 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:17:07.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:07.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:07.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:07.466 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:07.560 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:07.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:07.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:07.698 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:07.738 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:07.835 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T10:17:07.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:07.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:07.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:07.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:07.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:08.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:08.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:08.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:08.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[29993]: 4.e scrub starts 2024-09-16T10:17:08.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[29993]: 4.e scrub ok 2024-09-16T10:17:08.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[29993]: pgmap v155: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 646 op/s 2024-09-16T10:17:08.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:08.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1405030323' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:17:08.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[29993]: from='client.25018 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:17:08.174 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[39470]: 4.e scrub starts 2024-09-16T10:17:08.174 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[39470]: 4.e scrub ok 2024-09-16T10:17:08.174 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[39470]: pgmap v155: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 646 op/s 2024-09-16T10:17:08.174 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:08.174 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1405030323' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:17:08.174 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:07 smithi063 ceph-mon[39470]: from='client.25018 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:17:08.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:17:08.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:07 smithi165 ceph-mon[32696]: 4.e scrub starts 2024-09-16T10:17:08.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:07 smithi165 ceph-mon[32696]: 4.e scrub ok 2024-09-16T10:17:08.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:07 smithi165 ceph-mon[32696]: pgmap v155: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 646 op/s 2024-09-16T10:17:08.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:08.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:07 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1405030323' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:17:08.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:07 smithi165 ceph-mon[32696]: from='client.25018 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:17:08.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:08.265 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:08.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:08.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:08.414 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:08.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:08.516 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:08.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:17:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:17:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:08.557 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:08.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:08.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:08.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:08.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:08.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:17:08.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:08.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:08.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:08.922 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:08.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:09.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:09.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:09.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:09.169 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:08 smithi063 ceph-mon[39470]: from='client.25018 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:17:09.169 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:08 smithi063 ceph-mon[39470]: osdmap e136: 8 total, 8 up, 8 in 2024-09-16T10:17:09.170 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:08 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:09.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:08 smithi063 ceph-mon[29993]: from='client.25018 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:17:09.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:08 smithi063 ceph-mon[29993]: osdmap e136: 8 total, 8 up, 8 in 2024-09-16T10:17:09.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:08 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:09.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:09.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T10:17:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:09.285 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:09.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:08 smithi165 ceph-mon[32696]: from='client.25018 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:17:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:08 smithi165 ceph-mon[32696]: osdmap e136: 8 total, 8 up, 8 in 2024-09-16T10:17:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:08 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:09.336 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:09.342 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:09.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:09.408 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:09.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:09.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:09.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:09.553 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:09.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:09.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:09.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:09.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:09.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:09.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:09.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:09.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:09.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:09.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:10.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:10.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:09 smithi165 ceph-mon[32696]: 6.8 scrub starts 2024-09-16T10:17:10.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:09 smithi165 ceph-mon[32696]: 6.8 scrub ok 2024-09-16T10:17:10.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:09 smithi165 ceph-mon[32696]: pgmap v157: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-16T10:17:10.078 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:10.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:10.094 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:09 smithi063 ceph-mon[39470]: 6.8 scrub starts 2024-09-16T10:17:10.094 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:09 smithi063 ceph-mon[39470]: 6.8 scrub ok 2024-09-16T10:17:10.094 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:09 smithi063 ceph-mon[39470]: pgmap v157: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-16T10:17:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:09 smithi063 ceph-mon[29993]: 6.8 scrub starts 2024-09-16T10:17:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:09 smithi063 ceph-mon[29993]: 6.8 scrub ok 2024-09-16T10:17:10.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:09 smithi063 ceph-mon[29993]: pgmap v157: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-16T10:17:10.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:10.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:10.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:10.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:10.344 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:10.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:10.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:10.491 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:10.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:10.522 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:10.522 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:10.566 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:10.567 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:10.593 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:10.606 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:10.607 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:10.631 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:10.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:10.700 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:10.700 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:10.733 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:10.737 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:10.737 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:10.779 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:10.814 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:10.814 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:10.821 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:10.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:10.885 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:17:10.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:10.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:10.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:10.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/223174357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:10.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3635504531' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:10.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:10 smithi063 ceph-mon[29993]: from='client.25024 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:10.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/223174357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:10.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3635504531' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:10.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:10 smithi063 ceph-mon[39470]: from='client.25024 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:11.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:11.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:11.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:11.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:11.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:11.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:11.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/223174357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:11.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3635504531' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:11.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:10 smithi165 ceph-mon[32696]: from='client.25024 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:11.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:11.345 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:11.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:11.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:11.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:11.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:11.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:11.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:11.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:11.704 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:11.754 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:11.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:11.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:11.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:11.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:11.985 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.025 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:12.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:12.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:12.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:12.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[29993]: from='client.25024 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[29993]: osdmap e137: 8 total, 8 up, 8 in 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[29993]: 3.6 scrub starts 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[29993]: 3.6 scrub ok 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[29993]: pgmap v159: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:17:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:17:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[39470]: from='client.25024 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[39470]: osdmap e137: 8 total, 8 up, 8 in 2024-09-16T10:17:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[39470]: 3.6 scrub starts 2024-09-16T10:17:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[39470]: 3.6 scrub ok 2024-09-16T10:17:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:11 smithi063 ceph-mon[39470]: pgmap v159: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:17:12.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:12.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:11 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:17:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:11 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:17:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:11 smithi165 ceph-mon[32696]: from='client.25024 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:11 smithi165 ceph-mon[32696]: osdmap e137: 8 total, 8 up, 8 in 2024-09-16T10:17:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:11 smithi165 ceph-mon[32696]: 3.6 scrub starts 2024-09-16T10:17:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:11 smithi165 ceph-mon[32696]: 3.6 scrub ok 2024-09-16T10:17:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:11 smithi165 ceph-mon[32696]: pgmap v159: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:17:12.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:12.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:12.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:12.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:12.646 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:12.682 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:12.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:12.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:12.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:12.955 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:12.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:12.987 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:17:12.987 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:17:12.987 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:17:12.987 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:17:12.987 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:17:13.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:13.097 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:13.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:13.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:12 smithi063 ceph-mon[29993]: 6.18 scrub starts 2024-09-16T10:17:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:12 smithi063 ceph-mon[29993]: 6.18 scrub ok 2024-09-16T10:17:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:12 smithi063 ceph-mon[39470]: 6.18 scrub starts 2024-09-16T10:17:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:12 smithi063 ceph-mon[39470]: 6.18 scrub ok 2024-09-16T10:17:13.247 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:13.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:13.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:12 smithi165 ceph-mon[32696]: 6.18 scrub starts 2024-09-16T10:17:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:12 smithi165 ceph-mon[32696]: 6.18 scrub ok 2024-09-16T10:17:13.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:13.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:13.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:13.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:13.522 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:13.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:13.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:13.655 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:13.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:13.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:13.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:17:13.797 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:13.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:13.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:13.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:13.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:14.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:14.073 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:14.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:14.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:17:14.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:17:14.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[29993]: 3.1e scrub starts 2024-09-16T10:17:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[29993]: 3.1e scrub ok 2024-09-16T10:17:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[29993]: pgmap v160: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s 2024-09-16T10:17:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[29993]: 3.19 scrub starts 2024-09-16T10:17:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[29993]: 3.19 scrub ok 2024-09-16T10:17:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:17:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:17:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[39470]: 3.1e scrub starts 2024-09-16T10:17:14.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[39470]: 3.1e scrub ok 2024-09-16T10:17:14.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[39470]: pgmap v160: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s 2024-09-16T10:17:14.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[39470]: 3.19 scrub starts 2024-09-16T10:17:14.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:13 smithi063 ceph-mon[39470]: 3.19 scrub ok 2024-09-16T10:17:14.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:14.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:13 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:17:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:13 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:17:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:13 smithi165 ceph-mon[32696]: 3.1e scrub starts 2024-09-16T10:17:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:13 smithi165 ceph-mon[32696]: 3.1e scrub ok 2024-09-16T10:17:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:13 smithi165 ceph-mon[32696]: pgmap v160: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s 2024-09-16T10:17:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:13 smithi165 ceph-mon[32696]: 3.19 scrub starts 2024-09-16T10:17:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:13 smithi165 ceph-mon[32696]: 3.19 scrub ok 2024-09-16T10:17:14.320 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:14.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:14.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:14.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:14.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:14.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:14.689 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:14.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:14.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:17:14.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:14.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:14.862 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:17:14.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:14.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:17:14.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:17:14.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[29993]: 5.12 scrub starts 2024-09-16T10:17:14.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[29993]: 5.12 scrub ok 2024-09-16T10:17:14.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:17:14.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:17:14.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[39470]: 5.12 scrub starts 2024-09-16T10:17:14.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:14 smithi063 ceph-mon[39470]: 5.12 scrub ok 2024-09-16T10:17:14.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:14.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:14.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:15.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:15.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:15.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:15.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:15.223 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:15.268 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:15.278 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:15.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:15.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:14 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:17:15.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:14 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:17:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:14 smithi165 ceph-mon[32696]: 5.12 scrub starts 2024-09-16T10:17:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:14 smithi165 ceph-mon[32696]: 5.12 scrub ok 2024-09-16T10:17:15.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:15.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:15.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:15.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:15.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:15.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:15.633 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:15.704 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:15.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:15.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:15.857 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:15.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:15.862 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:15.924 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:15.943 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T10:17:15.945 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:17:15.945 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:17:15.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: pgmap v161: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-16T10:17:15.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:17:15.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:17:15.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:17:15.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:17:15.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2766855519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:17:15.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[29993]: from='client.44981 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:17:15.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:17:15.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:17:15.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: pgmap v161: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-16T10:17:15.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:17:15.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:17:15.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:17:15.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:17:15.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2766855519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:17:15.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:15 smithi063 ceph-mon[39470]: from='client.44981 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:17:15.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:16.030 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:16.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:16.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:16.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:16.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: pgmap v161: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2766855519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:17:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:15 smithi165 ceph-mon[32696]: from='client.44981 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:17:16.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:16.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:16.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:16.551 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:16.653 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:16.678 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:16.678 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:16.693 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:16.723 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:16.724 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:16.765 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:16.766 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:16.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:16.805 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:16.805 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:16.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:16.845 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:16.845 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:16.889 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:16.889 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:16.924 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:16.925 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:16.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:16.951 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:17:16.968 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:16.968 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:16.998 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:16.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:17.099 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:17.138 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:17.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:17.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: 5.9 scrub starts 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: 5.9 scrub ok 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: from='client.44981 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: osdmap e138: 8 total, 8 up, 8 in 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1424257845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: from='client.25036 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2204546214' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:17.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: 5.9 scrub starts 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: 5.9 scrub ok 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: from='client.44981 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: osdmap e138: 8 total, 8 up, 8 in 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1424257845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: from='client.25036 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2204546214' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:16 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:17:17.203 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:17:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:17:17.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:17.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:17.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: 5.9 scrub starts 2024-09-16T10:17:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: 5.9 scrub ok 2024-09-16T10:17:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: from='client.44981 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:17:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: osdmap e138: 8 total, 8 up, 8 in 2024-09-16T10:17:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:17:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:17:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1424257845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:17.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: from='client.25036 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:17.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2204546214' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:17.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:16 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:17:17.373 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:17.412 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:17.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:17.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:17.557 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:17.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:17.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:17.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:17.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:17.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:17.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:17.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:17.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:17.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:17.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:18.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:18.099 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:18.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:18.201 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:18.203 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T10:17:18.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:17:18.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:17 smithi165 ceph-mon[32696]: 4.1b scrub starts 2024-09-16T10:17:18.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:17 smithi165 ceph-mon[32696]: 4.1b scrub ok 2024-09-16T10:17:18.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:17 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:18.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:17 smithi165 ceph-mon[32696]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:17:18.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:17 smithi165 ceph-mon[32696]: osdmap e139: 8 total, 8 up, 8 in 2024-09-16T10:17:18.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:17 smithi165 ceph-mon[32696]: pgmap v164: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 826 B/s rd, 0 B/s wr, 847 op/s 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[29993]: 4.1b scrub starts 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[29993]: 4.1b scrub ok 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[29993]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[29993]: osdmap e139: 8 total, 8 up, 8 in 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[29993]: pgmap v164: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 826 B/s rd, 0 B/s wr, 847 op/s 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[39470]: 4.1b scrub starts 2024-09-16T10:17:18.254 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[39470]: 4.1b scrub ok 2024-09-16T10:17:18.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:18.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[39470]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:17:18.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[39470]: osdmap e139: 8 total, 8 up, 8 in 2024-09-16T10:17:18.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:17 smithi063 ceph-mon[39470]: pgmap v164: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 826 B/s rd, 0 B/s wr, 847 op/s 2024-09-16T10:17:18.290 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:18.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:18.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:18.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:18.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:18.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:18.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:17:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:17:18.521 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:18.570 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:18.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:18.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:18.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:18.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:18.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:18.827 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:18.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:18.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:18.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:18.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:19.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:18 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:19.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:19.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:19.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:19.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:19.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:18 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:19.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:18 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:19.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:19.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:19.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:19.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:19.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:19.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:19.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:19.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:19.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:19.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:19.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:19.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:19.733 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:19.733 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:19.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:19.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:19.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:19.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:19.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:20.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:20.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:20.110 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:20.146 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:20.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.279 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:20.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:20.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:19 smithi165 ceph-mon[32696]: pgmap v165: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-16T10:17:20.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:20.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:20.407 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:19 smithi063 ceph-mon[39470]: pgmap v165: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-16T10:17:20.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:20.409 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:19 smithi063 ceph-mon[29993]: pgmap v165: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-16T10:17:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:20.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:17:20.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:20.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:20.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:20.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:20.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:20.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:20.833 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:20.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:20.947 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:17:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:20.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:20.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:21.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:21.109 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:20 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:17:21.109 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:21 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:17:21.109 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:21 smithi063 ceph-mon[29993]: osdmap e140: 8 total, 8 up, 8 in 2024-09-16T10:17:21.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:21 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:17:21.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:21 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:17:21.110 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:21 smithi063 ceph-mon[39470]: osdmap e140: 8 total, 8 up, 8 in 2024-09-16T10:17:21.115 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:21.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:21.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:20 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:17:21.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:20 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:17:21.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:20 smithi165 ceph-mon[32696]: osdmap e140: 8 total, 8 up, 8 in 2024-09-16T10:17:21.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:21.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:21.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:21.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:21.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:21.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:21.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:21.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:21.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:21.544 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:17:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:17:21.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:21.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:17:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:21.633 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:21.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:21.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:21.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:21.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:21.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:21.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:21.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:21.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.030 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:22.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:22.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:22.157 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:22.165 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:22.166 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:22 smithi063 ceph-mon[29993]: pgmap v167: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 829 op/s 2024-09-16T10:17:22.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:22 smithi063 ceph-mon[29993]: from='client.44993 v1:172.21.15.63:0/2157777534' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:22 smithi063 ceph-mon[39470]: pgmap v167: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 829 op/s 2024-09-16T10:17:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:22 smithi063 ceph-mon[39470]: from='client.44993 v1:172.21.15.63:0/2157777534' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:22.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:22.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:22.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:22 smithi165 ceph-mon[32696]: pgmap v167: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 829 op/s 2024-09-16T10:17:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:22 smithi165 ceph-mon[32696]: from='client.44993 v1:172.21.15.63:0/2157777534' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:22.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:22.429 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:22.532 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:22.586 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:22.688 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:22.730 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:22.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:22.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:22.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:22.997 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:22.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:22.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:23.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:23.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:23.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:23.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:23.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:23.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:23.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:23.397 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:23.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:23.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:23 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:23.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:23 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:23 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:23 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:23.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:23.537 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:23.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:23.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:23 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:23.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:23 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:23.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:23.638 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:23.688 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:23.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:23.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:23.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:23.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:23.923 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:23.958 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:23.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:24.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:24.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:24.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:24.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:24.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:24.220 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:24.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:24.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:24.358 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:24.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:24.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:24 smithi063 ceph-mon[29993]: pgmap v168: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 715 B/s rd, 0 B/s wr, 776 op/s 2024-09-16T10:17:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:24 smithi063 ceph-mon[39470]: pgmap v168: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 715 B/s rd, 0 B/s wr, 776 op/s 2024-09-16T10:17:24.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:24.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:24.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:24.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:24 smithi165 ceph-mon[32696]: pgmap v168: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 715 B/s rd, 0 B/s wr, 776 op/s 2024-09-16T10:17:24.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:24.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:24.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:24.722 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:24.751 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:24.751 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:24.762 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:24.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:24.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:24.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:24.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:24.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:25.041 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:25.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:25.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:25.143 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:25.189 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:25.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:25.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:25.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:25.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:25.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:25.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:17:25.477 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:25.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:25.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:25.635 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:25.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:25.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:25.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:25.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:25.862 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:25.909 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:25.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:25.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:26.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:26.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:26.063 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:26.126 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T10:17:21.083866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3894155827,"num_objects":59611,"num_object_clones":0,"num_object_copies":178833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59611,"num_whiteouts":0,"num_read":1064,"num_read_kb":921,"num_write":179691,"num_write_kb":7635338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90338,"ondisk_log_size":90338,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12299576,"kb_used_data":11475144,"kb_used_omap":12,"kb_used_meta":824051,"kb_avail":737661640,"statfs":{"total":767960285184,"available":755365519360,"internally_reserved":0,"allocated":11750547456,"data_stored":11747547521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":843828785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-326041600,"num_objects":-4975,"num_object_clones":0,"num_object_copies":-14925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4975,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4975,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002806"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.971923+0000","last_change":"2024-09-16T10:16:35.216054+0000","last_active":"2024-09-16T10:17:16.971923+0000","last_peered":"2024-09-16T10:17:16.971923+0000","last_clean":"2024-09-16T10:17:16.971923+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:17:16.971923+0000","last_undegraded":"2024-09-16T10:17:16.971923+0000","last_fullsized":"2024-09-16T10:17:16.971923+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:35.215992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.215992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:42.184249+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":217,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.947874+0000","last_change":"2024-09-16T10:16:35.813476+0000","last_active":"2024-09-16T10:17:16.947874+0000","last_peered":"2024-09-16T10:17:16.947874+0000","last_clean":"2024-09-16T10:17:16.947874+0000","last_became_active":"2024-09-16T10:15:08.925269+0000","last_became_peered":"2024-09-16T10:15:08.925269+0000","last_unstale":"2024-09-16T10:17:16.947874+0000","last_undegraded":"2024-09-16T10:17:16.947874+0000","last_fullsized":"2024-09-16T10:17:16.947874+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:35.813372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.813372+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:03.819055+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.984829+0000","last_change":"2024-09-16T10:17:04.062121+0000","last_active":"2024-09-16T10:17:16.984829+0000","last_peered":"2024-09-16T10:17:16.984829+0000","last_clean":"2024-09-16T10:17:16.984829+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:17:16.984829+0000","last_undegraded":"2024-09-16T10:17:16.984829+0000","last_fullsized":"2024-09-16T10:17:16.984829+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:04.062060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.062060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.222918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.947964+0000","last_change":"2024-09-16T10:16:16.760046+0000","last_active":"2024-09-16T10:17:16.947964+0000","last_peered":"2024-09-16T10:17:16.947964+0000","last_clean":"2024-09-16T10:17:16.947964+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:17:16.947964+0000","last_undegraded":"2024-09-16T10:17:16.947964+0000","last_fullsized":"2024-09-16T10:17:16.947964+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.759981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.759981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:41.660333+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959719+0000","last_change":"2024-09-16T10:16:42.246257+0000","last_active":"2024-09-16T10:17:16.959719+0000","last_peered":"2024-09-16T10:17:16.959719+0000","last_clean":"2024-09-16T10:17:16.959719+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:17:16.959719+0000","last_undegraded":"2024-09-16T10:17:16.959719+0000","last_fullsized":"2024-09-16T10:17:16.959719+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:42.246193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.246193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.520461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965120+0000","last_change":"2024-09-16T10:17:13.018487+0000","last_active":"2024-09-16T10:17:16.965120+0000","last_peered":"2024-09-16T10:17:16.965120+0000","last_clean":"2024-09-16T10:17:16.965120+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:17:16.965120+0000","last_undegraded":"2024-09-16T10:17:16.965120+0000","last_fullsized":"2024-09-16T10:17:16.965120+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:13.018412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.018412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.241294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":210,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960292+0000","last_change":"2024-09-16T10:16:31.217281+0000","last_active":"2024-09-16T10:17:16.960292+0000","last_peered":"2024-09-16T10:17:16.960292+0000","last_clean":"2024-09-16T10:17:16.960292+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:17:16.960292+0000","last_undegraded":"2024-09-16T10:17:16.960292+0000","last_fullsized":"2024-09-16T10:17:16.960292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:16:31.217192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.217192+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:32.608583+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.947753+0000","last_change":"2024-09-16T10:17:14.871814+0000","last_active":"2024-09-16T10:17:16.947753+0000","last_peered":"2024-09-16T10:17:16.947753+0000","last_clean":"2024-09-16T10:17:16.947753+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:17:16.947753+0000","last_undegraded":"2024-09-16T10:17:16.947753+0000","last_fullsized":"2024-09-16T10:17:16.947753+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.871750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.871750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:31.182111+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960094+0000","last_change":"2024-09-16T10:15:29.351071+0000","last_active":"2024-09-16T10:17:16.960094+0000","last_peered":"2024-09-16T10:17:16.960094+0000","last_clean":"2024-09-16T10:17:16.960094+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:17:16.960094+0000","last_undegraded":"2024-09-16T10:17:16.960094+0000","last_fullsized":"2024-09-16T10:17:16.960094+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:29.351000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.351000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:01.422071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":200,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948253+0000","last_change":"2024-09-16T10:16:32.833966+0000","last_active":"2024-09-16T10:17:16.948253+0000","last_peered":"2024-09-16T10:17:16.948253+0000","last_clean":"2024-09-16T10:17:16.948253+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:17:16.948253+0000","last_undegraded":"2024-09-16T10:17:16.948253+0000","last_fullsized":"2024-09-16T10:17:16.948253+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:32.833895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.833895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:43.494945+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959582+0000","last_change":"2024-09-16T10:15:22.271162+0000","last_active":"2024-09-16T10:17:16.959582+0000","last_peered":"2024-09-16T10:17:16.959582+0000","last_clean":"2024-09-16T10:17:16.959582+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:17:16.959582+0000","last_undegraded":"2024-09-16T10:17:16.959582+0000","last_fullsized":"2024-09-16T10:17:16.959582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:22.271101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:22.271101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.398679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.968529+0000","last_change":"2024-09-16T10:17:12.089641+0000","last_active":"2024-09-16T10:17:16.968529+0000","last_peered":"2024-09-16T10:17:16.968529+0000","last_clean":"2024-09-16T10:17:16.968529+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:17:16.968529+0000","last_undegraded":"2024-09-16T10:17:16.968529+0000","last_fullsized":"2024-09-16T10:17:16.968529+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.089577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.089577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.096958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959581+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:17:16.959581+0000","last_peered":"2024-09-16T10:17:16.959581+0000","last_clean":"2024-09-16T10:17:16.959581+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:17:16.959581+0000","last_undegraded":"2024-09-16T10:17:16.959581+0000","last_fullsized":"2024-09-16T10:17:16.959581+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:18.386128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.947952+0000","last_change":"2024-09-16T10:17:16.937447+0000","last_active":"2024-09-16T10:17:16.947952+0000","last_peered":"2024-09-16T10:17:16.947952+0000","last_clean":"2024-09-16T10:17:16.947952+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:17:16.947952+0000","last_undegraded":"2024-09-16T10:17:16.947952+0000","last_fullsized":"2024-09-16T10:17:16.947952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:17:16.937353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.937353+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.410589+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974604+0000","last_change":"2024-09-16T10:15:16.573685+0000","last_active":"2024-09-16T10:17:16.974604+0000","last_peered":"2024-09-16T10:17:16.974604+0000","last_clean":"2024-09-16T10:17:16.974604+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:17:16.974604+0000","last_undegraded":"2024-09-16T10:17:16.974604+0000","last_fullsized":"2024-09-16T10:17:16.974604+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.573614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.573614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.640699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.958997+0000","last_change":"2024-09-16T10:15:11.297702+0000","last_active":"2024-09-16T10:17:16.958997+0000","last_peered":"2024-09-16T10:17:16.958997+0000","last_clean":"2024-09-16T10:17:16.958997+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:17:16.958997+0000","last_undegraded":"2024-09-16T10:17:16.958997+0000","last_fullsized":"2024-09-16T10:17:16.958997+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.297599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.297599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:55.375459+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.949045+0000","last_change":"2024-09-16T10:16:15.776330+0000","last_active":"2024-09-16T10:17:16.949045+0000","last_peered":"2024-09-16T10:17:16.949045+0000","last_clean":"2024-09-16T10:17:16.949045+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:17:16.949045+0000","last_undegraded":"2024-09-16T10:17:16.949045+0000","last_fullsized":"2024-09-16T10:17:16.949045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:15.776284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:15.776284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:35.232885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.969867+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:17:16.969867+0000","last_peered":"2024-09-16T10:17:16.969867+0000","last_clean":"2024-09-16T10:17:16.969867+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:17:16.969867+0000","last_undegraded":"2024-09-16T10:17:16.969867+0000","last_fullsized":"2024-09-16T10:17:16.969867+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.208132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959560+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:17:16.959560+0000","last_peered":"2024-09-16T10:17:16.959560+0000","last_clean":"2024-09-16T10:17:16.959560+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:17:16.959560+0000","last_undegraded":"2024-09-16T10:17:16.959560+0000","last_fullsized":"2024-09-16T10:17:16.959560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:27.433002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960785+0000","last_change":"2024-09-16T10:16:16.067128+0000","last_active":"2024-09-16T10:17:16.960785+0000","last_peered":"2024-09-16T10:17:16.960785+0000","last_clean":"2024-09-16T10:17:16.960785+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:17:16.960785+0000","last_undegraded":"2024-09-16T10:17:16.960785+0000","last_fullsized":"2024-09-16T10:17:16.960785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.066855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.066855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:22.755065+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.990998+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:17:16.990998+0000","last_peered":"2024-09-16T10:17:16.990998+0000","last_clean":"2024-09-16T10:17:16.990998+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:17:16.990998+0000","last_undegraded":"2024-09-16T10:17:16.990998+0000","last_fullsized":"2024-09-16T10:17:16.990998+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.270038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948399+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:17:16.948399+0000","last_peered":"2024-09-16T10:17:16.948399+0000","last_clean":"2024-09-16T10:17:16.948399+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:17:16.948399+0000","last_undegraded":"2024-09-16T10:17:16.948399+0000","last_fullsized":"2024-09-16T10:17:16.948399+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.816041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":227,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.966942+0000","last_change":"2024-09-16T10:16:35.181066+0000","last_active":"2024-09-16T10:17:16.966942+0000","last_peered":"2024-09-16T10:17:16.966942+0000","last_clean":"2024-09-16T10:17:16.966942+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:17:16.966942+0000","last_undegraded":"2024-09-16T10:17:16.966942+0000","last_fullsized":"2024-09-16T10:17:16.966942+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:16:35.180977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.180977+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:35.365790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948372+0000","last_change":"2024-09-16T10:16:31.785080+0000","last_active":"2024-09-16T10:17:16.948372+0000","last_peered":"2024-09-16T10:17:16.948372+0000","last_clean":"2024-09-16T10:17:16.948372+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:17:16.948372+0000","last_undegraded":"2024-09-16T10:17:16.948372+0000","last_fullsized":"2024-09-16T10:17:16.948372+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:31.784724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.784724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.916223+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974998+0000","last_change":"2024-09-16T10:16:37.672462+0000","last_active":"2024-09-16T10:17:16.974998+0000","last_peered":"2024-09-16T10:17:16.974998+0000","last_clean":"2024-09-16T10:17:16.974998+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:17:16.974998+0000","last_undegraded":"2024-09-16T10:17:16.974998+0000","last_fullsized":"2024-09-16T10:17:16.974998+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.246050+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.985236+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:17:16.985236+0000","last_peered":"2024-09-16T10:17:16.985236+0000","last_clean":"2024-09-16T10:17:16.985236+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:17:16.985236+0000","last_undegraded":"2024-09-16T10:17:16.985236+0000","last_fullsized":"2024-09-16T10:17:16.985236+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.691808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":191,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.971150+0000","last_change":"2024-09-16T10:15:13.091323+0000","last_active":"2024-09-16T10:17:16.971150+0000","last_peered":"2024-09-16T10:17:16.971150+0000","last_clean":"2024-09-16T10:17:16.971150+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:17:16.971150+0000","last_undegraded":"2024-09-16T10:17:16.971150+0000","last_fullsized":"2024-09-16T10:17:16.971150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:15:13.091193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.091193+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:27.281624+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":236,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.966532+0000","last_change":"2024-09-16T10:16:37.249076+0000","last_active":"2024-09-16T10:17:16.966532+0000","last_peered":"2024-09-16T10:17:16.966532+0000","last_clean":"2024-09-16T10:17:16.966532+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:17:16.966532+0000","last_undegraded":"2024-09-16T10:17:16.966532+0000","last_fullsized":"2024-09-16T10:17:16.966532+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:16:37.249033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.249033+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.553977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"139'11637","reported_seq":11765,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.091614+0000","last_change":"2024-09-16T10:14:43.957473+0000","last_active":"2024-09-16T10:17:18.091614+0000","last_peered":"2024-09-16T10:17:18.091614+0000","last_clean":"2024-09-16T10:17:18.091614+0000","last_became_active":"2024-09-16T10:14:43.957174+0000","last_became_peered":"2024-09-16T10:14:43.957174+0000","last_unstale":"2024-09-16T10:17:18.091614+0000","last_undegraded":"2024-09-16T10:17:18.091614+0000","last_fullsized":"2024-09-16T10:17:18.091614+0000","mapping_epoch":81,"log_start":"110'5300","ondisk_log_start":"110'5300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.202365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256573440,"num_objects":3915,"num_object_clones":0,"num_object_copies":11745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":497664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":202,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.958031+0000","last_change":"2024-09-16T10:14:45.136557+0000","last_active":"2024-09-16T10:17:16.958031+0000","last_peered":"2024-09-16T10:17:16.958031+0000","last_clean":"2024-09-16T10:17:16.958031+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:17:16.958031+0000","last_undegraded":"2024-09-16T10:17:16.958031+0000","last_fullsized":"2024-09-16T10:17:16.958031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:14:45.136460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.136460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.941516+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965639+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:17:16.965639+0000","last_peered":"2024-09-16T10:17:16.965639+0000","last_clean":"2024-09-16T10:17:16.965639+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:17:16.965639+0000","last_undegraded":"2024-09-16T10:17:16.965639+0000","last_fullsized":"2024-09-16T10:17:16.965639+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:45.174072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":217,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948526+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:17:16.948526+0000","last_peered":"2024-09-16T10:17:16.948526+0000","last_clean":"2024-09-16T10:17:16.948526+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:17:16.948526+0000","last_undegraded":"2024-09-16T10:17:16.948526+0000","last_fullsized":"2024-09-16T10:17:16.948526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:21.787825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959549+0000","last_change":"2024-09-16T10:15:54.314741+0000","last_active":"2024-09-16T10:17:16.959549+0000","last_peered":"2024-09-16T10:17:16.959549+0000","last_clean":"2024-09-16T10:17:16.959549+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:17:16.959549+0000","last_undegraded":"2024-09-16T10:17:16.959549+0000","last_fullsized":"2024-09-16T10:17:16.959549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:54.314680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.314680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.427634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"139'11101","reported_seq":11308,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.234279+0000","last_change":"2024-09-16T10:16:16.640532+0000","last_active":"2024-09-16T10:17:18.234279+0000","last_peered":"2024-09-16T10:17:18.234279+0000","last_clean":"2024-09-16T10:17:18.234279+0000","last_became_active":"2024-09-16T10:14:43.952713+0000","last_became_peered":"2024-09-16T10:14:43.952713+0000","last_unstale":"2024-09-16T10:17:18.234279+0000","last_undegraded":"2024-09-16T10:17:18.234279+0000","last_fullsized":"2024-09-16T10:17:18.234279+0000","mapping_epoch":81,"log_start":"114'6500","ondisk_log_start":"114'6500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7146","last_scrub_stamp":"2024-09-16T10:16:16.640092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.640092+0000","objects_scrubbed":7079,"log_size":4601,"log_dups_size":0,"ondisk_log_size":4601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:32.050074+0000","scrub_duration":2558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243335168,"num_objects":3713,"num_object_clones":0,"num_object_copies":11139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11101,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":209,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.964588+0000","last_change":"2024-09-16T10:15:23.703368+0000","last_active":"2024-09-16T10:17:16.964588+0000","last_peered":"2024-09-16T10:17:16.964588+0000","last_clean":"2024-09-16T10:17:16.964588+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:17:16.964588+0000","last_undegraded":"2024-09-16T10:17:16.964588+0000","last_fullsized":"2024-09-16T10:17:16.964588+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:15:23.703278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.703278+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.166045+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.991343+0000","last_change":"2024-09-16T10:16:11.013486+0000","last_active":"2024-09-16T10:17:16.991343+0000","last_peered":"2024-09-16T10:17:16.991343+0000","last_clean":"2024-09-16T10:17:16.991343+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:17:16.991343+0000","last_undegraded":"2024-09-16T10:17:16.991343+0000","last_fullsized":"2024-09-16T10:17:16.991343+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:11.013414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:11.013414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:38.240615+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.947933+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:17:16.947933+0000","last_peered":"2024-09-16T10:17:16.947933+0000","last_clean":"2024-09-16T10:17:16.947933+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:17:16.947933+0000","last_undegraded":"2024-09-16T10:17:16.947933+0000","last_fullsized":"2024-09-16T10:17:16.947933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:17.745738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948159+0000","last_change":"2024-09-16T10:16:43.906280+0000","last_active":"2024-09-16T10:17:16.948159+0000","last_peered":"2024-09-16T10:17:16.948159+0000","last_clean":"2024-09-16T10:17:16.948159+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:17:16.948159+0000","last_undegraded":"2024-09-16T10:17:16.948159+0000","last_fullsized":"2024-09-16T10:17:16.948159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.906200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.906200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.512702+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"139'11117","reported_seq":11245,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.292510+0000","last_change":"2024-09-16T10:14:43.953541+0000","last_active":"2024-09-16T10:17:18.292510+0000","last_peered":"2024-09-16T10:17:18.292510+0000","last_clean":"2024-09-16T10:17:18.292510+0000","last_became_active":"2024-09-16T10:14:43.952908+0000","last_became_peered":"2024-09-16T10:14:43.952908+0000","last_unstale":"2024-09-16T10:17:18.292510+0000","last_undegraded":"2024-09-16T10:17:18.292510+0000","last_fullsized":"2024-09-16T10:17:18.292510+0000","mapping_epoch":81,"log_start":"114'6539","ondisk_log_start":"114'6539","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4578,"log_dups_size":0,"ondisk_log_size":4578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:22.067048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243073024,"num_objects":3709,"num_object_clones":0,"num_object_copies":11127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11117,"num_write_kb":474432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":205,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.947939+0000","last_change":"2024-09-16T10:16:37.847991+0000","last_active":"2024-09-16T10:17:16.947939+0000","last_peered":"2024-09-16T10:17:16.947939+0000","last_clean":"2024-09-16T10:17:16.947939+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:17:16.947939+0000","last_undegraded":"2024-09-16T10:17:16.947939+0000","last_fullsized":"2024-09-16T10:17:16.947939+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:16:37.847913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.847913+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.083962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974333+0000","last_change":"2024-09-16T10:17:11.050221+0000","last_active":"2024-09-16T10:17:16.974333+0000","last_peered":"2024-09-16T10:17:16.974333+0000","last_clean":"2024-09-16T10:17:16.974333+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:17:16.974333+0000","last_undegraded":"2024-09-16T10:17:16.974333+0000","last_fullsized":"2024-09-16T10:17:16.974333+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:17:11.050150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:11.050150+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:29.263200+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":229,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960573+0000","last_change":"2024-09-16T10:15:20.675679+0000","last_active":"2024-09-16T10:17:16.960573+0000","last_peered":"2024-09-16T10:17:16.960573+0000","last_clean":"2024-09-16T10:17:16.960573+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:17:16.960573+0000","last_undegraded":"2024-09-16T10:17:16.960573+0000","last_fullsized":"2024-09-16T10:17:16.960573+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:20.675600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:20.675600+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.107238+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974824+0000","last_change":"2024-09-16T10:16:03.498672+0000","last_active":"2024-09-16T10:17:16.974824+0000","last_peered":"2024-09-16T10:17:16.974824+0000","last_clean":"2024-09-16T10:17:16.974824+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:17:16.974824+0000","last_undegraded":"2024-09-16T10:17:16.974824+0000","last_fullsized":"2024-09-16T10:17:16.974824+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:17.759192+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"139'11324","reported_seq":11452,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.011584+0000","last_change":"2024-09-16T10:14:43.952332+0000","last_active":"2024-09-16T10:17:18.011584+0000","last_peered":"2024-09-16T10:17:18.011584+0000","last_clean":"2024-09-16T10:17:18.011584+0000","last_became_active":"2024-09-16T10:14:43.952043+0000","last_became_peered":"2024-09-16T10:14:43.952043+0000","last_unstale":"2024-09-16T10:17:18.011584+0000","last_undegraded":"2024-09-16T10:17:18.011584+0000","last_fullsized":"2024-09-16T10:17:18.011584+0000","mapping_epoch":81,"log_start":"114'6500","ondisk_log_start":"114'6500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:26.872890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243531776,"num_objects":3716,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":200,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.983897+0000","last_change":"2024-09-16T10:16:14.094325+0000","last_active":"2024-09-16T10:17:16.983897+0000","last_peered":"2024-09-16T10:17:16.983897+0000","last_clean":"2024-09-16T10:17:16.983897+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:17:16.983897+0000","last_undegraded":"2024-09-16T10:17:16.983897+0000","last_fullsized":"2024-09-16T10:17:16.983897+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:16:14.093976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.093976+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:37.672785+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960175+0000","last_change":"2024-09-16T10:15:23.304088+0000","last_active":"2024-09-16T10:17:16.960175+0000","last_peered":"2024-09-16T10:17:16.960175+0000","last_clean":"2024-09-16T10:17:16.960175+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:17:16.960175+0000","last_undegraded":"2024-09-16T10:17:16.960175+0000","last_fullsized":"2024-09-16T10:17:16.960175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:23.304033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.304033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.369253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":225,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.969464+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:17:16.969464+0000","last_peered":"2024-09-16T10:17:16.969464+0000","last_clean":"2024-09-16T10:17:16.969464+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:17:16.969464+0000","last_undegraded":"2024-09-16T10:17:16.969464+0000","last_fullsized":"2024-09-16T10:17:16.969464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.249438+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.968733+0000","last_change":"2024-09-16T10:15:16.104168+0000","last_active":"2024-09-16T10:17:16.968733+0000","last_peered":"2024-09-16T10:17:16.968733+0000","last_clean":"2024-09-16T10:17:16.968733+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:17:16.968733+0000","last_undegraded":"2024-09-16T10:17:16.968733+0000","last_fullsized":"2024-09-16T10:17:16.968733+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.104092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.104092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.259525+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"139'11019","reported_seq":11217,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.042064+0000","last_change":"2024-09-16T10:16:02.977482+0000","last_active":"2024-09-16T10:17:18.042064+0000","last_peered":"2024-09-16T10:17:18.042064+0000","last_clean":"2024-09-16T10:17:18.042064+0000","last_became_active":"2024-09-16T10:14:43.955808+0000","last_became_peered":"2024-09-16T10:14:43.955808+0000","last_unstale":"2024-09-16T10:17:18.042064+0000","last_undegraded":"2024-09-16T10:17:18.042064+0000","last_fullsized":"2024-09-16T10:17:18.042064+0000","mapping_epoch":81,"log_start":"111'5200","ondisk_log_start":"111'5200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6320","last_scrub_stamp":"2024-09-16T10:16:02.977318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.977318+0000","objects_scrubbed":6273,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:26.363900+0000","scrub_duration":1630,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247136256,"num_objects":3771,"num_object_clones":0,"num_object_copies":11313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11019,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":220,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.969601+0000","last_change":"2024-09-16T10:17:03.051621+0000","last_active":"2024-09-16T10:17:16.969601+0000","last_peered":"2024-09-16T10:17:16.969601+0000","last_clean":"2024-09-16T10:17:16.969601+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:17:16.969601+0000","last_undegraded":"2024-09-16T10:17:16.969601+0000","last_fullsized":"2024-09-16T10:17:16.969601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:17:03.050732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.050732+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.857216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.957448+0000","last_change":"2024-09-16T10:16:33.949944+0000","last_active":"2024-09-16T10:17:16.957448+0000","last_peered":"2024-09-16T10:17:16.957448+0000","last_clean":"2024-09-16T10:17:16.957448+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:17:16.957448+0000","last_undegraded":"2024-09-16T10:17:16.957448+0000","last_fullsized":"2024-09-16T10:17:16.957448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:33.949798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.949798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.958848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965559+0000","last_change":"2024-09-16T10:16:07.174774+0000","last_active":"2024-09-16T10:17:16.965559+0000","last_peered":"2024-09-16T10:17:16.965559+0000","last_clean":"2024-09-16T10:17:16.965559+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:17:16.965559+0000","last_undegraded":"2024-09-16T10:17:16.965559+0000","last_fullsized":"2024-09-16T10:17:16.965559+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:07.174690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:07.174690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:28.274560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.985060+0000","last_change":"2024-09-16T10:17:03.061305+0000","last_active":"2024-09-16T10:17:16.985060+0000","last_peered":"2024-09-16T10:17:16.985060+0000","last_clean":"2024-09-16T10:17:16.985060+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:17:16.985060+0000","last_undegraded":"2024-09-16T10:17:16.985060+0000","last_fullsized":"2024-09-16T10:17:16.985060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.257731+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"139'11314","reported_seq":11522,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.042244+0000","last_change":"2024-09-16T10:16:14.720883+0000","last_active":"2024-09-16T10:17:18.042244+0000","last_peered":"2024-09-16T10:17:18.042244+0000","last_clean":"2024-09-16T10:17:18.042244+0000","last_became_active":"2024-09-16T10:14:43.957195+0000","last_became_peered":"2024-09-16T10:14:43.957195+0000","last_unstale":"2024-09-16T10:17:18.042244+0000","last_undegraded":"2024-09-16T10:17:18.042244+0000","last_fullsized":"2024-09-16T10:17:18.042244+0000","mapping_epoch":81,"log_start":"108'5000","ondisk_log_start":"108'5000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7198","last_scrub_stamp":"2024-09-16T10:16:14.720775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.720775+0000","objects_scrubbed":7123,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:27.978026+0000","scrub_duration":2606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246022144,"num_objects":3754,"num_object_clones":0,"num_object_copies":11262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":286,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.957355+0000","last_change":"2024-09-16T10:17:15.133634+0000","last_active":"2024-09-16T10:17:16.957355+0000","last_peered":"2024-09-16T10:17:16.957355+0000","last_clean":"2024-09-16T10:17:16.957355+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:17:16.957355+0000","last_undegraded":"2024-09-16T10:17:16.957355+0000","last_fullsized":"2024-09-16T10:17:16.957355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:17:15.133542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.133542+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.091484+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948266+0000","last_change":"2024-09-16T10:14:44.986853+0000","last_active":"2024-09-16T10:17:16.948266+0000","last_peered":"2024-09-16T10:17:16.948266+0000","last_clean":"2024-09-16T10:17:16.948266+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:17:16.948266+0000","last_undegraded":"2024-09-16T10:17:16.948266+0000","last_fullsized":"2024-09-16T10:17:16.948266+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:44.986805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.986805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:25.864946+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":191,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959619+0000","last_change":"2024-09-16T10:16:16.430214+0000","last_active":"2024-09-16T10:17:16.959619+0000","last_peered":"2024-09-16T10:17:16.959619+0000","last_clean":"2024-09-16T10:17:16.959619+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:17:16.959619+0000","last_undegraded":"2024-09-16T10:17:16.959619+0000","last_fullsized":"2024-09-16T10:17:16.959619+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:16:16.430154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.430154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:38.184696+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.972226+0000","last_change":"2024-09-16T10:16:04.405318+0000","last_active":"2024-09-16T10:17:16.972226+0000","last_peered":"2024-09-16T10:17:16.972226+0000","last_clean":"2024-09-16T10:17:16.972226+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:17:16.972226+0000","last_undegraded":"2024-09-16T10:17:16.972226+0000","last_fullsized":"2024-09-16T10:17:16.972226+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:04.405243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.405243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:21.319583+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.964869+0000","last_change":"2024-09-16T10:17:14.029861+0000","last_active":"2024-09-16T10:17:16.964869+0000","last_peered":"2024-09-16T10:17:16.964869+0000","last_clean":"2024-09-16T10:17:16.964869+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:17:16.964869+0000","last_undegraded":"2024-09-16T10:17:16.964869+0000","last_fullsized":"2024-09-16T10:17:16.964869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.029754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.029754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.182687+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"139'11107","reported_seq":11235,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.229417+0000","last_change":"2024-09-16T10:14:43.956747+0000","last_active":"2024-09-16T10:17:18.229417+0000","last_peered":"2024-09-16T10:17:18.229417+0000","last_clean":"2024-09-16T10:17:18.229417+0000","last_became_active":"2024-09-16T10:14:43.956362+0000","last_became_peered":"2024-09-16T10:14:43.956362+0000","last_unstale":"2024-09-16T10:17:18.229417+0000","last_undegraded":"2024-09-16T10:17:18.229417+0000","last_fullsized":"2024-09-16T10:17:18.229417+0000","mapping_epoch":81,"log_start":"112'5500","ondisk_log_start":"112'5500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.282565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241500160,"num_objects":3685,"num_object_clones":0,"num_object_copies":11055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960304+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:17:16.960304+0000","last_peered":"2024-09-16T10:17:16.960304+0000","last_clean":"2024-09-16T10:17:16.960304+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:17:16.960304+0000","last_undegraded":"2024-09-16T10:17:16.960304+0000","last_fullsized":"2024-09-16T10:17:16.960304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.895956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974915+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:17:16.974915+0000","last_peered":"2024-09-16T10:17:16.974915+0000","last_clean":"2024-09-16T10:17:16.974915+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:17:16.974915+0000","last_undegraded":"2024-09-16T10:17:16.974915+0000","last_fullsized":"2024-09-16T10:17:16.974915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.883706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":190,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.955414+0000","last_change":"2024-09-16T10:15:13.676523+0000","last_active":"2024-09-16T10:17:16.955414+0000","last_peered":"2024-09-16T10:17:16.955414+0000","last_clean":"2024-09-16T10:17:16.955414+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:17:16.955414+0000","last_undegraded":"2024-09-16T10:17:16.955414+0000","last_fullsized":"2024-09-16T10:17:16.955414+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:13.676437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.676437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.412650+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":165,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.973337+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:17:16.973337+0000","last_peered":"2024-09-16T10:17:16.973337+0000","last_clean":"2024-09-16T10:17:16.973337+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:17:16.973337+0000","last_undegraded":"2024-09-16T10:17:16.973337+0000","last_fullsized":"2024-09-16T10:17:16.973337+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:22.331031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975135+0000","last_change":"2024-09-16T10:17:12.578269+0000","last_active":"2024-09-16T10:17:16.975135+0000","last_peered":"2024-09-16T10:17:16.975135+0000","last_clean":"2024-09-16T10:17:16.975135+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:17:16.975135+0000","last_undegraded":"2024-09-16T10:17:16.975135+0000","last_fullsized":"2024-09-16T10:17:16.975135+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.578203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.578203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.857634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"139'11294","reported_seq":11422,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.043381+0000","last_change":"2024-09-16T10:14:43.959854+0000","last_active":"2024-09-16T10:17:18.043381+0000","last_peered":"2024-09-16T10:17:18.043381+0000","last_clean":"2024-09-16T10:17:18.043381+0000","last_became_active":"2024-09-16T10:14:43.959612+0000","last_became_peered":"2024-09-16T10:14:43.959612+0000","last_unstale":"2024-09-16T10:17:18.043381+0000","last_undegraded":"2024-09-16T10:17:18.043381+0000","last_fullsized":"2024-09-16T10:17:18.043381+0000","mapping_epoch":81,"log_start":"113'6000","ondisk_log_start":"113'6000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:50.620169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":202,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.971236+0000","last_change":"2024-09-16T10:16:08.502530+0000","last_active":"2024-09-16T10:17:16.971236+0000","last_peered":"2024-09-16T10:17:16.971236+0000","last_clean":"2024-09-16T10:17:16.971236+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:17:16.971236+0000","last_undegraded":"2024-09-16T10:17:16.971236+0000","last_fullsized":"2024-09-16T10:17:16.971236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:16:08.502443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:08.502443+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:37.498249+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.968617+0000","last_change":"2024-09-16T10:17:15.102575+0000","last_active":"2024-09-16T10:17:16.968617+0000","last_peered":"2024-09-16T10:17:16.968617+0000","last_clean":"2024-09-16T10:17:16.968617+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:17:16.968617+0000","last_undegraded":"2024-09-16T10:17:16.968617+0000","last_fullsized":"2024-09-16T10:17:16.968617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:15.102512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.102512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.326101+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":415,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974661+0000","last_change":"2024-09-16T10:16:05.559992+0000","last_active":"2024-09-16T10:17:16.974661+0000","last_peered":"2024-09-16T10:17:16.974661+0000","last_clean":"2024-09-16T10:17:16.974661+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:17:16.974661+0000","last_undegraded":"2024-09-16T10:17:16.974661+0000","last_fullsized":"2024-09-16T10:17:16.974661+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:16:05.559913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.559913+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":215,"num_read_kb":210,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.962817+0000","last_change":"2024-09-16T10:16:14.370379+0000","last_active":"2024-09-16T10:17:16.962817+0000","last_peered":"2024-09-16T10:17:16.962817+0000","last_clean":"2024-09-16T10:17:16.962817+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:17:16.962817+0000","last_undegraded":"2024-09-16T10:17:16.962817+0000","last_fullsized":"2024-09-16T10:17:16.962817+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:14.370307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.370307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:21.981752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.956992+0000","last_change":"2024-09-16T10:17:16.162854+0000","last_active":"2024-09-16T10:17:16.956992+0000","last_peered":"2024-09-16T10:17:16.956992+0000","last_clean":"2024-09-16T10:17:16.956992+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:17:16.956992+0000","last_undegraded":"2024-09-16T10:17:16.956992+0000","last_fullsized":"2024-09-16T10:17:16.956992+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:16.162771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.162771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:36.273519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"139'11473","reported_seq":11601,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.237134+0000","last_change":"2024-09-16T10:14:43.957718+0000","last_active":"2024-09-16T10:17:18.237134+0000","last_peered":"2024-09-16T10:17:18.237134+0000","last_clean":"2024-09-16T10:17:18.237134+0000","last_became_active":"2024-09-16T10:14:43.957476+0000","last_became_peered":"2024-09-16T10:14:43.957476+0000","last_unstale":"2024-09-16T10:17:18.237134+0000","last_undegraded":"2024-09-16T10:17:18.237134+0000","last_fullsized":"2024-09-16T10:17:18.237134+0000","mapping_epoch":81,"log_start":"112'5900","ondisk_log_start":"112'5900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5573,"log_dups_size":0,"ondisk_log_size":5573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.193673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251854848,"num_objects":3843,"num_object_clones":0,"num_object_copies":11529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11473,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":243,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.964916+0000","last_change":"2024-09-16T10:16:10.182532+0000","last_active":"2024-09-16T10:17:16.964916+0000","last_peered":"2024-09-16T10:17:16.964916+0000","last_clean":"2024-09-16T10:17:16.964916+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:17:16.964916+0000","last_undegraded":"2024-09-16T10:17:16.964916+0000","last_fullsized":"2024-09-16T10:17:16.964916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:16:10.182468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:10.182468+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:24.089408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.966776+0000","last_change":"2024-09-16T10:16:16.115314+0000","last_active":"2024-09-16T10:17:16.966776+0000","last_peered":"2024-09-16T10:17:16.966776+0000","last_clean":"2024-09-16T10:17:16.966776+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:17:16.966776+0000","last_undegraded":"2024-09-16T10:17:16.966776+0000","last_fullsized":"2024-09-16T10:17:16.966776+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.115237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.115237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.909446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":226,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974711+0000","last_change":"2024-09-16T10:16:42.677394+0000","last_active":"2024-09-16T10:17:16.974711+0000","last_peered":"2024-09-16T10:17:16.974711+0000","last_clean":"2024-09-16T10:17:16.974711+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:17:16.974711+0000","last_undegraded":"2024-09-16T10:17:16.974711+0000","last_fullsized":"2024-09-16T10:17:16.974711+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:16:42.677309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.677309+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:10.457942+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975120+0000","last_change":"2024-09-16T10:16:43.622567+0000","last_active":"2024-09-16T10:17:16.975120+0000","last_peered":"2024-09-16T10:17:16.975120+0000","last_clean":"2024-09-16T10:17:16.975120+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:17:16.975120+0000","last_undegraded":"2024-09-16T10:17:16.975120+0000","last_fullsized":"2024-09-16T10:17:16.975120+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.773945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.962676+0000","last_change":"2024-09-16T10:16:34.311115+0000","last_active":"2024-09-16T10:17:16.962676+0000","last_peered":"2024-09-16T10:17:16.962676+0000","last_clean":"2024-09-16T10:17:16.962676+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:17:16.962676+0000","last_undegraded":"2024-09-16T10:17:16.962676+0000","last_fullsized":"2024-09-16T10:17:16.962676+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:34.311052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:34.311052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.994480+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"139'11303","reported_seq":11432,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.068784+0000","last_change":"2024-09-16T10:14:43.958855+0000","last_active":"2024-09-16T10:17:18.068784+0000","last_peered":"2024-09-16T10:17:18.068784+0000","last_clean":"2024-09-16T10:17:18.068784+0000","last_became_active":"2024-09-16T10:14:43.958616+0000","last_became_peered":"2024-09-16T10:14:43.958616+0000","last_unstale":"2024-09-16T10:17:18.068784+0000","last_undegraded":"2024-09-16T10:17:18.068784+0000","last_fullsized":"2024-09-16T10:17:18.068784+0000","mapping_epoch":81,"log_start":"108'5000","ondisk_log_start":"108'5000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.193911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246677528,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11303,"num_write_kb":482113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":203,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948576+0000","last_change":"2024-09-16T10:17:06.903885+0000","last_active":"2024-09-16T10:17:16.948576+0000","last_peered":"2024-09-16T10:17:16.948576+0000","last_clean":"2024-09-16T10:17:16.948576+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:17:16.948576+0000","last_undegraded":"2024-09-16T10:17:16.948576+0000","last_fullsized":"2024-09-16T10:17:16.948576+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:06.903808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.903808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:21.156884+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948936+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:17:16.948936+0000","last_peered":"2024-09-16T10:17:16.948936+0000","last_clean":"2024-09-16T10:17:16.948936+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:17:16.948936+0000","last_undegraded":"2024-09-16T10:17:16.948936+0000","last_fullsized":"2024-09-16T10:17:16.948936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.838918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.962640+0000","last_change":"2024-09-16T10:15:30.302921+0000","last_active":"2024-09-16T10:17:16.962640+0000","last_peered":"2024-09-16T10:17:16.962640+0000","last_clean":"2024-09-16T10:17:16.962640+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:17:16.962640+0000","last_undegraded":"2024-09-16T10:17:16.962640+0000","last_fullsized":"2024-09-16T10:17:16.962640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.302859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.302859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.382437+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965221+0000","last_change":"2024-09-16T10:15:25.689754+0000","last_active":"2024-09-16T10:17:16.965221+0000","last_peered":"2024-09-16T10:17:16.965221+0000","last_clean":"2024-09-16T10:17:16.965221+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:17:16.965221+0000","last_undegraded":"2024-09-16T10:17:16.965221+0000","last_fullsized":"2024-09-16T10:17:16.965221+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:25.689615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.689615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.184130+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"139'10941","reported_seq":11138,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.156291+0000","last_change":"2024-09-16T10:15:56.898348+0000","last_active":"2024-09-16T10:17:18.156291+0000","last_peered":"2024-09-16T10:17:18.156291+0000","last_clean":"2024-09-16T10:17:18.156291+0000","last_became_active":"2024-09-16T10:14:43.953871+0000","last_became_peered":"2024-09-16T10:14:43.953871+0000","last_unstale":"2024-09-16T10:17:18.156291+0000","last_undegraded":"2024-09-16T10:17:18.156291+0000","last_fullsized":"2024-09-16T10:17:18.156291+0000","mapping_epoch":81,"log_start":"106'4100","ondisk_log_start":"106'4100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'5944","last_scrub_stamp":"2024-09-16T10:15:56.897937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.897937+0000","objects_scrubbed":5880,"log_size":6841,"log_dups_size":0,"ondisk_log_size":6841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:25.773998+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230883328,"num_objects":3523,"num_object_clones":0,"num_object_copies":10569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948857+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:17:16.948857+0000","last_peered":"2024-09-16T10:17:16.948857+0000","last_clean":"2024-09-16T10:17:16.948857+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:17:16.948857+0000","last_undegraded":"2024-09-16T10:17:16.948857+0000","last_fullsized":"2024-09-16T10:17:16.948857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:20.795125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965765+0000","last_change":"2024-09-16T10:16:05.172896+0000","last_active":"2024-09-16T10:17:16.965765+0000","last_peered":"2024-09-16T10:17:16.965765+0000","last_clean":"2024-09-16T10:17:16.965765+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:17:16.965765+0000","last_undegraded":"2024-09-16T10:17:16.965765+0000","last_fullsized":"2024-09-16T10:17:16.965765+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:05.172834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.172834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:17.891481+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.957081+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:17:16.957081+0000","last_peered":"2024-09-16T10:17:16.957081+0000","last_clean":"2024-09-16T10:17:16.957081+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:17:16.957081+0000","last_undegraded":"2024-09-16T10:17:16.957081+0000","last_fullsized":"2024-09-16T10:17:16.957081+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:19.875908+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959847+0000","last_change":"2024-09-16T10:15:11.699384+0000","last_active":"2024-09-16T10:17:16.959847+0000","last_peered":"2024-09-16T10:17:16.959847+0000","last_clean":"2024-09-16T10:17:16.959847+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:17:16.959847+0000","last_undegraded":"2024-09-16T10:17:16.959847+0000","last_fullsized":"2024-09-16T10:17:16.959847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.699308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.699308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.368765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"139'10857","reported_seq":11063,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.242611+0000","last_change":"2024-09-16T10:16:16.432815+0000","last_active":"2024-09-16T10:17:18.242611+0000","last_peered":"2024-09-16T10:17:18.242611+0000","last_clean":"2024-09-16T10:17:18.242611+0000","last_became_active":"2024-09-16T10:14:43.954996+0000","last_became_peered":"2024-09-16T10:14:43.954996+0000","last_unstale":"2024-09-16T10:17:18.242611+0000","last_undegraded":"2024-09-16T10:17:18.242611+0000","last_fullsized":"2024-09-16T10:17:18.242611+0000","mapping_epoch":81,"log_start":"112'5300","ondisk_log_start":"112'5300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7052","last_scrub_stamp":"2024-09-16T10:16:16.432702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.432702+0000","objects_scrubbed":6983,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:32.391032+0000","scrub_duration":2995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243204096,"num_objects":3711,"num_object_clones":0,"num_object_copies":11133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":183,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948455+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:17:16.948455+0000","last_peered":"2024-09-16T10:17:16.948455+0000","last_clean":"2024-09-16T10:17:16.948455+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:17:16.948455+0000","last_undegraded":"2024-09-16T10:17:16.948455+0000","last_fullsized":"2024-09-16T10:17:16.948455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.917544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.964642+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:17:16.964642+0000","last_peered":"2024-09-16T10:17:16.964642+0000","last_clean":"2024-09-16T10:17:16.964642+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:17:16.964642+0000","last_undegraded":"2024-09-16T10:17:16.964642+0000","last_fullsized":"2024-09-16T10:17:16.964642+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.173364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":218,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974037+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:17:16.974037+0000","last_peered":"2024-09-16T10:17:16.974037+0000","last_clean":"2024-09-16T10:17:16.974037+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:17:16.974037+0000","last_undegraded":"2024-09-16T10:17:16.974037+0000","last_fullsized":"2024-09-16T10:17:16.974037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.316990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948233+0000","last_change":"2024-09-16T10:16:07.824858+0000","last_active":"2024-09-16T10:17:16.948233+0000","last_peered":"2024-09-16T10:17:16.948233+0000","last_clean":"2024-09-16T10:17:16.948233+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:17:16.948233+0000","last_undegraded":"2024-09-16T10:17:16.948233+0000","last_fullsized":"2024-09-16T10:17:16.948233+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:07.824800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:07.824800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"139'11168","reported_seq":11296,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.242486+0000","last_change":"2024-09-16T10:14:43.952726+0000","last_active":"2024-09-16T10:17:18.242486+0000","last_peered":"2024-09-16T10:17:18.242486+0000","last_clean":"2024-09-16T10:17:18.242486+0000","last_became_active":"2024-09-16T10:14:43.952378+0000","last_became_peered":"2024-09-16T10:14:43.952378+0000","last_unstale":"2024-09-16T10:17:18.242486+0000","last_undegraded":"2024-09-16T10:17:18.242486+0000","last_fullsized":"2024-09-16T10:17:18.242486+0000","mapping_epoch":81,"log_start":"106'4300","ondisk_log_start":"106'4300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:30.419152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240910336,"num_objects":3676,"num_object_clones":0,"num_object_copies":11028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":200,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.969511+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:17:16.969511+0000","last_peered":"2024-09-16T10:17:16.969511+0000","last_clean":"2024-09-16T10:17:16.969511+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:17:16.969511+0000","last_undegraded":"2024-09-16T10:17:16.969511+0000","last_fullsized":"2024-09-16T10:17:16.969511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.984845+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959675+0000","last_change":"2024-09-16T10:15:13.340979+0000","last_active":"2024-09-16T10:17:16.959675+0000","last_peered":"2024-09-16T10:17:16.959675+0000","last_clean":"2024-09-16T10:17:16.959675+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:17:16.959675+0000","last_undegraded":"2024-09-16T10:17:16.959675+0000","last_fullsized":"2024-09-16T10:17:16.959675+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.340929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.340929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:21.290765+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":165,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.957087+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:17:16.957087+0000","last_peered":"2024-09-16T10:17:16.957087+0000","last_clean":"2024-09-16T10:17:16.957087+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:17:16.957087+0000","last_undegraded":"2024-09-16T10:17:16.957087+0000","last_fullsized":"2024-09-16T10:17:16.957087+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.418972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.968524+0000","last_change":"2024-09-16T10:14:45.941203+0000","last_active":"2024-09-16T10:17:16.968524+0000","last_peered":"2024-09-16T10:17:16.968524+0000","last_clean":"2024-09-16T10:17:16.968524+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:17:16.968524+0000","last_undegraded":"2024-09-16T10:17:16.968524+0000","last_fullsized":"2024-09-16T10:17:16.968524+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:14:45.941146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.941146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.036920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"139'11024","reported_seq":11228,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.239855+0000","last_change":"2024-09-16T10:16:09.904652+0000","last_active":"2024-09-16T10:17:18.239855+0000","last_peered":"2024-09-16T10:17:18.239855+0000","last_clean":"2024-09-16T10:17:18.239855+0000","last_became_active":"2024-09-16T10:14:43.959635+0000","last_became_peered":"2024-09-16T10:14:43.959635+0000","last_unstale":"2024-09-16T10:17:18.239855+0000","last_undegraded":"2024-09-16T10:17:18.239855+0000","last_fullsized":"2024-09-16T10:17:18.239855+0000","mapping_epoch":81,"log_start":"112'5400","ondisk_log_start":"112'5400","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6706","last_scrub_stamp":"2024-09-16T10:16:09.904421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:09.904421+0000","objects_scrubbed":6634,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.136594+0000","scrub_duration":2479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237764608,"num_objects":3628,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":215,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965591+0000","last_change":"2024-09-16T10:16:42.258515+0000","last_active":"2024-09-16T10:17:16.965591+0000","last_peered":"2024-09-16T10:17:16.965591+0000","last_clean":"2024-09-16T10:17:16.965591+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:17:16.965591+0000","last_undegraded":"2024-09-16T10:17:16.965591+0000","last_fullsized":"2024-09-16T10:17:16.965591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:16:42.258433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.258433+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.356020+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975335+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:17:16.975335+0000","last_peered":"2024-09-16T10:17:16.975335+0000","last_clean":"2024-09-16T10:17:16.975335+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:17:16.975335+0000","last_undegraded":"2024-09-16T10:17:16.975335+0000","last_fullsized":"2024-09-16T10:17:16.975335+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.641444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965648+0000","last_change":"2024-09-16T10:16:43.420443+0000","last_active":"2024-09-16T10:17:16.965648+0000","last_peered":"2024-09-16T10:17:16.965648+0000","last_clean":"2024-09-16T10:17:16.965648+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:17:16.965648+0000","last_undegraded":"2024-09-16T10:17:16.965648+0000","last_fullsized":"2024-09-16T10:17:16.965648+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.420249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.420249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.309426+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974636+0000","last_change":"2024-09-16T10:17:08.596403+0000","last_active":"2024-09-16T10:17:16.974636+0000","last_peered":"2024-09-16T10:17:16.974636+0000","last_clean":"2024-09-16T10:17:16.974636+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:17:16.974636+0000","last_undegraded":"2024-09-16T10:17:16.974636+0000","last_fullsized":"2024-09-16T10:17:16.974636+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.176712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"139'11073","reported_seq":11201,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.275934+0000","last_change":"2024-09-16T10:14:43.957583+0000","last_active":"2024-09-16T10:17:18.275934+0000","last_peered":"2024-09-16T10:17:18.275934+0000","last_clean":"2024-09-16T10:17:18.275934+0000","last_became_active":"2024-09-16T10:14:43.957207+0000","last_became_peered":"2024-09-16T10:14:43.957207+0000","last_unstale":"2024-09-16T10:17:18.275934+0000","last_undegraded":"2024-09-16T10:17:18.275934+0000","last_fullsized":"2024-09-16T10:17:18.275934+0000","mapping_epoch":81,"log_start":"114'6517","ondisk_log_start":"114'6517","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:25.110106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":10659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":199,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948098+0000","last_change":"2024-09-16T10:15:15.038399+0000","last_active":"2024-09-16T10:17:16.948098+0000","last_peered":"2024-09-16T10:17:16.948098+0000","last_clean":"2024-09-16T10:17:16.948098+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:17:16.948098+0000","last_undegraded":"2024-09-16T10:17:16.948098+0000","last_fullsized":"2024-09-16T10:17:16.948098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:15.038322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.038322+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:30.819499+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975463+0000","last_change":"2024-09-16T10:15:11.631330+0000","last_active":"2024-09-16T10:17:16.975463+0000","last_peered":"2024-09-16T10:17:16.975463+0000","last_clean":"2024-09-16T10:17:16.975463+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:17:16.975463+0000","last_undegraded":"2024-09-16T10:17:16.975463+0000","last_fullsized":"2024-09-16T10:17:16.975463+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.631262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.631262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.656489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":165,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.958317+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:17:16.958317+0000","last_peered":"2024-09-16T10:17:16.958317+0000","last_clean":"2024-09-16T10:17:16.958317+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:17:16.958317+0000","last_undegraded":"2024-09-16T10:17:16.958317+0000","last_fullsized":"2024-09-16T10:17:16.958317+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:20.258623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.964374+0000","last_change":"2024-09-16T10:16:32.171799+0000","last_active":"2024-09-16T10:17:16.964374+0000","last_peered":"2024-09-16T10:17:16.964374+0000","last_clean":"2024-09-16T10:17:16.964374+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:17:16.964374+0000","last_undegraded":"2024-09-16T10:17:16.964374+0000","last_fullsized":"2024-09-16T10:17:16.964374+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.171738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.171738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:39.617239+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"139'11390","reported_seq":11591,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:18.119070+0000","last_change":"2024-09-16T10:16:02.172955+0000","last_active":"2024-09-16T10:17:18.119070+0000","last_peered":"2024-09-16T10:17:18.119070+0000","last_clean":"2024-09-16T10:17:18.119070+0000","last_became_active":"2024-09-16T10:14:43.954475+0000","last_became_peered":"2024-09-16T10:14:43.954475+0000","last_unstale":"2024-09-16T10:17:18.119070+0000","last_undegraded":"2024-09-16T10:17:18.119070+0000","last_fullsized":"2024-09-16T10:17:18.119070+0000","mapping_epoch":81,"log_start":"114'6300","ondisk_log_start":"114'6300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6489","last_scrub_stamp":"2024-09-16T10:16:02.172829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.172829+0000","objects_scrubbed":6427,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.789342+0000","scrub_duration":2056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243531776,"num_objects":3716,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11390,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.958938+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:17:16.958938+0000","last_peered":"2024-09-16T10:17:16.958938+0000","last_clean":"2024-09-16T10:17:16.958938+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:17:16.958938+0000","last_undegraded":"2024-09-16T10:17:16.958938+0000","last_fullsized":"2024-09-16T10:17:16.958938+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:17.430135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":193,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975386+0000","last_change":"2024-09-16T10:15:25.529229+0000","last_active":"2024-09-16T10:17:16.975386+0000","last_peered":"2024-09-16T10:17:16.975386+0000","last_clean":"2024-09-16T10:17:16.975386+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:17:16.975386+0000","last_undegraded":"2024-09-16T10:17:16.975386+0000","last_fullsized":"2024-09-16T10:17:16.975386+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:15:25.529178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.529178+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.592697+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":234,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974735+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:17:16.974735+0000","last_peered":"2024-09-16T10:17:16.974735+0000","last_clean":"2024-09-16T10:17:16.974735+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:17:16.974735+0000","last_undegraded":"2024-09-16T10:17:16.974735+0000","last_fullsized":"2024-09-16T10:17:16.974735+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.912935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959275+0000","last_change":"2024-09-16T10:15:17.306478+0000","last_active":"2024-09-16T10:17:16.959275+0000","last_peered":"2024-09-16T10:17:16.959275+0000","last_clean":"2024-09-16T10:17:16.959275+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:17:16.959275+0000","last_undegraded":"2024-09-16T10:17:16.959275+0000","last_fullsized":"2024-09-16T10:17:16.959275+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.306392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.306392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:45.397274+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965999+0000","last_change":"2024-09-16T10:15:55.157938+0000","last_active":"2024-09-16T10:17:16.965999+0000","last_peered":"2024-09-16T10:17:16.965999+0000","last_clean":"2024-09-16T10:17:16.965999+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:17:16.965999+0000","last_undegraded":"2024-09-16T10:17:16.965999+0000","last_fullsized":"2024-09-16T10:17:16.965999+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:55.157860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:55.157860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.136638+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.991212+0000","last_change":"2024-09-16T10:16:06.952654+0000","last_active":"2024-09-16T10:17:16.991212+0000","last_peered":"2024-09-16T10:17:16.991212+0000","last_clean":"2024-09-16T10:17:16.991212+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:17:16.991212+0000","last_undegraded":"2024-09-16T10:17:16.991212+0000","last_fullsized":"2024-09-16T10:17:16.991212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:16:06.952390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:06.952390+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.060640+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":165,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.962453+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:17:16.962453+0000","last_peered":"2024-09-16T10:17:16.962453+0000","last_clean":"2024-09-16T10:17:16.962453+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:17:16.962453+0000","last_undegraded":"2024-09-16T10:17:16.962453+0000","last_fullsized":"2024-09-16T10:17:16.962453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.398561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975070+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:17:16.975070+0000","last_peered":"2024-09-16T10:17:16.975070+0000","last_clean":"2024-09-16T10:17:16.975070+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:17:16.975070+0000","last_undegraded":"2024-09-16T10:17:16.975070+0000","last_fullsized":"2024-09-16T10:17:16.975070+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:06.985366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":200,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.971331+0000","last_change":"2024-09-16T10:15:54.369062+0000","last_active":"2024-09-16T10:17:16.971331+0000","last_peered":"2024-09-16T10:17:16.971331+0000","last_clean":"2024-09-16T10:17:16.971331+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:17:16.971331+0000","last_undegraded":"2024-09-16T10:17:16.971331+0000","last_fullsized":"2024-09-16T10:17:16.971331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:54.368987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.368987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974826+0000","last_change":"2024-09-16T10:15:28.537739+0000","last_active":"2024-09-16T10:17:16.974826+0000","last_peered":"2024-09-16T10:17:16.974826+0000","last_clean":"2024-09-16T10:17:16.974826+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:17:16.974826+0000","last_undegraded":"2024-09-16T10:17:16.974826+0000","last_fullsized":"2024-09-16T10:17:16.974826+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:28.537673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.537673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.624149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.991086+0000","last_change":"2024-09-16T10:16:37.237971+0000","last_active":"2024-09-16T10:17:16.991086+0000","last_peered":"2024-09-16T10:17:16.991086+0000","last_clean":"2024-09-16T10:17:16.991086+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:17:16.991086+0000","last_undegraded":"2024-09-16T10:17:16.991086+0000","last_fullsized":"2024-09-16T10:17:16.991086+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:37.237849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.237849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:39.020098+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959206+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:17:16.959206+0000","last_peered":"2024-09-16T10:17:16.959206+0000","last_clean":"2024-09-16T10:17:16.959206+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:17:16.959206+0000","last_undegraded":"2024-09-16T10:17:16.959206+0000","last_fullsized":"2024-09-16T10:17:16.959206+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.431103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":201,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960001+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:17:16.960001+0000","last_peered":"2024-09-16T10:17:16.960001+0000","last_clean":"2024-09-16T10:17:16.960001+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:17:16.960001+0000","last_undegraded":"2024-09-16T10:17:16.960001+0000","last_fullsized":"2024-09-16T10:17:16.960001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.347562+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.969138+0000","last_change":"2024-09-16T10:14:37.844181+0000","last_active":"2024-09-16T10:17:16.969138+0000","last_peered":"2024-09-16T10:17:16.969138+0000","last_clean":"2024-09-16T10:17:16.969138+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:17:16.969138+0000","last_undegraded":"2024-09-16T10:17:16.969138+0000","last_fullsized":"2024-09-16T10:17:16.969138+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.844113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.844113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.105572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":234,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965355+0000","last_change":"2024-09-16T10:17:06.052878+0000","last_active":"2024-09-16T10:17:16.965355+0000","last_peered":"2024-09-16T10:17:16.965355+0000","last_clean":"2024-09-16T10:17:16.965355+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:17:16.965355+0000","last_undegraded":"2024-09-16T10:17:16.965355+0000","last_fullsized":"2024-09-16T10:17:16.965355+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:06.050469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.050469+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.355575+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948346+0000","last_change":"2024-09-16T10:16:05.834955+0000","last_active":"2024-09-16T10:17:16.948346+0000","last_peered":"2024-09-16T10:17:16.948346+0000","last_clean":"2024-09-16T10:17:16.948346+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:17:16.948346+0000","last_undegraded":"2024-09-16T10:17:16.948346+0000","last_fullsized":"2024-09-16T10:17:16.948346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:05.834879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.834879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.058409+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":192,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.966017+0000","last_change":"2024-09-16T10:16:04.888881+0000","last_active":"2024-09-16T10:17:16.966017+0000","last_peered":"2024-09-16T10:17:16.966017+0000","last_clean":"2024-09-16T10:17:16.966017+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:17:16.966017+0000","last_undegraded":"2024-09-16T10:17:16.966017+0000","last_fullsized":"2024-09-16T10:17:16.966017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:16:04.888793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.888793+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:24.436960+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975002+0000","last_change":"2024-09-16T10:15:21.601115+0000","last_active":"2024-09-16T10:17:16.975002+0000","last_peered":"2024-09-16T10:17:16.975002+0000","last_clean":"2024-09-16T10:17:16.975002+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:17:16.975002+0000","last_undegraded":"2024-09-16T10:17:16.975002+0000","last_fullsized":"2024-09-16T10:17:16.975002+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.601066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.601066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.626050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":226,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959846+0000","last_change":"2024-09-16T10:16:00.378802+0000","last_active":"2024-09-16T10:17:16.959846+0000","last_peered":"2024-09-16T10:17:16.959846+0000","last_clean":"2024-09-16T10:17:16.959846+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:17:16.959846+0000","last_undegraded":"2024-09-16T10:17:16.959846+0000","last_fullsized":"2024-09-16T10:17:16.959846+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:16:00.378709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:00.378709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.970745+0000","last_change":"2024-09-16T10:16:43.291855+0000","last_active":"2024-09-16T10:17:16.970745+0000","last_peered":"2024-09-16T10:17:16.970745+0000","last_clean":"2024-09-16T10:17:16.970745+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:17:16.970745+0000","last_undegraded":"2024-09-16T10:17:16.970745+0000","last_fullsized":"2024-09-16T10:17:16.970745+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.291720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.291720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:59.249028+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":187,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.948070+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:17:16.948070+0000","last_peered":"2024-09-16T10:17:16.948070+0000","last_clean":"2024-09-16T10:17:16.948070+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:17:16.948070+0000","last_undegraded":"2024-09-16T10:17:16.948070+0000","last_fullsized":"2024-09-16T10:17:16.948070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.821021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.949263+0000","last_change":"2024-09-16T10:16:01.807900+0000","last_active":"2024-09-16T10:17:16.949263+0000","last_peered":"2024-09-16T10:17:16.949263+0000","last_clean":"2024-09-16T10:17:16.949263+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:17:16.949263+0000","last_undegraded":"2024-09-16T10:17:16.949263+0000","last_fullsized":"2024-09-16T10:17:16.949263+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:01.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:01.807849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.983691+0000","last_change":"2024-09-16T10:17:14.106033+0000","last_active":"2024-09-16T10:17:16.983691+0000","last_peered":"2024-09-16T10:17:16.983691+0000","last_clean":"2024-09-16T10:17:16.983691+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:17:16.983691+0000","last_undegraded":"2024-09-16T10:17:16.983691+0000","last_fullsized":"2024-09-16T10:17:16.983691+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.105971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.105971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.805018+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.959921+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:17:16.959921+0000","last_peered":"2024-09-16T10:17:16.959921+0000","last_clean":"2024-09-16T10:17:16.959921+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:17:16.959921+0000","last_undegraded":"2024-09-16T10:17:16.959921+0000","last_fullsized":"2024-09-16T10:17:16.959921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.252222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":193,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.983998+0000","last_change":"2024-09-16T10:15:15.010801+0000","last_active":"2024-09-16T10:17:16.983998+0000","last_peered":"2024-09-16T10:17:16.983998+0000","last_clean":"2024-09-16T10:17:16.983998+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:17:16.983998+0000","last_undegraded":"2024-09-16T10:17:16.983998+0000","last_fullsized":"2024-09-16T10:17:16.983998+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:15.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.010709+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:26.108799+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.975295+0000","last_change":"2024-09-16T10:16:11.528789+0000","last_active":"2024-09-16T10:17:16.975295+0000","last_peered":"2024-09-16T10:17:16.975295+0000","last_clean":"2024-09-16T10:17:16.975295+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:17:16.975295+0000","last_undegraded":"2024-09-16T10:17:16.975295+0000","last_fullsized":"2024-09-16T10:17:16.975295+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:11.528731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:11.528731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:39.041613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.964374+0000","last_change":"2024-09-16T10:15:59.987731+0000","last_active":"2024-09-16T10:17:16.964374+0000","last_peered":"2024-09-16T10:17:16.964374+0000","last_clean":"2024-09-16T10:17:16.964374+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:17:16.964374+0000","last_undegraded":"2024-09-16T10:17:16.964374+0000","last_fullsized":"2024-09-16T10:17:16.964374+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:59.987642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:59.987642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:20.901671+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":160,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.968383+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:17:16.968383+0000","last_peered":"2024-09-16T10:17:16.968383+0000","last_clean":"2024-09-16T10:17:16.968383+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:17:16.968383+0000","last_undegraded":"2024-09-16T10:17:16.968383+0000","last_fullsized":"2024-09-16T10:17:16.968383+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:44.291340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960064+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:17:16.960064+0000","last_peered":"2024-09-16T10:17:16.960064+0000","last_clean":"2024-09-16T10:17:16.960064+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:17:16.960064+0000","last_undegraded":"2024-09-16T10:17:16.960064+0000","last_fullsized":"2024-09-16T10:17:16.960064+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.138673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.962543+0000","last_change":"2024-09-16T10:15:58.378648+0000","last_active":"2024-09-16T10:17:16.962543+0000","last_peered":"2024-09-16T10:17:16.962543+0000","last_clean":"2024-09-16T10:17:16.962543+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:17:16.962543+0000","last_undegraded":"2024-09-16T10:17:16.962543+0000","last_fullsized":"2024-09-16T10:17:16.962543+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:58.378585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:58.378585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.749256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.974425+0000","last_change":"2024-09-16T10:14:45.829620+0000","last_active":"2024-09-16T10:17:16.974425+0000","last_peered":"2024-09-16T10:17:16.974425+0000","last_clean":"2024-09-16T10:17:16.974425+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:17:16.974425+0000","last_undegraded":"2024-09-16T10:17:16.974425+0000","last_fullsized":"2024-09-16T10:17:16.974425+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:45.829572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.829572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.663935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.964544+0000","last_change":"2024-09-16T10:15:30.762339+0000","last_active":"2024-09-16T10:17:16.964544+0000","last_peered":"2024-09-16T10:17:16.964544+0000","last_clean":"2024-09-16T10:17:16.964544+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:17:16.964544+0000","last_undegraded":"2024-09-16T10:17:16.964544+0000","last_fullsized":"2024-09-16T10:17:16.964544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.762137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.762137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.149742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965483+0000","last_change":"2024-09-16T10:15:08.953214+0000","last_active":"2024-09-16T10:17:16.965483+0000","last_peered":"2024-09-16T10:17:16.965483+0000","last_clean":"2024-09-16T10:17:16.965483+0000","last_became_active":"2024-09-16T10:15:08.930505+0000","last_became_peered":"2024-09-16T10:15:08.930505+0000","last_unstale":"2024-09-16T10:17:16.965483+0000","last_undegraded":"2024-09-16T10:17:16.965483+0000","last_fullsized":"2024-09-16T10:17:16.965483+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:20.032046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.972013+0000","last_change":"2024-09-16T10:16:39.271363+0000","last_active":"2024-09-16T10:17:16.972013+0000","last_peered":"2024-09-16T10:17:16.972013+0000","last_clean":"2024-09-16T10:17:16.972013+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:17:16.972013+0000","last_undegraded":"2024-09-16T10:17:16.972013+0000","last_fullsized":"2024-09-16T10:17:16.972013+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:16:39.271283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.271283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.732007+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.966375+0000","last_change":"2024-09-16T10:16:34.225741+0000","last_active":"2024-09-16T10:17:16.966375+0000","last_peered":"2024-09-16T10:17:16.966375+0000","last_clean":"2024-09-16T10:17:16.966375+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:17:16.966375+0000","last_undegraded":"2024-09-16T10:17:16.966375+0000","last_fullsized":"2024-09-16T10:17:16.966375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:34.225689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:34.225689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.982411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":54,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.966257+0000","last_change":"2024-09-16T10:16:14.001413+0000","last_active":"2024-09-16T10:17:16.966257+0000","last_peered":"2024-09-16T10:17:16.966257+0000","last_clean":"2024-09-16T10:17:16.966257+0000","last_became_active":"2024-09-16T10:16:14.000930+0000","last_became_peered":"2024-09-16T10:16:14.000930+0000","last_unstale":"2024-09-16T10:17:16.966257+0000","last_undegraded":"2024-09-16T10:17:16.966257+0000","last_fullsized":"2024-09-16T10:17:16.966257+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:01.542093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:01.542093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:23.323926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.960556+0000","last_change":"2024-09-16T10:16:32.919293+0000","last_active":"2024-09-16T10:17:16.960556+0000","last_peered":"2024-09-16T10:17:16.960556+0000","last_clean":"2024-09-16T10:17:16.960556+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:17:16.960556+0000","last_undegraded":"2024-09-16T10:17:16.960556+0000","last_fullsized":"2024-09-16T10:17:16.960556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.919094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.919094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:50.652293+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.965569+0000","last_change":"2024-09-16T10:15:24.663908+0000","last_active":"2024-09-16T10:17:16.965569+0000","last_peered":"2024-09-16T10:17:16.965569+0000","last_clean":"2024-09-16T10:17:16.965569+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:17:16.965569+0000","last_undegraded":"2024-09-16T10:17:16.965569+0000","last_fullsized":"2024-09-16T10:17:16.965569+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.663832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.663832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:55.163009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":203,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.966584+0000","last_change":"2024-09-16T10:16:11.160548+0000","last_active":"2024-09-16T10:17:16.966584+0000","last_peered":"2024-09-16T10:17:16.966584+0000","last_clean":"2024-09-16T10:17:16.966584+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:17:16.966584+0000","last_undegraded":"2024-09-16T10:17:16.966584+0000","last_fullsized":"2024-09-16T10:17:16.966584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:16:11.160489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:11.160489+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:28.467994+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T10:17:16.971074+0000","last_change":"2024-09-16T10:16:36.222414+0000","last_active":"2024-09-16T10:17:16.971074+0000","last_peered":"2024-09-16T10:17:16.971074+0000","last_clean":"2024-09-16T10:17:16.971074+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:17:16.971074+0000","last_undegraded":"2024-09-16T10:17:16.971074+0000","last_fullsized":"2024-09-16T10:17:16.971074+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:36.222281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:36.222281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:03.830974+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3893559320,"num_objects":59412,"num_object_clones":0,"num_object_copies":178236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59412,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179142,"num_write_kb":7633665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11682656256,"data_stored":11682644040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89786,"ondisk_log_size":89786,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":215,"num_read_kb":210,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234032,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1311476,"kb_used_data":1221984,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92433676,"statfs":{"total":95995035648,"available":94652084224,"internally_reserved":0,"allocated":1251311616,"data_stored":1250937828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528372,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1303076,"kb_used_data":1209356,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92442076,"statfs":{"total":95995035648,"available":94660685824,"internally_reserved":0,"allocated":1238380544,"data_stored":1238026162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790010,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017388,"kb_used_data":941716,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92727764,"statfs":{"total":95995035648,"available":94953230336,"internally_reserved":0,"allocated":964317184,"data_stored":963955614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986239,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1276796,"kb_used_data":1187300,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92468356,"statfs":{"total":95995035648,"available":94687596544,"internally_reserved":0,"allocated":1215795200,"data_stored":1215416715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215172,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2301128,"kb_used_data":2161584,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91444024,"statfs":{"total":95995035648,"available":93638680576,"internally_reserved":0,"allocated":2213462016,"data_stored":2213072522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":142801352},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1527132,"kb_used_data":1423752,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92218020,"statfs":{"total":95995035648,"available":94431252480,"internally_reserved":0,"allocated":1457922048,"data_stored":1457571465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1534932,"kb_used_data":1427392,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92210220,"statfs":{"total":95995035648,"available":94423265280,"internally_reserved":0,"allocated":1461649408,"data_stored":1461243125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2027648,"kb_used_data":1902060,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91717504,"statfs":{"total":95995035648,"available":93918724096,"internally_reserved":0,"allocated":1947709440,"data_stored":1947324090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1940197376,"data_stored":1940193304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1460015104,"data_stored":1460011032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1435238400,"data_stored":1435238400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2203123712,"data_stored":2203123712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1211432960,"data_stored":1211432960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":956432384,"data_stored":956432384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1236209664,"data_stored":1236205592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1240006656,"data_stored":1240006656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:17:26.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:17:26.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:26.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:26.167 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:17:26.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:26.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:26.380 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:26 smithi063 ceph-mon[29993]: pgmap v169: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-16T10:17:26.380 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3636223511' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:26.380 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:26 smithi063 ceph-mon[29993]: from='client.25051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:26.381 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:26 smithi063 ceph-mon[39470]: pgmap v169: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-16T10:17:26.381 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3636223511' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:26.381 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:26 smithi063 ceph-mon[39470]: from='client.25051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:26.394 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:26.458 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:26 smithi165 ceph-mon[32696]: pgmap v169: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-16T10:17:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3636223511' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:26 smithi165 ceph-mon[32696]: from='client.25051 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:26.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:26.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:26.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:26.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:26.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:26.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:26.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:26.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:26.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:27.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:27.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:27.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:27.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:27.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:27.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:27.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:27.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:27.356 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:17:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:17:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:17:27.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:27 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:27.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:27 smithi063 ceph-mon[29993]: from='client.25051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:27.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:27 smithi063 ceph-mon[29993]: osdmap e141: 8 total, 8 up, 8 in 2024-09-16T10:17:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:27.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:27 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:27.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:27 smithi063 ceph-mon[39470]: from='client.25051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:27.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:27 smithi063 ceph-mon[39470]: osdmap e141: 8 total, 8 up, 8 in 2024-09-16T10:17:27.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:27.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:27.506 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:27.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:27 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:27 smithi165 ceph-mon[32696]: from='client.25051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:27 smithi165 ceph-mon[32696]: osdmap e141: 8 total, 8 up, 8 in 2024-09-16T10:17:27.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:27.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:27.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:27.656 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:27.657 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:27.700 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:27.701 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:27.733 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:27.735 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:27.735 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:27.773 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:27.773 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:27.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:27.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:27.803 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:27.803 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:27.839 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:27.839 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:27.870 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:27.912 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:27.912 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:27.912 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:27.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:27.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:27.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:28.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:28.107 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:28.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:28.209 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:28.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:28.304 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:28.310 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:28.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:28.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:28.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[29993]: pgmap v171: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 592 op/s 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3719324328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[39470]: pgmap v171: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 592 op/s 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:17:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3719324328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:28.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:28 smithi165 ceph-mon[32696]: pgmap v171: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 592 op/s 2024-09-16T10:17:28.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:28 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:17:28.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:28 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:17:28.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3719324328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:28.453 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:17:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:17:28.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:28.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:28.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:28.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:28.653 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:28.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:28.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:28.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:28.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:28.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:28.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:28.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:29.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:29.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:29.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:29.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:29.104 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:17:29.167 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:29.234 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:29.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:29.335 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:29.377 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:29.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:29.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:29 smithi063 ceph-mon[29993]: osdmap e142: 8 total, 8 up, 8 in 2024-09-16T10:17:29.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:29.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:29 smithi063 ceph-mon[39470]: osdmap e142: 8 total, 8 up, 8 in 2024-09-16T10:17:29.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:29.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:29.533 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:29.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:29.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:29 smithi165 ceph-mon[32696]: osdmap e142: 8 total, 8 up, 8 in 2024-09-16T10:17:29.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:29.635 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:29.686 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:29.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:29.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:29.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:29.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:29.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:29.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:29.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:30.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:30.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:30.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:30.201 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:30.269 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:30.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:30.297 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:17:30.297 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:17:30.297 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:17:30.297 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:17:30.297 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:17:30.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:30.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:30.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:30.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:30 smithi063 ceph-mon[29993]: pgmap v173: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:17:30.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:30 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:30.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:30 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:30.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:30 smithi063 ceph-mon[39470]: pgmap v173: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:17:30.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:30 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:30.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:30 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:30.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:30.539 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:30.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:30.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:30 smithi165 ceph-mon[32696]: pgmap v173: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-09-16T10:17:30.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:30 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:30.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:30 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:30.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:30.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:30.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:30.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:30.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:30.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:30.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:30.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:30.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:30.986 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:17:31.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:31.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:31.076 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:31.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:31.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:31.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:31.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:31.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:31.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:31.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:31.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:31.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:31.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:31.713 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:17:31.715 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:17:31.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:31.808 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:17:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:31.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:31.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:31.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:32.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:32.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:32.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:32.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:32.165 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:32.227 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:32.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:32.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:32.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:32.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[39470]: pgmap v174: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-16T10:17:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:17:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:17:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[29993]: pgmap v174: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-16T10:17:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:17:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:17:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:32.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:32.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:32.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:32 smithi165 ceph-mon[32696]: pgmap v174: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-16T10:17:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:17:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:17:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:17:32.588 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-16T10:17:31.091319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3250788915,"num_objects":49794,"num_object_clones":0,"num_object_copies":149382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49794,"num_whiteouts":0,"num_read":1074,"num_read_kb":931,"num_write":189508,"num_write_kb":7635338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90355,"ondisk_log_size":90355,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10449580,"kb_used_data":9587796,"kb_used_omap":12,"kb_used_meta":861555,"kb_avail":739511636,"statfs":{"total":767960285184,"available":757259915264,"internally_reserved":0,"allocated":9817903104,"data_stored":9814811224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":882232881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-605880320,"num_objects":-9245,"num_object_clones":0,"num_object_copies":-27735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9245,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9245,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005188"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":194,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173421+0000","last_change":"2024-09-16T10:16:35.216054+0000","last_active":"2024-09-16T10:17:28.173421+0000","last_peered":"2024-09-16T10:17:28.173421+0000","last_clean":"2024-09-16T10:17:28.173421+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:17:28.173421+0000","last_undegraded":"2024-09-16T10:17:28.173421+0000","last_fullsized":"2024-09-16T10:17:28.173421+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:35.215992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.215992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:42.184249+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":221,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.161775+0000","last_change":"2024-09-16T10:16:35.813476+0000","last_active":"2024-09-16T10:17:26.161775+0000","last_peered":"2024-09-16T10:17:26.161775+0000","last_clean":"2024-09-16T10:17:26.161775+0000","last_became_active":"2024-09-16T10:15:08.925269+0000","last_became_peered":"2024-09-16T10:15:08.925269+0000","last_unstale":"2024-09-16T10:17:26.161775+0000","last_undegraded":"2024-09-16T10:17:26.161775+0000","last_fullsized":"2024-09-16T10:17:26.161775+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:35.813372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.813372+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:03.819055+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":187,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.170821+0000","last_change":"2024-09-16T10:17:04.062121+0000","last_active":"2024-09-16T10:17:28.170821+0000","last_peered":"2024-09-16T10:17:28.170821+0000","last_clean":"2024-09-16T10:17:28.170821+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:17:28.170821+0000","last_undegraded":"2024-09-16T10:17:28.170821+0000","last_fullsized":"2024-09-16T10:17:28.170821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:04.062060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.062060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.222918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":172,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.161662+0000","last_change":"2024-09-16T10:16:16.760046+0000","last_active":"2024-09-16T10:17:26.161662+0000","last_peered":"2024-09-16T10:17:26.161662+0000","last_clean":"2024-09-16T10:17:26.161662+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:17:26.161662+0000","last_undegraded":"2024-09-16T10:17:26.161662+0000","last_fullsized":"2024-09-16T10:17:26.161662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.759981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.759981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:41.660333+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":175,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173244+0000","last_change":"2024-09-16T10:16:42.246257+0000","last_active":"2024-09-16T10:17:28.173244+0000","last_peered":"2024-09-16T10:17:28.173244+0000","last_clean":"2024-09-16T10:17:28.173244+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:17:28.173244+0000","last_undegraded":"2024-09-16T10:17:28.173244+0000","last_fullsized":"2024-09-16T10:17:28.173244+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:42.246193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.246193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.520461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":195,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.174060+0000","last_change":"2024-09-16T10:17:13.018487+0000","last_active":"2024-09-16T10:17:28.174060+0000","last_peered":"2024-09-16T10:17:28.174060+0000","last_clean":"2024-09-16T10:17:28.174060+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:17:28.174060+0000","last_undegraded":"2024-09-16T10:17:28.174060+0000","last_fullsized":"2024-09-16T10:17:28.174060+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:13.018412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.018412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.241294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":216,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.175351+0000","last_change":"2024-09-16T10:16:31.217281+0000","last_active":"2024-09-16T10:17:28.175351+0000","last_peered":"2024-09-16T10:17:28.175351+0000","last_clean":"2024-09-16T10:17:28.175351+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:17:28.175351+0000","last_undegraded":"2024-09-16T10:17:28.175351+0000","last_fullsized":"2024-09-16T10:17:28.175351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:16:31.217192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.217192+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:32.608583+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.161636+0000","last_change":"2024-09-16T10:17:14.871814+0000","last_active":"2024-09-16T10:17:26.161636+0000","last_peered":"2024-09-16T10:17:26.161636+0000","last_clean":"2024-09-16T10:17:26.161636+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:17:26.161636+0000","last_undegraded":"2024-09-16T10:17:26.161636+0000","last_fullsized":"2024-09-16T10:17:26.161636+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.871750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.871750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:31.182111+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172890+0000","last_change":"2024-09-16T10:15:29.351071+0000","last_active":"2024-09-16T10:17:26.172890+0000","last_peered":"2024-09-16T10:17:26.172890+0000","last_clean":"2024-09-16T10:17:26.172890+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:17:26.172890+0000","last_undegraded":"2024-09-16T10:17:26.172890+0000","last_fullsized":"2024-09-16T10:17:26.172890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:29.351000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.351000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:01.422071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":204,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.162070+0000","last_change":"2024-09-16T10:16:32.833966+0000","last_active":"2024-09-16T10:17:26.162070+0000","last_peered":"2024-09-16T10:17:26.162070+0000","last_clean":"2024-09-16T10:17:26.162070+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:17:26.162070+0000","last_undegraded":"2024-09-16T10:17:26.162070+0000","last_fullsized":"2024-09-16T10:17:26.162070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:32.833895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.833895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:43.494945+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":177,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172448+0000","last_change":"2024-09-16T10:15:22.271162+0000","last_active":"2024-09-16T10:17:26.172448+0000","last_peered":"2024-09-16T10:17:26.172448+0000","last_clean":"2024-09-16T10:17:26.172448+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:17:26.172448+0000","last_undegraded":"2024-09-16T10:17:26.172448+0000","last_fullsized":"2024-09-16T10:17:26.172448+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:22.271101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:22.271101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.398679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":182,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.170514+0000","last_change":"2024-09-16T10:17:12.089641+0000","last_active":"2024-09-16T10:17:28.170514+0000","last_peered":"2024-09-16T10:17:28.170514+0000","last_clean":"2024-09-16T10:17:28.170514+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:17:28.170514+0000","last_undegraded":"2024-09-16T10:17:28.170514+0000","last_fullsized":"2024-09-16T10:17:28.170514+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.089577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.089577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.096958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":177,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172466+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:17:26.172466+0000","last_peered":"2024-09-16T10:17:26.172466+0000","last_clean":"2024-09-16T10:17:26.172466+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:17:26.172466+0000","last_undegraded":"2024-09-16T10:17:26.172466+0000","last_fullsized":"2024-09-16T10:17:26.172466+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.593997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":198,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.163807+0000","last_change":"2024-09-16T10:17:16.937447+0000","last_active":"2024-09-16T10:17:26.163807+0000","last_peered":"2024-09-16T10:17:26.163807+0000","last_clean":"2024-09-16T10:17:26.163807+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:17:26.163807+0000","last_undegraded":"2024-09-16T10:17:26.163807+0000","last_fullsized":"2024-09-16T10:17:26.163807+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:17:16.937353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.937353+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.410589+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":177,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174396+0000","last_change":"2024-09-16T10:15:16.573685+0000","last_active":"2024-09-16T10:17:26.174396+0000","last_peered":"2024-09-16T10:17:26.174396+0000","last_clean":"2024-09-16T10:17:26.174396+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:17:26.174396+0000","last_undegraded":"2024-09-16T10:17:26.174396+0000","last_fullsized":"2024-09-16T10:17:26.174396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.573614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.573614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.640699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172239+0000","last_change":"2024-09-16T10:15:11.297702+0000","last_active":"2024-09-16T10:17:26.172239+0000","last_peered":"2024-09-16T10:17:26.172239+0000","last_clean":"2024-09-16T10:17:26.172239+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:17:26.172239+0000","last_undegraded":"2024-09-16T10:17:26.172239+0000","last_fullsized":"2024-09-16T10:17:26.172239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.297599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.297599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:55.375459+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":172,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164027+0000","last_change":"2024-09-16T10:16:15.776330+0000","last_active":"2024-09-16T10:17:26.164027+0000","last_peered":"2024-09-16T10:17:26.164027+0000","last_clean":"2024-09-16T10:17:26.164027+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:17:26.164027+0000","last_undegraded":"2024-09-16T10:17:26.164027+0000","last_fullsized":"2024-09-16T10:17:26.164027+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:15.776284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:15.776284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:35.232885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":178,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171838+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:17:28.171838+0000","last_peered":"2024-09-16T10:17:28.171838+0000","last_clean":"2024-09-16T10:17:28.171838+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:17:28.171838+0000","last_undegraded":"2024-09-16T10:17:28.171838+0000","last_fullsized":"2024-09-16T10:17:28.171838+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.208132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":200,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.182790+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:17:28.182790+0000","last_peered":"2024-09-16T10:17:28.182790+0000","last_clean":"2024-09-16T10:17:28.182790+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:17:28.182790+0000","last_undegraded":"2024-09-16T10:17:28.182790+0000","last_fullsized":"2024-09-16T10:17:28.182790+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":187,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.188665+0000","last_change":"2024-09-16T10:16:16.067128+0000","last_active":"2024-09-16T10:17:28.188665+0000","last_peered":"2024-09-16T10:17:28.188665+0000","last_clean":"2024-09-16T10:17:28.188665+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:17:28.188665+0000","last_undegraded":"2024-09-16T10:17:28.188665+0000","last_fullsized":"2024-09-16T10:17:28.188665+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.066855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.066855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":167,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.179909+0000","last_change":"2024-09-16T10:13:43.283705+0000","last_active":"2024-09-16T10:17:28.179909+0000","last_peered":"2024-09-16T10:17:28.179909+0000","last_clean":"2024-09-16T10:17:28.179909+0000","last_became_active":"2024-09-16T10:13:43.283188+0000","last_became_peered":"2024-09-16T10:13:43.283188+0000","last_unstale":"2024-09-16T10:17:28.179909+0000","last_undegraded":"2024-09-16T10:17:28.179909+0000","last_fullsized":"2024-09-16T10:17:28.179909+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.270038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":176,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.162213+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:17:26.162213+0000","last_peered":"2024-09-16T10:17:26.162213+0000","last_clean":"2024-09-16T10:17:26.162213+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:17:26.162213+0000","last_undegraded":"2024-09-16T10:17:26.162213+0000","last_fullsized":"2024-09-16T10:17:26.162213+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.816041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":233,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173122+0000","last_change":"2024-09-16T10:16:35.181066+0000","last_active":"2024-09-16T10:17:28.173122+0000","last_peered":"2024-09-16T10:17:28.173122+0000","last_clean":"2024-09-16T10:17:28.173122+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:17:28.173122+0000","last_undegraded":"2024-09-16T10:17:28.173122+0000","last_fullsized":"2024-09-16T10:17:28.173122+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:16:35.180977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.180977+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:35.365790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164161+0000","last_change":"2024-09-16T10:16:31.785080+0000","last_active":"2024-09-16T10:17:26.164161+0000","last_peered":"2024-09-16T10:17:26.164161+0000","last_clean":"2024-09-16T10:17:26.164161+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:17:26.164161+0000","last_undegraded":"2024-09-16T10:17:26.164161+0000","last_fullsized":"2024-09-16T10:17:26.164161+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:31.784724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.784724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.916223+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174679+0000","last_change":"2024-09-16T10:16:37.672462+0000","last_active":"2024-09-16T10:17:26.174679+0000","last_peered":"2024-09-16T10:17:26.174679+0000","last_clean":"2024-09-16T10:17:26.174679+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:17:26.174679+0000","last_undegraded":"2024-09-16T10:17:26.174679+0000","last_fullsized":"2024-09-16T10:17:26.174679+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.246050+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":202,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.170961+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:17:28.170961+0000","last_peered":"2024-09-16T10:17:28.170961+0000","last_clean":"2024-09-16T10:17:28.170961+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:17:28.170961+0000","last_undegraded":"2024-09-16T10:17:28.170961+0000","last_fullsized":"2024-09-16T10:17:28.170961+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.691808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":197,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.177790+0000","last_change":"2024-09-16T10:15:13.091323+0000","last_active":"2024-09-16T10:17:28.177790+0000","last_peered":"2024-09-16T10:17:28.177790+0000","last_clean":"2024-09-16T10:17:28.177790+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:17:28.177790+0000","last_undegraded":"2024-09-16T10:17:28.177790+0000","last_fullsized":"2024-09-16T10:17:28.177790+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:15:13.091193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.091193+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":242,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.166419+0000","last_change":"2024-09-16T10:16:37.249076+0000","last_active":"2024-09-16T10:17:26.166419+0000","last_peered":"2024-09-16T10:17:26.166419+0000","last_clean":"2024-09-16T10:17:26.166419+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:17:26.166419+0000","last_undegraded":"2024-09-16T10:17:26.166419+0000","last_fullsized":"2024-09-16T10:17:26.166419+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:16:37.249033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.249033+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.553977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"141'12237","reported_seq":12369,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.057069+0000","last_change":"2024-09-16T10:14:43.957473+0000","last_active":"2024-09-16T10:17:28.057069+0000","last_peered":"2024-09-16T10:17:28.057069+0000","last_clean":"2024-09-16T10:17:28.057069+0000","last_became_active":"2024-09-16T10:14:43.957174+0000","last_became_peered":"2024-09-16T10:14:43.957174+0000","last_unstale":"2024-09-16T10:17:28.057069+0000","last_undegraded":"2024-09-16T10:17:28.057069+0000","last_fullsized":"2024-09-16T10:17:28.057069+0000","mapping_epoch":81,"log_start":"112'5900","ondisk_log_start":"112'5900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.202365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217251840,"num_objects":3315,"num_object_clones":0,"num_object_copies":9945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12237,"num_write_kb":497664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":208,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171233+0000","last_change":"2024-09-16T10:14:45.136557+0000","last_active":"2024-09-16T10:17:28.171233+0000","last_peered":"2024-09-16T10:17:28.171233+0000","last_clean":"2024-09-16T10:17:28.171233+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:17:28.171233+0000","last_undegraded":"2024-09-16T10:17:28.171233+0000","last_fullsized":"2024-09-16T10:17:28.171233+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:14:45.136460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.136460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.941516+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":179,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.176534+0000","last_change":"2024-09-16T10:13:37.089573+0000","last_active":"2024-09-16T10:17:28.176534+0000","last_peered":"2024-09-16T10:17:28.176534+0000","last_clean":"2024-09-16T10:17:28.176534+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:17:28.176534+0000","last_undegraded":"2024-09-16T10:17:28.176534+0000","last_fullsized":"2024-09-16T10:17:28.176534+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:45.174072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":223,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164163+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:17:26.164163+0000","last_peered":"2024-09-16T10:17:26.164163+0000","last_clean":"2024-09-16T10:17:26.164163+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:17:26.164163+0000","last_undegraded":"2024-09-16T10:17:26.164163+0000","last_fullsized":"2024-09-16T10:17:26.164163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172566+0000","last_change":"2024-09-16T10:15:54.314741+0000","last_active":"2024-09-16T10:17:26.172566+0000","last_peered":"2024-09-16T10:17:26.172566+0000","last_clean":"2024-09-16T10:17:26.172566+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:17:26.172566+0000","last_undegraded":"2024-09-16T10:17:26.172566+0000","last_fullsized":"2024-09-16T10:17:26.172566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:54.314680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.314680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.427634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"142'11712","reported_seq":11925,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.285800+0000","last_change":"2024-09-16T10:16:16.640532+0000","last_active":"2024-09-16T10:17:28.285800+0000","last_peered":"2024-09-16T10:17:28.285800+0000","last_clean":"2024-09-16T10:17:28.285800+0000","last_became_active":"2024-09-16T10:14:43.952713+0000","last_became_peered":"2024-09-16T10:14:43.952713+0000","last_unstale":"2024-09-16T10:17:28.285800+0000","last_undegraded":"2024-09-16T10:17:28.285800+0000","last_fullsized":"2024-09-16T10:17:28.285800+0000","mapping_epoch":81,"log_start":"118'7100","ondisk_log_start":"118'7100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7146","last_scrub_stamp":"2024-09-16T10:16:16.640092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.640092+0000","objects_scrubbed":7079,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:32.050074+0000","scrub_duration":2558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203292672,"num_objects":3102,"num_object_clones":0,"num_object_copies":9306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":215,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.176314+0000","last_change":"2024-09-16T10:15:23.703368+0000","last_active":"2024-09-16T10:17:28.176314+0000","last_peered":"2024-09-16T10:17:28.176314+0000","last_clean":"2024-09-16T10:17:28.176314+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:17:28.176314+0000","last_undegraded":"2024-09-16T10:17:28.176314+0000","last_fullsized":"2024-09-16T10:17:28.176314+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:15:23.703278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.703278+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.166045+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":187,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.180284+0000","last_change":"2024-09-16T10:16:11.013486+0000","last_active":"2024-09-16T10:17:28.180284+0000","last_peered":"2024-09-16T10:17:28.180284+0000","last_clean":"2024-09-16T10:17:28.180284+0000","last_became_active":"2024-09-16T10:13:37.092864+0000","last_became_peered":"2024-09-16T10:13:37.092864+0000","last_unstale":"2024-09-16T10:17:28.180284+0000","last_undegraded":"2024-09-16T10:17:28.180284+0000","last_fullsized":"2024-09-16T10:17:28.180284+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:11.013414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:11.013414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:38.240615+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":168,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.163609+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:17:26.163609+0000","last_peered":"2024-09-16T10:17:26.163609+0000","last_clean":"2024-09-16T10:17:26.163609+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:17:26.163609+0000","last_undegraded":"2024-09-16T10:17:26.163609+0000","last_fullsized":"2024-09-16T10:17:26.163609+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:17.909064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":180,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.161978+0000","last_change":"2024-09-16T10:16:43.906280+0000","last_active":"2024-09-16T10:17:26.161978+0000","last_peered":"2024-09-16T10:17:26.161978+0000","last_clean":"2024-09-16T10:17:26.161978+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:17:26.161978+0000","last_undegraded":"2024-09-16T10:17:26.161978+0000","last_fullsized":"2024-09-16T10:17:26.161978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.906200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.906200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.512702+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"142'11712","reported_seq":11846,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.279125+0000","last_change":"2024-09-16T10:14:43.953541+0000","last_active":"2024-09-16T10:17:28.279125+0000","last_peered":"2024-09-16T10:17:28.279125+0000","last_clean":"2024-09-16T10:17:28.279125+0000","last_became_active":"2024-09-16T10:14:43.952908+0000","last_became_peered":"2024-09-16T10:14:43.952908+0000","last_unstale":"2024-09-16T10:17:28.279125+0000","last_undegraded":"2024-09-16T10:17:28.279125+0000","last_fullsized":"2024-09-16T10:17:28.279125+0000","mapping_epoch":81,"log_start":"118'7139","ondisk_log_start":"118'7139","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.179329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204079104,"num_objects":3114,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":474432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":209,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.163720+0000","last_change":"2024-09-16T10:16:37.847991+0000","last_active":"2024-09-16T10:17:26.163720+0000","last_peered":"2024-09-16T10:17:26.163720+0000","last_clean":"2024-09-16T10:17:26.163720+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:17:26.163720+0000","last_undegraded":"2024-09-16T10:17:26.163720+0000","last_fullsized":"2024-09-16T10:17:26.163720+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:16:37.847913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.847913+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.083962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":195,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.178128+0000","last_change":"2024-09-16T10:17:11.050221+0000","last_active":"2024-09-16T10:17:28.178128+0000","last_peered":"2024-09-16T10:17:28.178128+0000","last_clean":"2024-09-16T10:17:28.178128+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:17:28.178128+0000","last_undegraded":"2024-09-16T10:17:28.178128+0000","last_fullsized":"2024-09-16T10:17:28.178128+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:17:11.050150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:11.050150+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:29.263200+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":237,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.176066+0000","last_change":"2024-09-16T10:15:20.675679+0000","last_active":"2024-09-16T10:17:28.176066+0000","last_peered":"2024-09-16T10:17:28.176066+0000","last_clean":"2024-09-16T10:17:28.176066+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:17:28.176066+0000","last_undegraded":"2024-09-16T10:17:28.176066+0000","last_fullsized":"2024-09-16T10:17:28.176066+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:20.675600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:20.675600+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.107238+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174117+0000","last_change":"2024-09-16T10:16:03.498672+0000","last_active":"2024-09-16T10:17:26.174117+0000","last_peered":"2024-09-16T10:17:26.174117+0000","last_clean":"2024-09-16T10:17:26.174117+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:17:26.174117+0000","last_undegraded":"2024-09-16T10:17:26.174117+0000","last_fullsized":"2024-09-16T10:17:26.174117+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:21.723374+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"141'11966","reported_seq":12098,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.028215+0000","last_change":"2024-09-16T10:14:43.952332+0000","last_active":"2024-09-16T10:17:28.028215+0000","last_peered":"2024-09-16T10:17:28.028215+0000","last_clean":"2024-09-16T10:17:28.028215+0000","last_became_active":"2024-09-16T10:14:43.952043+0000","last_became_peered":"2024-09-16T10:14:43.952043+0000","last_unstale":"2024-09-16T10:17:28.028215+0000","last_undegraded":"2024-09-16T10:17:28.028215+0000","last_fullsized":"2024-09-16T10:17:28.028215+0000","mapping_epoch":81,"log_start":"118'7200","ondisk_log_start":"118'7200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201457664,"num_objects":3074,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11966,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":206,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.180909+0000","last_change":"2024-09-16T10:16:14.094325+0000","last_active":"2024-09-16T10:17:28.180909+0000","last_peered":"2024-09-16T10:17:28.180909+0000","last_clean":"2024-09-16T10:17:28.180909+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:17:28.180909+0000","last_undegraded":"2024-09-16T10:17:28.180909+0000","last_fullsized":"2024-09-16T10:17:28.180909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:16:14.093976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.093976+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:37.672785+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.173007+0000","last_change":"2024-09-16T10:15:23.304088+0000","last_active":"2024-09-16T10:17:26.173007+0000","last_peered":"2024-09-16T10:17:26.173007+0000","last_clean":"2024-09-16T10:17:26.173007+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:17:26.173007+0000","last_undegraded":"2024-09-16T10:17:26.173007+0000","last_fullsized":"2024-09-16T10:17:26.173007+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:23.304033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.304033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.369253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":233,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171404+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:17:28.171404+0000","last_peered":"2024-09-16T10:17:28.171404+0000","last_clean":"2024-09-16T10:17:28.171404+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:17:28.171404+0000","last_undegraded":"2024-09-16T10:17:28.171404+0000","last_fullsized":"2024-09-16T10:17:28.171404+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.998943+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":174,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.170690+0000","last_change":"2024-09-16T10:15:16.104168+0000","last_active":"2024-09-16T10:17:28.170690+0000","last_peered":"2024-09-16T10:17:28.170690+0000","last_clean":"2024-09-16T10:17:28.170690+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:17:28.170690+0000","last_undegraded":"2024-09-16T10:17:28.170690+0000","last_fullsized":"2024-09-16T10:17:28.170690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.104092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.104092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.259525+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"141'11635","reported_seq":11839,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:27.994007+0000","last_change":"2024-09-16T10:16:02.977482+0000","last_active":"2024-09-16T10:17:27.994007+0000","last_peered":"2024-09-16T10:17:27.994007+0000","last_clean":"2024-09-16T10:17:27.994007+0000","last_became_active":"2024-09-16T10:14:43.955808+0000","last_became_peered":"2024-09-16T10:14:43.955808+0000","last_unstale":"2024-09-16T10:17:27.994007+0000","last_undegraded":"2024-09-16T10:17:27.994007+0000","last_fullsized":"2024-09-16T10:17:27.994007+0000","mapping_epoch":81,"log_start":"113'5800","ondisk_log_start":"113'5800","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6320","last_scrub_stamp":"2024-09-16T10:16:02.977318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.977318+0000","objects_scrubbed":6273,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1630,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206766080,"num_objects":3155,"num_object_clones":0,"num_object_copies":9465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":226,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171590+0000","last_change":"2024-09-16T10:17:03.051621+0000","last_active":"2024-09-16T10:17:28.171590+0000","last_peered":"2024-09-16T10:17:28.171590+0000","last_clean":"2024-09-16T10:17:28.171590+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:17:28.171590+0000","last_undegraded":"2024-09-16T10:17:28.171590+0000","last_fullsized":"2024-09-16T10:17:28.171590+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:17:03.050732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.050732+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.857216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":195,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.178686+0000","last_change":"2024-09-16T10:16:33.949944+0000","last_active":"2024-09-16T10:17:28.178686+0000","last_peered":"2024-09-16T10:17:28.178686+0000","last_clean":"2024-09-16T10:17:28.178686+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:17:28.178686+0000","last_undegraded":"2024-09-16T10:17:28.178686+0000","last_fullsized":"2024-09-16T10:17:28.178686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:33.949798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.949798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.958848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":176,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.165726+0000","last_change":"2024-09-16T10:16:07.174774+0000","last_active":"2024-09-16T10:17:26.165726+0000","last_peered":"2024-09-16T10:17:26.165726+0000","last_clean":"2024-09-16T10:17:26.165726+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:17:26.165726+0000","last_undegraded":"2024-09-16T10:17:26.165726+0000","last_fullsized":"2024-09-16T10:17:26.165726+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:07.174690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:07.174690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:28.274560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":183,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.170733+0000","last_change":"2024-09-16T10:17:03.061305+0000","last_active":"2024-09-16T10:17:28.170733+0000","last_peered":"2024-09-16T10:17:28.170733+0000","last_clean":"2024-09-16T10:17:28.170733+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:17:28.170733+0000","last_undegraded":"2024-09-16T10:17:28.170733+0000","last_fullsized":"2024-09-16T10:17:28.170733+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.257731+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"141'11908","reported_seq":12120,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.054959+0000","last_change":"2024-09-16T10:16:14.720883+0000","last_active":"2024-09-16T10:17:28.054959+0000","last_peered":"2024-09-16T10:17:28.054959+0000","last_clean":"2024-09-16T10:17:28.054959+0000","last_became_active":"2024-09-16T10:14:43.957195+0000","last_became_peered":"2024-09-16T10:14:43.957195+0000","last_unstale":"2024-09-16T10:17:28.054959+0000","last_undegraded":"2024-09-16T10:17:28.054959+0000","last_fullsized":"2024-09-16T10:17:28.054959+0000","mapping_epoch":81,"log_start":"112'5600","ondisk_log_start":"112'5600","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7198","last_scrub_stamp":"2024-09-16T10:16:14.720775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.720775+0000","objects_scrubbed":7123,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207093760,"num_objects":3160,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":294,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.178523+0000","last_change":"2024-09-16T10:17:15.133634+0000","last_active":"2024-09-16T10:17:28.178523+0000","last_peered":"2024-09-16T10:17:28.178523+0000","last_clean":"2024-09-16T10:17:28.178523+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:17:28.178523+0000","last_undegraded":"2024-09-16T10:17:28.178523+0000","last_fullsized":"2024-09-16T10:17:28.178523+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:17:15.133542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.133542+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.091484+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164311+0000","last_change":"2024-09-16T10:14:44.986853+0000","last_active":"2024-09-16T10:17:26.164311+0000","last_peered":"2024-09-16T10:17:26.164311+0000","last_clean":"2024-09-16T10:17:26.164311+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:17:26.164311+0000","last_undegraded":"2024-09-16T10:17:26.164311+0000","last_fullsized":"2024-09-16T10:17:26.164311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:44.986805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:44.986805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":195,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172447+0000","last_change":"2024-09-16T10:16:16.430214+0000","last_active":"2024-09-16T10:17:26.172447+0000","last_peered":"2024-09-16T10:17:26.172447+0000","last_clean":"2024-09-16T10:17:26.172447+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:17:26.172447+0000","last_undegraded":"2024-09-16T10:17:26.172447+0000","last_fullsized":"2024-09-16T10:17:26.172447+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:16:16.430154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.430154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:38.184696+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":178,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.183894+0000","last_change":"2024-09-16T10:16:04.405318+0000","last_active":"2024-09-16T10:17:28.183894+0000","last_peered":"2024-09-16T10:17:28.183894+0000","last_clean":"2024-09-16T10:17:28.183894+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:17:28.183894+0000","last_undegraded":"2024-09-16T10:17:28.183894+0000","last_fullsized":"2024-09-16T10:17:28.183894+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:04.405243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.405243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.095983+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":183,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173441+0000","last_change":"2024-09-16T10:17:14.029861+0000","last_active":"2024-09-16T10:17:28.173441+0000","last_peered":"2024-09-16T10:17:28.173441+0000","last_clean":"2024-09-16T10:17:28.173441+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:17:28.173441+0000","last_undegraded":"2024-09-16T10:17:28.173441+0000","last_fullsized":"2024-09-16T10:17:28.173441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.029754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.029754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.182687+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"142'11694","reported_seq":11828,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.280914+0000","last_change":"2024-09-16T10:14:43.956747+0000","last_active":"2024-09-16T10:17:28.280914+0000","last_peered":"2024-09-16T10:17:28.280914+0000","last_clean":"2024-09-16T10:17:28.280914+0000","last_became_active":"2024-09-16T10:14:43.956362+0000","last_became_peered":"2024-09-16T10:14:43.956362+0000","last_unstale":"2024-09-16T10:17:28.280914+0000","last_undegraded":"2024-09-16T10:17:28.280914+0000","last_fullsized":"2024-09-16T10:17:28.280914+0000","mapping_epoch":81,"log_start":"114'6100","ondisk_log_start":"114'6100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.282565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203030528,"num_objects":3098,"num_object_clones":0,"num_object_copies":9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11694,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":202,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171327+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:17:28.171327+0000","last_peered":"2024-09-16T10:17:28.171327+0000","last_clean":"2024-09-16T10:17:28.171327+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:17:28.171327+0000","last_undegraded":"2024-09-16T10:17:28.171327+0000","last_fullsized":"2024-09-16T10:17:28.171327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.895956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":179,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171314+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:17:28.171314+0000","last_peered":"2024-09-16T10:17:28.171314+0000","last_clean":"2024-09-16T10:17:28.171314+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:17:28.171314+0000","last_undegraded":"2024-09-16T10:17:28.171314+0000","last_fullsized":"2024-09-16T10:17:28.171314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.883706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":196,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.182354+0000","last_change":"2024-09-16T10:15:13.676523+0000","last_active":"2024-09-16T10:17:28.182354+0000","last_peered":"2024-09-16T10:17:28.182354+0000","last_clean":"2024-09-16T10:17:28.182354+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:17:28.182354+0000","last_undegraded":"2024-09-16T10:17:28.182354+0000","last_fullsized":"2024-09-16T10:17:28.182354+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:15:13.676437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.676437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.412650+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":171,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.182689+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:17:28.182689+0000","last_peered":"2024-09-16T10:17:28.182689+0000","last_clean":"2024-09-16T10:17:28.182689+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:17:28.182689+0000","last_undegraded":"2024-09-16T10:17:28.182689+0000","last_fullsized":"2024-09-16T10:17:28.182689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.471638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174544+0000","last_change":"2024-09-16T10:17:12.578269+0000","last_active":"2024-09-16T10:17:26.174544+0000","last_peered":"2024-09-16T10:17:26.174544+0000","last_clean":"2024-09-16T10:17:26.174544+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:17:26.174544+0000","last_undegraded":"2024-09-16T10:17:26.174544+0000","last_fullsized":"2024-09-16T10:17:26.174544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.578203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.578203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.857634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"141'11872","reported_seq":12004,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.040567+0000","last_change":"2024-09-16T10:14:43.959854+0000","last_active":"2024-09-16T10:17:28.040567+0000","last_peered":"2024-09-16T10:17:28.040567+0000","last_clean":"2024-09-16T10:17:28.040567+0000","last_became_active":"2024-09-16T10:14:43.959612+0000","last_became_peered":"2024-09-16T10:14:43.959612+0000","last_unstale":"2024-09-16T10:17:28.040567+0000","last_undegraded":"2024-09-16T10:17:28.040567+0000","last_fullsized":"2024-09-16T10:17:28.040567+0000","mapping_epoch":81,"log_start":"114'6600","ondisk_log_start":"114'6600","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:50.620169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206831616,"num_objects":3156,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11872,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":208,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.177934+0000","last_change":"2024-09-16T10:16:08.502530+0000","last_active":"2024-09-16T10:17:28.177934+0000","last_peered":"2024-09-16T10:17:28.177934+0000","last_clean":"2024-09-16T10:17:28.177934+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:17:28.177934+0000","last_undegraded":"2024-09-16T10:17:28.177934+0000","last_fullsized":"2024-09-16T10:17:28.177934+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:16:08.502443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:08.502443+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:37.498249+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":202,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171999+0000","last_change":"2024-09-16T10:17:15.102575+0000","last_active":"2024-09-16T10:17:28.171999+0000","last_peered":"2024-09-16T10:17:28.171999+0000","last_clean":"2024-09-16T10:17:28.171999+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:17:28.171999+0000","last_undegraded":"2024-09-16T10:17:28.171999+0000","last_fullsized":"2024-09-16T10:17:28.171999+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:15.102512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.102512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.326101+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":429,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.279189+0000","last_change":"2024-09-16T10:16:05.559992+0000","last_active":"2024-09-16T10:17:26.279189+0000","last_peered":"2024-09-16T10:17:26.279189+0000","last_clean":"2024-09-16T10:17:26.279189+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:17:26.279189+0000","last_undegraded":"2024-09-16T10:17:26.279189+0000","last_fullsized":"2024-09-16T10:17:26.279189+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:16:05.559913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.559913+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:20.685013+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":225,"num_read_kb":220,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":177,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.173010+0000","last_change":"2024-09-16T10:16:14.370379+0000","last_active":"2024-09-16T10:17:26.173010+0000","last_peered":"2024-09-16T10:17:26.173010+0000","last_clean":"2024-09-16T10:17:26.173010+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:17:26.173010+0000","last_undegraded":"2024-09-16T10:17:26.173010+0000","last_fullsized":"2024-09-16T10:17:26.173010+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:14.370307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.370307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":183,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.178108+0000","last_change":"2024-09-16T10:17:16.162854+0000","last_active":"2024-09-16T10:17:28.178108+0000","last_peered":"2024-09-16T10:17:28.178108+0000","last_clean":"2024-09-16T10:17:28.178108+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:17:28.178108+0000","last_undegraded":"2024-09-16T10:17:28.178108+0000","last_fullsized":"2024-09-16T10:17:28.178108+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:16.162771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.162771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:36.273519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"142'12151","reported_seq":12285,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.307915+0000","last_change":"2024-09-16T10:14:43.957718+0000","last_active":"2024-09-16T10:17:28.307915+0000","last_peered":"2024-09-16T10:17:28.307915+0000","last_clean":"2024-09-16T10:17:28.307915+0000","last_became_active":"2024-09-16T10:14:43.957476+0000","last_became_peered":"2024-09-16T10:14:43.957476+0000","last_unstale":"2024-09-16T10:17:28.307915+0000","last_undegraded":"2024-09-16T10:17:28.307915+0000","last_fullsized":"2024-09-16T10:17:28.307915+0000","mapping_epoch":81,"log_start":"114'6500","ondisk_log_start":"114'6500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.193673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207421440,"num_objects":3165,"num_object_clones":0,"num_object_copies":9495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12151,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":249,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.166364+0000","last_change":"2024-09-16T10:16:10.182532+0000","last_active":"2024-09-16T10:17:26.166364+0000","last_peered":"2024-09-16T10:17:26.166364+0000","last_clean":"2024-09-16T10:17:26.166364+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:17:26.166364+0000","last_undegraded":"2024-09-16T10:17:26.166364+0000","last_fullsized":"2024-09-16T10:17:26.166364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:16:10.182468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:10.182468+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.968626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":195,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.174706+0000","last_change":"2024-09-16T10:16:16.115314+0000","last_active":"2024-09-16T10:17:28.174706+0000","last_peered":"2024-09-16T10:17:28.174706+0000","last_clean":"2024-09-16T10:17:28.174706+0000","last_became_active":"2024-09-16T10:13:37.087611+0000","last_became_peered":"2024-09-16T10:13:37.087611+0000","last_unstale":"2024-09-16T10:17:28.174706+0000","last_undegraded":"2024-09-16T10:17:28.174706+0000","last_fullsized":"2024-09-16T10:17:28.174706+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.115237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.115237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.909446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":230,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.175954+0000","last_change":"2024-09-16T10:16:42.677394+0000","last_active":"2024-09-16T10:17:26.175954+0000","last_peered":"2024-09-16T10:17:26.175954+0000","last_clean":"2024-09-16T10:17:26.175954+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:17:26.175954+0000","last_undegraded":"2024-09-16T10:17:26.175954+0000","last_fullsized":"2024-09-16T10:17:26.175954+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:16:42.677309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.677309+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:10.457942+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174741+0000","last_change":"2024-09-16T10:16:43.622567+0000","last_active":"2024-09-16T10:17:26.174741+0000","last_peered":"2024-09-16T10:17:26.174741+0000","last_clean":"2024-09-16T10:17:26.174741+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:17:26.174741+0000","last_undegraded":"2024-09-16T10:17:26.174741+0000","last_fullsized":"2024-09-16T10:17:26.174741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.773945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172846+0000","last_change":"2024-09-16T10:16:34.311115+0000","last_active":"2024-09-16T10:17:26.172846+0000","last_peered":"2024-09-16T10:17:26.172846+0000","last_clean":"2024-09-16T10:17:26.172846+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:17:26.172846+0000","last_undegraded":"2024-09-16T10:17:26.172846+0000","last_fullsized":"2024-09-16T10:17:26.172846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:34.311052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:34.311052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.994480+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"141'11921","reported_seq":12054,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.075989+0000","last_change":"2024-09-16T10:14:43.958855+0000","last_active":"2024-09-16T10:17:28.075989+0000","last_peered":"2024-09-16T10:17:28.075989+0000","last_clean":"2024-09-16T10:17:28.075989+0000","last_became_active":"2024-09-16T10:14:43.958616+0000","last_became_peered":"2024-09-16T10:14:43.958616+0000","last_unstale":"2024-09-16T10:17:28.075989+0000","last_undegraded":"2024-09-16T10:17:28.075989+0000","last_fullsized":"2024-09-16T10:17:28.075989+0000","mapping_epoch":81,"log_start":"112'5600","ondisk_log_start":"112'5600","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.193911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206176280,"num_objects":3147,"num_object_clones":0,"num_object_copies":9441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11921,"num_write_kb":482113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":207,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164238+0000","last_change":"2024-09-16T10:17:06.903885+0000","last_active":"2024-09-16T10:17:26.164238+0000","last_peered":"2024-09-16T10:17:26.164238+0000","last_clean":"2024-09-16T10:17:26.164238+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:17:26.164238+0000","last_undegraded":"2024-09-16T10:17:26.164238+0000","last_fullsized":"2024-09-16T10:17:26.164238+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:06.903808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.903808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:21.156884+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":176,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.163965+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:17:26.163965+0000","last_peered":"2024-09-16T10:17:26.163965+0000","last_clean":"2024-09-16T10:17:26.163965+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:17:26.163965+0000","last_undegraded":"2024-09-16T10:17:26.163965+0000","last_fullsized":"2024-09-16T10:17:26.163965+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.838918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":177,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172777+0000","last_change":"2024-09-16T10:15:30.302921+0000","last_active":"2024-09-16T10:17:26.172777+0000","last_peered":"2024-09-16T10:17:26.172777+0000","last_clean":"2024-09-16T10:17:26.172777+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:17:26.172777+0000","last_undegraded":"2024-09-16T10:17:26.172777+0000","last_fullsized":"2024-09-16T10:17:26.172777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.302859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.302859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.382437+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":175,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.172816+0000","last_change":"2024-09-16T10:15:25.689754+0000","last_active":"2024-09-16T10:17:28.172816+0000","last_peered":"2024-09-16T10:17:28.172816+0000","last_clean":"2024-09-16T10:17:28.172816+0000","last_became_active":"2024-09-16T10:13:43.283190+0000","last_became_peered":"2024-09-16T10:13:43.283190+0000","last_unstale":"2024-09-16T10:17:28.172816+0000","last_undegraded":"2024-09-16T10:17:28.172816+0000","last_fullsized":"2024-09-16T10:17:28.172816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:25.689615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.689615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.184130+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"142'11555","reported_seq":11756,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.269335+0000","last_change":"2024-09-16T10:15:56.898348+0000","last_active":"2024-09-16T10:17:28.269335+0000","last_peered":"2024-09-16T10:17:28.269335+0000","last_clean":"2024-09-16T10:17:28.269335+0000","last_became_active":"2024-09-16T10:14:43.953871+0000","last_became_peered":"2024-09-16T10:14:43.953871+0000","last_unstale":"2024-09-16T10:17:28.269335+0000","last_undegraded":"2024-09-16T10:17:28.269335+0000","last_fullsized":"2024-09-16T10:17:28.269335+0000","mapping_epoch":81,"log_start":"108'4700","ondisk_log_start":"108'4700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'5944","last_scrub_stamp":"2024-09-16T10:15:56.897937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.897937+0000","objects_scrubbed":5880,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:26.380881+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190644224,"num_objects":2909,"num_object_clones":0,"num_object_copies":8727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11555,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":200,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164251+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:17:26.164251+0000","last_peered":"2024-09-16T10:17:26.164251+0000","last_clean":"2024-09-16T10:17:26.164251+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:17:26.164251+0000","last_undegraded":"2024-09-16T10:17:26.164251+0000","last_fullsized":"2024-09-16T10:17:26.164251+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.165940+0000","last_change":"2024-09-16T10:16:05.172896+0000","last_active":"2024-09-16T10:17:26.165940+0000","last_peered":"2024-09-16T10:17:26.165940+0000","last_clean":"2024-09-16T10:17:26.165940+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:17:26.165940+0000","last_undegraded":"2024-09-16T10:17:26.165940+0000","last_fullsized":"2024-09-16T10:17:26.165940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:05.172834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.172834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:18.062065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":179,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.178210+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:17:28.178210+0000","last_peered":"2024-09-16T10:17:28.178210+0000","last_clean":"2024-09-16T10:17:28.178210+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:17:28.178210+0000","last_undegraded":"2024-09-16T10:17:28.178210+0000","last_fullsized":"2024-09-16T10:17:28.178210+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.117933+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":175,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.182996+0000","last_change":"2024-09-16T10:15:11.699384+0000","last_active":"2024-09-16T10:17:28.182996+0000","last_peered":"2024-09-16T10:17:28.182996+0000","last_clean":"2024-09-16T10:17:28.182996+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:17:28.182996+0000","last_undegraded":"2024-09-16T10:17:28.182996+0000","last_fullsized":"2024-09-16T10:17:28.182996+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.699308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.699308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.368765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"142'11504","reported_seq":11716,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.273927+0000","last_change":"2024-09-16T10:16:16.432815+0000","last_active":"2024-09-16T10:17:28.273927+0000","last_peered":"2024-09-16T10:17:28.273927+0000","last_clean":"2024-09-16T10:17:28.273927+0000","last_became_active":"2024-09-16T10:14:43.954996+0000","last_became_peered":"2024-09-16T10:14:43.954996+0000","last_unstale":"2024-09-16T10:17:28.273927+0000","last_undegraded":"2024-09-16T10:17:28.273927+0000","last_fullsized":"2024-09-16T10:17:28.273927+0000","mapping_epoch":81,"log_start":"114'5900","ondisk_log_start":"114'5900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7052","last_scrub_stamp":"2024-09-16T10:16:16.432702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.432702+0000","objects_scrubbed":6983,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:32.391032+0000","scrub_duration":2995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200802304,"num_objects":3064,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11504,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":187,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164273+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:17:26.164273+0000","last_peered":"2024-09-16T10:17:26.164273+0000","last_clean":"2024-09-16T10:17:26.164273+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:17:26.164273+0000","last_undegraded":"2024-09-16T10:17:26.164273+0000","last_fullsized":"2024-09-16T10:17:26.164273+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.917544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":179,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173523+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:17:28.173523+0000","last_peered":"2024-09-16T10:17:28.173523+0000","last_clean":"2024-09-16T10:17:28.173523+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:17:28.173523+0000","last_undegraded":"2024-09-16T10:17:28.173523+0000","last_fullsized":"2024-09-16T10:17:28.173523+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.173364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":226,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.183218+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:17:28.183218+0000","last_peered":"2024-09-16T10:17:28.183218+0000","last_clean":"2024-09-16T10:17:28.183218+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:17:28.183218+0000","last_undegraded":"2024-09-16T10:17:28.183218+0000","last_fullsized":"2024-09-16T10:17:28.183218+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.430003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":172,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.163934+0000","last_change":"2024-09-16T10:16:07.824858+0000","last_active":"2024-09-16T10:17:26.163934+0000","last_peered":"2024-09-16T10:17:26.163934+0000","last_clean":"2024-09-16T10:17:26.163934+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:17:26.163934+0000","last_undegraded":"2024-09-16T10:17:26.163934+0000","last_fullsized":"2024-09-16T10:17:26.163934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:07.824800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:07.824800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"142'11757","reported_seq":11891,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.259825+0000","last_change":"2024-09-16T10:14:43.952726+0000","last_active":"2024-09-16T10:17:28.259825+0000","last_peered":"2024-09-16T10:17:28.259825+0000","last_clean":"2024-09-16T10:17:28.259825+0000","last_became_active":"2024-09-16T10:14:43.952378+0000","last_became_peered":"2024-09-16T10:14:43.952378+0000","last_unstale":"2024-09-16T10:17:28.259825+0000","last_undegraded":"2024-09-16T10:17:28.259825+0000","last_fullsized":"2024-09-16T10:17:28.259825+0000","mapping_epoch":81,"log_start":"108'4900","ondisk_log_start":"108'4900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:30.419152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202309632,"num_objects":3087,"num_object_clones":0,"num_object_copies":9261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":206,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171496+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:17:28.171496+0000","last_peered":"2024-09-16T10:17:28.171496+0000","last_clean":"2024-09-16T10:17:28.171496+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:17:28.171496+0000","last_undegraded":"2024-09-16T10:17:28.171496+0000","last_fullsized":"2024-09-16T10:17:28.171496+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.984845+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172599+0000","last_change":"2024-09-16T10:15:13.340979+0000","last_active":"2024-09-16T10:17:26.172599+0000","last_peered":"2024-09-16T10:17:26.172599+0000","last_clean":"2024-09-16T10:17:26.172599+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:17:26.172599+0000","last_undegraded":"2024-09-16T10:17:26.172599+0000","last_fullsized":"2024-09-16T10:17:26.172599+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.340929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.340929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":171,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.183073+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:17:28.183073+0000","last_peered":"2024-09-16T10:17:28.183073+0000","last_clean":"2024-09-16T10:17:28.183073+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:17:28.183073+0000","last_undegraded":"2024-09-16T10:17:28.183073+0000","last_fullsized":"2024-09-16T10:17:28.183073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.418972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":175,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171894+0000","last_change":"2024-09-16T10:14:45.941203+0000","last_active":"2024-09-16T10:17:28.171894+0000","last_peered":"2024-09-16T10:17:28.171894+0000","last_clean":"2024-09-16T10:17:28.171894+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:17:28.171894+0000","last_undegraded":"2024-09-16T10:17:28.171894+0000","last_fullsized":"2024-09-16T10:17:28.171894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:14:45.941146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.941146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.036920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"142'11642","reported_seq":11852,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.216799+0000","last_change":"2024-09-16T10:16:09.904652+0000","last_active":"2024-09-16T10:17:28.216799+0000","last_peered":"2024-09-16T10:17:28.216799+0000","last_clean":"2024-09-16T10:17:28.216799+0000","last_became_active":"2024-09-16T10:14:43.959635+0000","last_became_peered":"2024-09-16T10:14:43.959635+0000","last_unstale":"2024-09-16T10:17:28.216799+0000","last_undegraded":"2024-09-16T10:17:28.216799+0000","last_fullsized":"2024-09-16T10:17:28.216799+0000","mapping_epoch":81,"log_start":"114'6000","ondisk_log_start":"114'6000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'6706","last_scrub_stamp":"2024-09-16T10:16:09.904421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:09.904421+0000","objects_scrubbed":6634,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197263360,"num_objects":3010,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":219,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.165789+0000","last_change":"2024-09-16T10:16:42.258515+0000","last_active":"2024-09-16T10:17:26.165789+0000","last_peered":"2024-09-16T10:17:26.165789+0000","last_clean":"2024-09-16T10:17:26.165789+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:17:26.165789+0000","last_undegraded":"2024-09-16T10:17:26.165789+0000","last_fullsized":"2024-09-16T10:17:26.165789+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:16:42.258433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.258433+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.356020+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174734+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:17:26.174734+0000","last_peered":"2024-09-16T10:17:26.174734+0000","last_clean":"2024-09-16T10:17:26.174734+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:17:26.174734+0000","last_undegraded":"2024-09-16T10:17:26.174734+0000","last_fullsized":"2024-09-16T10:17:26.174734+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.641444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":187,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.185334+0000","last_change":"2024-09-16T10:16:43.420443+0000","last_active":"2024-09-16T10:17:28.185334+0000","last_peered":"2024-09-16T10:17:28.185334+0000","last_clean":"2024-09-16T10:17:28.185334+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:17:28.185334+0000","last_undegraded":"2024-09-16T10:17:28.185334+0000","last_fullsized":"2024-09-16T10:17:28.185334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.420249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.420249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.309426+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.175929+0000","last_change":"2024-09-16T10:17:08.596403+0000","last_active":"2024-09-16T10:17:26.175929+0000","last_peered":"2024-09-16T10:17:26.175929+0000","last_clean":"2024-09-16T10:17:26.175929+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:17:26.175929+0000","last_undegraded":"2024-09-16T10:17:26.175929+0000","last_fullsized":"2024-09-16T10:17:26.175929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.176712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"142'11670","reported_seq":11804,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.297801+0000","last_change":"2024-09-16T10:14:43.957583+0000","last_active":"2024-09-16T10:17:28.297801+0000","last_peered":"2024-09-16T10:17:28.297801+0000","last_clean":"2024-09-16T10:17:28.297801+0000","last_became_active":"2024-09-16T10:14:43.957207+0000","last_became_peered":"2024-09-16T10:14:43.957207+0000","last_unstale":"2024-09-16T10:17:28.297801+0000","last_undegraded":"2024-09-16T10:17:28.297801+0000","last_fullsized":"2024-09-16T10:17:28.297801+0000","mapping_epoch":81,"log_start":"119'7117","ondisk_log_start":"119'7117","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4553,"log_dups_size":0,"ondisk_log_size":4553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193724416,"num_objects":2956,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11670,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":203,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.163900+0000","last_change":"2024-09-16T10:15:15.038399+0000","last_active":"2024-09-16T10:17:26.163900+0000","last_peered":"2024-09-16T10:17:26.163900+0000","last_clean":"2024-09-16T10:17:26.163900+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:17:26.163900+0000","last_undegraded":"2024-09-16T10:17:26.163900+0000","last_fullsized":"2024-09-16T10:17:26.163900+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:15:15.038322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.038322+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:30.819499+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.175018+0000","last_change":"2024-09-16T10:15:11.631330+0000","last_active":"2024-09-16T10:17:26.175018+0000","last_peered":"2024-09-16T10:17:26.175018+0000","last_clean":"2024-09-16T10:17:26.175018+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:17:26.175018+0000","last_undegraded":"2024-09-16T10:17:26.175018+0000","last_fullsized":"2024-09-16T10:17:26.175018+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.631262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.631262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.656489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":171,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.182151+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:17:28.182151+0000","last_peered":"2024-09-16T10:17:28.182151+0000","last_clean":"2024-09-16T10:17:28.182151+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:17:28.182151+0000","last_undegraded":"2024-09-16T10:17:28.182151+0000","last_fullsized":"2024-09-16T10:17:28.182151+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:20.476397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":183,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173279+0000","last_change":"2024-09-16T10:16:32.171799+0000","last_active":"2024-09-16T10:17:28.173279+0000","last_peered":"2024-09-16T10:17:28.173279+0000","last_clean":"2024-09-16T10:17:28.173279+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:17:28.173279+0000","last_undegraded":"2024-09-16T10:17:28.173279+0000","last_fullsized":"2024-09-16T10:17:28.173279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.171738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.171738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:39.617239+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"142'12023","reported_seq":12230,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173906+0000","last_change":"2024-09-16T10:16:02.172955+0000","last_active":"2024-09-16T10:17:28.173906+0000","last_peered":"2024-09-16T10:17:28.173906+0000","last_clean":"2024-09-16T10:17:28.173906+0000","last_became_active":"2024-09-16T10:14:43.954475+0000","last_became_peered":"2024-09-16T10:14:43.954475+0000","last_unstale":"2024-09-16T10:17:28.173906+0000","last_undegraded":"2024-09-16T10:17:28.173906+0000","last_fullsized":"2024-09-16T10:17:28.173906+0000","mapping_epoch":81,"log_start":"116'7000","ondisk_log_start":"116'7000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6489","last_scrub_stamp":"2024-09-16T10:16:02.172829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.172829+0000","objects_scrubbed":6427,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.789342+0000","scrub_duration":2056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202047488,"num_objects":3083,"num_object_clones":0,"num_object_copies":9249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12023,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":198,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172182+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:17:26.172182+0000","last_peered":"2024-09-16T10:17:26.172182+0000","last_clean":"2024-09-16T10:17:26.172182+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:17:26.172182+0000","last_undegraded":"2024-09-16T10:17:26.172182+0000","last_fullsized":"2024-09-16T10:17:26.172182+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.521661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":197,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174966+0000","last_change":"2024-09-16T10:15:25.529229+0000","last_active":"2024-09-16T10:17:26.174966+0000","last_peered":"2024-09-16T10:17:26.174966+0000","last_clean":"2024-09-16T10:17:26.174966+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:17:26.174966+0000","last_undegraded":"2024-09-16T10:17:26.174966+0000","last_fullsized":"2024-09-16T10:17:26.174966+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:15:25.529178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.529178+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.592697+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":240,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174068+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:17:26.174068+0000","last_peered":"2024-09-16T10:17:26.174068+0000","last_clean":"2024-09-16T10:17:26.174068+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:17:26.174068+0000","last_undegraded":"2024-09-16T10:17:26.174068+0000","last_fullsized":"2024-09-16T10:17:26.174068+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.912935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172260+0000","last_change":"2024-09-16T10:15:17.306478+0000","last_active":"2024-09-16T10:17:26.172260+0000","last_peered":"2024-09-16T10:17:26.172260+0000","last_clean":"2024-09-16T10:17:26.172260+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:17:26.172260+0000","last_undegraded":"2024-09-16T10:17:26.172260+0000","last_fullsized":"2024-09-16T10:17:26.172260+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:17.306392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:17.306392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:45.397274+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.165712+0000","last_change":"2024-09-16T10:15:55.157938+0000","last_active":"2024-09-16T10:17:26.165712+0000","last_peered":"2024-09-16T10:17:26.165712+0000","last_clean":"2024-09-16T10:17:26.165712+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:17:26.165712+0000","last_undegraded":"2024-09-16T10:17:26.165712+0000","last_fullsized":"2024-09-16T10:17:26.165712+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:55.157860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:55.157860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.136638+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":200,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.180133+0000","last_change":"2024-09-16T10:16:06.952654+0000","last_active":"2024-09-16T10:17:28.180133+0000","last_peered":"2024-09-16T10:17:28.180133+0000","last_clean":"2024-09-16T10:17:28.180133+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:17:28.180133+0000","last_undegraded":"2024-09-16T10:17:28.180133+0000","last_fullsized":"2024-09-16T10:17:28.180133+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:16:06.952390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:06.952390+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":169,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172766+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:17:26.172766+0000","last_peered":"2024-09-16T10:17:26.172766+0000","last_clean":"2024-09-16T10:17:26.172766+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:17:26.172766+0000","last_undegraded":"2024-09-16T10:17:26.172766+0000","last_fullsized":"2024-09-16T10:17:26.172766+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.398561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":173,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174038+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:17:26.174038+0000","last_peered":"2024-09-16T10:17:26.174038+0000","last_clean":"2024-09-16T10:17:26.174038+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:17:26.174038+0000","last_undegraded":"2024-09-16T10:17:26.174038+0000","last_fullsized":"2024-09-16T10:17:26.174038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:06.985366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":206,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.178038+0000","last_change":"2024-09-16T10:15:54.369062+0000","last_active":"2024-09-16T10:17:28.178038+0000","last_peered":"2024-09-16T10:17:28.178038+0000","last_clean":"2024-09-16T10:17:28.178038+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:17:28.178038+0000","last_undegraded":"2024-09-16T10:17:28.178038+0000","last_fullsized":"2024-09-16T10:17:28.178038+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:54.368987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.368987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:17.060776+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174263+0000","last_change":"2024-09-16T10:15:28.537739+0000","last_active":"2024-09-16T10:17:26.174263+0000","last_peered":"2024-09-16T10:17:26.174263+0000","last_clean":"2024-09-16T10:17:26.174263+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:17:26.174263+0000","last_undegraded":"2024-09-16T10:17:26.174263+0000","last_fullsized":"2024-09-16T10:17:26.174263+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:28.537673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.537673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.624149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":179,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.179997+0000","last_change":"2024-09-16T10:16:37.237971+0000","last_active":"2024-09-16T10:17:28.179997+0000","last_peered":"2024-09-16T10:17:28.179997+0000","last_clean":"2024-09-16T10:17:28.179997+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:17:28.179997+0000","last_undegraded":"2024-09-16T10:17:28.179997+0000","last_fullsized":"2024-09-16T10:17:28.179997+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:37.237849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.237849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:39.020098+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":167,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.182455+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:17:28.182455+0000","last_peered":"2024-09-16T10:17:28.182455+0000","last_clean":"2024-09-16T10:17:28.182455+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:17:28.182455+0000","last_undegraded":"2024-09-16T10:17:28.182455+0000","last_fullsized":"2024-09-16T10:17:28.182455+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.431103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":205,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172830+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:17:26.172830+0000","last_peered":"2024-09-16T10:17:26.172830+0000","last_clean":"2024-09-16T10:17:26.172830+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:17:26.172830+0000","last_undegraded":"2024-09-16T10:17:26.172830+0000","last_fullsized":"2024-09-16T10:17:26.172830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.347562+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.171168+0000","last_change":"2024-09-16T10:14:37.844181+0000","last_active":"2024-09-16T10:17:28.171168+0000","last_peered":"2024-09-16T10:17:28.171168+0000","last_clean":"2024-09-16T10:17:28.171168+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:17:28.171168+0000","last_undegraded":"2024-09-16T10:17:28.171168+0000","last_fullsized":"2024-09-16T10:17:28.171168+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.844113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.844113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.105572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":242,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.172921+0000","last_change":"2024-09-16T10:17:06.052878+0000","last_active":"2024-09-16T10:17:28.172921+0000","last_peered":"2024-09-16T10:17:28.172921+0000","last_clean":"2024-09-16T10:17:28.172921+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:17:28.172921+0000","last_undegraded":"2024-09-16T10:17:28.172921+0000","last_fullsized":"2024-09-16T10:17:28.172921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:06.050469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.050469+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.355575+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":172,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164421+0000","last_change":"2024-09-16T10:16:05.834955+0000","last_active":"2024-09-16T10:17:26.164421+0000","last_peered":"2024-09-16T10:17:26.164421+0000","last_clean":"2024-09-16T10:17:26.164421+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:17:26.164421+0000","last_undegraded":"2024-09-16T10:17:26.164421+0000","last_fullsized":"2024-09-16T10:17:26.164421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:05.834879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.834879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:29.058409+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":198,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173670+0000","last_change":"2024-09-16T10:16:04.888881+0000","last_active":"2024-09-16T10:17:28.173670+0000","last_peered":"2024-09-16T10:17:28.173670+0000","last_clean":"2024-09-16T10:17:28.173670+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:17:28.173670+0000","last_undegraded":"2024-09-16T10:17:28.173670+0000","last_fullsized":"2024-09-16T10:17:28.173670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:16:04.888793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.888793+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174414+0000","last_change":"2024-09-16T10:15:21.601115+0000","last_active":"2024-09-16T10:17:26.174414+0000","last_peered":"2024-09-16T10:17:26.174414+0000","last_clean":"2024-09-16T10:17:26.174414+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:17:26.174414+0000","last_undegraded":"2024-09-16T10:17:26.174414+0000","last_fullsized":"2024-09-16T10:17:26.174414+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.601066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.601066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.626050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":232,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172738+0000","last_change":"2024-09-16T10:16:00.378802+0000","last_active":"2024-09-16T10:17:26.172738+0000","last_peered":"2024-09-16T10:17:26.172738+0000","last_clean":"2024-09-16T10:17:26.172738+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:17:26.172738+0000","last_undegraded":"2024-09-16T10:17:26.172738+0000","last_fullsized":"2024-09-16T10:17:26.172738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:16:00.378709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:00.378709+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":174,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.177354+0000","last_change":"2024-09-16T10:16:43.291855+0000","last_active":"2024-09-16T10:17:28.177354+0000","last_peered":"2024-09-16T10:17:28.177354+0000","last_clean":"2024-09-16T10:17:28.177354+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:17:28.177354+0000","last_undegraded":"2024-09-16T10:17:28.177354+0000","last_fullsized":"2024-09-16T10:17:28.177354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.291720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.291720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:59.249028+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":191,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.163913+0000","last_change":"2024-09-16T10:13:39.094292+0000","last_active":"2024-09-16T10:17:26.163913+0000","last_peered":"2024-09-16T10:17:26.163913+0000","last_clean":"2024-09-16T10:17:26.163913+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:17:26.163913+0000","last_undegraded":"2024-09-16T10:17:26.163913+0000","last_fullsized":"2024-09-16T10:17:26.163913+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.164531+0000","last_change":"2024-09-16T10:16:01.807900+0000","last_active":"2024-09-16T10:17:26.164531+0000","last_peered":"2024-09-16T10:17:26.164531+0000","last_clean":"2024-09-16T10:17:26.164531+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:17:26.164531+0000","last_undegraded":"2024-09-16T10:17:26.164531+0000","last_fullsized":"2024-09-16T10:17:26.164531+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:01.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:01.807849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":187,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.180704+0000","last_change":"2024-09-16T10:17:14.106033+0000","last_active":"2024-09-16T10:17:28.180704+0000","last_peered":"2024-09-16T10:17:28.180704+0000","last_clean":"2024-09-16T10:17:28.180704+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:17:28.180704+0000","last_undegraded":"2024-09-16T10:17:28.180704+0000","last_fullsized":"2024-09-16T10:17:28.180704+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.105971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.105971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.805018+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":167,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173563+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:17:28.173563+0000","last_peered":"2024-09-16T10:17:28.173563+0000","last_clean":"2024-09-16T10:17:28.173563+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:17:28.173563+0000","last_undegraded":"2024-09-16T10:17:28.173563+0000","last_fullsized":"2024-09-16T10:17:28.173563+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.252222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":199,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.180994+0000","last_change":"2024-09-16T10:15:15.010801+0000","last_active":"2024-09-16T10:17:28.180994+0000","last_peered":"2024-09-16T10:17:28.180994+0000","last_clean":"2024-09-16T10:17:28.180994+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:17:28.180994+0000","last_undegraded":"2024-09-16T10:17:28.180994+0000","last_fullsized":"2024-09-16T10:17:28.180994+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:15.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.010709+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.174661+0000","last_change":"2024-09-16T10:16:11.528789+0000","last_active":"2024-09-16T10:17:26.174661+0000","last_peered":"2024-09-16T10:17:26.174661+0000","last_clean":"2024-09-16T10:17:26.174661+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:17:26.174661+0000","last_undegraded":"2024-09-16T10:17:26.174661+0000","last_fullsized":"2024-09-16T10:17:26.174661+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:11.528731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:11.528731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:39.041613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":179,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173340+0000","last_change":"2024-09-16T10:15:59.987731+0000","last_active":"2024-09-16T10:17:28.173340+0000","last_peered":"2024-09-16T10:17:28.173340+0000","last_clean":"2024-09-16T10:17:28.173340+0000","last_became_active":"2024-09-16T10:13:41.113969+0000","last_became_peered":"2024-09-16T10:13:41.113969+0000","last_unstale":"2024-09-16T10:17:28.173340+0000","last_undegraded":"2024-09-16T10:17:28.173340+0000","last_fullsized":"2024-09-16T10:17:28.173340+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:59.987642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:59.987642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":166,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.170366+0000","last_change":"2024-09-16T10:13:43.117070+0000","last_active":"2024-09-16T10:17:28.170366+0000","last_peered":"2024-09-16T10:17:28.170366+0000","last_clean":"2024-09-16T10:17:28.170366+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:17:28.170366+0000","last_undegraded":"2024-09-16T10:17:28.170366+0000","last_fullsized":"2024-09-16T10:17:28.170366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:44.291340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":194,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.175901+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:17:28.175901+0000","last_peered":"2024-09-16T10:17:28.175901+0000","last_clean":"2024-09-16T10:17:28.175901+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:17:28.175901+0000","last_undegraded":"2024-09-16T10:17:28.175901+0000","last_fullsized":"2024-09-16T10:17:28.175901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.138673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.172867+0000","last_change":"2024-09-16T10:15:58.378648+0000","last_active":"2024-09-16T10:17:26.172867+0000","last_peered":"2024-09-16T10:17:26.172867+0000","last_clean":"2024-09-16T10:17:26.172867+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:17:26.172867+0000","last_undegraded":"2024-09-16T10:17:26.172867+0000","last_fullsized":"2024-09-16T10:17:26.172867+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:58.378585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:58.378585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":177,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.173702+0000","last_change":"2024-09-16T10:14:45.829620+0000","last_active":"2024-09-16T10:17:26.173702+0000","last_peered":"2024-09-16T10:17:26.173702+0000","last_clean":"2024-09-16T10:17:26.173702+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:17:26.173702+0000","last_undegraded":"2024-09-16T10:17:26.173702+0000","last_fullsized":"2024-09-16T10:17:26.173702+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:45.829572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.829572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.663935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":175,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173436+0000","last_change":"2024-09-16T10:15:30.762339+0000","last_active":"2024-09-16T10:17:28.173436+0000","last_peered":"2024-09-16T10:17:28.173436+0000","last_clean":"2024-09-16T10:17:28.173436+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:17:28.173436+0000","last_undegraded":"2024-09-16T10:17:28.173436+0000","last_fullsized":"2024-09-16T10:17:28.173436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.762137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.762137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.149742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":202,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.176222+0000","last_change":"2024-09-16T10:15:08.953214+0000","last_active":"2024-09-16T10:17:28.176222+0000","last_peered":"2024-09-16T10:17:28.176222+0000","last_clean":"2024-09-16T10:17:28.176222+0000","last_became_active":"2024-09-16T10:15:08.930505+0000","last_became_peered":"2024-09-16T10:15:08.930505+0000","last_unstale":"2024-09-16T10:17:28.176222+0000","last_undegraded":"2024-09-16T10:17:28.176222+0000","last_fullsized":"2024-09-16T10:17:28.176222+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.117953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":192,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.173495+0000","last_change":"2024-09-16T10:16:39.271363+0000","last_active":"2024-09-16T10:17:28.173495+0000","last_peered":"2024-09-16T10:17:28.173495+0000","last_clean":"2024-09-16T10:17:28.173495+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:17:28.173495+0000","last_undegraded":"2024-09-16T10:17:28.173495+0000","last_fullsized":"2024-09-16T10:17:28.173495+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:16:39.271283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.271283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.732007+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.165758+0000","last_change":"2024-09-16T10:16:34.225741+0000","last_active":"2024-09-16T10:17:26.165758+0000","last_peered":"2024-09-16T10:17:26.165758+0000","last_clean":"2024-09-16T10:17:26.165758+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:17:26.165758+0000","last_undegraded":"2024-09-16T10:17:26.165758+0000","last_fullsized":"2024-09-16T10:17:26.165758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:34.225689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:34.225689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.982411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":60,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.174184+0000","last_change":"2024-09-16T10:16:14.001413+0000","last_active":"2024-09-16T10:17:28.174184+0000","last_peered":"2024-09-16T10:17:28.174184+0000","last_clean":"2024-09-16T10:17:28.174184+0000","last_became_active":"2024-09-16T10:16:14.000930+0000","last_became_peered":"2024-09-16T10:16:14.000930+0000","last_unstale":"2024-09-16T10:17:28.174184+0000","last_undegraded":"2024-09-16T10:17:28.174184+0000","last_fullsized":"2024-09-16T10:17:28.174184+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:01.542093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:01.542093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":183,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.188387+0000","last_change":"2024-09-16T10:16:32.919293+0000","last_active":"2024-09-16T10:17:28.188387+0000","last_peered":"2024-09-16T10:17:28.188387+0000","last_clean":"2024-09-16T10:17:28.188387+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:17:28.188387+0000","last_undegraded":"2024-09-16T10:17:28.188387+0000","last_fullsized":"2024-09-16T10:17:28.188387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.919094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.919094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:50.652293+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":187,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.176453+0000","last_change":"2024-09-16T10:15:24.663908+0000","last_active":"2024-09-16T10:17:28.176453+0000","last_peered":"2024-09-16T10:17:28.176453+0000","last_clean":"2024-09-16T10:17:28.176453+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:17:28.176453+0000","last_undegraded":"2024-09-16T10:17:28.176453+0000","last_fullsized":"2024-09-16T10:17:28.176453+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.663832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.663832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:55.163009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":207,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T10:17:26.166431+0000","last_change":"2024-09-16T10:16:11.160548+0000","last_active":"2024-09-16T10:17:26.166431+0000","last_peered":"2024-09-16T10:17:26.166431+0000","last_clean":"2024-09-16T10:17:26.166431+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:17:26.166431+0000","last_undegraded":"2024-09-16T10:17:26.166431+0000","last_fullsized":"2024-09-16T10:17:26.166431+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:16:11.160489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:11.160489+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:28.467994+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":186,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T10:17:28.177706+0000","last_change":"2024-09-16T10:16:36.222414+0000","last_active":"2024-09-16T10:17:28.177706+0000","last_peered":"2024-09-16T10:17:28.177706+0000","last_clean":"2024-09-16T10:17:28.177706+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:17:28.177706+0000","last_undegraded":"2024-09-16T10:17:28.177706+0000","last_fullsized":"2024-09-16T10:17:28.177706+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:36.222281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:36.222281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:03.830974+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3250192408,"num_objects":49595,"num_object_clones":0,"num_object_copies":148785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49595,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188959,"num_write_kb":7633665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9749868544,"data_stored":9749856328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89803,"ondisk_log_size":89803,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":225,"num_read_kb":220,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234034,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1125900,"kb_used_data":1022520,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92619252,"statfs":{"total":95995035648,"available":94842114048,"internally_reserved":0,"allocated":1047060480,"data_stored":1046675872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528374,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120244,"kb_used_data":1012708,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92624908,"statfs":{"total":95995035648,"available":94847905792,"internally_reserved":0,"allocated":1037012992,"data_stored":1036647790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790012,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858304,"kb_used_data":786860,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92886848,"statfs":{"total":95995035648,"available":95116132352,"internally_reserved":0,"allocated":805744640,"data_stored":805372250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986241,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1094732,"kb_used_data":991356,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92650420,"statfs":{"total":95995035648,"available":94874030080,"internally_reserved":0,"allocated":1015148544,"data_stored":1014759239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215174,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1942440,"kb_used_data":1798740,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91802712,"statfs":{"total":95995035648,"available":94005977088,"internally_reserved":0,"allocated":1841909760,"data_stored":1841504061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":147061192},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1296308,"kb_used_data":1192928,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92448844,"statfs":{"total":95995035648,"available":94667616256,"internally_reserved":0,"allocated":1221558272,"data_stored":1221196869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1295852,"kb_used_data":1192472,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92449300,"statfs":{"total":95995035648,"available":94668083200,"internally_reserved":0,"allocated":1221091328,"data_stored":1220674225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1715800,"kb_used_data":1590212,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92029352,"statfs":{"total":95995035648,"available":94238056448,"internally_reserved":0,"allocated":1628377088,"data_stored":1627980918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1616187392,"data_stored":1616183320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1218777088,"data_stored":1218773016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1199964160,"data_stored":1199964160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1836843008,"data_stored":1836843008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1010630656,"data_stored":1010630656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":796131328,"data_stored":796131328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1034555392,"data_stored":1034551320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1036779520,"data_stored":1036779520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:17:32.595 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T10:17:32.595 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:17:32.595 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T10:17:32.595 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.893229 2024-09-16T10:17:32.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 0.8932287916510645 2024-09-16T10:17:32.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:32.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:32.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:32.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:32.765 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:32.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:32.866 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:32.907 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:32.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:33.009 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:33.070 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:33.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:33.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:33.199 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:33.204 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:33.240 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:33.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:33.262 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:33.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:33.346 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:33 smithi063 ceph-mon[29993]: from='client.45011 v1:172.21.15.63:0/3918940912' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:33.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:33 smithi063 ceph-mon[39470]: from='client.45011 v1:172.21.15.63:0/3918940912' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:33.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:33.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:33.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:33.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:33.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:33.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:33 smithi165 ceph-mon[32696]: from='client.45011 v1:172.21.15.63:0/3918940912' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:33.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:33.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:33.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:33.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:33.769 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:33.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:33.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:33.929 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:33.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:34.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:34.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:34.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:34.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:34.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:34.244 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:17:34.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:34.332 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:34.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:34.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:34.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:34.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: pgmap v175: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 757 op/s 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3203775474' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: from='client.45017 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: from='client.45017 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:17:34.528 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[29993]: osdmap e143: 8 total, 8 up, 8 in 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: pgmap v175: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 757 op/s 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3203775474' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: from='client.45017 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: from='client.45017 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:17:34.529 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:34 smithi063 ceph-mon[39470]: osdmap e143: 8 total, 8 up, 8 in 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: pgmap v175: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 757 op/s 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3203775474' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: from='client.45017 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: from='client.45017 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:17:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:34 smithi165 ceph-mon[32696]: osdmap e143: 8 total, 8 up, 8 in 2024-09-16T10:17:34.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:34.622 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:34.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:34.724 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:34.771 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:34.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:34.871 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:34.871 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:34.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:34.896 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:34.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:34.921 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:34.921 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:34.962 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:34.963 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:34.998 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:35.000 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:35.000 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:35.036 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:35.036 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:35.049 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:35.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:35.080 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:35.114 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:35.114 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:35.149 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:35.150 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:35.153 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:35.200 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:35.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:35.302 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:35.340 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:35.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:35.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:35.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/829618490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:35.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/829618490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:35.485 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:35.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:35.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/829618490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:35.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:35.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:35.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:35.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:35.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:35.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:35.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:35.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:35.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:35.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.009 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:36.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:36.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:36.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:36.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:36.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:36.269 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.3 primary-affinity to 0.893229 (8585382) 2024-09-16T10:17:36.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:36.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:36.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:36.518 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 5.15 scrub starts 2024-09-16T10:17:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 5.15 scrub ok 2024-09-16T10:17:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 4.13 scrub starts 2024-09-16T10:17:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 4.13 scrub ok 2024-09-16T10:17:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:17:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:17:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 4.1e scrub starts 2024-09-16T10:17:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 4.1e scrub ok 2024-09-16T10:17:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: pgmap v177: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-16T10:17:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 5.13 scrub starts 2024-09-16T10:17:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: 5.13 scrub ok 2024-09-16T10:17:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/29010665' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645} : dispatch 2024-09-16T10:17:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:36 smithi165 ceph-mon[32696]: from='client.25078 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645} : dispatch 2024-09-16T10:17:36.621 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:36.657 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 5.15 scrub starts 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 5.15 scrub ok 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 4.13 scrub starts 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 4.13 scrub ok 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 4.1e scrub starts 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 4.1e scrub ok 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: pgmap v177: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 5.13 scrub starts 2024-09-16T10:17:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: 5.13 scrub ok 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/29010665' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645} : dispatch 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[29993]: from='client.25078 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645} : dispatch 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 5.15 scrub starts 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 5.15 scrub ok 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 4.13 scrub starts 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 4.13 scrub ok 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:17:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 4.1e scrub starts 2024-09-16T10:17:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 4.1e scrub ok 2024-09-16T10:17:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: pgmap v177: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-16T10:17:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 5.13 scrub starts 2024-09-16T10:17:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: 5.13 scrub ok 2024-09-16T10:17:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/29010665' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645} : dispatch 2024-09-16T10:17:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:36 smithi063 ceph-mon[39470]: from='client.25078 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645} : dispatch 2024-09-16T10:17:36.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:36.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:36.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:36.924 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:36.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:37.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:37.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:37.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:37.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:37.194 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:37.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:37.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:17:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:17:37.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:37.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:37.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:37.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:37.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:37.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:37 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:17:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:37 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:17:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:37 smithi165 ceph-mon[32696]: 3.3 scrub starts 2024-09-16T10:17:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:37 smithi165 ceph-mon[32696]: 3.3 scrub ok 2024-09-16T10:17:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:37 smithi165 ceph-mon[32696]: from='client.25078 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645}]': finished 2024-09-16T10:17:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:37 smithi165 ceph-mon[32696]: osdmap e144: 8 total, 8 up, 8 in 2024-09-16T10:17:37.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:37.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:37.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:37.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:37.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:17:37.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:17:37.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[39470]: 3.3 scrub starts 2024-09-16T10:17:37.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[39470]: 3.3 scrub ok 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[39470]: from='client.25078 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645}]': finished 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[39470]: osdmap e144: 8 total, 8 up, 8 in 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[29993]: 3.3 scrub starts 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[29993]: 3.3 scrub ok 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[29993]: from='client.25078 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8932287916510645}]': finished 2024-09-16T10:17:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:37 smithi063 ceph-mon[29993]: osdmap e144: 8 total, 8 up, 8 in 2024-09-16T10:17:37.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:37.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:37.818 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:37.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:37.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:17:37.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:38.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:38.128 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:38.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:38.276 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:38.287 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[29993]: 3.16 scrub starts 2024-09-16T10:17:38.287 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[29993]: 3.16 scrub ok 2024-09-16T10:17:38.288 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[39470]: 3.16 scrub starts 2024-09-16T10:17:38.327 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:38.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:38.357 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:38.357 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:38.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:38.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:38.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:17:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-mon[32696]: 3.16 scrub starts 2024-09-16T10:17:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-mon[32696]: 3.16 scrub ok 2024-09-16T10:17:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-mon[32696]: 7.1 scrub starts 2024-09-16T10:17:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-mon[32696]: pgmap v179: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 615 op/s 2024-09-16T10:17:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-mon[32696]: osdmap e145: 8 total, 8 up, 8 in 2024-09-16T10:17:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-mon[32696]: 7.1 scrub ok 2024-09-16T10:17:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:38.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:38.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:38.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:38.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[39470]: 3.16 scrub ok 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[39470]: 7.1 scrub starts 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[39470]: pgmap v179: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 615 op/s 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[39470]: osdmap e145: 8 total, 8 up, 8 in 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[39470]: 7.1 scrub ok 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[29993]: 7.1 scrub starts 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[29993]: pgmap v179: 148 pgs: 148 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 615 op/s 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[29993]: osdmap e145: 8 total, 8 up, 8 in 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[29993]: 7.1 scrub ok 2024-09-16T10:17:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:38.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:38.748 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:38.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:38.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:38.896 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:38.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:38.999 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:39.045 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:39.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:39.126 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:39.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:39.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:17:39.211 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:39.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:39.313 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:39.357 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:39.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:39.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:17:39.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[39470]: 4.17 scrub starts 2024-09-16T10:17:39.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[39470]: 4.17 scrub ok 2024-09-16T10:17:39.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:39.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:17:39.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:17:39.437 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[29993]: 4.17 scrub starts 2024-09-16T10:17:39.437 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[29993]: 4.17 scrub ok 2024-09-16T10:17:39.437 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:39.437 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:17:39.437 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:39 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:17:39.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:39.492 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:39 smithi165 ceph-mon[32696]: 4.17 scrub starts 2024-09-16T10:17:39.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:39 smithi165 ceph-mon[32696]: 4.17 scrub ok 2024-09-16T10:17:39.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:39.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:39 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:17:39.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:39 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:17:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:39.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:39.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:39.536 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:39.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:39.638 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:39.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:39.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:39.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:39.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:39.776 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:39.776 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:39.812 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:39.813 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:39.849 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:39.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:39.884 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:39.885 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:39.923 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:39.927 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:39.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:39.944 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:39.944 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:39.976 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:40.017 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:40.018 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:40.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:40.051 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:40.052 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:40.077 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:40.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:200665 7.8 e146/81 7:190a54d5:::benchmark_data_smithi063_73086_object81386:head v 146'12538, mlcod=146'12535)", 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.130496+0000", 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0012862069999999999, 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001321528, 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:40.135 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130496+0000", 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:40.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130496+0000", 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130497+0000", 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4869999999999999e-06 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130504+0000", 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9519999999999997e-06 2024-09-16T10:17:40.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130516+0000", 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1331999999999999e-05 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130523+0000", 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1439999999999997e-06 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:40.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130595+0000", 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2131000000000004e-05 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.130619+0000", 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3652999999999999e-05 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-16T10:17:40.139 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:17:40.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:40.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:40.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:40.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:40.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:40.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:40.491 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:40.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: 7.0 scrub starts 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: 7.0 scrub ok 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: pgmap v181: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: osdmap e146: 8 total, 8 up, 8 in 2024-09-16T10:17:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2176288254' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:40.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: 7.0 scrub starts 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: 7.0 scrub ok 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: pgmap v181: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: osdmap e146: 8 total, 8 up, 8 in 2024-09-16T10:17:40.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2176288254' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:40.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: 7.0 scrub starts 2024-09-16T10:17:40.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:17:40.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:17:40.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: 7.0 scrub ok 2024-09-16T10:17:40.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: pgmap v181: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-16T10:17:40.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:17:40.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:17:40.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: osdmap e146: 8 total, 8 up, 8 in 2024-09-16T10:17:40.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2176288254' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:40.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:40.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:40.718 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:40.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:40.862 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:40.889 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:40.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:40.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:41.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:41.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:41.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:41.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:41.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:41.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:41.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:41.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.571 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:41.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:41.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:41.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:41.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:41.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:41.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:41.974 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:41.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:41 smithi165 ceph-mon[32696]: 2.2 scrub starts 2024-09-16T10:17:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:41 smithi165 ceph-mon[32696]: 2.2 scrub ok 2024-09-16T10:17:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:41 smithi165 ceph-mon[32696]: pgmap v183: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 812 op/s 2024-09-16T10:17:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:41 smithi165 ceph-mon[32696]: 7.9 scrub starts 2024-09-16T10:17:42.076 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:42.119 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:42.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:42.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[29993]: 2.2 scrub starts 2024-09-16T10:17:42.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[29993]: 2.2 scrub ok 2024-09-16T10:17:42.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[29993]: pgmap v183: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 812 op/s 2024-09-16T10:17:42.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[29993]: 7.9 scrub starts 2024-09-16T10:17:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[39470]: 2.2 scrub starts 2024-09-16T10:17:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[39470]: 2.2 scrub ok 2024-09-16T10:17:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[39470]: pgmap v183: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 812 op/s 2024-09-16T10:17:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:41 smithi063 ceph-mon[39470]: 7.9 scrub starts 2024-09-16T10:17:42.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:42.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:42.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:42.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:42.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:42.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:42.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:17:42.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:42.535 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:42.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:42.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:42.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:42.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:42.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:42.801 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:42.810 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:17:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:42.811 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:42.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:42.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:42.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:42.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:43.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:43.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:43.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:43.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:42 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:17:43.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:42 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:17:43.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:42 smithi165 ceph-mon[32696]: osdmap e147: 8 total, 8 up, 8 in 2024-09-16T10:17:43.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3043866946' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:43.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:42 smithi165 ceph-mon[32696]: from='client.25090 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:43.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:42 smithi165 ceph-mon[32696]: 7.9 scrub ok 2024-09-16T10:17:43.106 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:17:43.107 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:17:43.107 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T10:17:43.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:17:43.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:17:43.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:17:43.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[29993]: osdmap e147: 8 total, 8 up, 8 in 2024-09-16T10:17:43.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3043866946' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:43.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[29993]: from='client.25090 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:43.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[29993]: 7.9 scrub ok 2024-09-16T10:17:43.144 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:17:43.144 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:17:43.144 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[39470]: osdmap e147: 8 total, 8 up, 8 in 2024-09-16T10:17:43.144 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3043866946' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:43.144 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[39470]: from='client.25090 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:43.144 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:42 smithi063 ceph-mon[39470]: 7.9 scrub ok 2024-09-16T10:17:43.171 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:43.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:43.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:43.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:43.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:43.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:43.358 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:17:43.358 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:17:43.358 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:17:43.358 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:17:43.358 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:17:43.451 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:43.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:43.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:43.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:17:43.600 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:43.636 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:43.717 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:43.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:43.818 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:43.867 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:43.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:43.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:44.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:17:44.040 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:44.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: from='client.25090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: osdmap e148: 8 total, 8 up, 8 in 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: 4.9 scrub starts 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: 4.9 scrub ok 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: 6.1f scrub starts 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: 6.1f scrub ok 2024-09-16T10:17:44.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: pgmap v186: 148 pgs: 148 active+clean; 2.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 999 op/s 2024-09-16T10:17:44.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:43 smithi165 ceph-mon[32696]: 7.f scrub starts 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: from='client.25090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: osdmap e148: 8 total, 8 up, 8 in 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: 4.9 scrub starts 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: 4.9 scrub ok 2024-09-16T10:17:44.135 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: 6.1f scrub starts 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: 6.1f scrub ok 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: pgmap v186: 148 pgs: 148 active+clean; 2.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 999 op/s 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[29993]: 7.f scrub starts 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: from='client.25090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: osdmap e148: 8 total, 8 up, 8 in 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: 4.9 scrub starts 2024-09-16T10:17:44.136 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: 4.9 scrub ok 2024-09-16T10:17:44.137 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: 6.1f scrub starts 2024-09-16T10:17:44.137 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: 6.1f scrub ok 2024-09-16T10:17:44.137 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: pgmap v186: 148 pgs: 148 active+clean; 2.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 999 op/s 2024-09-16T10:17:44.137 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:43 smithi063 ceph-mon[39470]: 7.f scrub starts 2024-09-16T10:17:44.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:44.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:44.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:44.277 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:44.326 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:44.334 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:44.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:44.407 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:44.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:44.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:44.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:44.572 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:44.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:44.693 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:17:44.693 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 148, 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:17:42.798230+0000", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:13:06.309218+0000", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5799936, 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:17:44.694 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 7, 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:17:44.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:17:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 1, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 1, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 1, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 1, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 1, 2024-09-16T10:17:44.697 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "21", 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:17:44.698 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:17:44.699 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:17:44.700 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.701 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 7.8899998664855957, 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 7.8899998664855957, 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.37999999523162842, 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 3, 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 3, 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:17:44.702 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:17:44.703 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:17:44.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:17:44.705 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:17:44.706 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:17:44.707 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:17:44.708 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:17:44.709 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:17:44.710 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:17:44.711 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:17:44.712 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:17:44.713 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:17:44.714 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:17:44.715 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:17:44.716 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:17:44.717 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:17:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:17:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:17:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:17:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:17:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:17:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:17:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:17:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:17:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.25, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.25, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.25, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.25, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:17:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:17:44.724 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:17:44.725 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:17:44.726 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:17:44.727 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:17:44.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.728 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:17:44.728 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:17:44.728 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:17:44.728 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:17:44.728 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:17:44.728 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.729 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:17:44.730 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:17:44.730 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.25, 2024-09-16T10:17:44.730 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.25, 2024-09-16T10:17:44.730 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:17:44.730 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.25, 2024-09-16T10:17:44.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:44.731 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.25, 2024-09-16T10:17:44.731 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:17:44.731 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:17:44.731 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:17:44.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:17:44.732 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:17:44.733 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:17:44.734 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:17:44.735 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:17:44.736 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:17:44.737 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:17:44.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 7, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_0", 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:14:42.862958+0000", 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:17:44.739 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:17:44.740 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "86", 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:17:44.741 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:17:44.742 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:17:44.743 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2, 2024-09-16T10:17:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 144, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:17:44.746 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.747 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:17:44.748 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.750 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:17:44.751 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.752 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.753 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.754 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.755 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.756 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:17:44.758 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0.893218994140625, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 117, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.759 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.761 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 96, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.763 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.764 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.765 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.766 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:17:44.767 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 144, 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.769 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.769 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:17:44.769 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.770 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:17:44.771 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.772 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 144, 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.773 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:17:44.774 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:17:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:17:44.776 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 144, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:17:44.777 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:17:44.778 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:17:44.779 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:17:44.780 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:33.234888+0000", 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:17:44.781 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:16:54.555092+0000", 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.782 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:16:43.568282+0000", 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:38.584488+0000", 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:16:52.704360+0000", 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.784 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:19.764081+0000", 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.785 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:27.384891+0000", 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:41.335130+0000", 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.786 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [], 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.10", 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.18", 2024-09-16T10:17:44.787 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:17:44.788 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.12", 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 7, 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 3 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:17:44.789 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:17:44.790 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:17:44.792 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:17:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:44.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:44.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:44.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:44.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:45.021 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:17:45.021 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 7.f scrub ok 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 4.14 scrub starts 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 4.14 scrub ok 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 4.2 scrub starts 2024-09-16T10:17:45.022 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: 4.2 scrub ok 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2652363080' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 7.f scrub ok 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 4.14 scrub starts 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 4.14 scrub ok 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:17:45.023 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:17:45.024 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 4.2 scrub starts 2024-09-16T10:17:45.024 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: 4.2 scrub ok 2024-09-16T10:17:45.024 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:44 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2652363080' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:17:45.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:45.068 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:45.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:45.089 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:17:45.089 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:13791 7.1 7:8959df09:::benchmark_data_smithi063_73086_object13790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:14:55.132518+0000", 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 169.90482325599999, 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069425162999999998, 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 13791 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132518+0000", 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132518+0000", 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132520+0000", 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3850000000000001e-06 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132547+0000", 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6900999999999999e-05 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132558+0000", 2024-09-16T10:17:45.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1109e-05 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132563+0000", 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4659999999999998e-06 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132620+0000", 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6916999999999997e-05 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132731+0000", 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011111 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132887+0000", 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015598200000000001 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.133901+0000", 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001013262 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.133961+0000", 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0639e-05 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.201854+0000", 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.067893091000000003 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.201901+0000", 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6792000000000002e-05 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.201944+0000", 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2509000000000002e-05 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39780 7.1 7:87ba97cd:::benchmark_data_smithi063_73086_object39779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.255004+0000", 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 155.78233731, 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.132848357, 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.096 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39780 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255004+0000", 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255004+0000", 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255007+0000", 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4930000000000001e-06 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255051+0000", 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4141999999999999e-05 2024-09-16T10:17:45.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255072+0000", 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0594e-05 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255084+0000", 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2075999999999999e-05 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255147+0000", 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2867000000000006e-05 2024-09-16T10:17:45.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255233+0000", 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6434999999999995e-05 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255375+0000", 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014167199999999999 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.256696+0000", 2024-09-16T10:17:45.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001321572 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.257082+0000", 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000385521 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.387743+0000", 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13066166400000001 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.387809+0000", 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5554000000000003e-05 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.387853+0000", 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3767000000000003e-05 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70291 7.1 7:8090c1a5:::benchmark_data_smithi063_73086_object70290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:17:45.102 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.13915526700001, 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12180724800000001, 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70291 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898188+0000", 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898240+0000", 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1740999999999998e-05 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898249+0000", 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4490000000000005e-06 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898254+0000", 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2569999999999998e-06 2024-09-16T10:17:45.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898300+0000", 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5454000000000002e-05 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898383+0000", 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3005999999999998e-05 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898476+0000", 2024-09-16T10:17:45.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3029000000000007e-05 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.905816+0000", 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073403699999999997 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906673+0000", 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085627000000000001 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019900+0000", 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11322701 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019950+0000", 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0862999999999997e-05 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019994+0000", 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3169e-05 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70294 7.1 7:8c6bcbeb:::benchmark_data_smithi063_73086_object70293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.13709205399999, 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119864807, 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70294 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900252+0000", 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1119999999999999e-06 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900304+0000", 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1965999999999997e-05 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900313+0000", 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4690000000000003e-06 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900320+0000", 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.404e-06 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900758+0000", 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000438023 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900864+0000", 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106434 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900994+0000", 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129567 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906642+0000", 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056482299999999997 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906681+0000", 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9443000000000002e-05 2024-09-16T10:17:45.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020041+0000", 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113360006 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020084+0000", 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2623999999999997e-05 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020114+0000", 2024-09-16T10:17:45.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0528999999999999e-05 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70297 7.1 7:8aa306c3:::benchmark_data_smithi063_73086_object70296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.13682362599999, 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119738053, 2024-09-16T10:17:45.115 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70297 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900519+0000", 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1799999999999999e-06 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900602+0000", 2024-09-16T10:17:45.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2973000000000007e-05 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900612+0000", 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3230000000000004e-06 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900618+0000", 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9489999999999999e-06 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903224+0000", 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026055459999999998 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903382+0000", 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015748800000000001 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903501+0000", 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119727 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906766+0000", 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032648949999999999 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019591+0000", 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112824517 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020144+0000", 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055325600000000002 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020224+0000", 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9878999999999999e-05 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020256+0000", 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2320000000000002e-05 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70298 7.3 e106/81 7:c8a39fda:::benchmark_data_smithi063_73086_object70297:head v 106'4332, mlcod=106'4328)", 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.134908749, 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.117018094, 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902434+0000", 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5570000000000001e-06 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902490+0000", 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5559000000000001e-05 2024-09-16T10:17:45.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902501+0000", 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0879999999999999e-05 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902508+0000", 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.204e-06 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902555+0000", 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6430000000000001e-05 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902566+0000", 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1922999999999999e-05 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906782+0000", 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042155300000000003 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019451+0000", 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112669011 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70300 7.8 e106/81 7:10588a9d:::benchmark_data_smithi063_73086_object70299:head v 106'4365, mlcod=106'4363)", 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.133733084, 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118272198, 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.126 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903610+0000", 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0640000000000001e-06 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903703+0000", 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3548000000000006e-05 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903721+0000", 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8036000000000001e-05 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903727+0000", 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.156e-06 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903787+0000", 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9658999999999999e-05 2024-09-16T10:17:45.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903816+0000", 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8988000000000001e-05 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.021844+0000", 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11802747700000001 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.021881+0000", 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7270000000000001e-05 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70292 7.9 e106/81 7:99cfcedf:::benchmark_data_smithi063_73086_object70291:head v 106'4342, mlcod=106'4340)", 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.904195+0000", 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.13314659, 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11532339799999999, 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.131 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904195+0000", 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904195+0000", 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904196+0000", 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1230000000000001e-06 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904230+0000", 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4264000000000003e-05 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904239+0000", 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6260000000000001e-06 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904245+0000", 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5470000000000004e-06 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.905845+0000", 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016002760000000001 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.905862+0000", 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6835000000000001e-05 2024-09-16T10:17:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019483+0000", 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11362138200000001 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019518+0000", 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5345000000000001e-05 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.135 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70296 7.4 e106/81 7:2c63ac7c:::benchmark_data_smithi063_73086_object70295:head v 106'4337, mlcod=106'4335)", 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.904254+0000", 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.133087609, 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11525693200000001, 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904254+0000", 2024-09-16T10:17:45.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904254+0000", 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904255+0000", 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904283+0000", 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8755000000000002e-05 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904289+0000", 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2889999999999997e-06 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904293+0000", 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1559999999999997e-06 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904355+0000", 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1920000000000003e-05 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904372+0000", 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6883000000000001e-05 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906808+0000", 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024367820000000002 2024-09-16T10:17:45.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019511+0000", 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112702711 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70303 7.8 e106/81 7:176924a2:::benchmark_data_smithi063_73086_object70302:head v 106'4366, mlcod=106'4363)", 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.130703141, 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115509458, 2024-09-16T10:17:45.140 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906640+0000", 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.215e-06 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906678+0000", 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8124000000000003e-05 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906691+0000", 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3244999999999999e-05 2024-09-16T10:17:45.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906698+0000", 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2949999999999996e-06 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906760+0000", 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1163999999999999e-05 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906778+0000", 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8695e-05 2024-09-16T10:17:45.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.022123+0000", 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115344601 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.022148+0000", 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5119000000000001e-05 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.44750.0:70297 7.1 e106/81 ondisk, result = 0)", 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.906846+0000", 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.13049593599999, 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112753247, 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906846+0000", 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906846+0000", 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906847+0000", 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9999999999999996e-07 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906851+0000", 2024-09-16T10:17:45.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2570000000000001e-06 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019522+0000", 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112670991 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019529+0000", 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019576+0000", 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7012000000000003e-05 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019584+0000", 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2840000000000006e-06 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019599+0000", 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4664000000000001e-05 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70304 7.3 e106/81 7:c3ecd6bf:::benchmark_data_smithi063_73086_object70303:head v 106'4333, mlcod=106'4331)", 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.909677+0000", 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.12766425199999, 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.114141138, 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909677+0000", 2024-09-16T10:17:45.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909677+0000", 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909679+0000", 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6270000000000001e-06 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909762+0000", 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3071999999999993e-05 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019488+0000", 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10972603 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019500+0000", 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1464999999999999e-05 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019619+0000", 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118899 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019639+0000", 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200999999999999e-05 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.023760+0000", 2024-09-16T10:17:45.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041216530000000003 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.023819+0000", 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8190999999999999e-05 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:91533 7.1 7:89733e8c:::benchmark_data_smithi063_73086_object91532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:17:45.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:51.076341+0000", 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 113.961000813, 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080046422000000006, 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 91533 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076341+0000", 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076341+0000", 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076343+0000", 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2000000000000001e-06 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076390+0000", 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7349999999999999e-05 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076403+0000", 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2258e-05 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076408+0000", 2024-09-16T10:17:45.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4670000000000003e-06 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076460+0000", 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1959999999999997e-05 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076561+0000", 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100449 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.076650+0000", 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9791999999999997e-05 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.078955+0000", 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023045650000000002 2024-09-16T10:17:45.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.079081+0000", 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012565300000000001 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.156282+0000", 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.077201109000000004 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.156340+0000", 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8062000000000001e-05 2024-09-16T10:17:45.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.156387+0000", 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7556999999999999e-05 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174929 7.1 7:83c2bca2:::benchmark_data_smithi063_73086_object55650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.168864173999999, 2024-09-16T10:17:45.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12126697, 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174929 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868479+0000", 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2869999999999999e-06 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868482+0000", 2024-09-16T10:17:45.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6799999999999999e-06 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868491+0000", 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4889999999999995e-06 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868498+0000", 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9450000000000004e-06 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868544+0000", 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5828000000000002e-05 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868647+0000", 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102902 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.869028+0000", 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038142800000000001 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.871769+0000", 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027412370000000001 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.871892+0000", 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012271299999999999 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989639+0000", 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.117746694 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989694+0000", 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5634000000000003e-05 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989744+0000", 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0133000000000003e-05 2024-09-16T10:17:45.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174942 7.1 7:89bad1de:::benchmark_data_smithi063_73086_object55663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.878911+0000", 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.158430784, 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110946397, 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174942 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878911+0000", 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878911+0000", 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878913+0000", 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9069999999999999e-06 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878918+0000", 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4829999999999999e-06 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878928+0000", 2024-09-16T10:17:45.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5070000000000006e-06 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878935+0000", 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0520000000000004e-06 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.879041+0000", 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106211 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.879153+0000", 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011212 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.879254+0000", 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010049500000000001 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.882256+0000", 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030022820000000002 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.882267+0000", 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0906e-05 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989780+0000", 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.107513172 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989822+0000", 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2045000000000001e-05 2024-09-16T10:17:45.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989857+0000", 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5216999999999998e-05 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174960 7.1 7:84626464:::benchmark_data_smithi063_73086_object55681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.895856+0000", 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.141485707000001, 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099689238999999999, 2024-09-16T10:17:45.173 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174960 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895856+0000", 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895856+0000", 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895857+0000", 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.508e-06 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895863+0000", 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4929999999999998e-06 2024-09-16T10:17:45.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895874+0000", 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0817e-05 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895880+0000", 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0719999999999996e-06 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895932+0000", 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2456000000000001e-05 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896053+0000", 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120671 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896127+0000", 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4060000000000006e-05 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.898045+0000", 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001917693 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.898305+0000", 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026003799999999998 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.995469+0000", 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097164392000000002 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.995500+0000", 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1139999999999997e-05 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.995545+0000", 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4898999999999998e-05 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.179 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:201328 7.1 7:8fb42478:::benchmark_data_smithi063_73086_object82049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.853935+0000", 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.183406261, 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072541727, 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 201328 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.853935+0000", 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.853935+0000", 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.853937+0000", 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9530000000000002e-06 2024-09-16T10:17:45.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.853943+0000", 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2290000000000002e-06 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.853951+0000", 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1990000000000006e-06 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.853956+0000", 2024-09-16T10:17:45.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7620000000000003e-06 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.854003+0000", 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7905999999999998e-05 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.854112+0000", 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010875200000000001 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.854219+0000", 2024-09-16T10:17:45.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010672599999999999 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.856557+0000", 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023383430000000001 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.857804+0000", 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012468830000000001 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.926388+0000", 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.068583928000000002 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.926436+0000", 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7564000000000001e-05 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.926477+0000", 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1482e-05 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.185 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:201826 7.4 e146/81 7:2bb12e82:::benchmark_data_smithi063_73086_object82547:head v 146'12527, mlcod=146'12525)", 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.6738214930000002, 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13052909300000001, 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363522+0000", 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.686e-06 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363529+0000", 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5209999999999997e-06 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363541+0000", 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.163e-05 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363549+0000", 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2870000000000004e-06 2024-09-16T10:17:45.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363609+0000", 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0059000000000002e-05 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363629+0000", 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9953999999999999e-05 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.494008+0000", 2024-09-16T10:17:45.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13037901800000001 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.494049+0000", 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0938e-05 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:201829 7.1 7:86c5e7dd:::benchmark_data_smithi063_73086_object82550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.67347468, 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13543617699999999, 2024-09-16T10:17:45.190 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 201829 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363869+0000", 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.748e-06 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363874+0000", 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4380000000000003e-06 2024-09-16T10:17:45.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489286+0000", 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12541170800000001 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489301+0000", 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4732999999999999e-05 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489378+0000", 2024-09-16T10:17:45.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7651000000000006e-05 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489473+0000", 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4846999999999995e-05 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489580+0000", 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106932 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.497200+0000", 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076199780000000003 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.497235+0000", 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4955e-05 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499237+0000", 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020019410000000001 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499273+0000", 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6531e-05 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499303+0000", 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9714999999999999e-05 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:201820 7.9 e146/81 7:907729a7:::benchmark_data_smithi063_73086_object82541:head v 146'12370, mlcod=146'12367)", 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:17:45.196 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.6688770320000001, 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13076132600000001, 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:17:45.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368466+0000", 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5910000000000001e-06 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368474+0000", 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4989999999999997e-06 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:45.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489290+0000", 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120816008 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489300+0000", 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0530000000000001e-05 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489358+0000", 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7380999999999998e-05 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489379+0000", 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1596000000000001e-05 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499198+0000", 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098188400000000006 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499226+0000", 2024-09-16T10:17:45.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7881e-05 2024-09-16T10:17:45.201 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.201 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.201 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.201 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:45.201 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:45.201 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:17:45.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:17:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:17:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:17:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:17:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 7.f scrub ok 2024-09-16T10:17:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 4.14 scrub starts 2024-09-16T10:17:45.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 4.14 scrub ok 2024-09-16T10:17:45.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:17:45.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:17:45.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 4.2 scrub starts 2024-09-16T10:17:45.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: 4.2 scrub ok 2024-09-16T10:17:45.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:44 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2652363080' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:17:45.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:45.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:45.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:45.369 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:45.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:45.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:45.514 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:45.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:45.538 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:17:45.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:45.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:45.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:45.743 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:45.787 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:45.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:45.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: 3.14 scrub starts 2024-09-16T10:17:45.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: 3.14 scrub ok 2024-09-16T10:17:45.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2721115969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:45.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: pgmap v187: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s 2024-09-16T10:17:45.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: 6.10 scrub starts 2024-09-16T10:17:45.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: 6.10 scrub ok 2024-09-16T10:17:45.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: 3.8 scrub starts 2024-09-16T10:17:45.862 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[29993]: 3.8 scrub ok 2024-09-16T10:17:45.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:45.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:45.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:46.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:46.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:46.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: 3.14 scrub starts 2024-09-16T10:17:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: 3.14 scrub ok 2024-09-16T10:17:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2721115969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: pgmap v187: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s 2024-09-16T10:17:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: 6.10 scrub starts 2024-09-16T10:17:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: 6.10 scrub ok 2024-09-16T10:17:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: 3.8 scrub starts 2024-09-16T10:17:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:45 smithi165 ceph-mon[32696]: 3.8 scrub ok 2024-09-16T10:17:46.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:46.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:46.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: 3.14 scrub starts 2024-09-16T10:17:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: 3.14 scrub ok 2024-09-16T10:17:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2721115969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: pgmap v187: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s 2024-09-16T10:17:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: 6.10 scrub starts 2024-09-16T10:17:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: 6.10 scrub ok 2024-09-16T10:17:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: 3.8 scrub starts 2024-09-16T10:17:46.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:45 smithi063 ceph-mon[39470]: 3.8 scrub ok 2024-09-16T10:17:46.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:46.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:46.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:17:46.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:46.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:46.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:46.565 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:46.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:46.696 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:46.711 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:46.774 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:46.775 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:46.776 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:46.776 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:46.776 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:46.776 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:46.776 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:46.776 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:46.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.854 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:46.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:46.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:46.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:46.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:47.075 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:46 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:17:47.075 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:46 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:17:47.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:46 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:17:47.076 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:46 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:17:47.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:47.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:47.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:47.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:46 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:17:47.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:46 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:17:47.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:47.244 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:47.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:47.248 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:47.248 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:47.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:47.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:47.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:47.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:17:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:17:47.478 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:47.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:47.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:47.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:47.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:47.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:47.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:47.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:47.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:47.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:47.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:47.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:47.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:48.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:48.126 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:48.148 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:47 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:17:48.149 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:47 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:17:48.149 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:47 smithi165 ceph-mon[32696]: 6.c scrub starts 2024-09-16T10:17:48.149 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:47 smithi165 ceph-mon[32696]: 6.c scrub ok 2024-09-16T10:17:48.149 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:47 smithi165 ceph-mon[32696]: pgmap v188: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 635 op/s 2024-09-16T10:17:48.149 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:47 smithi165 ceph-mon[32696]: 2.1 scrub starts 2024-09-16T10:17:48.149 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:47 smithi165 ceph-mon[32696]: 2.1 scrub ok 2024-09-16T10:17:48.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:17:48.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:48.168 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:17:48.169 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:17:48.169 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[29993]: 6.c scrub starts 2024-09-16T10:17:48.169 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[29993]: 6.c scrub ok 2024-09-16T10:17:48.169 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[29993]: pgmap v188: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 635 op/s 2024-09-16T10:17:48.169 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[29993]: 2.1 scrub starts 2024-09-16T10:17:48.169 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[29993]: 2.1 scrub ok 2024-09-16T10:17:48.170 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:17:48.170 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:17:48.171 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[39470]: 6.c scrub starts 2024-09-16T10:17:48.171 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[39470]: 6.c scrub ok 2024-09-16T10:17:48.171 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[39470]: pgmap v188: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 635 op/s 2024-09-16T10:17:48.171 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[39470]: 2.1 scrub starts 2024-09-16T10:17:48.171 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:47 smithi063 ceph-mon[39470]: 2.1 scrub ok 2024-09-16T10:17:48.225 INFO:tasks.thrashosds.thrasher:j is {'epoch': 148, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:17:42.798230+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:13:06.309218+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T10:14:42.862958+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.893218994140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 117, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:33.234888+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:16:54.555092+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:16:43.568282+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:38.584488+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:16:52.704360+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:19.764081+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:27.384891+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:41.335130+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '6.12', 'mappings': [{'from': 7, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:17:48.225 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T10:17:48.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:48.307 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:48.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:48.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:48.498 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:48.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:48.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:17:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:17:48.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:48.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:48.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:48.714 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:48.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:48.864 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:48.953 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:48.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:48.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:48 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:48.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:48.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:49.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:48 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:49.088 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:48 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:49.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:49.107 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:49.107 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:49.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:49.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:49.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:49.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:49.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:49.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:49.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:49.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:49.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:49.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:49.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:49.675 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:49.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:49.778 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:49.824 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:49.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:49.926 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:49.966 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:49.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:50.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:50.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:17:50.111 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:50.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:50.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[29993]: 6.1e scrub starts 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[29993]: 6.1e scrub ok 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[29993]: pgmap v189: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[29993]: 7.4 scrub starts 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[29993]: 4.f scrub starts 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[29993]: 4.f scrub ok 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[39470]: 6.1e scrub starts 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[39470]: 6.1e scrub ok 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[39470]: pgmap v189: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[39470]: 7.4 scrub starts 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[39470]: 4.f scrub starts 2024-09-16T10:17:50.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:49 smithi063 ceph-mon[39470]: 4.f scrub ok 2024-09-16T10:17:50.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:50.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:50.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:49 smithi165 ceph-mon[32696]: 6.1e scrub starts 2024-09-16T10:17:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:49 smithi165 ceph-mon[32696]: 6.1e scrub ok 2024-09-16T10:17:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:49 smithi165 ceph-mon[32696]: pgmap v189: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-16T10:17:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:49 smithi165 ceph-mon[32696]: 7.4 scrub starts 2024-09-16T10:17:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:49 smithi165 ceph-mon[32696]: 4.f scrub starts 2024-09-16T10:17:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:49 smithi165 ceph-mon[32696]: 4.f scrub ok 2024-09-16T10:17:50.347 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:50.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:50.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:50.426 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:50.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:50.526 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:50.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:50.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:50.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:50.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:50.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:50.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:50.798 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:50.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:50.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:17:50.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:50.908 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:17:50.908 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: 5.10 scrub starts 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: 5.10 scrub ok 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1982202159' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: from='client.25105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: 7.4 scrub ok 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: 4.7 scrub starts 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: 4.7 scrub ok 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: 6.f scrub starts 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[29993]: 6.f scrub ok 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: 5.10 scrub starts 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: 5.10 scrub ok 2024-09-16T10:17:50.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1982202159' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:50.910 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: from='client.25105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:50.910 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: 7.4 scrub ok 2024-09-16T10:17:50.910 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: 4.7 scrub starts 2024-09-16T10:17:50.910 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: 4.7 scrub ok 2024-09-16T10:17:50.910 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: 6.f scrub starts 2024-09-16T10:17:50.910 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:50 smithi063 ceph-mon[39470]: 6.f scrub ok 2024-09-16T10:17:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:50.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:50.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:50.994 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:50.994 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:51.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: 5.10 scrub starts 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: 5.10 scrub ok 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1982202159' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: from='client.25105 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: 7.4 scrub ok 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: 4.7 scrub starts 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: 4.7 scrub ok 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: 6.f scrub starts 2024-09-16T10:17:51.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:50 smithi165 ceph-mon[32696]: 6.f scrub ok 2024-09-16T10:17:51.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:51.075 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:51.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:51.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:51.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:51.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:51.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:51.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:51.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:51.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:51.472 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:51.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:51.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:51.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:51.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:51.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:51.727 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:51.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:51.829 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:51.855 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:51.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:51.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:51.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:52.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:52.105 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:52.147 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:52.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:52.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[29993]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:17:52.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[29993]: osdmap e149: 8 total, 8 up, 8 in 2024-09-16T10:17:52.171 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[29993]: pgmap v191: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:17:52.171 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:52.171 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[39470]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:17:52.172 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[39470]: osdmap e149: 8 total, 8 up, 8 in 2024-09-16T10:17:52.172 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:51 smithi063 ceph-mon[39470]: pgmap v191: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:17:52.248 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:52.281 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:52.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:51 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:51 smithi165 ceph-mon[32696]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:17:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:51 smithi165 ceph-mon[32696]: osdmap e149: 8 total, 8 up, 8 in 2024-09-16T10:17:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:51 smithi165 ceph-mon[32696]: pgmap v191: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:17:52.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:52.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:52.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:52.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:17:52.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:52.638 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:52.676 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:52.763 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:17:52.764 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:17:52.764 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:17:52.764 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:17:52.764 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:17:52.764 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:17:52.764 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:17:52.765 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:17:52.766 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:17:52.778 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:52.812 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:52.846 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:52.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:52.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:52.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:53.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:53.080 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:53.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:53.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:52 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1558796664' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:53.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:52 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:53.176 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:52 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1558796664' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:53.176 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:52 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:53.176 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:52 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1558796664' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:17:53.177 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:52 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:17:53.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:53.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:53.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:53.226 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:17:53.226 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:17:53.226 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T10:17:53.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:17:53.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:53.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:53.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:53.426 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:17:53.426 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:17:53.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.426 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:213654 7.d 7:b996a622:::benchmark_data_smithi063_73086_object94375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T10:17:53.426 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:53.421397+0000", 2024-09-16T10:17:53.426 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00093128299999999998, 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095922300000000004, 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213654 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:53.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421397+0000", 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421397+0000", 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421399+0000", 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.68e-06 2024-09-16T10:17:53.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421404+0000", 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9409999999999998e-06 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421416+0000", 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1887e-05 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:53.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421423+0000", 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421473+0000", 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9577999999999999e-05 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421585+0000", 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112359 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421698+0000", 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011349900000000001 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:213655 7.7 7:edae6fdd:::benchmark_data_smithi063_73086_object94376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:53.422106+0000", 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00022218699999999999, 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033173000000000002, 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:53.431 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213655 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422106+0000", 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422106+0000", 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422108+0000", 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.105e-06 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422112+0000", 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0040000000000001e-06 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422121+0000", 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7000000000000003e-06 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:53.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422128+0000", 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1210000000000001e-06 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422176+0000", 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7460000000000003e-05 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422243+0000", 2024-09-16T10:17:53.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7372000000000005e-05 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422345+0000", 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010161 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:213656 7.c 7:3bd28819:::benchmark_data_smithi063_73086_object94377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:53.422315+0000", 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.348e-05, 2024-09-16T10:17:53.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018890000000000001, 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213656 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422315+0000", 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422315+0000", 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422317+0000", 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3909999999999999e-06 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422321+0000", 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3919999999999996e-06 2024-09-16T10:17:53.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422330+0000", 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1789999999999997e-06 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422337+0000", 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4629999999999998e-06 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.422455+0000", 2024-09-16T10:17:53.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117914 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:213638 7.d 7:bb6ec029:::benchmark_data_smithi063_73086_object94359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:53.406993+0000", 2024-09-16T10:17:53.440 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.015335847, 2024-09-16T10:17:53.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015564269, 2024-09-16T10:17:53.441 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:53.441 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:53.441 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:17:53.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:53.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:53.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:53.442 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213638 2024-09-16T10:17:53.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.442 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:53.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:53.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.406993+0000", 2024-09-16T10:17:53.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:53.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.406993+0000", 2024-09-16T10:17:53.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:53.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.406995+0000", 2024-09-16T10:17:53.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3499999999999999e-06 2024-09-16T10:17:53.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:53.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.407001+0000", 2024-09-16T10:17:53.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-16T10:17:53.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:53.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.407016+0000", 2024-09-16T10:17:53.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5096000000000001e-05 2024-09-16T10:17:53.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:53.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.407025+0000", 2024-09-16T10:17:53.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5099999999999998e-06 2024-09-16T10:17:53.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:53.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.407097+0000", 2024-09-16T10:17:53.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2533000000000005e-05 2024-09-16T10:17:53.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:53.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.407228+0000", 2024-09-16T10:17:53.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013108799999999999 2024-09-16T10:17:53.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:53.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.407361+0000", 2024-09-16T10:17:53.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013280400000000001 2024-09-16T10:17:53.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:53.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.412219+0000", 2024-09-16T10:17:53.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004857452 2024-09-16T10:17:53.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:53.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.416800+0000", 2024-09-16T10:17:53.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045818949999999999 2024-09-16T10:17:53.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.452 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:53.452 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.452 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:213651 7.d 7:b7f34056:::benchmark_data_smithi063_73086_object94372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T10:17:53.452 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:53.417818+0000", 2024-09-16T10:17:53.453 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0045103840000000001, 2024-09-16T10:17:53.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048099340000000001, 2024-09-16T10:17:53.453 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:53.453 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:53.453 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:17:53.453 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:53.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:53.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:53.454 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213651 2024-09-16T10:17:53.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.454 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:53.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:53.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.417818+0000", 2024-09-16T10:17:53.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:53.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.417818+0000", 2024-09-16T10:17:53.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:53.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:53.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.417820+0000", 2024-09-16T10:17:53.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.609e-06 2024-09-16T10:17:53.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:53.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.417825+0000", 2024-09-16T10:17:53.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9069999999999998e-06 2024-09-16T10:17:53.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.417835+0000", 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0644e-05 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.417842+0000", 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5150000000000003e-06 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.418068+0000", 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022634099999999999 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.418156+0000", 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8126000000000003e-05 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.418221+0000", 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4362999999999993e-05 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:53.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:53.421047+0000", 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002826435 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 5 2024-09-16T10:17:53.460 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:17:53.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:53.485 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:53.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:53.588 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:53.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:53.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:53.679 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:17:53.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:53.755 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:53.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:53.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:53.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:53.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:54.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:54.054 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:53 smithi063 ceph-mon[29993]: pgmap v192: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 939 op/s 2024-09-16T10:17:54.055 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:53 smithi063 ceph-mon[39470]: pgmap v192: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 939 op/s 2024-09-16T10:17:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:54.094 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:54.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:54.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:54.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:54.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:54.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:53 smithi165 ceph-mon[32696]: pgmap v192: 148 pgs: 148 active+clean; 1.8 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 939 op/s 2024-09-16T10:17:54.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:54.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:17:54.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:54.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:54.423 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:54.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:54.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:54.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:54.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:54.639 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:54.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:54.706 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:17:54.736 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:17:54.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:54.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:54.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:54.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:54.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:54.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:55.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:55.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:55.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:55.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:55.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:55.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:55.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:55.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:55.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:55.473 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:17:55.473 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:17:55.473 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:17:55.473 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:17:55.473 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:17:55.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:17:55.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:55.580 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:55.682 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:55.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:55.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:55.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:55.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:55.977 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:55.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:56.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:56.120 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:56.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:56.165 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:17:56.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:56.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:56.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:56.254 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T10:17:53.099314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1933187635,"num_objects":29689,"num_object_clones":0,"num_object_copies":89067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29689,"num_whiteouts":0,"num_read":1094,"num_read_kb":951,"num_write":209613,"num_write_kb":7635338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":590698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90560,"ondisk_log_size":90560,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6588192,"kb_used_data":5683296,"kb_used_omap":12,"kb_used_meta":904563,"kb_avail":743373024,"statfs":{"total":767960285184,"available":761213976576,"internally_reserved":0,"allocated":5819695104,"data_stored":5816781123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":926273073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-615383040,"num_objects":-9390,"num_object_clones":0,"num_object_copies":-28170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9390,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9390,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003142"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.821146+0000","last_change":"2024-09-16T10:16:35.216054+0000","last_active":"2024-09-16T10:17:42.821146+0000","last_peered":"2024-09-16T10:17:42.821146+0000","last_clean":"2024-09-16T10:17:42.821146+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:17:42.821146+0000","last_undegraded":"2024-09-16T10:17:42.821146+0000","last_fullsized":"2024-09-16T10:17:42.821146+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:35.215992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.215992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":237,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.903548+0000","last_change":"2024-09-16T10:16:35.813476+0000","last_active":"2024-09-16T10:17:50.903548+0000","last_peered":"2024-09-16T10:17:50.903548+0000","last_clean":"2024-09-16T10:17:50.903548+0000","last_became_active":"2024-09-16T10:15:08.925269+0000","last_became_peered":"2024-09-16T10:15:08.925269+0000","last_unstale":"2024-09-16T10:17:50.903548+0000","last_undegraded":"2024-09-16T10:17:50.903548+0000","last_fullsized":"2024-09-16T10:17:50.903548+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:35.813372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.813372+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:03.819055+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.807505+0000","last_change":"2024-09-16T10:17:04.062121+0000","last_active":"2024-09-16T10:17:42.807505+0000","last_peered":"2024-09-16T10:17:42.807505+0000","last_clean":"2024-09-16T10:17:42.807505+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:17:42.807505+0000","last_undegraded":"2024-09-16T10:17:42.807505+0000","last_fullsized":"2024-09-16T10:17:42.807505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:04.062060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.062060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.222918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.905368+0000","last_change":"2024-09-16T10:16:16.760046+0000","last_active":"2024-09-16T10:17:50.905368+0000","last_peered":"2024-09-16T10:17:50.905368+0000","last_clean":"2024-09-16T10:17:50.905368+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:17:50.905368+0000","last_undegraded":"2024-09-16T10:17:50.905368+0000","last_fullsized":"2024-09-16T10:17:50.905368+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.759981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.759981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811006+0000","last_change":"2024-09-16T10:16:42.246257+0000","last_active":"2024-09-16T10:17:42.811006+0000","last_peered":"2024-09-16T10:17:42.811006+0000","last_clean":"2024-09-16T10:17:42.811006+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:17:42.811006+0000","last_undegraded":"2024-09-16T10:17:42.811006+0000","last_fullsized":"2024-09-16T10:17:42.811006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:42.246193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.246193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.520461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":208,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810799+0000","last_change":"2024-09-16T10:17:13.018487+0000","last_active":"2024-09-16T10:17:42.810799+0000","last_peered":"2024-09-16T10:17:42.810799+0000","last_clean":"2024-09-16T10:17:42.810799+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:17:42.810799+0000","last_undegraded":"2024-09-16T10:17:42.810799+0000","last_fullsized":"2024-09-16T10:17:42.810799+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:13.018412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.018412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.241294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":228,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811348+0000","last_change":"2024-09-16T10:16:31.217281+0000","last_active":"2024-09-16T10:17:42.811348+0000","last_peered":"2024-09-16T10:17:42.811348+0000","last_clean":"2024-09-16T10:17:42.811348+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:17:42.811348+0000","last_undegraded":"2024-09-16T10:17:42.811348+0000","last_fullsized":"2024-09-16T10:17:42.811348+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:16:31.217192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.217192+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.903396+0000","last_change":"2024-09-16T10:17:14.871814+0000","last_active":"2024-09-16T10:17:50.903396+0000","last_peered":"2024-09-16T10:17:50.903396+0000","last_clean":"2024-09-16T10:17:50.903396+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:17:50.903396+0000","last_undegraded":"2024-09-16T10:17:50.903396+0000","last_fullsized":"2024-09-16T10:17:50.903396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.871750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.871750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:31.182111+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.925046+0000","last_change":"2024-09-16T10:15:29.351071+0000","last_active":"2024-09-16T10:17:50.925046+0000","last_peered":"2024-09-16T10:17:50.925046+0000","last_clean":"2024-09-16T10:17:50.925046+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:17:50.925046+0000","last_undegraded":"2024-09-16T10:17:50.925046+0000","last_fullsized":"2024-09-16T10:17:50.925046+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:29.351000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.351000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:01.422071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":220,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904489+0000","last_change":"2024-09-16T10:16:32.833966+0000","last_active":"2024-09-16T10:17:50.904489+0000","last_peered":"2024-09-16T10:17:50.904489+0000","last_clean":"2024-09-16T10:17:50.904489+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:17:50.904489+0000","last_undegraded":"2024-09-16T10:17:50.904489+0000","last_fullsized":"2024-09-16T10:17:50.904489+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:32.833895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.833895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":193,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924803+0000","last_change":"2024-09-16T10:15:22.271162+0000","last_active":"2024-09-16T10:17:50.924803+0000","last_peered":"2024-09-16T10:17:50.924803+0000","last_clean":"2024-09-16T10:17:50.924803+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:17:50.924803+0000","last_undegraded":"2024-09-16T10:17:50.924803+0000","last_fullsized":"2024-09-16T10:17:50.924803+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:22.271101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:22.271101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.398679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":194,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814319+0000","last_change":"2024-09-16T10:17:12.089641+0000","last_active":"2024-09-16T10:17:42.814319+0000","last_peered":"2024-09-16T10:17:42.814319+0000","last_clean":"2024-09-16T10:17:42.814319+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:17:42.814319+0000","last_undegraded":"2024-09-16T10:17:42.814319+0000","last_fullsized":"2024-09-16T10:17:42.814319+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.089577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.089577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.096958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":193,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924263+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:17:50.924263+0000","last_peered":"2024-09-16T10:17:50.924263+0000","last_clean":"2024-09-16T10:17:50.924263+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:17:50.924263+0000","last_undegraded":"2024-09-16T10:17:50.924263+0000","last_fullsized":"2024-09-16T10:17:50.924263+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.593997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":214,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904747+0000","last_change":"2024-09-16T10:17:16.937447+0000","last_active":"2024-09-16T10:17:50.904747+0000","last_peered":"2024-09-16T10:17:50.904747+0000","last_clean":"2024-09-16T10:17:50.904747+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:17:50.904747+0000","last_undegraded":"2024-09-16T10:17:50.904747+0000","last_fullsized":"2024-09-16T10:17:50.904747+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:17:16.937353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.937353+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.410589+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.812685+0000","last_change":"2024-09-16T10:15:16.573685+0000","last_active":"2024-09-16T10:17:42.812685+0000","last_peered":"2024-09-16T10:17:42.812685+0000","last_clean":"2024-09-16T10:17:42.812685+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:17:42.812685+0000","last_undegraded":"2024-09-16T10:17:42.812685+0000","last_fullsized":"2024-09-16T10:17:42.812685+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.573614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.573614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.640699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.923475+0000","last_change":"2024-09-16T10:15:11.297702+0000","last_active":"2024-09-16T10:17:50.923475+0000","last_peered":"2024-09-16T10:17:50.923475+0000","last_clean":"2024-09-16T10:17:50.923475+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:17:50.923475+0000","last_undegraded":"2024-09-16T10:17:50.923475+0000","last_fullsized":"2024-09-16T10:17:50.923475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.297599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.297599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:55.375459+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.905288+0000","last_change":"2024-09-16T10:17:48.893103+0000","last_active":"2024-09-16T10:17:50.905288+0000","last_peered":"2024-09-16T10:17:50.905288+0000","last_clean":"2024-09-16T10:17:50.905288+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:17:50.905288+0000","last_undegraded":"2024-09-16T10:17:50.905288+0000","last_fullsized":"2024-09-16T10:17:50.905288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:48.893042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:48.893042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:18.645144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815692+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:17:42.815692+0000","last_peered":"2024-09-16T10:17:42.815692+0000","last_clean":"2024-09-16T10:17:42.815692+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:17:42.815692+0000","last_undegraded":"2024-09-16T10:17:42.815692+0000","last_fullsized":"2024-09-16T10:17:42.815692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.208132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":212,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815110+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:17:42.815110+0000","last_peered":"2024-09-16T10:17:42.815110+0000","last_clean":"2024-09-16T10:17:42.815110+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:17:42.815110+0000","last_undegraded":"2024-09-16T10:17:42.815110+0000","last_fullsized":"2024-09-16T10:17:42.815110+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:28.390262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.819013+0000","last_change":"2024-09-16T10:16:16.067128+0000","last_active":"2024-09-16T10:17:42.819013+0000","last_peered":"2024-09-16T10:17:42.819013+0000","last_clean":"2024-09-16T10:17:42.819013+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:17:42.819013+0000","last_undegraded":"2024-09-16T10:17:42.819013+0000","last_fullsized":"2024-09-16T10:17:42.819013+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.066855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.066855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.192100+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":27,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.978180+0000","last_change":"2024-09-16T10:17:42.978180+0000","last_active":"2024-09-16T10:17:42.978180+0000","last_peered":"2024-09-16T10:17:42.978180+0000","last_clean":"2024-09-16T10:17:42.978180+0000","last_became_active":"2024-09-16T10:17:37.309204+0000","last_became_peered":"2024-09-16T10:17:37.309204+0000","last_unstale":"2024-09-16T10:17:42.978180+0000","last_undegraded":"2024-09-16T10:17:42.978180+0000","last_fullsized":"2024-09-16T10:17:42.978180+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:42.978105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.978105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:49.488053+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904634+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:17:50.904634+0000","last_peered":"2024-09-16T10:17:50.904634+0000","last_clean":"2024-09-16T10:17:50.904634+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:17:50.904634+0000","last_undegraded":"2024-09-16T10:17:50.904634+0000","last_fullsized":"2024-09-16T10:17:50.904634+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":245,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.809821+0000","last_change":"2024-09-16T10:16:35.181066+0000","last_active":"2024-09-16T10:17:42.809821+0000","last_peered":"2024-09-16T10:17:42.809821+0000","last_clean":"2024-09-16T10:17:42.809821+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:17:42.809821+0000","last_undegraded":"2024-09-16T10:17:42.809821+0000","last_fullsized":"2024-09-16T10:17:42.809821+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:16:35.180977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.180977+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904241+0000","last_change":"2024-09-16T10:16:31.785080+0000","last_active":"2024-09-16T10:17:50.904241+0000","last_peered":"2024-09-16T10:17:50.904241+0000","last_clean":"2024-09-16T10:17:50.904241+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:17:50.904241+0000","last_undegraded":"2024-09-16T10:17:50.904241+0000","last_fullsized":"2024-09-16T10:17:50.904241+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:31.784724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.784724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":188,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813755+0000","last_change":"2024-09-16T10:16:37.672462+0000","last_active":"2024-09-16T10:17:42.813755+0000","last_peered":"2024-09-16T10:17:42.813755+0000","last_clean":"2024-09-16T10:17:42.813755+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:17:42.813755+0000","last_undegraded":"2024-09-16T10:17:42.813755+0000","last_fullsized":"2024-09-16T10:17:42.813755+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.246050+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":214,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.807648+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:17:42.807648+0000","last_peered":"2024-09-16T10:17:42.807648+0000","last_clean":"2024-09-16T10:17:42.807648+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:17:42.807648+0000","last_undegraded":"2024-09-16T10:17:42.807648+0000","last_fullsized":"2024-09-16T10:17:42.807648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.691808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":218,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.818454+0000","last_change":"2024-09-16T10:17:35.057981+0000","last_active":"2024-09-16T10:17:42.818454+0000","last_peered":"2024-09-16T10:17:42.818454+0000","last_clean":"2024-09-16T10:17:42.818454+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:17:42.818454+0000","last_undegraded":"2024-09-16T10:17:42.818454+0000","last_fullsized":"2024-09-16T10:17:42.818454+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:35.057906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.057906+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.808000+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":260,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810787+0000","last_change":"2024-09-16T10:16:37.249076+0000","last_active":"2024-09-16T10:17:42.810787+0000","last_peered":"2024-09-16T10:17:42.810787+0000","last_clean":"2024-09-16T10:17:42.810787+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:17:42.810787+0000","last_undegraded":"2024-09-16T10:17:42.810787+0000","last_fullsized":"2024-09-16T10:17:42.810787+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:16:37.249033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.249033+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.553977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"148'13537","reported_seq":13683,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.106291+0000","last_change":"2024-09-16T10:14:43.957473+0000","last_active":"2024-09-16T10:17:48.106291+0000","last_peered":"2024-09-16T10:17:48.106291+0000","last_clean":"2024-09-16T10:17:48.106291+0000","last_became_active":"2024-09-16T10:14:43.957174+0000","last_became_peered":"2024-09-16T10:14:43.957174+0000","last_unstale":"2024-09-16T10:17:48.106291+0000","last_undegraded":"2024-09-16T10:17:48.106291+0000","last_fullsized":"2024-09-16T10:17:48.106291+0000","mapping_epoch":81,"log_start":"116'7200","ondisk_log_start":"116'7200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.202365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132055040,"num_objects":2015,"num_object_clones":0,"num_object_copies":6045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13537,"num_write_kb":497664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":220,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.807792+0000","last_change":"2024-09-16T10:14:45.136557+0000","last_active":"2024-09-16T10:17:42.807792+0000","last_peered":"2024-09-16T10:17:42.807792+0000","last_clean":"2024-09-16T10:17:42.807792+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:17:42.807792+0000","last_undegraded":"2024-09-16T10:17:42.807792+0000","last_fullsized":"2024-09-16T10:17:42.807792+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:14:45.136460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.136460+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.941516+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:45.221342+0000","last_change":"2024-09-16T10:17:45.221342+0000","last_active":"2024-09-16T10:17:45.221342+0000","last_peered":"2024-09-16T10:17:45.221342+0000","last_clean":"2024-09-16T10:17:45.221342+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:17:45.221342+0000","last_undegraded":"2024-09-16T10:17:45.221342+0000","last_fullsized":"2024-09-16T10:17:45.221342+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.221281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.221281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:57.786575+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":244,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904991+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:17:50.904991+0000","last_peered":"2024-09-16T10:17:50.904991+0000","last_clean":"2024-09-16T10:17:50.904991+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:17:50.904991+0000","last_undegraded":"2024-09-16T10:17:50.904991+0000","last_fullsized":"2024-09-16T10:17:50.904991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.935259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924574+0000","last_change":"2024-09-16T10:15:54.314741+0000","last_active":"2024-09-16T10:17:50.924574+0000","last_peered":"2024-09-16T10:17:50.924574+0000","last_clean":"2024-09-16T10:17:50.924574+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:17:50.924574+0000","last_undegraded":"2024-09-16T10:17:50.924574+0000","last_fullsized":"2024-09-16T10:17:50.924574+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:54.314680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.314680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.427634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"148'12958","reported_seq":13183,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.370243+0000","last_change":"2024-09-16T10:16:16.640532+0000","last_active":"2024-09-16T10:17:48.370243+0000","last_peered":"2024-09-16T10:17:48.370243+0000","last_clean":"2024-09-16T10:17:48.370243+0000","last_became_active":"2024-09-16T10:14:43.952713+0000","last_became_peered":"2024-09-16T10:14:43.952713+0000","last_unstale":"2024-09-16T10:17:48.370243+0000","last_undegraded":"2024-09-16T10:17:48.370243+0000","last_fullsized":"2024-09-16T10:17:48.370243+0000","mapping_epoch":81,"log_start":"125'8400","ondisk_log_start":"125'8400","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7146","last_scrub_stamp":"2024-09-16T10:16:16.640092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.640092+0000","objects_scrubbed":7079,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":1856,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12958,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":228,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811771+0000","last_change":"2024-09-16T10:15:23.703368+0000","last_active":"2024-09-16T10:17:42.811771+0000","last_peered":"2024-09-16T10:17:42.811771+0000","last_clean":"2024-09-16T10:17:42.811771+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:17:42.811771+0000","last_undegraded":"2024-09-16T10:17:42.811771+0000","last_fullsized":"2024-09-16T10:17:42.811771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:15:23.703278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.703278+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.166045+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.812916+0000","last_change":"2024-09-16T10:17:38.748281+0000","last_active":"2024-09-16T10:17:42.812916+0000","last_peered":"2024-09-16T10:17:42.812916+0000","last_clean":"2024-09-16T10:17:42.812916+0000","last_became_active":"2024-09-16T10:17:37.306205+0000","last_became_peered":"2024-09-16T10:17:37.306205+0000","last_unstale":"2024-09-16T10:17:42.812916+0000","last_undegraded":"2024-09-16T10:17:42.812916+0000","last_fullsized":"2024-09-16T10:17:42.812916+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:38.748219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.748219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:05.492417+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":185,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904232+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:17:50.904232+0000","last_peered":"2024-09-16T10:17:50.904232+0000","last_clean":"2024-09-16T10:17:50.904232+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:17:50.904232+0000","last_undegraded":"2024-09-16T10:17:50.904232+0000","last_fullsized":"2024-09-16T10:17:50.904232+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:17.909064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904438+0000","last_change":"2024-09-16T10:16:43.906280+0000","last_active":"2024-09-16T10:17:50.904438+0000","last_peered":"2024-09-16T10:17:50.904438+0000","last_clean":"2024-09-16T10:17:50.904438+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:17:50.904438+0000","last_undegraded":"2024-09-16T10:17:50.904438+0000","last_fullsized":"2024-09-16T10:17:50.904438+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.906200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.906200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"148'12974","reported_seq":13120,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.366577+0000","last_change":"2024-09-16T10:14:43.953541+0000","last_active":"2024-09-16T10:17:48.366577+0000","last_peered":"2024-09-16T10:17:48.366577+0000","last_clean":"2024-09-16T10:17:48.366577+0000","last_became_active":"2024-09-16T10:14:43.952908+0000","last_became_peered":"2024-09-16T10:14:43.952908+0000","last_unstale":"2024-09-16T10:17:48.366577+0000","last_undegraded":"2024-09-16T10:17:48.366577+0000","last_fullsized":"2024-09-16T10:17:48.366577+0000","mapping_epoch":81,"log_start":"125'8339","ondisk_log_start":"125'8339","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.179329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121372672,"num_objects":1852,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12974,"num_write_kb":474432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904279+0000","last_change":"2024-09-16T10:16:37.847991+0000","last_active":"2024-09-16T10:17:50.904279+0000","last_peered":"2024-09-16T10:17:50.904279+0000","last_clean":"2024-09-16T10:17:50.904279+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:17:50.904279+0000","last_undegraded":"2024-09-16T10:17:50.904279+0000","last_fullsized":"2024-09-16T10:17:50.904279+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:16:37.847913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.847913+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.083962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.818675+0000","last_change":"2024-09-16T10:17:11.050221+0000","last_active":"2024-09-16T10:17:42.818675+0000","last_peered":"2024-09-16T10:17:42.818675+0000","last_clean":"2024-09-16T10:17:42.818675+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:17:42.818675+0000","last_undegraded":"2024-09-16T10:17:42.818675+0000","last_fullsized":"2024-09-16T10:17:42.818675+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:17:11.050150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:11.050150+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:29.263200+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":254,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:43.377416+0000","last_change":"2024-09-16T10:15:20.675679+0000","last_active":"2024-09-16T10:17:43.377416+0000","last_peered":"2024-09-16T10:17:43.377416+0000","last_clean":"2024-09-16T10:17:43.377416+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:17:43.377416+0000","last_undegraded":"2024-09-16T10:17:43.377416+0000","last_fullsized":"2024-09-16T10:17:43.377416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:20.675600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:20.675600+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.107238+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.812963+0000","last_change":"2024-09-16T10:16:03.498672+0000","last_active":"2024-09-16T10:17:42.812963+0000","last_peered":"2024-09-16T10:17:42.812963+0000","last_clean":"2024-09-16T10:17:42.812963+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:17:42.812963+0000","last_undegraded":"2024-09-16T10:17:42.812963+0000","last_fullsized":"2024-09-16T10:17:42.812963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:21.723374+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"149'13458","reported_seq":13639,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:53.044228+0000","last_change":"2024-09-16T10:17:37.439115+0000","last_active":"2024-09-16T10:17:53.044228+0000","last_peered":"2024-09-16T10:17:53.044228+0000","last_clean":"2024-09-16T10:17:53.044228+0000","last_became_active":"2024-09-16T10:14:43.952043+0000","last_became_peered":"2024-09-16T10:14:43.952043+0000","last_unstale":"2024-09-16T10:17:53.044228+0000","last_undegraded":"2024-09-16T10:17:53.044228+0000","last_fullsized":"2024-09-16T10:17:53.044228+0000","mapping_epoch":81,"log_start":"126'8600","ondisk_log_start":"126'8600","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'12535","last_scrub_stamp":"2024-09-16T10:17:37.438837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:37.438837+0000","objects_scrubbed":2521,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.510192+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103677952,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13458,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":226,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:44.394734+0000","last_change":"2024-09-16T10:17:44.394734+0000","last_active":"2024-09-16T10:17:44.394734+0000","last_peered":"2024-09-16T10:17:44.394734+0000","last_clean":"2024-09-16T10:17:44.394734+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:17:44.394734+0000","last_undegraded":"2024-09-16T10:17:44.394734+0000","last_fullsized":"2024-09-16T10:17:44.394734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:44.392303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.392303+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:47.877424+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":202,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.925172+0000","last_change":"2024-09-16T10:15:23.304088+0000","last_active":"2024-09-16T10:17:50.925172+0000","last_peered":"2024-09-16T10:17:50.925172+0000","last_clean":"2024-09-16T10:17:50.925172+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:17:50.925172+0000","last_undegraded":"2024-09-16T10:17:50.925172+0000","last_fullsized":"2024-09-16T10:17:50.925172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:23.304033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:23.304033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":249,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815200+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:17:42.815200+0000","last_peered":"2024-09-16T10:17:42.815200+0000","last_clean":"2024-09-16T10:17:42.815200+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:17:42.815200+0000","last_undegraded":"2024-09-16T10:17:42.815200+0000","last_fullsized":"2024-09-16T10:17:42.815200+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.998943+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814540+0000","last_change":"2024-09-16T10:15:16.104168+0000","last_active":"2024-09-16T10:17:42.814540+0000","last_peered":"2024-09-16T10:17:42.814540+0000","last_clean":"2024-09-16T10:17:42.814540+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:17:42.814540+0000","last_undegraded":"2024-09-16T10:17:42.814540+0000","last_fullsized":"2024-09-16T10:17:42.814540+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.104092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.104092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"149'13125","reported_seq":13377,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:53.019821+0000","last_change":"2024-09-16T10:17:39.034260+0000","last_active":"2024-09-16T10:17:53.019821+0000","last_peered":"2024-09-16T10:17:53.019821+0000","last_clean":"2024-09-16T10:17:53.019821+0000","last_became_active":"2024-09-16T10:14:43.955808+0000","last_became_peered":"2024-09-16T10:14:43.955808+0000","last_unstale":"2024-09-16T10:17:53.019821+0000","last_undegraded":"2024-09-16T10:17:53.019821+0000","last_fullsized":"2024-09-16T10:17:53.019821+0000","mapping_epoch":81,"log_start":"119'7300","ondisk_log_start":"119'7300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'12331","last_scrub_stamp":"2024-09-16T10:17:39.034057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:39.034057+0000","objects_scrubbed":2480,"log_size":5825,"log_dups_size":0,"ondisk_log_size":5825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:57.622104+0000","scrub_duration":582,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109117440,"num_objects":1665,"num_object_clones":0,"num_object_copies":4995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13125,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":239,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815362+0000","last_change":"2024-09-16T10:17:03.051621+0000","last_active":"2024-09-16T10:17:42.815362+0000","last_peered":"2024-09-16T10:17:42.815362+0000","last_clean":"2024-09-16T10:17:42.815362+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:17:42.815362+0000","last_undegraded":"2024-09-16T10:17:42.815362+0000","last_fullsized":"2024-09-16T10:17:42.815362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:17:03.050732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.050732+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.857216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":208,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.816163+0000","last_change":"2024-09-16T10:16:33.949944+0000","last_active":"2024-09-16T10:17:42.816163+0000","last_peered":"2024-09-16T10:17:42.816163+0000","last_clean":"2024-09-16T10:17:42.816163+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:17:42.816163+0000","last_undegraded":"2024-09-16T10:17:42.816163+0000","last_fullsized":"2024-09-16T10:17:42.816163+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:33.949798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.949798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.958848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810202+0000","last_change":"2024-09-16T10:16:07.174774+0000","last_active":"2024-09-16T10:17:42.810202+0000","last_peered":"2024-09-16T10:17:42.810202+0000","last_clean":"2024-09-16T10:17:42.810202+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:17:42.810202+0000","last_undegraded":"2024-09-16T10:17:42.810202+0000","last_fullsized":"2024-09-16T10:17:42.810202+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:07.174690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:07.174690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:34.045269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.807075+0000","last_change":"2024-09-16T10:17:03.061305+0000","last_active":"2024-09-16T10:17:42.807075+0000","last_peered":"2024-09-16T10:17:42.807075+0000","last_clean":"2024-09-16T10:17:42.807075+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:17:42.807075+0000","last_undegraded":"2024-09-16T10:17:42.807075+0000","last_fullsized":"2024-09-16T10:17:42.807075+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.257731+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"148'13136","reported_seq":13360,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.081503+0000","last_change":"2024-09-16T10:16:14.720883+0000","last_active":"2024-09-16T10:17:48.081503+0000","last_peered":"2024-09-16T10:17:48.081503+0000","last_clean":"2024-09-16T10:17:48.081503+0000","last_became_active":"2024-09-16T10:14:43.957195+0000","last_became_peered":"2024-09-16T10:14:43.957195+0000","last_unstale":"2024-09-16T10:17:48.081503+0000","last_undegraded":"2024-09-16T10:17:48.081503+0000","last_fullsized":"2024-09-16T10:17:48.081503+0000","mapping_epoch":81,"log_start":"115'6800","ondisk_log_start":"115'6800","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7198","last_scrub_stamp":"2024-09-16T10:16:14.720775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.720775+0000","objects_scrubbed":7123,"log_size":6336,"log_dups_size":0,"ondisk_log_size":6336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:33.046935+0000","scrub_duration":2606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126615552,"num_objects":1932,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13136,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":310,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815106+0000","last_change":"2024-09-16T10:17:15.133634+0000","last_active":"2024-09-16T10:17:42.815106+0000","last_peered":"2024-09-16T10:17:42.815106+0000","last_clean":"2024-09-16T10:17:42.815106+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:17:42.815106+0000","last_undegraded":"2024-09-16T10:17:42.815106+0000","last_fullsized":"2024-09-16T10:17:42.815106+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:17:15.133542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.133542+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.091484+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.905053+0000","last_change":"2024-09-16T10:17:35.928758+0000","last_active":"2024-09-16T10:17:50.905053+0000","last_peered":"2024-09-16T10:17:50.905053+0000","last_clean":"2024-09-16T10:17:50.905053+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:17:50.905053+0000","last_undegraded":"2024-09-16T10:17:50.905053+0000","last_fullsized":"2024-09-16T10:17:50.905053+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.928701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.928701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:03.464982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":220,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924186+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:17:50.924186+0000","last_peered":"2024-09-16T10:17:50.924186+0000","last_clean":"2024-09-16T10:17:50.924186+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:17:50.924186+0000","last_undegraded":"2024-09-16T10:17:50.924186+0000","last_fullsized":"2024-09-16T10:17:50.924186+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:02.800871+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.820549+0000","last_change":"2024-09-16T10:16:04.405318+0000","last_active":"2024-09-16T10:17:42.820549+0000","last_peered":"2024-09-16T10:17:42.820549+0000","last_clean":"2024-09-16T10:17:42.820549+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:17:42.820549+0000","last_undegraded":"2024-09-16T10:17:42.820549+0000","last_fullsized":"2024-09-16T10:17:42.820549+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:04.405243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.405243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.095983+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810544+0000","last_change":"2024-09-16T10:17:14.029861+0000","last_active":"2024-09-16T10:17:42.810544+0000","last_peered":"2024-09-16T10:17:42.810544+0000","last_clean":"2024-09-16T10:17:42.810544+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:17:42.810544+0000","last_undegraded":"2024-09-16T10:17:42.810544+0000","last_fullsized":"2024-09-16T10:17:42.810544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.029754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.029754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.182687+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"148'12942","reported_seq":13088,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.326189+0000","last_change":"2024-09-16T10:14:43.956747+0000","last_active":"2024-09-16T10:17:48.326189+0000","last_peered":"2024-09-16T10:17:48.326189+0000","last_clean":"2024-09-16T10:17:48.326189+0000","last_became_active":"2024-09-16T10:14:43.956362+0000","last_became_peered":"2024-09-16T10:14:43.956362+0000","last_unstale":"2024-09-16T10:17:48.326189+0000","last_undegraded":"2024-09-16T10:17:48.326189+0000","last_fullsized":"2024-09-16T10:17:48.326189+0000","mapping_epoch":81,"log_start":"119'7300","ondisk_log_start":"119'7300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12942,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":214,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.807893+0000","last_change":"2024-09-16T10:13:39.094685+0000","last_active":"2024-09-16T10:17:42.807893+0000","last_peered":"2024-09-16T10:17:42.807893+0000","last_clean":"2024-09-16T10:17:42.807893+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:17:42.807893+0000","last_undegraded":"2024-09-16T10:17:42.807893+0000","last_fullsized":"2024-09-16T10:17:42.807893+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:49.895956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":192,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.807957+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:17:42.807957+0000","last_peered":"2024-09-16T10:17:42.807957+0000","last_clean":"2024-09-16T10:17:42.807957+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:17:42.807957+0000","last_undegraded":"2024-09-16T10:17:42.807957+0000","last_fullsized":"2024-09-16T10:17:42.807957+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.883706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":217,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:47.455302+0000","last_change":"2024-09-16T10:17:47.455218+0000","last_active":"2024-09-16T10:17:47.455302+0000","last_peered":"2024-09-16T10:17:47.455302+0000","last_clean":"2024-09-16T10:17:47.455302+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:17:47.455302+0000","last_undegraded":"2024-09-16T10:17:47.455302+0000","last_fullsized":"2024-09-16T10:17:47.455302+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:17:47.455142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:47.455142+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:15.928277+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":183,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815359+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:17:42.815359+0000","last_peered":"2024-09-16T10:17:42.815359+0000","last_clean":"2024-09-16T10:17:42.815359+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:17:42.815359+0000","last_undegraded":"2024-09-16T10:17:42.815359+0000","last_fullsized":"2024-09-16T10:17:42.815359+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.471638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813959+0000","last_change":"2024-09-16T10:17:12.578269+0000","last_active":"2024-09-16T10:17:42.813959+0000","last_peered":"2024-09-16T10:17:42.813959+0000","last_clean":"2024-09-16T10:17:42.813959+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:17:42.813959+0000","last_undegraded":"2024-09-16T10:17:42.813959+0000","last_fullsized":"2024-09-16T10:17:42.813959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.578203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.578203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.857634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"148'13158","reported_seq":13303,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.091978+0000","last_change":"2024-09-16T10:14:43.959854+0000","last_active":"2024-09-16T10:17:48.091978+0000","last_peered":"2024-09-16T10:17:48.091978+0000","last_clean":"2024-09-16T10:17:48.091978+0000","last_became_active":"2024-09-16T10:14:43.959612+0000","last_became_peered":"2024-09-16T10:14:43.959612+0000","last_unstale":"2024-09-16T10:17:48.091978+0000","last_undegraded":"2024-09-16T10:17:48.091978+0000","last_fullsized":"2024-09-16T10:17:48.091978+0000","mapping_epoch":81,"log_start":"122'7800","ondisk_log_start":"122'7800","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5358,"log_dups_size":0,"ondisk_log_size":5358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:50.620169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122552320,"num_objects":1870,"num_object_clones":0,"num_object_copies":5610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13158,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":228,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:44.234787+0000","last_change":"2024-09-16T10:17:44.234787+0000","last_active":"2024-09-16T10:17:44.234787+0000","last_peered":"2024-09-16T10:17:44.234787+0000","last_clean":"2024-09-16T10:17:44.234787+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:17:44.234787+0000","last_undegraded":"2024-09-16T10:17:44.234787+0000","last_fullsized":"2024-09-16T10:17:44.234787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:44.234584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.234584+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:05.324187+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":215,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814809+0000","last_change":"2024-09-16T10:17:15.102575+0000","last_active":"2024-09-16T10:17:42.814809+0000","last_peered":"2024-09-16T10:17:42.814809+0000","last_clean":"2024-09-16T10:17:42.814809+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:17:42.814809+0000","last_undegraded":"2024-09-16T10:17:42.814809+0000","last_fullsized":"2024-09-16T10:17:42.814809+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:15.102512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.102512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.326101+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":464,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:46.304109+0000","last_change":"2024-09-16T10:16:05.559992+0000","last_active":"2024-09-16T10:17:46.304109+0000","last_peered":"2024-09-16T10:17:46.304109+0000","last_clean":"2024-09-16T10:17:46.304109+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:17:46.304109+0000","last_undegraded":"2024-09-16T10:17:46.304109+0000","last_fullsized":"2024-09-16T10:17:46.304109+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:16:05.559913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.559913+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:20.685013+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.925647+0000","last_change":"2024-09-16T10:17:35.559111+0000","last_active":"2024-09-16T10:17:50.925647+0000","last_peered":"2024-09-16T10:17:50.925647+0000","last_clean":"2024-09-16T10:17:50.925647+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:17:50.925647+0000","last_undegraded":"2024-09-16T10:17:50.925647+0000","last_fullsized":"2024-09-16T10:17:50.925647+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.559045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.559045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:44.355200+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.807352+0000","last_change":"2024-09-16T10:17:16.162854+0000","last_active":"2024-09-16T10:17:42.807352+0000","last_peered":"2024-09-16T10:17:42.807352+0000","last_clean":"2024-09-16T10:17:42.807352+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:17:42.807352+0000","last_undegraded":"2024-09-16T10:17:42.807352+0000","last_fullsized":"2024-09-16T10:17:42.807352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:16.162771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.162771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:36.273519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"148'13346","reported_seq":13487,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.323333+0000","last_change":"2024-09-16T10:14:43.957718+0000","last_active":"2024-09-16T10:17:48.323333+0000","last_peered":"2024-09-16T10:17:48.323333+0000","last_clean":"2024-09-16T10:17:48.323333+0000","last_became_active":"2024-09-16T10:14:43.957476+0000","last_became_peered":"2024-09-16T10:14:43.957476+0000","last_unstale":"2024-09-16T10:17:48.323333+0000","last_undegraded":"2024-09-16T10:17:48.323333+0000","last_fullsized":"2024-09-16T10:17:48.323333+0000","mapping_epoch":81,"log_start":"120'7700","ondisk_log_start":"120'7700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.193673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129105920,"num_objects":1970,"num_object_clones":0,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13346,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":267,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813231+0000","last_change":"2024-09-16T10:16:10.182532+0000","last_active":"2024-09-16T10:17:42.813231+0000","last_peered":"2024-09-16T10:17:42.813231+0000","last_clean":"2024-09-16T10:17:42.813231+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:17:42.813231+0000","last_undegraded":"2024-09-16T10:17:42.813231+0000","last_fullsized":"2024-09-16T10:17:42.813231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:16:10.182468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:10.182468+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.968626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924025+0000","last_change":"2024-09-16T10:17:43.413171+0000","last_active":"2024-09-16T10:17:50.924025+0000","last_peered":"2024-09-16T10:17:50.924025+0000","last_clean":"2024-09-16T10:17:50.924025+0000","last_became_active":"2024-09-16T10:17:37.298525+0000","last_became_peered":"2024-09-16T10:17:37.298525+0000","last_unstale":"2024-09-16T10:17:50.924025+0000","last_undegraded":"2024-09-16T10:17:50.924025+0000","last_fullsized":"2024-09-16T10:17:50.924025+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.413088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.413088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:54.071484+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":245,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814335+0000","last_change":"2024-09-16T10:16:42.677394+0000","last_active":"2024-09-16T10:17:42.814335+0000","last_peered":"2024-09-16T10:17:42.814335+0000","last_clean":"2024-09-16T10:17:42.814335+0000","last_became_active":"2024-09-16T10:13:30.532072+0000","last_became_peered":"2024-09-16T10:13:30.532072+0000","last_unstale":"2024-09-16T10:17:42.814335+0000","last_undegraded":"2024-09-16T10:17:42.814335+0000","last_fullsized":"2024-09-16T10:17:42.814335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:16:42.677309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.677309+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:10.457942+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813910+0000","last_change":"2024-09-16T10:16:43.622567+0000","last_active":"2024-09-16T10:17:42.813910+0000","last_peered":"2024-09-16T10:17:42.813910+0000","last_clean":"2024-09-16T10:17:42.813910+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:17:42.813910+0000","last_undegraded":"2024-09-16T10:17:42.813910+0000","last_fullsized":"2024-09-16T10:17:42.813910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.773945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":189,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.925436+0000","last_change":"2024-09-16T10:16:34.311115+0000","last_active":"2024-09-16T10:17:50.925436+0000","last_peered":"2024-09-16T10:17:50.925436+0000","last_clean":"2024-09-16T10:17:50.925436+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:17:50.925436+0000","last_undegraded":"2024-09-16T10:17:50.925436+0000","last_fullsized":"2024-09-16T10:17:50.925436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:34.311052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:34.311052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.994480+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"148'13198","reported_seq":13345,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.075668+0000","last_change":"2024-09-16T10:14:43.958855+0000","last_active":"2024-09-16T10:17:48.075668+0000","last_peered":"2024-09-16T10:17:48.075668+0000","last_clean":"2024-09-16T10:17:48.075668+0000","last_became_active":"2024-09-16T10:14:43.958616+0000","last_became_peered":"2024-09-16T10:14:43.958616+0000","last_unstale":"2024-09-16T10:17:48.075668+0000","last_undegraded":"2024-09-16T10:17:48.075668+0000","last_fullsized":"2024-09-16T10:17:48.075668+0000","mapping_epoch":81,"log_start":"115'6900","ondisk_log_start":"115'6900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.193911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122486808,"num_objects":1870,"num_object_clones":0,"num_object_copies":5610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13198,"num_write_kb":482113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":223,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.905044+0000","last_change":"2024-09-16T10:17:06.903885+0000","last_active":"2024-09-16T10:17:50.905044+0000","last_peered":"2024-09-16T10:17:50.905044+0000","last_clean":"2024-09-16T10:17:50.905044+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:17:50.905044+0000","last_undegraded":"2024-09-16T10:17:50.905044+0000","last_fullsized":"2024-09-16T10:17:50.905044+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:06.903808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.903808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:21.156884+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904593+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:17:50.904593+0000","last_peered":"2024-09-16T10:17:50.904593+0000","last_clean":"2024-09-16T10:17:50.904593+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:17:50.904593+0000","last_undegraded":"2024-09-16T10:17:50.904593+0000","last_fullsized":"2024-09-16T10:17:50.904593+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.838918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":193,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.925349+0000","last_change":"2024-09-16T10:15:30.302921+0000","last_active":"2024-09-16T10:17:50.925349+0000","last_peered":"2024-09-16T10:17:50.925349+0000","last_clean":"2024-09-16T10:17:50.925349+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:17:50.925349+0000","last_undegraded":"2024-09-16T10:17:50.925349+0000","last_fullsized":"2024-09-16T10:17:50.925349+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.302859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.302859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":27,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:47.013243+0000","last_change":"2024-09-16T10:17:47.013243+0000","last_active":"2024-09-16T10:17:47.013243+0000","last_peered":"2024-09-16T10:17:47.013243+0000","last_clean":"2024-09-16T10:17:47.013243+0000","last_became_active":"2024-09-16T10:17:37.302404+0000","last_became_peered":"2024-09-16T10:17:37.302404+0000","last_unstale":"2024-09-16T10:17:47.013243+0000","last_undegraded":"2024-09-16T10:17:47.013243+0000","last_fullsized":"2024-09-16T10:17:47.013243+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:47.013173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:47.013173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:07.781255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"148'12728","reported_seq":12943,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.281364+0000","last_change":"2024-09-16T10:15:56.898348+0000","last_active":"2024-09-16T10:17:48.281364+0000","last_peered":"2024-09-16T10:17:48.281364+0000","last_clean":"2024-09-16T10:17:48.281364+0000","last_became_active":"2024-09-16T10:14:43.953871+0000","last_became_peered":"2024-09-16T10:14:43.953871+0000","last_unstale":"2024-09-16T10:17:48.281364+0000","last_undegraded":"2024-09-16T10:17:48.281364+0000","last_fullsized":"2024-09-16T10:17:48.281364+0000","mapping_epoch":81,"log_start":"114'5900","ondisk_log_start":"114'5900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'5944","last_scrub_stamp":"2024-09-16T10:15:56.897937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.897937+0000","objects_scrubbed":5880,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:26.380881+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113770496,"num_objects":1736,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12728,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":216,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904543+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:17:50.904543+0000","last_peered":"2024-09-16T10:17:50.904543+0000","last_clean":"2024-09-16T10:17:50.904543+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:17:50.904543+0000","last_undegraded":"2024-09-16T10:17:50.904543+0000","last_fullsized":"2024-09-16T10:17:50.904543+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:31.892247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810416+0000","last_change":"2024-09-16T10:16:05.172896+0000","last_active":"2024-09-16T10:17:42.810416+0000","last_peered":"2024-09-16T10:17:42.810416+0000","last_clean":"2024-09-16T10:17:42.810416+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:17:42.810416+0000","last_undegraded":"2024-09-16T10:17:42.810416+0000","last_fullsized":"2024-09-16T10:17:42.810416+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:05.172834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.172834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:18.062065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813286+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:17:42.813286+0000","last_peered":"2024-09-16T10:17:42.813286+0000","last_clean":"2024-09-16T10:17:42.813286+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:17:42.813286+0000","last_undegraded":"2024-09-16T10:17:42.813286+0000","last_fullsized":"2024-09-16T10:17:42.813286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.117933+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814810+0000","last_change":"2024-09-16T10:15:11.699384+0000","last_active":"2024-09-16T10:17:42.814810+0000","last_peered":"2024-09-16T10:17:42.814810+0000","last_clean":"2024-09-16T10:17:42.814810+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:17:42.814810+0000","last_undegraded":"2024-09-16T10:17:42.814810+0000","last_fullsized":"2024-09-16T10:17:42.814810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.699308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.699308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.368765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"148'12705","reported_seq":12958,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.359668+0000","last_change":"2024-09-16T10:17:44.054053+0000","last_active":"2024-09-16T10:17:48.359668+0000","last_peered":"2024-09-16T10:17:48.359668+0000","last_clean":"2024-09-16T10:17:48.359668+0000","last_became_active":"2024-09-16T10:14:43.954996+0000","last_became_peered":"2024-09-16T10:14:43.954996+0000","last_unstale":"2024-09-16T10:17:48.359668+0000","last_undegraded":"2024-09-16T10:17:48.359668+0000","last_fullsized":"2024-09-16T10:17:48.359668+0000","mapping_epoch":81,"log_start":"119'7100","ondisk_log_start":"119'7100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'12448","last_scrub_stamp":"2024-09-16T10:17:44.053912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.053912+0000","objects_scrubbed":2146,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:52.599103+0000","scrub_duration":827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122093568,"num_objects":1863,"num_object_clones":0,"num_object_copies":5589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12705,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904326+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:17:50.904326+0000","last_peered":"2024-09-16T10:17:50.904326+0000","last_clean":"2024-09-16T10:17:50.904326+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:17:50.904326+0000","last_undegraded":"2024-09-16T10:17:50.904326+0000","last_fullsized":"2024-09-16T10:17:50.904326+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.917544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811377+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:17:42.811377+0000","last_peered":"2024-09-16T10:17:42.811377+0000","last_clean":"2024-09-16T10:17:42.811377+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:17:42.811377+0000","last_undegraded":"2024-09-16T10:17:42.811377+0000","last_fullsized":"2024-09-16T10:17:42.811377+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.173364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":242,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814945+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:17:42.814945+0000","last_peered":"2024-09-16T10:17:42.814945+0000","last_clean":"2024-09-16T10:17:42.814945+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:17:42.814945+0000","last_undegraded":"2024-09-16T10:17:42.814945+0000","last_fullsized":"2024-09-16T10:17:42.814945+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.430003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904847+0000","last_change":"2024-09-16T10:17:43.963680+0000","last_active":"2024-09-16T10:17:50.904847+0000","last_peered":"2024-09-16T10:17:50.904847+0000","last_clean":"2024-09-16T10:17:50.904847+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:17:50.904847+0000","last_undegraded":"2024-09-16T10:17:50.904847+0000","last_fullsized":"2024-09-16T10:17:50.904847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.963621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.963621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:13.887722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"148'13043","reported_seq":13189,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.304954+0000","last_change":"2024-09-16T10:14:43.952726+0000","last_active":"2024-09-16T10:17:48.304954+0000","last_peered":"2024-09-16T10:17:48.304954+0000","last_clean":"2024-09-16T10:17:48.304954+0000","last_became_active":"2024-09-16T10:14:43.952378+0000","last_became_peered":"2024-09-16T10:14:43.952378+0000","last_unstale":"2024-09-16T10:17:48.304954+0000","last_undegraded":"2024-09-16T10:17:48.304954+0000","last_fullsized":"2024-09-16T10:17:48.304954+0000","mapping_epoch":81,"log_start":"114'6200","ondisk_log_start":"114'6200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6843,"log_dups_size":0,"ondisk_log_size":6843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:33.407300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118030336,"num_objects":1801,"num_object_clones":0,"num_object_copies":5403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13043,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":218,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815243+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:17:42.815243+0000","last_peered":"2024-09-16T10:17:42.815243+0000","last_clean":"2024-09-16T10:17:42.815243+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:17:42.815243+0000","last_undegraded":"2024-09-16T10:17:42.815243+0000","last_fullsized":"2024-09-16T10:17:42.815243+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.984845+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924408+0000","last_change":"2024-09-16T10:15:13.340979+0000","last_active":"2024-09-16T10:17:50.924408+0000","last_peered":"2024-09-16T10:17:50.924408+0000","last_clean":"2024-09-16T10:17:50.924408+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:17:50.924408+0000","last_undegraded":"2024-09-16T10:17:50.924408+0000","last_fullsized":"2024-09-16T10:17:50.924408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.340929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.340929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.449973+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":183,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.817105+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:17:42.817105+0000","last_peered":"2024-09-16T10:17:42.817105+0000","last_clean":"2024-09-16T10:17:42.817105+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:17:42.817105+0000","last_undegraded":"2024-09-16T10:17:42.817105+0000","last_fullsized":"2024-09-16T10:17:42.817105+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.418972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814664+0000","last_change":"2024-09-16T10:14:45.941203+0000","last_active":"2024-09-16T10:17:42.814664+0000","last_peered":"2024-09-16T10:17:42.814664+0000","last_clean":"2024-09-16T10:17:42.814664+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:17:42.814664+0000","last_undegraded":"2024-09-16T10:17:42.814664+0000","last_fullsized":"2024-09-16T10:17:42.814664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:14:45.941146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.941146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:00.036920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"148'12762","reported_seq":13015,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.369180+0000","last_change":"2024-09-16T10:17:42.364230+0000","last_active":"2024-09-16T10:17:48.369180+0000","last_peered":"2024-09-16T10:17:48.369180+0000","last_clean":"2024-09-16T10:17:48.369180+0000","last_became_active":"2024-09-16T10:14:43.959635+0000","last_became_peered":"2024-09-16T10:14:43.959635+0000","last_unstale":"2024-09-16T10:17:48.369180+0000","last_undegraded":"2024-09-16T10:17:48.369180+0000","last_fullsized":"2024-09-16T10:17:48.369180+0000","mapping_epoch":81,"log_start":"119'7200","ondisk_log_start":"119'7200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'12433","last_scrub_stamp":"2024-09-16T10:17:42.364000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.364000+0000","objects_scrubbed":2254,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:48.430009+0000","scrub_duration":1172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123863040,"num_objects":1890,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12762,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":233,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810312+0000","last_change":"2024-09-16T10:16:42.258515+0000","last_active":"2024-09-16T10:17:42.810312+0000","last_peered":"2024-09-16T10:17:42.810312+0000","last_clean":"2024-09-16T10:17:42.810312+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:17:42.810312+0000","last_undegraded":"2024-09-16T10:17:42.810312+0000","last_fullsized":"2024-09-16T10:17:42.810312+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:16:42.258433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.258433+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.356020+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814357+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:17:42.814357+0000","last_peered":"2024-09-16T10:17:42.814357+0000","last_clean":"2024-09-16T10:17:42.814357+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:17:42.814357+0000","last_undegraded":"2024-09-16T10:17:42.814357+0000","last_fullsized":"2024-09-16T10:17:42.814357+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:53.641444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815608+0000","last_change":"2024-09-16T10:16:43.420443+0000","last_active":"2024-09-16T10:17:42.815608+0000","last_peered":"2024-09-16T10:17:42.815608+0000","last_clean":"2024-09-16T10:17:42.815608+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:17:42.815608+0000","last_undegraded":"2024-09-16T10:17:42.815608+0000","last_fullsized":"2024-09-16T10:17:42.815608+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.420249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.420249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:47.309426+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814265+0000","last_change":"2024-09-16T10:17:08.596403+0000","last_active":"2024-09-16T10:17:42.814265+0000","last_peered":"2024-09-16T10:17:42.814265+0000","last_clean":"2024-09-16T10:17:42.814265+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:17:42.814265+0000","last_undegraded":"2024-09-16T10:17:42.814265+0000","last_fullsized":"2024-09-16T10:17:42.814265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.176712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"148'12762","reported_seq":12908,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.384269+0000","last_change":"2024-09-16T10:14:43.957583+0000","last_active":"2024-09-16T10:17:48.384269+0000","last_peered":"2024-09-16T10:17:48.384269+0000","last_clean":"2024-09-16T10:17:48.384269+0000","last_became_active":"2024-09-16T10:14:43.957207+0000","last_became_peered":"2024-09-16T10:14:43.957207+0000","last_unstale":"2024-09-16T10:17:48.384269+0000","last_undegraded":"2024-09-16T10:17:48.384269+0000","last_fullsized":"2024-09-16T10:17:48.384269+0000","mapping_epoch":81,"log_start":"124'8117","ondisk_log_start":"124'8117","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.156614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122159104,"num_objects":1864,"num_object_clones":0,"num_object_copies":5592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12762,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":227,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904460+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:17:50.904460+0000","last_peered":"2024-09-16T10:17:50.904460+0000","last_clean":"2024-09-16T10:17:50.904460+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:17:50.904460+0000","last_undegraded":"2024-09-16T10:17:50.904460+0000","last_fullsized":"2024-09-16T10:17:50.904460+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:49.150565+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814286+0000","last_change":"2024-09-16T10:15:11.631330+0000","last_active":"2024-09-16T10:17:42.814286+0000","last_peered":"2024-09-16T10:17:42.814286+0000","last_clean":"2024-09-16T10:17:42.814286+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:17:42.814286+0000","last_undegraded":"2024-09-16T10:17:42.814286+0000","last_fullsized":"2024-09-16T10:17:42.814286+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.631262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.631262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:46.656489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":183,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.818535+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:17:42.818535+0000","last_peered":"2024-09-16T10:17:42.818535+0000","last_clean":"2024-09-16T10:17:42.818535+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:17:42.818535+0000","last_undegraded":"2024-09-16T10:17:42.818535+0000","last_fullsized":"2024-09-16T10:17:42.818535+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:20.476397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811098+0000","last_change":"2024-09-16T10:16:32.171799+0000","last_active":"2024-09-16T10:17:42.811098+0000","last_peered":"2024-09-16T10:17:42.811098+0000","last_clean":"2024-09-16T10:17:42.811098+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:17:42.811098+0000","last_undegraded":"2024-09-16T10:17:42.811098+0000","last_fullsized":"2024-09-16T10:17:42.811098+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.171738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.171738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"148'13232","reported_seq":13451,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:48.245095+0000","last_change":"2024-09-16T10:16:02.172955+0000","last_active":"2024-09-16T10:17:48.245095+0000","last_peered":"2024-09-16T10:17:48.245095+0000","last_clean":"2024-09-16T10:17:48.245095+0000","last_became_active":"2024-09-16T10:14:43.954475+0000","last_became_peered":"2024-09-16T10:14:43.954475+0000","last_unstale":"2024-09-16T10:17:48.245095+0000","last_undegraded":"2024-09-16T10:17:48.245095+0000","last_fullsized":"2024-09-16T10:17:48.245095+0000","mapping_epoch":81,"log_start":"123'8200","ondisk_log_start":"123'8200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6489","last_scrub_stamp":"2024-09-16T10:16:02.172829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.172829+0000","objects_scrubbed":6427,"log_size":5032,"log_dups_size":0,"ondisk_log_size":5032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:34.148686+0000","scrub_duration":2056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122814464,"num_objects":1874,"num_object_clones":0,"num_object_copies":5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13232,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":214,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.923270+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:17:50.923270+0000","last_peered":"2024-09-16T10:17:50.923270+0000","last_clean":"2024-09-16T10:17:50.923270+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:17:50.923270+0000","last_undegraded":"2024-09-16T10:17:50.923270+0000","last_fullsized":"2024-09-16T10:17:50.923270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.521661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.812946+0000","last_change":"2024-09-16T10:15:25.529229+0000","last_active":"2024-09-16T10:17:42.812946+0000","last_peered":"2024-09-16T10:17:42.812946+0000","last_clean":"2024-09-16T10:17:42.812946+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:17:42.812946+0000","last_undegraded":"2024-09-16T10:17:42.812946+0000","last_fullsized":"2024-09-16T10:17:42.812946+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:15:25.529178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.529178+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:57.592697+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":258,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.812602+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:17:42.812602+0000","last_peered":"2024-09-16T10:17:42.812602+0000","last_clean":"2024-09-16T10:17:42.812602+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:17:42.812602+0000","last_undegraded":"2024-09-16T10:17:42.812602+0000","last_fullsized":"2024-09-16T10:17:42.812602+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.912935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.923647+0000","last_change":"2024-09-16T10:17:46.443359+0000","last_active":"2024-09-16T10:17:50.923647+0000","last_peered":"2024-09-16T10:17:50.923647+0000","last_clean":"2024-09-16T10:17:50.923647+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:17:50.923647+0000","last_undegraded":"2024-09-16T10:17:50.923647+0000","last_fullsized":"2024-09-16T10:17:50.923647+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:46.443295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:46.443295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:59.510329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810219+0000","last_change":"2024-09-16T10:15:55.157938+0000","last_active":"2024-09-16T10:17:42.810219+0000","last_peered":"2024-09-16T10:17:42.810219+0000","last_clean":"2024-09-16T10:17:42.810219+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:17:42.810219+0000","last_undegraded":"2024-09-16T10:17:42.810219+0000","last_fullsized":"2024-09-16T10:17:42.810219+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:55.157860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:55.157860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.136638+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":220,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.822155+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:17:42.822155+0000","last_peered":"2024-09-16T10:17:42.822155+0000","last_clean":"2024-09-16T10:17:42.822155+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:17:42.822155+0000","last_undegraded":"2024-09-16T10:17:42.822155+0000","last_fullsized":"2024-09-16T10:17:42.822155+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:47.968995+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":185,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.925503+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:17:50.925503+0000","last_peered":"2024-09-16T10:17:50.925503+0000","last_clean":"2024-09-16T10:17:50.925503+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:17:50.925503+0000","last_undegraded":"2024-09-16T10:17:50.925503+0000","last_fullsized":"2024-09-16T10:17:50.925503+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.398561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813849+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:17:42.813849+0000","last_peered":"2024-09-16T10:17:42.813849+0000","last_clean":"2024-09-16T10:17:42.813849+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:17:42.813849+0000","last_undegraded":"2024-09-16T10:17:42.813849+0000","last_fullsized":"2024-09-16T10:17:42.813849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:06.985366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":218,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.818591+0000","last_change":"2024-09-16T10:15:54.369062+0000","last_active":"2024-09-16T10:17:42.818591+0000","last_peered":"2024-09-16T10:17:42.818591+0000","last_clean":"2024-09-16T10:17:42.818591+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:17:42.818591+0000","last_undegraded":"2024-09-16T10:17:42.818591+0000","last_fullsized":"2024-09-16T10:17:42.818591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:54.368987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.368987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:17.060776+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.812679+0000","last_change":"2024-09-16T10:15:28.537739+0000","last_active":"2024-09-16T10:17:42.812679+0000","last_peered":"2024-09-16T10:17:42.812679+0000","last_clean":"2024-09-16T10:17:42.812679+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:17:42.812679+0000","last_undegraded":"2024-09-16T10:17:42.812679+0000","last_fullsized":"2024-09-16T10:17:42.812679+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:28.537673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.537673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:51.624149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.821706+0000","last_change":"2024-09-16T10:16:37.237971+0000","last_active":"2024-09-16T10:17:42.821706+0000","last_peered":"2024-09-16T10:17:42.821706+0000","last_clean":"2024-09-16T10:17:42.821706+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:17:42.821706+0000","last_undegraded":"2024-09-16T10:17:42.821706+0000","last_fullsized":"2024-09-16T10:17:42.821706+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:37.237849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.237849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814672+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:17:42.814672+0000","last_peered":"2024-09-16T10:17:42.814672+0000","last_clean":"2024-09-16T10:17:42.814672+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:17:42.814672+0000","last_undegraded":"2024-09-16T10:17:42.814672+0000","last_fullsized":"2024-09-16T10:17:42.814672+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:30.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":221,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924947+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:17:50.924947+0000","last_peered":"2024-09-16T10:17:50.924947+0000","last_clean":"2024-09-16T10:17:50.924947+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:17:50.924947+0000","last_undegraded":"2024-09-16T10:17:50.924947+0000","last_fullsized":"2024-09-16T10:17:50.924947+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.347562+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814900+0000","last_change":"2024-09-16T10:14:37.844181+0000","last_active":"2024-09-16T10:17:42.814900+0000","last_peered":"2024-09-16T10:17:42.814900+0000","last_clean":"2024-09-16T10:17:42.814900+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:17:42.814900+0000","last_undegraded":"2024-09-16T10:17:42.814900+0000","last_fullsized":"2024-09-16T10:17:42.814900+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:37.844113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:14:37.844113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:58.105572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":258,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813316+0000","last_change":"2024-09-16T10:17:06.052878+0000","last_active":"2024-09-16T10:17:42.813316+0000","last_peered":"2024-09-16T10:17:42.813316+0000","last_clean":"2024-09-16T10:17:42.813316+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:17:42.813316+0000","last_undegraded":"2024-09-16T10:17:42.813316+0000","last_fullsized":"2024-09-16T10:17:42.813316+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:06.050469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.050469+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.355575+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.905579+0000","last_change":"2024-09-16T10:17:45.915676+0000","last_active":"2024-09-16T10:17:50.905579+0000","last_peered":"2024-09-16T10:17:50.905579+0000","last_clean":"2024-09-16T10:17:50.905579+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:17:50.905579+0000","last_undegraded":"2024-09-16T10:17:50.905579+0000","last_fullsized":"2024-09-16T10:17:50.905579+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.915613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.915613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:57.303412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":218,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:44.208606+0000","last_change":"2024-09-16T10:17:44.208606+0000","last_active":"2024-09-16T10:17:44.208606+0000","last_peered":"2024-09-16T10:17:44.208606+0000","last_clean":"2024-09-16T10:17:44.208606+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:17:44.208606+0000","last_undegraded":"2024-09-16T10:17:44.208606+0000","last_fullsized":"2024-09-16T10:17:44.208606+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:44.208496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.208496+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:02.974829+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813248+0000","last_change":"2024-09-16T10:15:21.601115+0000","last_active":"2024-09-16T10:17:42.813248+0000","last_peered":"2024-09-16T10:17:42.813248+0000","last_clean":"2024-09-16T10:17:42.813248+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:17:42.813248+0000","last_undegraded":"2024-09-16T10:17:42.813248+0000","last_fullsized":"2024-09-16T10:17:42.813248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:21.601066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:21.601066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:54.626050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":261,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.924779+0000","last_change":"2024-09-16T10:17:34.543214+0000","last_active":"2024-09-16T10:17:50.924779+0000","last_peered":"2024-09-16T10:17:50.924779+0000","last_clean":"2024-09-16T10:17:50.924779+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:17:50.924779+0000","last_undegraded":"2024-09-16T10:17:50.924779+0000","last_fullsized":"2024-09-16T10:17:50.924779+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:34.543121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.543121+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:51.702679+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815058+0000","last_change":"2024-09-16T10:16:43.291855+0000","last_active":"2024-09-16T10:17:42.815058+0000","last_peered":"2024-09-16T10:17:42.815058+0000","last_clean":"2024-09-16T10:17:42.815058+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:17:42.815058+0000","last_undegraded":"2024-09-16T10:17:42.815058+0000","last_fullsized":"2024-09-16T10:17:42.815058+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.291720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.291720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:59.249028+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":215,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904811+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:17:50.904811+0000","last_peered":"2024-09-16T10:17:50.904811+0000","last_clean":"2024-09-16T10:17:50.904811+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:17:50.904811+0000","last_undegraded":"2024-09-16T10:17:50.904811+0000","last_fullsized":"2024-09-16T10:17:50.904811+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:56.914977+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":208,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.904514+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:17:50.904514+0000","last_peered":"2024-09-16T10:17:50.904514+0000","last_clean":"2024-09-16T10:17:50.904514+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:17:50.904514+0000","last_undegraded":"2024-09-16T10:17:50.904514+0000","last_fullsized":"2024-09-16T10:17:50.904514+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:45.277710+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.808354+0000","last_change":"2024-09-16T10:17:14.106033+0000","last_active":"2024-09-16T10:17:42.808354+0000","last_peered":"2024-09-16T10:17:42.808354+0000","last_clean":"2024-09-16T10:17:42.808354+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:17:42.808354+0000","last_undegraded":"2024-09-16T10:17:42.808354+0000","last_fullsized":"2024-09-16T10:17:42.808354+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.105971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.105971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.805018+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810751+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:17:42.810751+0000","last_peered":"2024-09-16T10:17:42.810751+0000","last_clean":"2024-09-16T10:17:42.810751+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:17:42.810751+0000","last_undegraded":"2024-09-16T10:17:42.810751+0000","last_fullsized":"2024-09-16T10:17:42.810751+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.252222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.808544+0000","last_change":"2024-09-16T10:15:15.010801+0000","last_active":"2024-09-16T10:17:42.808544+0000","last_peered":"2024-09-16T10:17:42.808544+0000","last_clean":"2024-09-16T10:17:42.808544+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:17:42.808544+0000","last_undegraded":"2024-09-16T10:17:42.808544+0000","last_fullsized":"2024-09-16T10:17:42.808544+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:15.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.010709+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:30.212036+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.814318+0000","last_change":"2024-09-16T10:17:42.811582+0000","last_active":"2024-09-16T10:17:42.814318+0000","last_peered":"2024-09-16T10:17:42.814318+0000","last_clean":"2024-09-16T10:17:42.814318+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:17:42.814318+0000","last_undegraded":"2024-09-16T10:17:42.814318+0000","last_fullsized":"2024-09-16T10:17:42.814318+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:42.811515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.811515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:45.586424+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":19,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.815039+0000","last_change":"2024-09-16T10:17:37.303380+0000","last_active":"2024-09-16T10:17:42.815039+0000","last_peered":"2024-09-16T10:17:42.815039+0000","last_clean":"2024-09-16T10:17:42.815039+0000","last_became_active":"2024-09-16T10:17:37.302939+0000","last_became_peered":"2024-09-16T10:17:37.302939+0000","last_unstale":"2024-09-16T10:17:42.815039+0000","last_undegraded":"2024-09-16T10:17:42.815039+0000","last_fullsized":"2024-09-16T10:17:42.815039+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:40.762324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:45.206938+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:17:45.206938+0000","last_peered":"2024-09-16T10:17:45.206938+0000","last_clean":"2024-09-16T10:17:45.206938+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:17:45.206938+0000","last_undegraded":"2024-09-16T10:17:45.206938+0000","last_fullsized":"2024-09-16T10:17:45.206938+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:49.801010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810893+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:17:42.810893+0000","last_peered":"2024-09-16T10:17:42.810893+0000","last_clean":"2024-09-16T10:17:42.810893+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:17:42.810893+0000","last_undegraded":"2024-09-16T10:17:42.810893+0000","last_fullsized":"2024-09-16T10:17:42.810893+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:52.138673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T10:17:50.925592+0000","last_change":"2024-09-16T10:17:36.539962+0000","last_active":"2024-09-16T10:17:50.925592+0000","last_peered":"2024-09-16T10:17:50.925592+0000","last_clean":"2024-09-16T10:17:50.925592+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:17:50.925592+0000","last_undegraded":"2024-09-16T10:17:50.925592+0000","last_fullsized":"2024-09-16T10:17:50.925592+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:36.539894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:36.539894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:41.349797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.813944+0000","last_change":"2024-09-16T10:14:45.829620+0000","last_active":"2024-09-16T10:17:42.813944+0000","last_peered":"2024-09-16T10:17:42.813944+0000","last_clean":"2024-09-16T10:17:42.813944+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:17:42.813944+0000","last_undegraded":"2024-09-16T10:17:42.813944+0000","last_fullsized":"2024-09-16T10:17:42.813944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:45.829572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:14:45.829572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:48.663935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811289+0000","last_change":"2024-09-16T10:15:30.762339+0000","last_active":"2024-09-16T10:17:42.811289+0000","last_peered":"2024-09-16T10:17:42.811289+0000","last_clean":"2024-09-16T10:17:42.811289+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:17:42.811289+0000","last_undegraded":"2024-09-16T10:17:42.811289+0000","last_fullsized":"2024-09-16T10:17:42.811289+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.762137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.762137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:56.149742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":214,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811633+0000","last_change":"2024-09-16T10:15:08.953214+0000","last_active":"2024-09-16T10:17:42.811633+0000","last_peered":"2024-09-16T10:17:42.811633+0000","last_clean":"2024-09-16T10:17:42.811633+0000","last_became_active":"2024-09-16T10:15:08.930505+0000","last_became_peered":"2024-09-16T10:15:08.930505+0000","last_unstale":"2024-09-16T10:17:42.811633+0000","last_undegraded":"2024-09-16T10:17:42.811633+0000","last_fullsized":"2024-09-16T10:17:42.811633+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.117953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":204,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.821196+0000","last_change":"2024-09-16T10:16:39.271363+0000","last_active":"2024-09-16T10:17:42.821196+0000","last_peered":"2024-09-16T10:17:42.821196+0000","last_clean":"2024-09-16T10:17:42.821196+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:17:42.821196+0000","last_undegraded":"2024-09-16T10:17:42.821196+0000","last_fullsized":"2024-09-16T10:17:42.821196+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:16:39.271283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.271283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.732007+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810461+0000","last_change":"2024-09-16T10:16:34.225741+0000","last_active":"2024-09-16T10:17:42.810461+0000","last_peered":"2024-09-16T10:17:42.810461+0000","last_clean":"2024-09-16T10:17:42.810461+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:17:42.810461+0000","last_undegraded":"2024-09-16T10:17:42.810461+0000","last_fullsized":"2024-09-16T10:17:42.810461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:34.225689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:34.225689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:02.982411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":80,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811495+0000","last_change":"2024-09-16T10:17:39.193790+0000","last_active":"2024-09-16T10:17:42.811495+0000","last_peered":"2024-09-16T10:17:42.811495+0000","last_clean":"2024-09-16T10:17:42.811495+0000","last_became_active":"2024-09-16T10:16:14.000930+0000","last_became_peered":"2024-09-16T10:16:14.000930+0000","last_unstale":"2024-09-16T10:17:42.811495+0000","last_undegraded":"2024-09-16T10:17:42.811495+0000","last_fullsized":"2024-09-16T10:17:42.811495+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:39.193466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:39.193466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:47.428002+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.818715+0000","last_change":"2024-09-16T10:16:32.919293+0000","last_active":"2024-09-16T10:17:42.818715+0000","last_peered":"2024-09-16T10:17:42.818715+0000","last_clean":"2024-09-16T10:17:42.818715+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:17:42.818715+0000","last_undegraded":"2024-09-16T10:17:42.818715+0000","last_fullsized":"2024-09-16T10:17:42.818715+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.919094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.919094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:50.652293+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.811881+0000","last_change":"2024-09-16T10:15:24.663908+0000","last_active":"2024-09-16T10:17:42.811881+0000","last_peered":"2024-09-16T10:17:42.811881+0000","last_clean":"2024-09-16T10:17:42.811881+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:17:42.811881+0000","last_undegraded":"2024-09-16T10:17:42.811881+0000","last_fullsized":"2024-09-16T10:17:42.811881+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.663832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.663832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:55.163009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":230,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.810935+0000","last_change":"2024-09-16T10:17:35.029288+0000","last_active":"2024-09-16T10:17:42.810935+0000","last_peered":"2024-09-16T10:17:42.810935+0000","last_clean":"2024-09-16T10:17:42.810935+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:17:42.810935+0000","last_undegraded":"2024-09-16T10:17:42.810935+0000","last_fullsized":"2024-09-16T10:17:42.810935+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:35.029210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.029210+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:38.152396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T10:17:42.818367+0000","last_change":"2024-09-16T10:16:36.222414+0000","last_active":"2024-09-16T10:17:42.818367+0000","last_peered":"2024-09-16T10:17:42.818367+0000","last_clean":"2024-09-16T10:17:42.818367+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:17:42.818367+0000","last_undegraded":"2024-09-16T10:17:42.818367+0000","last_fullsized":"2024-09-16T10:17:42.818367+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:36.222281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:36.222281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:03.830974+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1932591128,"num_objects":29490,"num_object_clones":0,"num_object_copies":88470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29490,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209064,"num_write_kb":7633665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5729366016,"data_stored":5729353800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90008,"ondisk_log_size":90008,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234038,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736664,"kb_used_data":620808,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93008488,"statfs":{"total":95995035648,"available":95240691712,"internally_reserved":0,"allocated":635707392,"data_stored":635345699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":118618570},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528378,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712836,"kb_used_data":613620,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93032316,"statfs":{"total":95995035648,"available":95265091584,"internally_reserved":0,"allocated":628346880,"data_stored":628004593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790017,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":488280,"kb_used_data":412680,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93256872,"statfs":{"total":95995035648,"available":95495036928,"internally_reserved":0,"allocated":422584320,"data_stored":422229442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986246,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":608488,"kb_used_data":505112,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93136664,"statfs":{"total":95995035648,"available":95371943936,"internally_reserved":0,"allocated":517234688,"data_stored":516862895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215178,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1259944,"kb_used_data":1098264,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92485208,"statfs":{"total":95995035648,"available":94704852992,"internally_reserved":0,"allocated":1124622336,"data_stored":1124244937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":165542344},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":838720,"kb_used_data":727024,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92906432,"statfs":{"total":95995035648,"available":95136186368,"internally_reserved":0,"allocated":744472576,"data_stored":744134088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855544,"kb_used_data":743720,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92889608,"statfs":{"total":95995035648,"available":95118958592,"internally_reserved":0,"allocated":761569280,"data_stored":761175092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087716,"kb_used_data":962068,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92657436,"statfs":{"total":95995035648,"available":94881214464,"internally_reserved":0,"allocated":985157632,"data_stored":984784377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":977342464,"data_stored":977338392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":738398208,"data_stored":738394136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":719454208,"data_stored":719454208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1117519872,"data_stored":1117519872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":511705088,"data_stored":511705088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":411107328,"data_stored":411107328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":625872896,"data_stored":625868824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":627965952,"data_stored":627965952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:17:56.261 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T10:17:56.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-16T10:17:56.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:17:56.352 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:56.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:56 smithi165 ceph-mon[32696]: from='client.25117 v1:172.21.15.63:0/349013125' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:56.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:56 smithi165 ceph-mon[32696]: pgmap v193: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-09-16T10:17:56.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3151764044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:56.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:56 smithi165 ceph-mon[32696]: from='client.25123 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:56.398 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:56.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:56.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[29993]: from='client.25117 v1:172.21.15.63:0/349013125' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:56.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[29993]: pgmap v193: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-09-16T10:17:56.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3151764044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:56.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[29993]: from='client.25123 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[39470]: from='client.25117 v1:172.21.15.63:0/349013125' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:17:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[39470]: pgmap v193: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-09-16T10:17:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3151764044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:56 smithi063 ceph-mon[39470]: from='client.25123 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:17:56.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:56.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:56.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:56.662 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:56.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:56.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:56.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:56.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:56.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:56.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:17:56.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:56.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:17:57.046 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:57.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:57.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:57.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:17:57.184 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:17:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:17:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:17:57.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:57.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:57.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:57.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:57.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:57.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:57.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:57.413 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:17:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:17:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:17:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39765 7.c 7:361d2464:::benchmark_data_smithi063_73086_object39764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 168.008111904, 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13511668299999999, 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39765 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247952+0000", 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.889e-06 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248012+0000", 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0418000000000003e-05 2024-09-16T10:17:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248023+0000", 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0852e-05 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248030+0000", 2024-09-16T10:17:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9489999999999999e-06 2024-09-16T10:17:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248067+0000", 2024-09-16T10:17:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7120999999999999e-05 2024-09-16T10:17:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248138+0000", 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0885999999999999e-05 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248217+0000", 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9058000000000001e-05 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250443+0000", 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022264720000000002 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250487+0000", 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3823999999999999e-05 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.382955+0000", 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13246761100000001 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383013+0000", 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7670000000000002e-05 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383066+0000", 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3933e-05 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39767 7.c 7:3eeafd38:::benchmark_data_smithi063_73086_object39766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 168.00801663799999, 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13517669099999999, 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39767 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248046+0000", 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5199999999999996e-07 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248153+0000", 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106803 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248170+0000", 2024-09-16T10:17:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.721e-05 2024-09-16T10:17:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248175+0000", 2024-09-16T10:17:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5609999999999998e-06 2024-09-16T10:17:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249333+0000", 2024-09-16T10:17:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011573200000000001 2024-09-16T10:17:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249455+0000", 2024-09-16T10:17:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012275400000000001 2024-09-16T10:17:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:17:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249590+0000", 2024-09-16T10:17:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013500099999999999 2024-09-16T10:17:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250643+0000", 2024-09-16T10:17:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001052966 2024-09-16T10:17:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252177+0000", 2024-09-16T10:17:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015336340000000001 2024-09-16T10:17:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383126+0000", 2024-09-16T10:17:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13094891 2024-09-16T10:17:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383178+0000", 2024-09-16T10:17:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2141000000000001e-05 2024-09-16T10:17:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383222+0000", 2024-09-16T10:17:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3739000000000003e-05 2024-09-16T10:17:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39768 7.c 7:3b0be1c9:::benchmark_data_smithi063_73086_object39767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:17:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 168.00576293, 2024-09-16T10:17:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13302192400000001, 2024-09-16T10:17:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39768 2024-09-16T10:17:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:17:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:17:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250301+0000", 2024-09-16T10:17:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2529999999999999e-06 2024-09-16T10:17:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250409+0000", 2024-09-16T10:17:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108145 2024-09-16T10:17:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250635+0000", 2024-09-16T10:17:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225582 2024-09-16T10:17:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250641+0000", 2024-09-16T10:17:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.63e-06 2024-09-16T10:17:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250684+0000", 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3006000000000001e-05 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250790+0000", 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105831 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:17:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250923+0000", 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013304199999999999 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252242+0000", 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013184189999999999 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252758+0000", 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051628000000000004 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383257+0000", 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13049871900000001 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383300+0000", 2024-09-16T10:17:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3223999999999998e-05 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383321+0000", 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0792999999999999e-05 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39770 7.7 7:ec0b9b23:::benchmark_data_smithi063_73086_object39769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:17:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 168.00540737200001, 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13275636499999999, 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39770 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250655+0000", 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7999999999999993e-07 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250688+0000", 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2687000000000001e-05 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250695+0000", 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0110000000000003e-06 2024-09-16T10:17:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250700+0000", 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250739+0000", 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8665999999999999e-05 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250820+0000", 2024-09-16T10:17:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1252000000000001e-05 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250936+0000", 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116036 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.253100+0000", 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002163674 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255541+0000", 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024417549999999999 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383318+0000", 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.127776905 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383372+0000", 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3826000000000003e-05 2024-09-16T10:17:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383411+0000", 2024-09-16T10:17:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8621000000000002e-05 2024-09-16T10:17:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39786 7.c 7:3b648db5:::benchmark_data_smithi063_73086_object39785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:17:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 167.99454941600001, 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124519465, 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39786 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261514+0000", 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3430000000000001e-06 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261577+0000", 2024-09-16T10:17:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3646999999999996e-05 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261587+0000", 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9499999999999996e-06 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261593+0000", 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6810000000000003e-06 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261633+0000", 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0262999999999997e-05 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261709+0000", 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.551e-05 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:17:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261799+0000", 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0437000000000002e-05 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.262947+0000", 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001148224 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.263126+0000", 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000178529 2024-09-16T10:17:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.385937+0000", 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122810896 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.385988+0000", 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0961000000000002e-05 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.386032+0000", 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4023999999999997e-05 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40098 7.7 7:e51c64ce:::benchmark_data_smithi063_73086_object40097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 167.71069796699999, 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118499387, 2024-09-16T10:17:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40098 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:17:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545366+0000", 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0839999999999999e-06 2024-09-16T10:17:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545435+0000", 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9424999999999993e-05 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545449+0000", 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3283e-05 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545456+0000", 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0650000000000001e-06 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545522+0000", 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.614e-05 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545636+0000", 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011427800000000001 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545759+0000", 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012285799999999999 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.547169+0000", 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001410251 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.550261+0000", 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030920029999999999 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663775+0000", 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113513832 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663832+0000", 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6771999999999998e-05 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663863+0000", 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1396000000000002e-05 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40106 7.d 7:bffd718f:::benchmark_data_smithi063_73086_object40105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 167.706451246, 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112057936, 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40106 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:17:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549612+0000", 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3289999999999999e-06 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549663+0000", 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1014999999999999e-05 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549673+0000", 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9229999999999997e-06 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549678+0000", 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1120000000000004e-06 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549728+0000", 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9700999999999997e-05 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549857+0000", 2024-09-16T10:17:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012977400000000001 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549995+0000", 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137498 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551274+0000", 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012796140000000001 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551903+0000", 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062848199999999998 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661566+0000", 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109663317 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661615+0000", 2024-09-16T10:17:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9193000000000001e-05 2024-09-16T10:17:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661668+0000", 2024-09-16T10:17:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2978e-05 2024-09-16T10:17:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40108 7.7 7:e3eeca68:::benchmark_data_smithi063_73086_object40107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 167.704590365, 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12682985199999999, 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40108 2024-09-16T10:17:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551473+0000", 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3090000000000001e-06 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551526+0000", 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999999e-05 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551536+0000", 2024-09-16T10:17:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6120000000000005e-06 2024-09-16T10:17:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551541+0000", 2024-09-16T10:17:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8790000000000002e-06 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551580+0000", 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9727000000000001e-05 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551660+0000", 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9480999999999993e-05 2024-09-16T10:17:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551754+0000", 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4548000000000003e-05 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.552878+0000", 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011237459999999999 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663896+0000", 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11101776200000001 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678241+0000", 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014344894 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678275+0000", 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4551000000000002e-05 2024-09-16T10:17:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678301+0000", 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5843e-05 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:43489 7.d 7:bb6aa884:::benchmark_data_smithi063_73086_object43488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:11.749872+0000", 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 165.50619016100001, 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111586563, 2024-09-16T10:17:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 43489 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749872+0000", 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749872+0000", 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749873+0000", 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4500000000000001e-06 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749922+0000", 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9478999999999998e-05 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749933+0000", 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0097e-05 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749938+0000", 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2410000000000003e-06 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.749987+0000", 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8896000000000001e-05 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.750067+0000", 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0623000000000004e-05 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.750150+0000", 2024-09-16T10:17:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2922000000000003e-05 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.751459+0000", 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013091610000000001 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754957+0000", 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034977670000000002 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.861381+0000", 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10642346 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.861416+0000", 2024-09-16T10:17:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5337999999999999e-05 2024-09-16T10:17:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.861458+0000", 2024-09-16T10:17:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2129000000000003e-05 2024-09-16T10:17:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:43504 7.7 7:eb5ac715:::benchmark_data_smithi063_73086_object43503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:17:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:11.756623+0000", 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 165.499438277, 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111441769, 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 43504 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756623+0000", 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756623+0000", 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756625+0000", 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.742e-06 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756655+0000", 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9899e-05 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756665+0000", 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9769999999999995e-06 2024-09-16T10:17:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.756672+0000", 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7309999999999996e-06 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.757092+0000", 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042053700000000002 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.757230+0000", 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013731600000000001 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.757359+0000", 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012892200000000001 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.758660+0000", 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013009510000000001 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.865343+0000", 2024-09-16T10:17:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10668319499999999 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.867941+0000", 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025982100000000001 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.868000+0000", 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8640000000000001e-05 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.868065+0000", 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5648999999999994e-05 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65089 7.d 7:bff27887:::benchmark_data_smithi063_73086_object65088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 149.69950107700001, 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12756952299999999, 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65089 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556562+0000", 2024-09-16T10:17:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.686e-06 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556698+0000", 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013535199999999999 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556711+0000", 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3067e-05 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556719+0000", 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7919999999999993e-06 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556773+0000", 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4647e-05 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556924+0000", 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015124 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.557015+0000", 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0681999999999997e-05 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.558770+0000", 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001754395 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.562719+0000", 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039499050000000001 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684014+0000", 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.1212949 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684082+0000", 2024-09-16T10:17:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7417999999999997e-05 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684130+0000", 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8439000000000002e-05 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65092 7.d 7:b5d3643d:::benchmark_data_smithi063_73086_object65091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 149.696411273, 2024-09-16T10:17:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12473208600000001, 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65092 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:17:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559652+0000", 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4109999999999999e-06 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559730+0000", 2024-09-16T10:17:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8343000000000005e-05 2024-09-16T10:17:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559741+0000", 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-05 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559747+0000", 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.417e-06 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559788+0000", 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1597999999999997e-05 2024-09-16T10:17:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559877+0000", 2024-09-16T10:17:57.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9018999999999996e-05 2024-09-16T10:17:57.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:57.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559960+0000", 2024-09-16T10:17:57.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2101000000000006e-05 2024-09-16T10:17:57.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.561615+0000", 2024-09-16T10:17:57.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00165525 2024-09-16T10:17:57.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.562754+0000", 2024-09-16T10:17:57.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00113956 2024-09-16T10:17:57.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684280+0000", 2024-09-16T10:17:57.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121525444 2024-09-16T10:17:57.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684345+0000", 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684383+0000", 2024-09-16T10:17:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7639999999999999e-05 2024-09-16T10:17:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72694 7.d 7:b02443fe:::benchmark_data_smithi063_73086_object72693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 143.385872828, 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11432945999999999, 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72694 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870190+0000", 2024-09-16T10:17:57.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0499999999999999e-06 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870260+0000", 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9900000000000005e-05 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870269+0000", 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9959999999999999e-06 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870273+0000", 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6210000000000003e-06 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872430+0000", 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002156748 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872556+0000", 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125859 2024-09-16T10:17:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872660+0000", 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103859 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.874597+0000", 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373229999999999 2024-09-16T10:17:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.877583+0000", 2024-09-16T10:17:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002986247 2024-09-16T10:17:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984443+0000", 2024-09-16T10:17:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10685934800000001 2024-09-16T10:17:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984481+0000", 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7718e-05 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984518+0000", 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7790999999999998e-05 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72705 7.7 7:eddfad96:::benchmark_data_smithi063_73086_object72704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.878700+0000", 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 143.377361834, 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110056815, 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72705 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878700+0000", 2024-09-16T10:17:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878700+0000", 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878702+0000", 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9039999999999999e-06 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878874+0000", 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017175799999999999 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878885+0000", 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1523e-05 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878892+0000", 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0729999999999999e-06 2024-09-16T10:17:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878939+0000", 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6609e-05 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879055+0000", 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116026 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879181+0000", 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000126305 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.881701+0000", 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025194390000000001 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.884850+0000", 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031490649999999999 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988668+0000", 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103817933 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988720+0000", 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2361999999999998e-05 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.988757+0000", 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6817999999999999e-05 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90054 7.c 7:3cae452b:::benchmark_data_smithi063_73086_object90053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 127.742624515, 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12751799899999999, 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90054 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513440+0000", 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.452e-06 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513492+0000", 2024-09-16T10:17:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1950999999999998e-05 2024-09-16T10:17:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513503+0000", 2024-09-16T10:17:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1232e-05 2024-09-16T10:17:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513509+0000", 2024-09-16T10:17:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0680000000000002e-06 2024-09-16T10:17:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513571+0000", 2024-09-16T10:17:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2434999999999994e-05 2024-09-16T10:17:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513694+0000", 2024-09-16T10:17:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012256499999999999 2024-09-16T10:17:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:17:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513865+0000", 2024-09-16T10:17:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017080599999999999 2024-09-16T10:17:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.517203+0000", 2024-09-16T10:17:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033382860000000002 2024-09-16T10:17:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.518058+0000", 2024-09-16T10:17:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085528199999999996 2024-09-16T10:17:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640875+0000", 2024-09-16T10:17:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12281645400000001 2024-09-16T10:17:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640919+0000", 2024-09-16T10:17:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.401e-05 2024-09-16T10:17:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640955+0000", 2024-09-16T10:17:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6458000000000002e-05 2024-09-16T10:17:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90065 7.d 7:bffa18df:::benchmark_data_smithi063_73086_object90064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:17:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:17:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 127.734191088, 2024-09-16T10:17:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124785905, 2024-09-16T10:17:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.549 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90065 2024-09-16T10:17:57.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.549 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:17:57.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:17:57.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521873+0000", 2024-09-16T10:17:57.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.497e-06 2024-09-16T10:17:57.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521903+0000", 2024-09-16T10:17:57.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9407000000000002e-05 2024-09-16T10:17:57.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521916+0000", 2024-09-16T10:17:57.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3597999999999999e-05 2024-09-16T10:17:57.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521924+0000", 2024-09-16T10:17:57.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521991+0000", 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7318999999999997e-05 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522105+0000", 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113467 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522205+0000", 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9829999999999998e-05 2024-09-16T10:17:57.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525304+0000", 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030995089999999999 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525567+0000", 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026322999999999999 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646542+0000", 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120974217 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646604+0000", 2024-09-16T10:17:57.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2543000000000001e-05 2024-09-16T10:17:57.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646657+0000", 2024-09-16T10:17:57.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2438000000000002e-05 2024-09-16T10:17:57.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.560 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174938 7.d 7:bc5264d9:::benchmark_data_smithi063_73086_object55659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:17:57.561 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.878108+0000", 2024-09-16T10:17:57.561 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.377953996000002, 2024-09-16T10:17:57.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10993975, 2024-09-16T10:17:57.561 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.562 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.562 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.562 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174938 2024-09-16T10:17:57.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878108+0000", 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878108+0000", 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878109+0000", 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2550000000000001e-06 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878113+0000", 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4410000000000003e-06 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878127+0000", 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4004e-05 2024-09-16T10:17:57.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878134+0000", 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6749999999999996e-06 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878196+0000", 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2184e-05 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878297+0000", 2024-09-16T10:17:57.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010108200000000001 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878404+0000", 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106575 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.882057+0000", 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003653398 2024-09-16T10:17:57.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888307+0000", 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062492829999999996 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.987929+0000", 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099622205000000005 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.987996+0000", 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6895000000000002e-05 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.988047+0000", 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1752999999999997e-05 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174948 7.7 7:e876b2a9:::benchmark_data_smithi063_73086_object55669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.884541+0000", 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.371521026000003, 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106541999, 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174948 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884541+0000", 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884541+0000", 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:57 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:57.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:57 smithi165 ceph-mon[32696]: from='client.25123 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:57.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:57 smithi165 ceph-mon[32696]: osdmap e150: 8 total, 8 up, 8 in 2024-09-16T10:17:57.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884542+0000", 2024-09-16T10:17:57.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.271e-06 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884547+0000", 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6090000000000003e-06 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884558+0000", 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1141999999999999e-05 2024-09-16T10:17:57.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884563+0000", 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3870000000000003e-06 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884615+0000", 2024-09-16T10:17:57.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2237999999999997e-05 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884731+0000", 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115957 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.884873+0000", 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014157699999999999 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.887894+0000", 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030214299999999999 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989581+0000", 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10168667300000001 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991008+0000", 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014269529999999999 2024-09-16T10:17:57.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991050+0000", 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2001e-05 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991083+0000", 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2761e-05 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174956 7.7 7:eae3f4a6:::benchmark_data_smithi063_73086_object55677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.366373031000002, 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12889425400000001, 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174956 2024-09-16T10:17:57.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889690+0000", 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.562e-06 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889695+0000", 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8149999999999996e-06 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889708+0000", 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2943e-05 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889715+0000", 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.272e-06 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889765+0000", 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9412000000000001e-05 2024-09-16T10:17:57.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889881+0000", 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116789 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889985+0000", 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103195 2024-09-16T10:17:57.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895390+0000", 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054051689999999996 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991131+0000", 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095740960999999999 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018515+0000", 2024-09-16T10:17:57.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027384196 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018547+0000", 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1973e-05 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018583+0000", 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5967000000000003e-05 2024-09-16T10:17:57.583 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174957 7.7 7:e71673f7:::benchmark_data_smithi063_73086_object55678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.363713185999998, 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.126386786, 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:17:57.584 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174957 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:17:57.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892351+0000", 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0310000000000001e-06 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892356+0000", 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2689999999999999e-06 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:17:57.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892368+0000", 2024-09-16T10:17:57.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1848000000000001e-05 2024-09-16T10:17:57.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:17:57.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892376+0000", 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0649999999999999e-06 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892738+0000", 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036210000000000002 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892862+0000", 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012463099999999999 2024-09-16T10:17:57.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892975+0000", 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112526 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895478+0000", 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002503211 2024-09-16T10:17:57.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991158+0000", 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095680116999999995 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018680+0000", 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027522007000000001 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018705+0000", 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4802000000000001e-05 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018735+0000", 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0179e-05 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.591 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.592 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:17:57.592 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:17:57.592 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:17:57.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:57.593 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:57 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:57.593 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:57 smithi063 ceph-mon[29993]: from='client.25123 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:57.593 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:57 smithi063 ceph-mon[29993]: osdmap e150: 8 total, 8 up, 8 in 2024-09-16T10:17:57.593 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:57 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:17:57.594 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:57 smithi063 ceph-mon[39470]: from='client.25123 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:17:57.594 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:57 smithi063 ceph-mon[39470]: osdmap e150: 8 total, 8 up, 8 in 2024-09-16T10:17:57.633 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:57.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:57.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:57.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:57.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:57.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:57.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:57.900 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:57.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:58.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:17:58.039 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:58.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:58.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:17:58.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:17:58.196 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:58.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:17:58.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:58 smithi165 ceph-mon[32696]: pgmap v195: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-09-16T10:17:58.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:58.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:58.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:58.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:58.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:58 smithi063 ceph-mon[39470]: pgmap v195: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-09-16T10:17:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:58 smithi063 ceph-mon[29993]: pgmap v195: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-09-16T10:17:58.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:58.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:58.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:17:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:17:58.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:17:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:58.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:58.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:17:58.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:17:58.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:58.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:58.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:58.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:58.877 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:58.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:58.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:17:58.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:59.006 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:17:59.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:59.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:17:59.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:17:59.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:59.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:59.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:59 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:17:59.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:59 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:17:59.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:59 smithi165 ceph-mon[32696]: osdmap e151: 8 total, 8 up, 8 in 2024-09-16T10:17:59.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:17:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:59.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:59.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[29993]: osdmap e151: 8 total, 8 up, 8 in 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[39470]: osdmap e151: 8 total, 8 up, 8 in 2024-09-16T10:17:59.466 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:17:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:17:59.503 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:17:59.543 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:17:59.579 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:17:59.579 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:17:59.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:17:59.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:17:59.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:17:59.776 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:59.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:17:59.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:17:59.935 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:17:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:17:59.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:00.037 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:00.072 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:00.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:00.174 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:00.210 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:00.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:00.258 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:00.258 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:00.259 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:00.259 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:00.260 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:00.313 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:00.350 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:00.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:00.451 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:00.498 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:00.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:00 smithi165 ceph-mon[32696]: pgmap v197: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-16T10:18:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3643526835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:00.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:00.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:00.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:00.654 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:00 smithi063 ceph-mon[29993]: pgmap v197: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-16T10:18:00.654 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3643526835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:00.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:00 smithi063 ceph-mon[39470]: pgmap v197: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-16T10:18:00.654 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3643526835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:00.733 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:00.773 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:00.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:00.874 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:00.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:01.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:01.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:01.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:01.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:01.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:18:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:01.194 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:01.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:01.295 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:01.354 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:01.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:01.366 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:01.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:01.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:01.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:01.524 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:01.590 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:01.630 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:01.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:01.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:01.778 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:01.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:01.803 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 1 pg_num to 11 2024-09-16T10:18:01.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:01.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:01 smithi063 ceph-mon[39470]: pgmap v198: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:18:01.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2516380247' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:18:01.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:01 smithi063 ceph-mon[39470]: from='client.25135 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:18:01.904 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:01 smithi063 ceph-mon[29993]: pgmap v198: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:18:01.904 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2516380247' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:18:01.904 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:01 smithi063 ceph-mon[29993]: from='client.25135 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:18:01.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:01.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:01.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:01 smithi165 ceph-mon[32696]: pgmap v198: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:18:01.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2516380247' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:18:01.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:01 smithi165 ceph-mon[32696]: from='client.25135 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:18:02.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:02.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:02.119 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:02.119 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:02.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:02.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:02.274 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.318 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:02.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:02.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:02.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:02.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:02.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:02.676 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:02.712 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:02.809 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:18:02.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:02.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:02.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:02.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[39470]: from='client.25135 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:18:02.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[39470]: osdmap e152: 8 total, 8 up, 8 in 2024-09-16T10:18:02.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/177344614' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:02.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[39470]: from='client.45080 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[29993]: from='client.25135 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:18:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[29993]: osdmap e152: 8 total, 8 up, 8 in 2024-09-16T10:18:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/177344614' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:02 smithi063 ceph-mon[29993]: from='client.45080 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:02.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:02.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:02.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:02 smithi165 ceph-mon[32696]: from='client.25135 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:18:02.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:02 smithi165 ceph-mon[32696]: osdmap e152: 8 total, 8 up, 8 in 2024-09-16T10:18:02.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/177344614' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:02.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:02 smithi165 ceph-mon[32696]: from='client.45080 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:02.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:18:03.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:03.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:03.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:03.213 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:03.253 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:03.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:03.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:03.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:18:03.369 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:03.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:03.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:03.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:03.547 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:03.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:03.648 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:03.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:03.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:03.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:03.817 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:03.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:03.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:03.852 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:03.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:03.931 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:03.931 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:03.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:03.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: from='client.45080 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: osdmap e153: 8 total, 8 up, 8 in 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: pgmap v201: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: 3.12 scrub starts 2024-09-16T10:18:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:03 smithi165 ceph-mon[32696]: 3.12 scrub ok 2024-09-16T10:18:04.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:04.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:04.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:04.166 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: from='client.45080 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:18:04.166 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: osdmap e153: 8 total, 8 up, 8 in 2024-09-16T10:18:04.166 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:18:04.166 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:18:04.166 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: pgmap v201: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: 3.12 scrub starts 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[29993]: 3.12 scrub ok 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: from='client.45080 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: osdmap e153: 8 total, 8 up, 8 in 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: pgmap v201: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-16T10:18:04.167 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:04.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:04.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: 3.12 scrub starts 2024-09-16T10:18:04.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:03 smithi063 ceph-mon[39470]: 3.12 scrub ok 2024-09-16T10:18:04.194 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:04.227 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:04.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:04.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:04.382 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:04.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:04.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:04.528 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:04.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:04.530 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:04.530 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:04.569 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:04.570 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:04.609 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:04.609 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:04.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:04.651 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:04.651 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:04.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:04.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:04.686 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:04.687 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:04.717 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:04.717 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:04.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:18:04.756 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:04.756 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:04.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:04.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:04.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:04.790 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:04.791 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:04.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:04.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:04.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:04.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: osdmap e154: 8 total, 8 up, 8 in 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: 5.0 scrub starts 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: 5.0 scrub ok 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: 3.5 scrub starts 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: 3.5 scrub ok 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3127285592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:04.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: osdmap e154: 8 total, 8 up, 8 in 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: 5.0 scrub starts 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: 5.0 scrub ok 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: 3.5 scrub starts 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: 3.5 scrub ok 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:18:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:04 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3127285592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:05.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:05.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:05.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:05.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T10:18:05.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: osdmap e154: 8 total, 8 up, 8 in 2024-09-16T10:18:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: 5.0 scrub starts 2024-09-16T10:18:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: 5.0 scrub ok 2024-09-16T10:18:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: 3.5 scrub starts 2024-09-16T10:18:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: 3.5 scrub ok 2024-09-16T10:18:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:18:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:18:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:04 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3127285592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:05.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:05.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:05.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:05.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:05.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:05.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:05.397 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:05.437 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:05.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:05.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:05.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:05.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:05.597 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:18:05.646 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:05.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:05.724 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:05.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:05.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:05.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:05.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:05.952 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:06.001 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:06.041 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:18:06.042 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: 6.2 scrub ok 2024-09-16T10:18:06.042 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: osdmap e155: 8 total, 8 up, 8 in 2024-09-16T10:18:06.042 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: 6.9 scrub starts 2024-09-16T10:18:06.042 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: 6.9 scrub ok 2024-09-16T10:18:06.042 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: pgmap v204: 158 pgs: 10 unknown, 148 active+clean; 940 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-16T10:18:06.042 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:06.042 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:06.043 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:18:06.043 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: 6.2 scrub ok 2024-09-16T10:18:06.043 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: osdmap e155: 8 total, 8 up, 8 in 2024-09-16T10:18:06.043 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: 6.9 scrub starts 2024-09-16T10:18:06.043 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: 6.9 scrub ok 2024-09-16T10:18:06.044 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: pgmap v204: 158 pgs: 10 unknown, 148 active+clean; 940 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-16T10:18:06.044 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:06.044 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:05 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:06.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:06.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:06.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:06.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:06.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:18:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: 6.2 scrub ok 2024-09-16T10:18:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: osdmap e155: 8 total, 8 up, 8 in 2024-09-16T10:18:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: 6.9 scrub starts 2024-09-16T10:18:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: 6.9 scrub ok 2024-09-16T10:18:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: pgmap v204: 158 pgs: 10 unknown, 148 active+clean; 940 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-16T10:18:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:05 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:18:06.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:06.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:06.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:06.518 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:06.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:06.649 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:226467 7.e e156/81 7:7ca93d88:::benchmark_data_smithi063_73086_object107188:head v 156'13737, mlcod=156'13735)", 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.317854+0000", 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.014979492000000001, 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015007405999999999, 2024-09-16T10:18:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.317854+0000", 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.317854+0000", 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.317855+0000", 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2410000000000001e-06 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.317863+0000", 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5859999999999999e-06 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.317878+0000", 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4793999999999999e-05 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.317884+0000", 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7490000000000001e-06 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.317983+0000", 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8653000000000006e-05 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.318002+0000", 2024-09-16T10:18:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.908e-05 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:226471 7.6 7:692e9831:::benchmark_data_smithi063_73086_object107192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.319024+0000", 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.013809078000000001, 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013897797, 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 226471 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319024+0000", 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319024+0000", 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319026+0000", 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.697e-06 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319031+0000", 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.831e-06 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319044+0000", 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3491e-05 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319051+0000", 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9689999999999997e-06 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.325622+0000", 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0065706130000000003 2024-09-16T10:18:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.325731+0000", 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010861699999999999 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.325847+0000", 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011629999999999999 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:226472 7.9 7:921927e1:::benchmark_data_smithi063_73086_object107193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.319074+0000", 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.013759103, 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013917374, 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 226472 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319074+0000", 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319074+0000", 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319075+0000", 2024-09-16T10:18:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.06e-07 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319078+0000", 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1870000000000001e-06 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319084+0000", 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1820000000000003e-06 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319090+0000", 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3120000000000001e-06 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319139+0000", 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8922999999999999e-05 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319243+0000", 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000104332 2024-09-16T10:18:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.319352+0000", 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000109448 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.325437+0000", 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060849069999999996 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.328895+0000", 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034572299999999999 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:226473 7.9 7:9d078b72:::benchmark_data_smithi063_73086_object107194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.321885+0000", 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.010948787, 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011180677, 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 226473 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.321885+0000", 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.321885+0000", 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.321887+0000", 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3520000000000001e-06 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.321893+0000", 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8239999999999996e-06 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.321905+0000", 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2099999999999999e-05 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.321914+0000", 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4099999999999997e-06 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.321966+0000", 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1439e-05 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.322075+0000", 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010903700000000001 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.322185+0000", 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000110639 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327238+0000", 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0050521480000000002 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:226468 7.e e156/81)", 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.327251+0000", 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.005582282, 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058748899999999998, 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327251+0000", 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327251+0000", 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327252+0000", 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7199999999999998e-07 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327256+0000", 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5090000000000004e-06 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327262+0000", 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0739999999999998e-06 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327267+0000", 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0159999999999999e-06 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:226478 7.d e156/81 7:b017cdce:::benchmark_data_smithi063_73086_object107199:head v 156'14315, mlcod=156'14313)", 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.327338+0000", 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0054950889999999999, 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058275640000000004, 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327338+0000", 2024-09-16T10:18:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327338+0000", 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327339+0000", 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9699999999999995e-07 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327343+0000", 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2069999999999997e-06 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327350+0000", 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7970000000000003e-06 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327355+0000", 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1100000000000002e-06 2024-09-16T10:18:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.330098+0000", 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027428109999999999 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.330119+0000", 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1061e-05 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:226479 7.5 e156/81 7:a906113b:::benchmark_data_smithi063_73086_object107200:head v 156'14288, mlcod=156'14286)", 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.327419+0000", 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0054140389999999998, 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0057962780000000002, 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327419+0000", 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327419+0000", 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327421+0000", 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.35e-06 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327426+0000", 2024-09-16T10:18:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7559999999999996e-06 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327436+0000", 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0920000000000004e-06 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.327442+0000", 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4790000000000001e-06 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.331802+0000", 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043597330000000002 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.331837+0000", 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5145999999999999e-05 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:226456 7.6 7:6758dfb7:::benchmark_data_smithi063_73086_object107177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-16T10:18:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:06.306060+0000", 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.026773604999999999, 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.027208604000000001, 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 226456 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.306060+0000", 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.306060+0000", 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.306065+0000", 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9799999999999998e-06 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.306071+0000", 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4099999999999996e-06 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.306080+0000", 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0569999999999999e-06 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.306088+0000", 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3309999999999998e-06 2024-09-16T10:18:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.309365+0000", 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003277191 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.309476+0000", 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111694 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.309684+0000", 2024-09-16T10:18:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020772400000000001 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.312587+0000", 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029030359999999999 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:06.318660+0000", 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060726629999999998 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:18:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 8 2024-09-16T10:18:06.752 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:18:06.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:06.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:06.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:06.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:18:06.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:18:06.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: 5.1c scrub starts 2024-09-16T10:18:06.919 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:06.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:07.021 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:07.061 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:07.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:07.105 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:07.163 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:07.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:18:07.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:18:07.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: 5.1c scrub starts 2024-09-16T10:18:07.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: 5.1c scrub ok 2024-09-16T10:18:07.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:18:07.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: osdmap e156: 8 total, 8 up, 8 in 2024-09-16T10:18:07.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: 6.0 scrub starts 2024-09-16T10:18:07.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[29993]: 6.0 scrub ok 2024-09-16T10:18:07.204 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:18:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:18:07.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: 5.1c scrub ok 2024-09-16T10:18:07.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:18:07.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: osdmap e156: 8 total, 8 up, 8 in 2024-09-16T10:18:07.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: 6.0 scrub starts 2024-09-16T10:18:07.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:06 smithi063 ceph-mon[39470]: 6.0 scrub ok 2024-09-16T10:18:07.215 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:07.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:07.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: 5.1c scrub starts 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: 5.1c scrub ok 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: osdmap e156: 8 total, 8 up, 8 in 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: 6.0 scrub starts 2024-09-16T10:18:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:06 smithi165 ceph-mon[32696]: 6.0 scrub ok 2024-09-16T10:18:07.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:07.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:07.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:07.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:07.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:07.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:07.528 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:18:07.544 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:18:07.544 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:18:07.544 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:18:07.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:18:07.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:07.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:07.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:07.657 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:18:07.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:07.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:07.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:07.826 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:07.888 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:07.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:07.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:08.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:08.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: osdmap e157: 8 total, 8 up, 8 in 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.4 scrub ok 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: pgmap v207: 158 pgs: 10 unknown, 148 active+clean; 940 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 3.1f scrub starts 2024-09-16T10:18:08.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 3.1f scrub ok 2024-09-16T10:18:08.154 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:18:08.154 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.1 scrub ok 2024-09-16T10:18:08.154 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:18:08.154 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.7 scrub ok 2024-09-16T10:18:08.154 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:18:08.154 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:18:08.154 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: osdmap e157: 8 total, 8 up, 8 in 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.4 scrub ok 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: pgmap v207: 158 pgs: 10 unknown, 148 active+clean; 940 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 3.1f scrub starts 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 3.1f scrub ok 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.1 scrub ok 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.7 scrub ok 2024-09-16T10:18:08.156 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:18:08.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:18:08.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:07 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:08.165 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:08.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:08.283 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:08.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: osdmap e157: 8 total, 8 up, 8 in 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.4 scrub ok 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: pgmap v207: 158 pgs: 10 unknown, 148 active+clean; 940 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 3.1f scrub starts 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 3.1f scrub ok 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.1 scrub ok 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:18:08.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.7 scrub ok 2024-09-16T10:18:08.293 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:18:08.293 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:18:08.293 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:07 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:08.303 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:08.315 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:08.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:08.448 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:08.551 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:08.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:18:08.592 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:08.696 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.733 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:08.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:08.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:08.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:08.943 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:18:08.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:08.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:08.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:09.055 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.5 scrub ok 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 4.0 scrub starts 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 4.0 scrub ok 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.a scrub starts 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.a scrub ok 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:09.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/157851631' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[29993]: from='client.25150 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.5 scrub ok 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 4.0 scrub starts 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 4.0 scrub ok 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.a scrub starts 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.a scrub ok 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:18:09.057 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:18:09.058 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:09.058 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/157851631' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:09.058 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:08 smithi063 ceph-mon[39470]: from='client.25150 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:09.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:09.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:09.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:09.216 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:09.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.5 scrub ok 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 4.0 scrub starts 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 4.0 scrub ok 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.a scrub starts 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.a scrub ok 2024-09-16T10:18:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:18:09.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:18:09.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:09.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/157851631' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:09.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:08 smithi165 ceph-mon[32696]: from='client.25150 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:09.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:09.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:09.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:09.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:09.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:09.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:09.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:09.670 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:09.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:09.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:09.813 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:09.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:09.816 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:09.817 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:09.882 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:09.882 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:09.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:09.926 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:09.927 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 158, 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:18:08.916825+0000", 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:13:06.309218+0000", 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:18:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5799936, 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 7, 2024-09-16T10:18:09.950 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:18:09.951 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 11, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 11, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 11, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 11, 2024-09-16T10:18:09.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 11, 2024-09-16T10:18:09.953 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:18:09.953 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:18:09.953 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:18:09.953 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:18:09.953 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "156", 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "154", 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:18:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:18:09.955 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:18:09.955 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:18:09.955 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:18:09.955 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:18:09.955 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:18:09.955 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:18:09.955 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:18:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:18:09.957 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:18:09.958 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.1800000667572021, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.1800000667572021, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:09.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:18:09.960 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:18:09.961 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:18:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:18:09.963 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:18:09.964 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:18:09.965 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:18:09.966 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:18:09.967 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:18:09.968 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.969 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:18:09.970 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:18:09.971 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:18:09.972 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:18:09.973 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:18:09.974 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:18:09.975 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:18:09.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:18:09.977 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:18:09.978 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:18:09.979 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.25, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.25, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.25, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.25, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:18:09.980 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:18:09.981 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:18:09.982 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:18:09.983 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:18:09.984 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:18:09.985 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:18:09.986 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.25, 2024-09-16T10:18:09.987 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.25, 2024-09-16T10:18:09.987 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:18:09.987 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.25, 2024-09-16T10:18:09.987 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.25, 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:18:09.988 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:18:09.989 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:18:09.990 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:18:09.991 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:18:09.992 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:18:09.993 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 7, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_0", 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:14:42.862958+0000", 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:18:09.995 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:18:09.996 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "86", 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:18:09.997 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:18:09.998 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:18:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.98665237426757812, 2024-09-16T10:18:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.99000000953674316, 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.99000000953674316 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 144, 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:18:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 156, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.006 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.007 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.008 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:18:10.011 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.012 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:18:10.013 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0.893218994140625, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.014 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 156, 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.015 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:18:10.016 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:18:10.017 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.018 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 156, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:18:10.019 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.021 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.022 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:18:10.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 156, 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:10.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:10.027 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.027 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:18:10.027 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:18:10.027 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.027 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.027 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:18:10.029 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.030 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 156, 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.031 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:18:10.032 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.033 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:18:10.034 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 156, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:18:10.035 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:18:10.036 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.037 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:18:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:33.234888+0000", 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.040 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:16:54.555092+0000", 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.041 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:18:04.255008+0000", 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.042 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:38.584488+0000", 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:57.972003+0000", 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:19.764081+0000", 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.044 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:27.384891+0000", 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:18:10.045 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:17:41.335130+0000", 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [], 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.10", 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.046 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.18", 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:18:10.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.12", 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 7, 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 3 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:10.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.049 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:18:10.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:18:10.049 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:10.049 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:18:10.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.051 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:18:10.052 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:18:10.053 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:10.054 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:10.054 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:10.054 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:10.084 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:10.084 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:10.121 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:10.122 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:10.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:10.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:10.162 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:10.162 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:10.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:10.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: osdmap e158: 8 total, 8 up, 8 in 2024-09-16T10:18:10.291 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: pgmap v209: 158 pgs: 158 active+clean; 629 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 1.8 scrub starts 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 1.8 scrub ok 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 6.16 scrub starts 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 6.16 scrub ok 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 5.1f scrub starts 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: 5.1f scrub ok 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3323702822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/863577030' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:18:10.292 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: osdmap e158: 8 total, 8 up, 8 in 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: pgmap v209: 158 pgs: 158 active+clean; 629 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 1.8 scrub starts 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 1.8 scrub ok 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 6.16 scrub starts 2024-09-16T10:18:10.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 6.16 scrub ok 2024-09-16T10:18:10.294 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 5.1f scrub starts 2024-09-16T10:18:10.294 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: 5.1f scrub ok 2024-09-16T10:18:10.294 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3323702822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:10.294 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/863577030' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:18:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:18:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:18:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:18:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:18:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:18:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:18:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: osdmap e158: 8 total, 8 up, 8 in 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: pgmap v209: 158 pgs: 158 active+clean; 629 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 1.8 scrub starts 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 1.8 scrub ok 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 6.16 scrub starts 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 6.16 scrub ok 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 5.1f scrub starts 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: 5.1f scrub ok 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3323702822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:10.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/863577030' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:18:10.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:10.427 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:10.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:10.571 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:10.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:10.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:10.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:10.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:10.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:10.973 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:10.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:11.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:11.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:11.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:11.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:11.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:11.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:11.335 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:11.370 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:11.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:11.472 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:11.515 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:11.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:11.617 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:11.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:11.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:11.762 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:11.801 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:11.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:11.903 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:11.953 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:11.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:12.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:12.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:12.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:12.185 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:12.227 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:12.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:12.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:12.364 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:12.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:12.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:18:12.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:12 smithi063 ceph-mon[29993]: pgmap v210: 158 pgs: 158 active+clean; 629 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-16T10:18:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:12 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:18:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:12 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:18:12.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:12 smithi063 ceph-mon[39470]: pgmap v210: 158 pgs: 158 active+clean; 629 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-16T10:18:12.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:12 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:18:12.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:12 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:18:12.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:12.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:12.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:12 smithi165 ceph-mon[32696]: pgmap v210: 158 pgs: 158 active+clean; 629 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-16T10:18:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:12 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:18:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:12 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:18:12.593 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:12.636 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:12.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:12.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:12.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:12.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:12.849 INFO:tasks.thrashosds.thrasher:j is {'epoch': 158, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:18:08.916825+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:13:06.309218+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '154', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T10:14:42.862958+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9866523742675781, 'average_primary_affinity': 0.9900000095367432, 'average_primary_affinity_weighted': 0.9900000095367432}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 156, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.893218994140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 156, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 156, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 156, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 156, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 156, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:33.234888+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:16:54.555092+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:18:04.255008+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:38.584488+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:57.972003+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:19.764081+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:27.384891+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:17:41.335130+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '6.12', 'mappings': [{'from': 7, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:18:12.849 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:18:12.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:12.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:12.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:12.970 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:12.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:13.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:13.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.170 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:13.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:13.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:18:13.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:13.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:13.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:13.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 7.3 scrub starts 2024-09-16T10:18:13.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:18:13.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 7.3 scrub ok 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 5.1b scrub starts 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 5.1b scrub ok 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: osdmap e159: 8 total, 8 up, 8 in 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 5.11 scrub starts 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 5.11 scrub ok 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 3.13 scrub starts 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[29993]: 3.13 scrub ok 2024-09-16T10:18:13.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 7.3 scrub starts 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 7.3 scrub ok 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 5.1b scrub starts 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 5.1b scrub ok 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: osdmap e159: 8 total, 8 up, 8 in 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 5.11 scrub starts 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 5.11 scrub ok 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 3.13 scrub starts 2024-09-16T10:18:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:13 smithi063 ceph-mon[39470]: 3.13 scrub ok 2024-09-16T10:18:13.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 7.3 scrub starts 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 7.3 scrub ok 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 5.1b scrub starts 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 5.1b scrub ok 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: osdmap e159: 8 total, 8 up, 8 in 2024-09-16T10:18:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 5.11 scrub starts 2024-09-16T10:18:13.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 5.11 scrub ok 2024-09-16T10:18:13.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 3.13 scrub starts 2024-09-16T10:18:13.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:13 smithi165 ceph-mon[32696]: 3.13 scrub ok 2024-09-16T10:18:13.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:13.655 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:13.697 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:13.799 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:13.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:13.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:13.944 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:18:13.960 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:13.960 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:13.982 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:13.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:13.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:14.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:14.136 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:14.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:14.238 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:14.292 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:14.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:14.306 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:14 smithi063 ceph-mon[39470]: pgmap v212: 158 pgs: 158 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-16T10:18:14.306 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:14 smithi063 ceph-mon[29993]: pgmap v212: 158 pgs: 158 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-16T10:18:14.393 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:14.438 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:14.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:14.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:14.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:14 smithi165 ceph-mon[32696]: pgmap v212: 158 pgs: 158 active+clean; 564 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:14.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:14.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:18:14.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:14.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:14.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:14.704 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:14.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:14.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:14.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:14.919 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:14.962 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:14.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:15.034 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:15.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:15.141 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:15.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:15.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:15.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:15.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:15.338 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:15.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:15.425 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:15.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:15.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:15 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:18:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:15 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:18:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:15 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:18:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:15 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:18:15.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:15.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:15 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:18:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:15 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:18:15.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:15.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:15.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:15.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:15.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:18:15.766 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:18:15.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:15.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:15.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:15.914 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:15.953 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:15.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:16.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:16.097 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:16.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:16.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:16.238 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:16.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:16.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:16.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:16.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:16.453 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:16.454 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:16.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:16.488 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:16.488 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:16.488 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:16 smithi063 ceph-mon[29993]: pgmap v213: 158 pgs: 158 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-09-16T10:18:16.488 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:16 smithi063 ceph-mon[29993]: 3.17 scrub starts 2024-09-16T10:18:16.489 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:16 smithi063 ceph-mon[29993]: 3.17 scrub ok 2024-09-16T10:18:16.489 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:16 smithi063 ceph-mon[39470]: pgmap v213: 158 pgs: 158 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-09-16T10:18:16.489 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:16 smithi063 ceph-mon[39470]: 3.17 scrub starts 2024-09-16T10:18:16.489 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:16 smithi063 ceph-mon[39470]: 3.17 scrub ok 2024-09-16T10:18:16.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:16.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:16.523 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:16.523 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:16.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:16 smithi165 ceph-mon[32696]: pgmap v213: 158 pgs: 158 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.30k op/s 2024-09-16T10:18:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:16 smithi165 ceph-mon[32696]: 3.17 scrub starts 2024-09-16T10:18:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:16 smithi165 ceph-mon[32696]: 3.17 scrub ok 2024-09-16T10:18:16.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:16.558 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:16.592 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:16.592 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:16.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:16.625 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:16.626 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:16.646 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:16.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:16.670 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:16.671 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:16.749 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:16.790 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:16.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:16.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:16.933 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:16.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:17.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:17.080 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:17.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:17.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:17.228 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-16T10:18:15.115294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297474611,"num_objects":4730,"num_object_clones":0,"num_object_copies":14190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":993,"num_read_kb":745,"num_write":234459,"num_write_kb":7633710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84047,"ondisk_log_size":84047,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1947060,"kb_used_data":995036,"kb_used_omap":12,"kb_used_meta":951795,"kb_avail":748014156,"statfs":{"total":767960285184,"available":765966495744,"internally_reserved":0,"allocated":1018916864,"data_stored":1015554170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":974638641},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,7,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-688062464,"num_objects":-10499,"num_object_clones":0,"num_object_copies":-31497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10499,"num_whiteouts":0,"num_read":-116,"num_read_kb":-221,"num_write":10386,"num_write_kb":-1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002352"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":237,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.203539+0000","last_change":"2024-09-16T10:18:07.119362+0000","last_active":"2024-09-16T10:18:12.203539+0000","last_peered":"2024-09-16T10:18:12.203539+0000","last_clean":"2024-09-16T10:18:12.203539+0000","last_became_active":"2024-09-16T10:13:37.096707+0000","last_became_peered":"2024-09-16T10:13:37.096707+0000","last_unstale":"2024-09-16T10:18:12.203539+0000","last_undegraded":"2024-09-16T10:18:12.203539+0000","last_fullsized":"2024-09-16T10:18:12.203539+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:07.119072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.119072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:27.494366+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":265,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190291+0000","last_change":"2024-09-16T10:18:08.743312+0000","last_active":"2024-09-16T10:18:12.190291+0000","last_peered":"2024-09-16T10:18:12.190291+0000","last_clean":"2024-09-16T10:18:12.190291+0000","last_became_active":"2024-09-16T10:15:08.925269+0000","last_became_peered":"2024-09-16T10:15:08.925269+0000","last_unstale":"2024-09-16T10:18:12.190291+0000","last_undegraded":"2024-09-16T10:18:12.190291+0000","last_fullsized":"2024-09-16T10:18:12.190291+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:08.743259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.743259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:27.839953+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202599+0000","last_change":"2024-09-16T10:17:04.062121+0000","last_active":"2024-09-16T10:18:12.202599+0000","last_peered":"2024-09-16T10:18:12.202599+0000","last_clean":"2024-09-16T10:18:12.202599+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:18:12.202599+0000","last_undegraded":"2024-09-16T10:18:12.202599+0000","last_fullsized":"2024-09-16T10:18:12.202599+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:04.062060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.062060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.222918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":208,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190411+0000","last_change":"2024-09-16T10:16:16.760046+0000","last_active":"2024-09-16T10:18:12.190411+0000","last_peered":"2024-09-16T10:18:12.190411+0000","last_clean":"2024-09-16T10:18:12.190411+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:18:12.190411+0000","last_undegraded":"2024-09-16T10:18:12.190411+0000","last_fullsized":"2024-09-16T10:18:12.190411+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.759981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.759981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:58.834889+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":209,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.210140+0000","last_change":"2024-09-16T10:16:42.246257+0000","last_active":"2024-09-16T10:18:12.210140+0000","last_peered":"2024-09-16T10:18:12.210140+0000","last_clean":"2024-09-16T10:18:12.210140+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:18:12.210140+0000","last_undegraded":"2024-09-16T10:18:12.210140+0000","last_fullsized":"2024-09-16T10:18:12.210140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:42.246193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.246193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":230,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.211552+0000","last_change":"2024-09-16T10:17:13.018487+0000","last_active":"2024-09-16T10:18:12.211552+0000","last_peered":"2024-09-16T10:18:12.211552+0000","last_clean":"2024-09-16T10:18:12.211552+0000","last_became_active":"2024-09-16T10:13:37.092087+0000","last_became_peered":"2024-09-16T10:13:37.092087+0000","last_unstale":"2024-09-16T10:18:12.211552+0000","last_undegraded":"2024-09-16T10:18:12.211552+0000","last_fullsized":"2024-09-16T10:18:12.211552+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:13.018412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.018412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.241294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":250,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213098+0000","last_change":"2024-09-16T10:16:31.217281+0000","last_active":"2024-09-16T10:18:12.213098+0000","last_peered":"2024-09-16T10:18:12.213098+0000","last_clean":"2024-09-16T10:18:12.213098+0000","last_became_active":"2024-09-16T10:13:39.093667+0000","last_became_peered":"2024-09-16T10:13:39.093667+0000","last_unstale":"2024-09-16T10:18:12.213098+0000","last_undegraded":"2024-09-16T10:18:12.213098+0000","last_fullsized":"2024-09-16T10:18:12.213098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:16:31.217192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.217192+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":220,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190221+0000","last_change":"2024-09-16T10:17:14.871814+0000","last_active":"2024-09-16T10:18:12.190221+0000","last_peered":"2024-09-16T10:18:12.190221+0000","last_clean":"2024-09-16T10:18:12.190221+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:18:12.190221+0000","last_undegraded":"2024-09-16T10:18:12.190221+0000","last_fullsized":"2024-09-16T10:18:12.190221+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.871750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.871750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:31.182111+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208650+0000","last_change":"2024-09-16T10:15:29.351071+0000","last_active":"2024-09-16T10:18:12.208650+0000","last_peered":"2024-09-16T10:18:12.208650+0000","last_clean":"2024-09-16T10:18:12.208650+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:18:12.208650+0000","last_undegraded":"2024-09-16T10:18:12.208650+0000","last_fullsized":"2024-09-16T10:18:12.208650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:29.351000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:29.351000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":240,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190587+0000","last_change":"2024-09-16T10:16:32.833966+0000","last_active":"2024-09-16T10:18:12.190587+0000","last_peered":"2024-09-16T10:18:12.190587+0000","last_clean":"2024-09-16T10:18:12.190587+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:18:12.190587+0000","last_undegraded":"2024-09-16T10:18:12.190587+0000","last_fullsized":"2024-09-16T10:18:12.190587+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:32.833895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.833895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:00.865075+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208896+0000","last_change":"2024-09-16T10:18:12.133375+0000","last_active":"2024-09-16T10:18:12.208896+0000","last_peered":"2024-09-16T10:18:12.208896+0000","last_clean":"2024-09-16T10:18:12.208896+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:18:12.208896+0000","last_undegraded":"2024-09-16T10:18:12.208896+0000","last_fullsized":"2024-09-16T10:18:12.208896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.133300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.133300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:18.560125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":216,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202942+0000","last_change":"2024-09-16T10:17:12.089641+0000","last_active":"2024-09-16T10:18:12.202942+0000","last_peered":"2024-09-16T10:18:12.202942+0000","last_clean":"2024-09-16T10:18:12.202942+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:18:12.202942+0000","last_undegraded":"2024-09-16T10:18:12.202942+0000","last_fullsized":"2024-09-16T10:18:12.202942+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.089577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.089577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.096958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":213,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208378+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:18:12.208378+0000","last_peered":"2024-09-16T10:18:12.208378+0000","last_clean":"2024-09-16T10:18:12.208378+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:18:12.208378+0000","last_undegraded":"2024-09-16T10:18:12.208378+0000","last_fullsized":"2024-09-16T10:18:12.208378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.593997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":234,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190548+0000","last_change":"2024-09-16T10:17:16.937447+0000","last_active":"2024-09-16T10:18:12.190548+0000","last_peered":"2024-09-16T10:18:12.190548+0000","last_clean":"2024-09-16T10:18:12.190548+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:18:12.190548+0000","last_undegraded":"2024-09-16T10:18:12.190548+0000","last_fullsized":"2024-09-16T10:18:12.190548+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:17:16.937353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.937353+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.410589+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":213,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213489+0000","last_change":"2024-09-16T10:15:16.573685+0000","last_active":"2024-09-16T10:18:12.213489+0000","last_peered":"2024-09-16T10:18:12.213489+0000","last_clean":"2024-09-16T10:18:12.213489+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:18:12.213489+0000","last_undegraded":"2024-09-16T10:18:12.213489+0000","last_fullsized":"2024-09-16T10:18:12.213489+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:16.573614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:16.573614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:57.677714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":217,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207891+0000","last_change":"2024-09-16T10:18:11.125675+0000","last_active":"2024-09-16T10:18:12.207891+0000","last_peered":"2024-09-16T10:18:12.207891+0000","last_clean":"2024-09-16T10:18:12.207891+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:18:12.207891+0000","last_undegraded":"2024-09-16T10:18:12.207891+0000","last_fullsized":"2024-09-16T10:18:12.207891+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:11.125619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:11.125619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:18.260480+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":217,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190740+0000","last_change":"2024-09-16T10:17:48.893103+0000","last_active":"2024-09-16T10:18:12.190740+0000","last_peered":"2024-09-16T10:18:12.190740+0000","last_clean":"2024-09-16T10:18:12.190740+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:18:12.190740+0000","last_undegraded":"2024-09-16T10:18:12.190740+0000","last_fullsized":"2024-09-16T10:18:12.190740+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:48.893042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:48.893042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:18.645144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204818+0000","last_change":"2024-09-16T10:13:37.098871+0000","last_active":"2024-09-16T10:18:12.204818+0000","last_peered":"2024-09-16T10:18:12.204818+0000","last_clean":"2024-09-16T10:18:12.204818+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:18:12.204818+0000","last_undegraded":"2024-09-16T10:18:12.204818+0000","last_fullsized":"2024-09-16T10:18:12.204818+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:53.279867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":234,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204422+0000","last_change":"2024-09-16T10:13:39.099840+0000","last_active":"2024-09-16T10:18:12.204422+0000","last_peered":"2024-09-16T10:18:12.204422+0000","last_clean":"2024-09-16T10:18:12.204422+0000","last_became_active":"2024-09-16T10:13:39.099416+0000","last_became_peered":"2024-09-16T10:13:39.099416+0000","last_unstale":"2024-09-16T10:18:12.204422+0000","last_undegraded":"2024-09-16T10:18:12.204422+0000","last_fullsized":"2024-09-16T10:18:12.204422+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:28.390262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207844+0000","last_change":"2024-09-16T10:16:16.067128+0000","last_active":"2024-09-16T10:18:12.207844+0000","last_peered":"2024-09-16T10:18:12.207844+0000","last_clean":"2024-09-16T10:18:12.207844+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:18:12.207844+0000","last_undegraded":"2024-09-16T10:18:12.207844+0000","last_fullsized":"2024-09-16T10:18:12.207844+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:16.066855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:16.066855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.192100+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":49,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207359+0000","last_change":"2024-09-16T10:17:42.978180+0000","last_active":"2024-09-16T10:18:12.207359+0000","last_peered":"2024-09-16T10:18:12.207359+0000","last_clean":"2024-09-16T10:18:12.207359+0000","last_became_active":"2024-09-16T10:17:37.309204+0000","last_became_peered":"2024-09-16T10:17:37.309204+0000","last_unstale":"2024-09-16T10:18:12.207359+0000","last_undegraded":"2024-09-16T10:18:12.207359+0000","last_fullsized":"2024-09-16T10:18:12.207359+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:42.978105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.978105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:49.488053+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":212,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190699+0000","last_change":"2024-09-16T10:13:37.094376+0000","last_active":"2024-09-16T10:18:12.190699+0000","last_peered":"2024-09-16T10:18:12.190699+0000","last_clean":"2024-09-16T10:18:12.190699+0000","last_became_active":"2024-09-16T10:13:37.093737+0000","last_became_peered":"2024-09-16T10:13:37.093737+0000","last_unstale":"2024-09-16T10:18:12.190699+0000","last_undegraded":"2024-09-16T10:18:12.190699+0000","last_fullsized":"2024-09-16T10:18:12.190699+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:50.915856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":267,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208912+0000","last_change":"2024-09-16T10:16:35.181066+0000","last_active":"2024-09-16T10:18:12.208912+0000","last_peered":"2024-09-16T10:18:12.208912+0000","last_clean":"2024-09-16T10:18:12.208912+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:18:12.208912+0000","last_undegraded":"2024-09-16T10:18:12.208912+0000","last_fullsized":"2024-09-16T10:18:12.208912+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:16:35.180977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:35.180977+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":228,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.191375+0000","last_change":"2024-09-16T10:18:05.796009+0000","last_active":"2024-09-16T10:18:12.191375+0000","last_peered":"2024-09-16T10:18:12.191375+0000","last_clean":"2024-09-16T10:18:12.191375+0000","last_became_active":"2024-09-16T10:13:41.112417+0000","last_became_peered":"2024-09-16T10:13:41.112417+0000","last_unstale":"2024-09-16T10:18:12.191375+0000","last_undegraded":"2024-09-16T10:18:12.191375+0000","last_fullsized":"2024-09-16T10:18:12.191375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:05.795954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.795954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:19.243409+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.215520+0000","last_change":"2024-09-16T10:16:37.672462+0000","last_active":"2024-09-16T10:18:12.215520+0000","last_peered":"2024-09-16T10:18:12.215520+0000","last_clean":"2024-09-16T10:18:12.215520+0000","last_became_active":"2024-09-16T10:13:43.119390+0000","last_became_peered":"2024-09-16T10:13:43.119390+0000","last_unstale":"2024-09-16T10:18:12.215520+0000","last_undegraded":"2024-09-16T10:18:12.215520+0000","last_fullsized":"2024-09-16T10:18:12.215520+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":236,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202741+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:18:12.202741+0000","last_peered":"2024-09-16T10:18:12.202741+0000","last_clean":"2024-09-16T10:18:12.202741+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:18:12.202741+0000","last_undegraded":"2024-09-16T10:18:12.202741+0000","last_fullsized":"2024-09-16T10:18:12.202741+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.691808+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":240,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205751+0000","last_change":"2024-09-16T10:17:35.057981+0000","last_active":"2024-09-16T10:18:12.205751+0000","last_peered":"2024-09-16T10:18:12.205751+0000","last_clean":"2024-09-16T10:18:12.205751+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:18:12.205751+0000","last_undegraded":"2024-09-16T10:18:12.205751+0000","last_fullsized":"2024-09-16T10:18:12.205751+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:35.057906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.057906+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.808000+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":296,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207507+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:18:12.207507+0000","last_peered":"2024-09-16T10:18:12.207507+0000","last_clean":"2024-09-16T10:18:12.207507+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:18:12.207507+0000","last_undegraded":"2024-09-16T10:18:12.207507+0000","last_fullsized":"2024-09-16T10:18:12.207507+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:36.152890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":24,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207892+0000","last_change":"2024-09-16T10:18:08.115768+0000","last_active":"2024-09-16T10:18:12.207892+0000","last_peered":"2024-09-16T10:18:12.207892+0000","last_clean":"2024-09-16T10:18:12.207892+0000","last_became_active":"2024-09-16T10:18:06.889088+0000","last_became_peered":"2024-09-16T10:18:06.889088+0000","last_unstale":"2024-09-16T10:18:12.207892+0000","last_undegraded":"2024-09-16T10:18:12.207892+0000","last_fullsized":"2024-09-16T10:18:12.207892+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.115716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.115716+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:33.843494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"159'15250","reported_seq":15418,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.099896+0000","last_change":"2024-09-16T10:14:43.957473+0000","last_active":"2024-09-16T10:18:13.099896+0000","last_peered":"2024-09-16T10:18:13.099896+0000","last_clean":"2024-09-16T10:18:13.099896+0000","last_became_active":"2024-09-16T10:14:43.957174+0000","last_became_peered":"2024-09-16T10:14:43.957174+0000","last_unstale":"2024-09-16T10:18:13.099896+0000","last_undegraded":"2024-09-16T10:18:13.099896+0000","last_fullsized":"2024-09-16T10:18:13.099896+0000","mapping_epoch":81,"log_start":"130'9521","ondisk_log_start":"130'9521","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:53.078606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19791872,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15250,"num_write_kb":497664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202164+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:18:12.202164+0000","last_peered":"2024-09-16T10:18:12.202164+0000","last_clean":"2024-09-16T10:18:12.202164+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:18:12.202164+0000","last_undegraded":"2024-09-16T10:18:12.202164+0000","last_fullsized":"2024-09-16T10:18:12.202164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:12.078495+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212448+0000","last_change":"2024-09-16T10:17:45.221342+0000","last_active":"2024-09-16T10:18:12.212448+0000","last_peered":"2024-09-16T10:18:12.212448+0000","last_clean":"2024-09-16T10:18:12.212448+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:18:12.212448+0000","last_undegraded":"2024-09-16T10:18:12.212448+0000","last_fullsized":"2024-09-16T10:18:12.212448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.221281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.221281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:57.786575+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":268,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190876+0000","last_change":"2024-09-16T10:13:41.118989+0000","last_active":"2024-09-16T10:18:12.190876+0000","last_peered":"2024-09-16T10:18:12.190876+0000","last_clean":"2024-09-16T10:18:12.190876+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:18:12.190876+0000","last_undegraded":"2024-09-16T10:18:12.190876+0000","last_fullsized":"2024-09-16T10:18:12.190876+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.935259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":209,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208558+0000","last_change":"2024-09-16T10:15:54.314741+0000","last_active":"2024-09-16T10:18:12.208558+0000","last_peered":"2024-09-16T10:18:12.208558+0000","last_clean":"2024-09-16T10:18:12.208558+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:18:12.208558+0000","last_undegraded":"2024-09-16T10:18:12.208558+0000","last_fullsized":"2024-09-16T10:18:12.208558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:54.314680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.314680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":24,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.191091+0000","last_change":"2024-09-16T10:18:07.752771+0000","last_active":"2024-09-16T10:18:12.191091+0000","last_peered":"2024-09-16T10:18:12.191091+0000","last_clean":"2024-09-16T10:18:12.191091+0000","last_became_active":"2024-09-16T10:18:06.885242+0000","last_became_peered":"2024-09-16T10:18:06.885242+0000","last_unstale":"2024-09-16T10:18:12.191091+0000","last_undegraded":"2024-09-16T10:18:12.191091+0000","last_fullsized":"2024-09-16T10:18:12.191091+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.752706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.752706+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:17.947313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"159'14518","reported_seq":14777,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.406301+0000","last_change":"2024-09-16T10:18:12.115201+0000","last_active":"2024-09-16T10:18:13.406301+0000","last_peered":"2024-09-16T10:18:13.406301+0000","last_clean":"2024-09-16T10:18:13.406301+0000","last_became_active":"2024-09-16T10:14:43.952713+0000","last_became_peered":"2024-09-16T10:14:43.952713+0000","last_unstale":"2024-09-16T10:18:13.406301+0000","last_undegraded":"2024-09-16T10:18:13.406301+0000","last_fullsized":"2024-09-16T10:18:13.406301+0000","mapping_epoch":81,"log_start":"135'10254","ondisk_log_start":"135'10254","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'14432","last_scrub_stamp":"2024-09-16T10:18:12.115069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.115069+0000","objects_scrubbed":387,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:30.680206+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19398656,"num_objects":296,"num_object_clones":0,"num_object_copies":888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14518,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":258,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212225+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:18:12.212225+0000","last_peered":"2024-09-16T10:18:12.212225+0000","last_clean":"2024-09-16T10:18:12.212225+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:18:12.212225+0000","last_undegraded":"2024-09-16T10:18:12.212225+0000","last_fullsized":"2024-09-16T10:18:12.212225+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:27.688406+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":50,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.214556+0000","last_change":"2024-09-16T10:17:38.748281+0000","last_active":"2024-09-16T10:18:12.214556+0000","last_peered":"2024-09-16T10:18:12.214556+0000","last_clean":"2024-09-16T10:18:12.214556+0000","last_became_active":"2024-09-16T10:17:37.306205+0000","last_became_peered":"2024-09-16T10:17:37.306205+0000","last_unstale":"2024-09-16T10:18:12.214556+0000","last_undegraded":"2024-09-16T10:18:12.214556+0000","last_fullsized":"2024-09-16T10:18:12.214556+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:38.748219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.748219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:05.492417+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":205,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190358+0000","last_change":"2024-09-16T10:13:41.118898+0000","last_active":"2024-09-16T10:18:12.190358+0000","last_peered":"2024-09-16T10:18:12.190358+0000","last_clean":"2024-09-16T10:18:12.190358+0000","last_became_active":"2024-09-16T10:13:41.118533+0000","last_became_peered":"2024-09-16T10:13:41.118533+0000","last_unstale":"2024-09-16T10:18:12.190358+0000","last_undegraded":"2024-09-16T10:18:12.190358+0000","last_fullsized":"2024-09-16T10:18:12.190358+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:17.909064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":225,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190552+0000","last_change":"2024-09-16T10:18:04.793611+0000","last_active":"2024-09-16T10:18:12.190552+0000","last_peered":"2024-09-16T10:18:12.190552+0000","last_clean":"2024-09-16T10:18:12.190552+0000","last_became_active":"2024-09-16T10:13:43.122457+0000","last_became_peered":"2024-09-16T10:13:43.122457+0000","last_unstale":"2024-09-16T10:18:12.190552+0000","last_undegraded":"2024-09-16T10:18:12.190552+0000","last_fullsized":"2024-09-16T10:18:12.190552+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.793540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.793540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:11.539428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":29,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.209141+0000","last_change":"2024-09-16T10:18:07.024979+0000","last_active":"2024-09-16T10:18:12.209141+0000","last_peered":"2024-09-16T10:18:12.209141+0000","last_clean":"2024-09-16T10:18:12.209141+0000","last_became_active":"2024-09-16T10:18:06.896380+0000","last_became_peered":"2024-09-16T10:18:06.896380+0000","last_unstale":"2024-09-16T10:18:12.209141+0000","last_undegraded":"2024-09-16T10:18:12.209141+0000","last_fullsized":"2024-09-16T10:18:12.209141+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.024848+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:26.258314+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"159'14572","reported_seq":14736,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.406755+0000","last_change":"2024-09-16T10:14:43.953541+0000","last_active":"2024-09-16T10:18:13.406755+0000","last_peered":"2024-09-16T10:18:13.406755+0000","last_clean":"2024-09-16T10:18:13.406755+0000","last_became_active":"2024-09-16T10:14:43.952908+0000","last_became_peered":"2024-09-16T10:14:43.952908+0000","last_unstale":"2024-09-16T10:18:13.406755+0000","last_undegraded":"2024-09-16T10:18:13.406755+0000","last_fullsized":"2024-09-16T10:18:13.406755+0000","mapping_epoch":81,"log_start":"135'10309","ondisk_log_start":"135'10309","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.179329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16646144,"num_objects":254,"num_object_clones":0,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14572,"num_write_kb":474432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":254,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190394+0000","last_change":"2024-09-16T10:18:06.739000+0000","last_active":"2024-09-16T10:18:12.190394+0000","last_peered":"2024-09-16T10:18:12.190394+0000","last_clean":"2024-09-16T10:18:12.190394+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:18:12.190394+0000","last_undegraded":"2024-09-16T10:18:12.190394+0000","last_fullsized":"2024-09-16T10:18:12.190394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:18:06.738912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:06.738912+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:13.229418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":230,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205923+0000","last_change":"2024-09-16T10:17:11.050221+0000","last_active":"2024-09-16T10:18:12.205923+0000","last_peered":"2024-09-16T10:18:12.205923+0000","last_clean":"2024-09-16T10:18:12.205923+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:18:12.205923+0000","last_undegraded":"2024-09-16T10:18:12.205923+0000","last_fullsized":"2024-09-16T10:18:12.205923+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:17:11.050150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:11.050150+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:29.263200+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":289,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212001+0000","last_change":"2024-09-16T10:18:04.023115+0000","last_active":"2024-09-16T10:18:12.212001+0000","last_peered":"2024-09-16T10:18:12.212001+0000","last_clean":"2024-09-16T10:18:12.212001+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:18:12.212001+0000","last_undegraded":"2024-09-16T10:18:12.212001+0000","last_fullsized":"2024-09-16T10:18:12.212001+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:04.022579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.022579+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:15.555787+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213122+0000","last_change":"2024-09-16T10:16:03.498672+0000","last_active":"2024-09-16T10:18:12.213122+0000","last_peered":"2024-09-16T10:18:12.213122+0000","last_clean":"2024-09-16T10:18:12.213122+0000","last_became_active":"2024-09-16T10:13:43.284355+0000","last_became_peered":"2024-09-16T10:13:43.284355+0000","last_unstale":"2024-09-16T10:18:12.213122+0000","last_undegraded":"2024-09-16T10:18:12.213122+0000","last_fullsized":"2024-09-16T10:18:12.213122+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:21.723374+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":100,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207287+0000","last_change":"2024-09-16T10:18:07.192860+0000","last_active":"2024-09-16T10:18:12.207287+0000","last_peered":"2024-09-16T10:18:12.207287+0000","last_clean":"2024-09-16T10:18:12.207287+0000","last_became_active":"2024-09-16T10:18:06.896951+0000","last_became_peered":"2024-09-16T10:18:06.896951+0000","last_unstale":"2024-09-16T10:18:12.207287+0000","last_undegraded":"2024-09-16T10:18:12.207287+0000","last_fullsized":"2024-09-16T10:18:12.207287+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.192777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.192777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:32.150544+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"159'14751","reported_seq":14952,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.074066+0000","last_change":"2024-09-16T10:17:37.439115+0000","last_active":"2024-09-16T10:18:13.074066+0000","last_peered":"2024-09-16T10:18:13.074066+0000","last_clean":"2024-09-16T10:18:13.074066+0000","last_became_active":"2024-09-16T10:14:43.952043+0000","last_became_peered":"2024-09-16T10:14:43.952043+0000","last_unstale":"2024-09-16T10:18:13.074066+0000","last_undegraded":"2024-09-16T10:18:13.074066+0000","last_fullsized":"2024-09-16T10:18:13.074066+0000","mapping_epoch":81,"log_start":"133'10192","ondisk_log_start":"133'10192","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'12535","last_scrub_stamp":"2024-09-16T10:17:37.438837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:37.438837+0000","objects_scrubbed":2521,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.510192+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18939904,"num_objects":289,"num_object_clones":0,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14751,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":249,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.201655+0000","last_change":"2024-09-16T10:17:44.394734+0000","last_active":"2024-09-16T10:18:12.201655+0000","last_peered":"2024-09-16T10:18:12.201655+0000","last_clean":"2024-09-16T10:18:12.201655+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:18:12.201655+0000","last_undegraded":"2024-09-16T10:18:12.201655+0000","last_fullsized":"2024-09-16T10:18:12.201655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:44.392303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.392303+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:47.877424+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":230,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208803+0000","last_change":"2024-09-16T10:18:04.187225+0000","last_active":"2024-09-16T10:18:12.208803+0000","last_peered":"2024-09-16T10:18:12.208803+0000","last_clean":"2024-09-16T10:18:12.208803+0000","last_became_active":"2024-09-16T10:13:37.092762+0000","last_became_peered":"2024-09-16T10:13:37.092762+0000","last_unstale":"2024-09-16T10:18:12.208803+0000","last_undegraded":"2024-09-16T10:18:12.208803+0000","last_fullsized":"2024-09-16T10:18:12.208803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.187181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.187181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:21.749892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":277,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204401+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:18:12.204401+0000","last_peered":"2024-09-16T10:18:12.204401+0000","last_clean":"2024-09-16T10:18:12.204401+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:18:12.204401+0000","last_undegraded":"2024-09-16T10:18:12.204401+0000","last_fullsized":"2024-09-16T10:18:12.204401+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.998943+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":217,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.203153+0000","last_change":"2024-09-16T10:18:06.111219+0000","last_active":"2024-09-16T10:18:12.203153+0000","last_peered":"2024-09-16T10:18:12.203153+0000","last_clean":"2024-09-16T10:18:12.203153+0000","last_became_active":"2024-09-16T10:13:43.118670+0000","last_became_peered":"2024-09-16T10:13:43.118670+0000","last_unstale":"2024-09-16T10:18:12.203153+0000","last_undegraded":"2024-09-16T10:18:12.203153+0000","last_fullsized":"2024-09-16T10:18:12.203153+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:06.111161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:06.111161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:34.669953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":100,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206904+0000","last_change":"2024-09-16T10:18:08.171074+0000","last_active":"2024-09-16T10:18:12.206904+0000","last_peered":"2024-09-16T10:18:12.206904+0000","last_clean":"2024-09-16T10:18:12.206904+0000","last_became_active":"2024-09-16T10:18:06.910799+0000","last_became_peered":"2024-09-16T10:18:06.910799+0000","last_unstale":"2024-09-16T10:18:12.206904+0000","last_undegraded":"2024-09-16T10:18:12.206904+0000","last_fullsized":"2024-09-16T10:18:12.206904+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.171021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.171021+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:35.593678+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"159'14491","reported_seq":14763,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.034014+0000","last_change":"2024-09-16T10:17:39.034260+0000","last_active":"2024-09-16T10:18:13.034014+0000","last_peered":"2024-09-16T10:18:13.034014+0000","last_clean":"2024-09-16T10:18:13.034014+0000","last_became_active":"2024-09-16T10:14:43.955808+0000","last_became_peered":"2024-09-16T10:14:43.955808+0000","last_unstale":"2024-09-16T10:18:13.034014+0000","last_undegraded":"2024-09-16T10:18:13.034014+0000","last_fullsized":"2024-09-16T10:18:13.034014+0000","mapping_epoch":81,"log_start":"130'9074","ondisk_log_start":"130'9074","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'12331","last_scrub_stamp":"2024-09-16T10:17:39.034057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:39.034057+0000","objects_scrubbed":2480,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:57.622104+0000","scrub_duration":582,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":299,"num_object_clones":0,"num_object_copies":897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14491,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":261,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204546+0000","last_change":"2024-09-16T10:17:03.051621+0000","last_active":"2024-09-16T10:18:12.204546+0000","last_peered":"2024-09-16T10:18:12.204546+0000","last_clean":"2024-09-16T10:18:12.204546+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:18:12.204546+0000","last_undegraded":"2024-09-16T10:18:12.204546+0000","last_fullsized":"2024-09-16T10:18:12.204546+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:17:03.050732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.050732+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:15.857216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":230,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202381+0000","last_change":"2024-09-16T10:16:33.949944+0000","last_active":"2024-09-16T10:18:12.202381+0000","last_peered":"2024-09-16T10:18:12.202381+0000","last_clean":"2024-09-16T10:18:12.202381+0000","last_became_active":"2024-09-16T10:13:37.092450+0000","last_became_peered":"2024-09-16T10:13:37.092450+0000","last_unstale":"2024-09-16T10:18:12.202381+0000","last_undegraded":"2024-09-16T10:18:12.202381+0000","last_fullsized":"2024-09-16T10:18:12.202381+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:33.949798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.949798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:02.476284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":213,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206529+0000","last_change":"2024-09-16T10:16:07.174774+0000","last_active":"2024-09-16T10:18:12.206529+0000","last_peered":"2024-09-16T10:18:12.206529+0000","last_clean":"2024-09-16T10:18:12.206529+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:18:12.206529+0000","last_undegraded":"2024-09-16T10:18:12.206529+0000","last_fullsized":"2024-09-16T10:18:12.206529+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:07.174690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:07.174690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:34.045269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202505+0000","last_change":"2024-09-16T10:17:03.061305+0000","last_active":"2024-09-16T10:18:12.202505+0000","last_peered":"2024-09-16T10:18:12.202505+0000","last_clean":"2024-09-16T10:18:12.202505+0000","last_became_active":"2024-09-16T10:13:43.283171+0000","last_became_peered":"2024-09-16T10:13:43.283171+0000","last_unstale":"2024-09-16T10:18:12.202505+0000","last_undegraded":"2024-09-16T10:18:12.202505+0000","last_fullsized":"2024-09-16T10:18:12.202505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.257731+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":24,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207760+0000","last_change":"2024-09-16T10:18:07.131191+0000","last_active":"2024-09-16T10:18:12.207760+0000","last_peered":"2024-09-16T10:18:12.207760+0000","last_clean":"2024-09-16T10:18:12.207760+0000","last_became_active":"2024-09-16T10:18:06.887486+0000","last_became_peered":"2024-09-16T10:18:06.887486+0000","last_unstale":"2024-09-16T10:18:12.207760+0000","last_undegraded":"2024-09-16T10:18:12.207760+0000","last_fullsized":"2024-09-16T10:18:12.207760+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.131122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.131122+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:17.390723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"159'14765","reported_seq":15013,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.116661+0000","last_change":"2024-09-16T10:16:14.720883+0000","last_active":"2024-09-16T10:18:13.116661+0000","last_peered":"2024-09-16T10:18:13.116661+0000","last_clean":"2024-09-16T10:18:13.116661+0000","last_became_active":"2024-09-16T10:14:43.957195+0000","last_became_peered":"2024-09-16T10:14:43.957195+0000","last_unstale":"2024-09-16T10:18:13.116661+0000","last_undegraded":"2024-09-16T10:18:13.116661+0000","last_fullsized":"2024-09-16T10:18:13.116661+0000","mapping_epoch":81,"log_start":"129'9148","ondisk_log_start":"129'9148","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'7198","last_scrub_stamp":"2024-09-16T10:16:14.720775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:14.720775+0000","objects_scrubbed":7123,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:33.046935+0000","scrub_duration":2606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14765,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":338,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.201788+0000","last_change":"2024-09-16T10:17:15.133634+0000","last_active":"2024-09-16T10:18:12.201788+0000","last_peered":"2024-09-16T10:18:12.201788+0000","last_clean":"2024-09-16T10:18:12.201788+0000","last_became_active":"2024-09-16T10:13:39.098231+0000","last_became_peered":"2024-09-16T10:13:39.098231+0000","last_unstale":"2024-09-16T10:18:12.201788+0000","last_undegraded":"2024-09-16T10:18:12.201788+0000","last_fullsized":"2024-09-16T10:18:12.201788+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:17:15.133542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.133542+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:42.091484+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.191252+0000","last_change":"2024-09-16T10:17:35.928758+0000","last_active":"2024-09-16T10:18:12.191252+0000","last_peered":"2024-09-16T10:18:12.191252+0000","last_clean":"2024-09-16T10:18:12.191252+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:18:12.191252+0000","last_undegraded":"2024-09-16T10:18:12.191252+0000","last_fullsized":"2024-09-16T10:18:12.191252+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.928701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.928701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:03.464982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":240,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208507+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:18:12.208507+0000","last_peered":"2024-09-16T10:18:12.208507+0000","last_clean":"2024-09-16T10:18:12.208507+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:18:12.208507+0000","last_undegraded":"2024-09-16T10:18:12.208507+0000","last_fullsized":"2024-09-16T10:18:12.208507+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:02.800871+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":213,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205516+0000","last_change":"2024-09-16T10:16:04.405318+0000","last_active":"2024-09-16T10:18:12.205516+0000","last_peered":"2024-09-16T10:18:12.205516+0000","last_clean":"2024-09-16T10:18:12.205516+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:18:12.205516+0000","last_undegraded":"2024-09-16T10:18:12.205516+0000","last_fullsized":"2024-09-16T10:18:12.205516+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:04.405243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.405243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.095983+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.211284+0000","last_change":"2024-09-16T10:17:14.029861+0000","last_active":"2024-09-16T10:18:12.211284+0000","last_peered":"2024-09-16T10:18:12.211284+0000","last_clean":"2024-09-16T10:18:12.211284+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:18:12.211284+0000","last_undegraded":"2024-09-16T10:18:12.211284+0000","last_fullsized":"2024-09-16T10:18:12.211284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.029754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.029754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.182687+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":291,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212798+0000","last_change":"2024-09-16T10:18:07.653542+0000","last_active":"2024-09-16T10:18:12.212798+0000","last_peered":"2024-09-16T10:18:12.212798+0000","last_clean":"2024-09-16T10:18:12.212798+0000","last_became_active":"2024-09-16T10:18:06.878175+0000","last_became_peered":"2024-09-16T10:18:06.878175+0000","last_unstale":"2024-09-16T10:18:12.212798+0000","last_undegraded":"2024-09-16T10:18:12.212798+0000","last_fullsized":"2024-09-16T10:18:12.212798+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.653477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.653477+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:22.389692+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"159'14534","reported_seq":14728,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.407744+0000","last_change":"2024-09-16T10:17:49.973218+0000","last_active":"2024-09-16T10:18:13.407744+0000","last_peered":"2024-09-16T10:18:13.407744+0000","last_clean":"2024-09-16T10:18:13.407744+0000","last_became_active":"2024-09-16T10:14:43.956362+0000","last_became_peered":"2024-09-16T10:14:43.956362+0000","last_unstale":"2024-09-16T10:18:13.407744+0000","last_undegraded":"2024-09-16T10:18:13.407744+0000","last_fullsized":"2024-09-16T10:18:13.407744+0000","mapping_epoch":81,"log_start":"130'9318","ondisk_log_start":"130'9318","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'13050","last_scrub_stamp":"2024-09-16T10:17:49.973131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.973131+0000","objects_scrubbed":1765,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:03.107363+0000","scrub_duration":742,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14534,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":245,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202210+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:18:12.202210+0000","last_peered":"2024-09-16T10:18:12.202210+0000","last_clean":"2024-09-16T10:18:12.202210+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:18:12.202210+0000","last_undegraded":"2024-09-16T10:18:12.202210+0000","last_fullsized":"2024-09-16T10:18:12.202210+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:14.062329+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206521+0000","last_change":"2024-09-16T10:13:37.093128+0000","last_active":"2024-09-16T10:18:12.206521+0000","last_peered":"2024-09-16T10:18:12.206521+0000","last_clean":"2024-09-16T10:18:12.206521+0000","last_became_active":"2024-09-16T10:13:37.092738+0000","last_became_peered":"2024-09-16T10:13:37.092738+0000","last_unstale":"2024-09-16T10:18:12.206521+0000","last_undegraded":"2024-09-16T10:18:12.206521+0000","last_fullsized":"2024-09-16T10:18:12.206521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:53.417622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":239,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.203997+0000","last_change":"2024-09-16T10:17:47.455218+0000","last_active":"2024-09-16T10:18:12.203997+0000","last_peered":"2024-09-16T10:18:12.203997+0000","last_clean":"2024-09-16T10:18:12.203997+0000","last_became_active":"2024-09-16T10:13:36.082979+0000","last_became_peered":"2024-09-16T10:13:36.082979+0000","last_unstale":"2024-09-16T10:18:12.203997+0000","last_undegraded":"2024-09-16T10:18:12.203997+0000","last_fullsized":"2024-09-16T10:18:12.203997+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:17:47.455142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:47.455142+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:15.928277+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0.001379061,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":206,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205292+0000","last_change":"2024-09-16T10:13:41.112698+0000","last_active":"2024-09-16T10:18:12.205292+0000","last_peered":"2024-09-16T10:18:12.205292+0000","last_clean":"2024-09-16T10:18:12.205292+0000","last_became_active":"2024-09-16T10:13:41.112378+0000","last_became_peered":"2024-09-16T10:13:41.112378+0000","last_unstale":"2024-09-16T10:18:12.205292+0000","last_undegraded":"2024-09-16T10:18:12.205292+0000","last_fullsized":"2024-09-16T10:18:12.205292+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.471638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213941+0000","last_change":"2024-09-16T10:17:12.578269+0000","last_active":"2024-09-16T10:18:12.213941+0000","last_peered":"2024-09-16T10:18:12.213941+0000","last_clean":"2024-09-16T10:18:12.213941+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:18:12.213941+0000","last_undegraded":"2024-09-16T10:18:12.213941+0000","last_fullsized":"2024-09-16T10:18:12.213941+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.578203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.578203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.857634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":24,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202970+0000","last_change":"2024-09-16T10:18:07.544995+0000","last_active":"2024-09-16T10:18:12.202970+0000","last_peered":"2024-09-16T10:18:12.202970+0000","last_clean":"2024-09-16T10:18:12.202970+0000","last_became_active":"2024-09-16T10:18:06.877808+0000","last_became_peered":"2024-09-16T10:18:06.877808+0000","last_unstale":"2024-09-16T10:18:12.202970+0000","last_undegraded":"2024-09-16T10:18:12.202970+0000","last_fullsized":"2024-09-16T10:18:12.202970+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.544697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.544697+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:26.750354+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"159'14724","reported_seq":14892,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.096850+0000","last_change":"2024-09-16T10:14:43.959854+0000","last_active":"2024-09-16T10:18:13.096850+0000","last_peered":"2024-09-16T10:18:13.096850+0000","last_clean":"2024-09-16T10:18:13.096850+0000","last_became_active":"2024-09-16T10:14:43.959612+0000","last_became_peered":"2024-09-16T10:14:43.959612+0000","last_unstale":"2024-09-16T10:18:13.096850+0000","last_undegraded":"2024-09-16T10:18:13.096850+0000","last_fullsized":"2024-09-16T10:18:13.096850+0000","mapping_epoch":81,"log_start":"132'9772","ondisk_log_start":"132'9772","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:51.697189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":304,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":251,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205839+0000","last_change":"2024-09-16T10:17:44.234787+0000","last_active":"2024-09-16T10:18:12.205839+0000","last_peered":"2024-09-16T10:18:12.205839+0000","last_clean":"2024-09-16T10:18:12.205839+0000","last_became_active":"2024-09-16T10:13:39.091465+0000","last_became_peered":"2024-09-16T10:13:39.091465+0000","last_unstale":"2024-09-16T10:18:12.205839+0000","last_undegraded":"2024-09-16T10:18:12.205839+0000","last_fullsized":"2024-09-16T10:18:12.205839+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:44.234584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.234584+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:05.324187+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":237,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212768+0000","last_change":"2024-09-16T10:17:15.102575+0000","last_active":"2024-09-16T10:18:12.212768+0000","last_peered":"2024-09-16T10:18:12.212768+0000","last_clean":"2024-09-16T10:18:12.212768+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:18:12.212768+0000","last_undegraded":"2024-09-16T10:18:12.212768+0000","last_fullsized":"2024-09-16T10:18:12.212768+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:15.102512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.102512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:43.326101+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":511,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213456+0000","last_change":"2024-09-16T10:16:05.559992+0000","last_active":"2024-09-16T10:18:12.213456+0000","last_peered":"2024-09-16T10:18:12.213456+0000","last_clean":"2024-09-16T10:18:12.213456+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:18:12.213456+0000","last_undegraded":"2024-09-16T10:18:12.213456+0000","last_fullsized":"2024-09-16T10:18:12.213456+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:16:05.559913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.559913+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:20.685013+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.209112+0000","last_change":"2024-09-16T10:17:35.559111+0000","last_active":"2024-09-16T10:18:12.209112+0000","last_peered":"2024-09-16T10:18:12.209112+0000","last_clean":"2024-09-16T10:18:12.209112+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:18:12.209112+0000","last_undegraded":"2024-09-16T10:18:12.209112+0000","last_fullsized":"2024-09-16T10:18:12.209112+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.559045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.559045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:44.355200+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.201291+0000","last_change":"2024-09-16T10:17:16.162854+0000","last_active":"2024-09-16T10:18:12.201291+0000","last_peered":"2024-09-16T10:18:12.201291+0000","last_clean":"2024-09-16T10:18:12.201291+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:18:12.201291+0000","last_undegraded":"2024-09-16T10:18:12.201291+0000","last_fullsized":"2024-09-16T10:18:12.201291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:16.162771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.162771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:36.273519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"159'15036","reported_seq":15204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.379595+0000","last_change":"2024-09-16T10:14:43.957718+0000","last_active":"2024-09-16T10:18:13.379595+0000","last_peered":"2024-09-16T10:18:13.379595+0000","last_clean":"2024-09-16T10:18:13.379595+0000","last_became_active":"2024-09-16T10:14:43.957476+0000","last_became_peered":"2024-09-16T10:14:43.957476+0000","last_unstale":"2024-09-16T10:18:13.379595+0000","last_undegraded":"2024-09-16T10:18:13.379595+0000","last_fullsized":"2024-09-16T10:18:13.379595+0000","mapping_epoch":81,"log_start":"131'9889","ondisk_log_start":"131'9889","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:02.051180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15036,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":295,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207128+0000","last_change":"2024-09-16T10:16:10.182532+0000","last_active":"2024-09-16T10:18:12.207128+0000","last_peered":"2024-09-16T10:18:12.207128+0000","last_clean":"2024-09-16T10:18:12.207128+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:18:12.207128+0000","last_undegraded":"2024-09-16T10:18:12.207128+0000","last_fullsized":"2024-09-16T10:18:12.207128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:16:10.182468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:10.182468+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:25.968626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":50,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208335+0000","last_change":"2024-09-16T10:17:43.413171+0000","last_active":"2024-09-16T10:18:12.208335+0000","last_peered":"2024-09-16T10:18:12.208335+0000","last_clean":"2024-09-16T10:18:12.208335+0000","last_became_active":"2024-09-16T10:17:37.298525+0000","last_became_peered":"2024-09-16T10:17:37.298525+0000","last_unstale":"2024-09-16T10:18:12.208335+0000","last_undegraded":"2024-09-16T10:18:12.208335+0000","last_fullsized":"2024-09-16T10:18:12.208335+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.413088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.413088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:54.071484+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":285,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213597+0000","last_change":"2024-09-16T10:18:05.620756+0000","last_active":"2024-09-16T10:18:12.213597+0000","last_peered":"2024-09-16T10:18:12.213597+0000","last_clean":"2024-09-16T10:18:12.213597+0000","last_became_active":"2024-09-16T10:18:04.861294+0000","last_became_peered":"2024-09-16T10:18:04.861294+0000","last_unstale":"2024-09-16T10:18:12.213597+0000","last_undegraded":"2024-09-16T10:18:12.213597+0000","last_fullsized":"2024-09-16T10:18:12.213597+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:05.620685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.620685+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:32.901988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.215612+0000","last_change":"2024-09-16T10:16:43.622567+0000","last_active":"2024-09-16T10:18:12.215612+0000","last_peered":"2024-09-16T10:18:12.215612+0000","last_clean":"2024-09-16T10:18:12.215612+0000","last_became_active":"2024-09-16T10:13:41.107906+0000","last_became_peered":"2024-09-16T10:13:41.107906+0000","last_unstale":"2024-09-16T10:18:12.215612+0000","last_undegraded":"2024-09-16T10:18:12.215612+0000","last_fullsized":"2024-09-16T10:18:12.215612+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208993+0000","last_change":"2024-09-16T10:16:34.311115+0000","last_active":"2024-09-16T10:18:12.208993+0000","last_peered":"2024-09-16T10:18:12.208993+0000","last_clean":"2024-09-16T10:18:12.208993+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:18:12.208993+0000","last_undegraded":"2024-09-16T10:18:12.208993+0000","last_fullsized":"2024-09-16T10:18:12.208993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:34.311052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:34.311052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"159'14766","reported_seq":14935,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.065097+0000","last_change":"2024-09-16T10:14:43.958855+0000","last_active":"2024-09-16T10:18:13.065097+0000","last_peered":"2024-09-16T10:18:13.065097+0000","last_clean":"2024-09-16T10:18:13.065097+0000","last_became_active":"2024-09-16T10:14:43.958616+0000","last_became_peered":"2024-09-16T10:14:43.958616+0000","last_unstale":"2024-09-16T10:18:13.065097+0000","last_undegraded":"2024-09-16T10:18:13.065097+0000","last_fullsized":"2024-09-16T10:18:13.065097+0000","mapping_epoch":81,"log_start":"128'9096","ondisk_log_start":"128'9096","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":5670,"log_dups_size":0,"ondisk_log_size":5670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:55.128443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19726360,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14766,"num_write_kb":482113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":244,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190986+0000","last_change":"2024-09-16T10:17:06.903885+0000","last_active":"2024-09-16T10:18:12.190986+0000","last_peered":"2024-09-16T10:18:12.190986+0000","last_clean":"2024-09-16T10:18:12.190986+0000","last_became_active":"2024-09-16T10:13:39.094558+0000","last_became_peered":"2024-09-16T10:13:39.094558+0000","last_unstale":"2024-09-16T10:18:12.190986+0000","last_undegraded":"2024-09-16T10:18:12.190986+0000","last_fullsized":"2024-09-16T10:18:12.190986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:06.903808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.903808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:21.156884+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":213,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190698+0000","last_change":"2024-09-16T10:13:37.095376+0000","last_active":"2024-09-16T10:18:12.190698+0000","last_peered":"2024-09-16T10:18:12.190698+0000","last_clean":"2024-09-16T10:18:12.190698+0000","last_became_active":"2024-09-16T10:13:37.095024+0000","last_became_peered":"2024-09-16T10:13:37.095024+0000","last_unstale":"2024-09-16T10:18:12.190698+0000","last_undegraded":"2024-09-16T10:18:12.190698+0000","last_fullsized":"2024-09-16T10:18:12.190698+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:54.809271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.209019+0000","last_change":"2024-09-16T10:15:30.302921+0000","last_active":"2024-09-16T10:18:12.209019+0000","last_peered":"2024-09-16T10:18:12.209019+0000","last_clean":"2024-09-16T10:18:12.209019+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:18:12.209019+0000","last_undegraded":"2024-09-16T10:18:12.209019+0000","last_fullsized":"2024-09-16T10:18:12.209019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:30.302859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:30.302859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:58.258006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":49,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207028+0000","last_change":"2024-09-16T10:17:47.013243+0000","last_active":"2024-09-16T10:18:12.207028+0000","last_peered":"2024-09-16T10:18:12.207028+0000","last_clean":"2024-09-16T10:18:12.207028+0000","last_became_active":"2024-09-16T10:17:37.302404+0000","last_became_peered":"2024-09-16T10:17:37.302404+0000","last_unstale":"2024-09-16T10:18:12.207028+0000","last_undegraded":"2024-09-16T10:18:12.207028+0000","last_fullsized":"2024-09-16T10:18:12.207028+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:47.013173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:47.013173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:07.781255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":31,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207710+0000","last_change":"2024-09-16T10:18:09.128140+0000","last_active":"2024-09-16T10:18:12.207710+0000","last_peered":"2024-09-16T10:18:12.207710+0000","last_clean":"2024-09-16T10:18:12.207710+0000","last_became_active":"2024-09-16T10:18:06.890806+0000","last_became_peered":"2024-09-16T10:18:06.890806+0000","last_unstale":"2024-09-16T10:18:12.207710+0000","last_undegraded":"2024-09-16T10:18:12.207710+0000","last_fullsized":"2024-09-16T10:18:12.207710+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:09.128070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.128070+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:31.190804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"159'14206","reported_seq":14443,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.305021+0000","last_change":"2024-09-16T10:15:56.898348+0000","last_active":"2024-09-16T10:18:13.305021+0000","last_peered":"2024-09-16T10:18:13.305021+0000","last_clean":"2024-09-16T10:18:13.305021+0000","last_became_active":"2024-09-16T10:14:43.953871+0000","last_became_peered":"2024-09-16T10:14:43.953871+0000","last_unstale":"2024-09-16T10:18:13.305021+0000","last_undegraded":"2024-09-16T10:18:13.305021+0000","last_fullsized":"2024-09-16T10:18:13.305021+0000","mapping_epoch":81,"log_start":"124'8048","ondisk_log_start":"124'8048","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'5944","last_scrub_stamp":"2024-09-16T10:15:56.897937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.897937+0000","objects_scrubbed":5880,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:26.380881+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14206,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":237,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190957+0000","last_change":"2024-09-16T10:13:39.094034+0000","last_active":"2024-09-16T10:18:12.190957+0000","last_peered":"2024-09-16T10:18:12.190957+0000","last_clean":"2024-09-16T10:18:12.190957+0000","last_became_active":"2024-09-16T10:13:39.093252+0000","last_became_peered":"2024-09-16T10:13:39.093252+0000","last_unstale":"2024-09-16T10:18:12.190957+0000","last_undegraded":"2024-09-16T10:18:12.190957+0000","last_fullsized":"2024-09-16T10:18:12.190957+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:31.892247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206685+0000","last_change":"2024-09-16T10:16:05.172896+0000","last_active":"2024-09-16T10:18:12.206685+0000","last_peered":"2024-09-16T10:18:12.206685+0000","last_clean":"2024-09-16T10:18:12.206685+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:18:12.206685+0000","last_undegraded":"2024-09-16T10:18:12.206685+0000","last_fullsized":"2024-09-16T10:18:12.206685+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:05.172834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.172834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:18.062065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.201453+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:18:12.201453+0000","last_peered":"2024-09-16T10:18:12.201453+0000","last_clean":"2024-09-16T10:18:12.201453+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:18:12.201453+0000","last_undegraded":"2024-09-16T10:18:12.201453+0000","last_fullsized":"2024-09-16T10:18:12.201453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.117933+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204741+0000","last_change":"2024-09-16T10:17:50.491329+0000","last_active":"2024-09-16T10:18:12.204741+0000","last_peered":"2024-09-16T10:18:12.204741+0000","last_clean":"2024-09-16T10:18:12.204741+0000","last_became_active":"2024-09-16T10:13:43.125537+0000","last_became_peered":"2024-09-16T10:13:43.125537+0000","last_unstale":"2024-09-16T10:18:12.204741+0000","last_undegraded":"2024-09-16T10:18:12.204741+0000","last_fullsized":"2024-09-16T10:18:12.204741+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:50.491234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.491234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:03.062143+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":291,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213678+0000","last_change":"2024-09-16T10:18:08.644230+0000","last_active":"2024-09-16T10:18:12.213678+0000","last_peered":"2024-09-16T10:18:12.213678+0000","last_clean":"2024-09-16T10:18:12.213678+0000","last_became_active":"2024-09-16T10:18:06.898533+0000","last_became_peered":"2024-09-16T10:18:06.898533+0000","last_unstale":"2024-09-16T10:18:12.213678+0000","last_undegraded":"2024-09-16T10:18:12.213678+0000","last_fullsized":"2024-09-16T10:18:12.213678+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.644158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.644158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:35.444567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"159'14312","reported_seq":14588,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.401282+0000","last_change":"2024-09-16T10:17:44.054053+0000","last_active":"2024-09-16T10:18:13.401282+0000","last_peered":"2024-09-16T10:18:13.401282+0000","last_clean":"2024-09-16T10:18:13.401282+0000","last_became_active":"2024-09-16T10:14:43.954996+0000","last_became_peered":"2024-09-16T10:14:43.954996+0000","last_unstale":"2024-09-16T10:18:13.401282+0000","last_undegraded":"2024-09-16T10:18:13.401282+0000","last_fullsized":"2024-09-16T10:18:13.401282+0000","mapping_epoch":81,"log_start":"130'9106","ondisk_log_start":"130'9106","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'12448","last_scrub_stamp":"2024-09-16T10:17:44.053912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.053912+0000","objects_scrubbed":2146,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:52.599103+0000","scrub_duration":827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":256,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14312,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":224,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.191323+0000","last_change":"2024-09-16T10:13:39.094005+0000","last_active":"2024-09-16T10:18:12.191323+0000","last_peered":"2024-09-16T10:18:12.191323+0000","last_clean":"2024-09-16T10:18:12.191323+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:18:12.191323+0000","last_undegraded":"2024-09-16T10:18:12.191323+0000","last_fullsized":"2024-09-16T10:18:12.191323+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:59.821228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":214,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.210518+0000","last_change":"2024-09-16T10:13:37.091058+0000","last_active":"2024-09-16T10:18:12.210518+0000","last_peered":"2024-09-16T10:18:12.210518+0000","last_clean":"2024-09-16T10:18:12.210518+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:18:12.210518+0000","last_undegraded":"2024-09-16T10:18:12.210518+0000","last_fullsized":"2024-09-16T10:18:12.210518+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:56.218583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":270,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.215340+0000","last_change":"2024-09-16T10:13:41.112146+0000","last_active":"2024-09-16T10:18:12.215340+0000","last_peered":"2024-09-16T10:18:12.215340+0000","last_clean":"2024-09-16T10:18:12.215340+0000","last_became_active":"2024-09-16T10:13:41.110513+0000","last_became_peered":"2024-09-16T10:13:41.110513+0000","last_unstale":"2024-09-16T10:18:12.215340+0000","last_undegraded":"2024-09-16T10:18:12.215340+0000","last_fullsized":"2024-09-16T10:18:12.215340+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.430003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":216,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190654+0000","last_change":"2024-09-16T10:17:43.963680+0000","last_active":"2024-09-16T10:18:12.190654+0000","last_peered":"2024-09-16T10:18:12.190654+0000","last_clean":"2024-09-16T10:18:12.190654+0000","last_became_active":"2024-09-16T10:13:43.119258+0000","last_became_peered":"2024-09-16T10:13:43.119258+0000","last_unstale":"2024-09-16T10:18:12.190654+0000","last_undegraded":"2024-09-16T10:18:12.190654+0000","last_fullsized":"2024-09-16T10:18:12.190654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.963621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.963621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:13.887722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"159'14557","reported_seq":14725,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.242498+0000","last_change":"2024-09-16T10:14:43.952726+0000","last_active":"2024-09-16T10:18:13.242498+0000","last_peered":"2024-09-16T10:18:13.242498+0000","last_clean":"2024-09-16T10:18:13.242498+0000","last_became_active":"2024-09-16T10:14:43.952378+0000","last_became_peered":"2024-09-16T10:14:43.952378+0000","last_unstale":"2024-09-16T10:18:13.242498+0000","last_undegraded":"2024-09-16T10:18:13.242498+0000","last_fullsized":"2024-09-16T10:18:13.242498+0000","mapping_epoch":81,"log_start":"125'8430","ondisk_log_start":"125'8430","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:33.407300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14557,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":241,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204451+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:18:12.204451+0000","last_peered":"2024-09-16T10:18:12.204451+0000","last_clean":"2024-09-16T10:18:12.204451+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:18:12.204451+0000","last_undegraded":"2024-09-16T10:18:12.204451+0000","last_fullsized":"2024-09-16T10:18:12.204451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:24.984845+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208473+0000","last_change":"2024-09-16T10:15:13.340979+0000","last_active":"2024-09-16T10:18:12.208473+0000","last_peered":"2024-09-16T10:18:12.208473+0000","last_clean":"2024-09-16T10:18:12.208473+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:18:12.208473+0000","last_undegraded":"2024-09-16T10:18:12.208473+0000","last_fullsized":"2024-09-16T10:18:12.208473+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:13.340929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:13.340929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.449973+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":206,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206291+0000","last_change":"2024-09-16T10:13:41.117371+0000","last_active":"2024-09-16T10:18:12.206291+0000","last_peered":"2024-09-16T10:18:12.206291+0000","last_clean":"2024-09-16T10:18:12.206291+0000","last_became_active":"2024-09-16T10:13:41.117058+0000","last_became_peered":"2024-09-16T10:13:41.117058+0000","last_unstale":"2024-09-16T10:18:12.206291+0000","last_undegraded":"2024-09-16T10:18:12.206291+0000","last_fullsized":"2024-09-16T10:18:12.206291+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:56.543849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212683+0000","last_change":"2024-09-16T10:18:05.104074+0000","last_active":"2024-09-16T10:18:12.212683+0000","last_peered":"2024-09-16T10:18:12.212683+0000","last_clean":"2024-09-16T10:18:12.212683+0000","last_became_active":"2024-09-16T10:13:43.122284+0000","last_became_peered":"2024-09-16T10:13:43.122284+0000","last_unstale":"2024-09-16T10:18:12.212683+0000","last_undegraded":"2024-09-16T10:18:12.212683+0000","last_fullsized":"2024-09-16T10:18:12.212683+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:15.354336+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"159'14361","reported_seq":14636,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.405458+0000","last_change":"2024-09-16T10:17:42.364230+0000","last_active":"2024-09-16T10:18:13.405458+0000","last_peered":"2024-09-16T10:18:13.405458+0000","last_clean":"2024-09-16T10:18:13.405458+0000","last_became_active":"2024-09-16T10:14:43.959635+0000","last_became_peered":"2024-09-16T10:14:43.959635+0000","last_unstale":"2024-09-16T10:18:13.405458+0000","last_undegraded":"2024-09-16T10:18:13.405458+0000","last_fullsized":"2024-09-16T10:18:13.405458+0000","mapping_epoch":81,"log_start":"130'9141","ondisk_log_start":"130'9141","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'12433","last_scrub_stamp":"2024-09-16T10:17:42.364000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.364000+0000","objects_scrubbed":2254,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:48.430009+0000","scrub_duration":1172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14361,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":256,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206540+0000","last_change":"2024-09-16T10:16:42.258515+0000","last_active":"2024-09-16T10:18:12.206540+0000","last_peered":"2024-09-16T10:18:12.206540+0000","last_clean":"2024-09-16T10:18:12.206540+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:18:12.206540+0000","last_undegraded":"2024-09-16T10:18:12.206540+0000","last_fullsized":"2024-09-16T10:18:12.206540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:16:42.258433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:42.258433+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:00.169260+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.214186+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:18:12.214186+0000","last_peered":"2024-09-16T10:18:12.214186+0000","last_clean":"2024-09-16T10:18:12.214186+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:18:12.214186+0000","last_undegraded":"2024-09-16T10:18:12.214186+0000","last_fullsized":"2024-09-16T10:18:12.214186+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:59.700257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206397+0000","last_change":"2024-09-16T10:16:43.420443+0000","last_active":"2024-09-16T10:18:12.206397+0000","last_peered":"2024-09-16T10:18:12.206397+0000","last_clean":"2024-09-16T10:18:12.206397+0000","last_became_active":"2024-09-16T10:13:41.113035+0000","last_became_peered":"2024-09-16T10:13:41.113035+0000","last_unstale":"2024-09-16T10:18:12.206397+0000","last_undegraded":"2024-09-16T10:18:12.206397+0000","last_fullsized":"2024-09-16T10:18:12.206397+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.420249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.420249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:52.513152+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213519+0000","last_change":"2024-09-16T10:17:08.596403+0000","last_active":"2024-09-16T10:18:12.213519+0000","last_peered":"2024-09-16T10:18:12.213519+0000","last_clean":"2024-09-16T10:18:12.213519+0000","last_became_active":"2024-09-16T10:13:43.121274+0000","last_became_peered":"2024-09-16T10:13:43.121274+0000","last_unstale":"2024-09-16T10:18:12.213519+0000","last_undegraded":"2024-09-16T10:18:12.213519+0000","last_fullsized":"2024-09-16T10:18:12.213519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.176712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"159'14337","reported_seq":14505,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.380060+0000","last_change":"2024-09-16T10:14:43.957583+0000","last_active":"2024-09-16T10:18:13.380060+0000","last_peered":"2024-09-16T10:18:13.380060+0000","last_clean":"2024-09-16T10:18:13.380060+0000","last_became_active":"2024-09-16T10:14:43.957207+0000","last_became_peered":"2024-09-16T10:14:43.957207+0000","last_unstale":"2024-09-16T10:18:13.380060+0000","last_undegraded":"2024-09-16T10:18:13.380060+0000","last_fullsized":"2024-09-16T10:18:13.380060+0000","mapping_epoch":81,"log_start":"135'10098","ondisk_log_start":"135'10098","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:14:42.920859+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:32.156614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18939904,"num_objects":289,"num_object_clones":0,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14337,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":248,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190539+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:18:12.190539+0000","last_peered":"2024-09-16T10:18:12.190539+0000","last_clean":"2024-09-16T10:18:12.190539+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:18:12.190539+0000","last_undegraded":"2024-09-16T10:18:12.190539+0000","last_fullsized":"2024-09-16T10:18:12.190539+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:49.150565+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":222,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.214340+0000","last_change":"2024-09-16T10:15:11.631330+0000","last_active":"2024-09-16T10:18:12.214340+0000","last_peered":"2024-09-16T10:18:12.214340+0000","last_clean":"2024-09-16T10:18:12.214340+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:18:12.214340+0000","last_undegraded":"2024-09-16T10:18:12.214340+0000","last_fullsized":"2024-09-16T10:18:12.214340+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:11.631262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:11.631262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:52.716987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":206,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206867+0000","last_change":"2024-09-16T10:13:41.114195+0000","last_active":"2024-09-16T10:18:12.206867+0000","last_peered":"2024-09-16T10:18:12.206867+0000","last_clean":"2024-09-16T10:18:12.206867+0000","last_became_active":"2024-09-16T10:13:41.111933+0000","last_became_peered":"2024-09-16T10:13:41.111933+0000","last_unstale":"2024-09-16T10:18:12.206867+0000","last_undegraded":"2024-09-16T10:18:12.206867+0000","last_fullsized":"2024-09-16T10:18:12.206867+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:20.476397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.210252+0000","last_change":"2024-09-16T10:16:32.171799+0000","last_active":"2024-09-16T10:18:12.210252+0000","last_peered":"2024-09-16T10:18:12.210252+0000","last_clean":"2024-09-16T10:18:12.210252+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:18:12.210252+0000","last_undegraded":"2024-09-16T10:18:12.210252+0000","last_fullsized":"2024-09-16T10:18:12.210252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.171738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.171738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"159'14843","reported_seq":15084,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:13.281449+0000","last_change":"2024-09-16T10:16:02.172955+0000","last_active":"2024-09-16T10:18:13.281449+0000","last_peered":"2024-09-16T10:18:13.281449+0000","last_clean":"2024-09-16T10:18:13.281449+0000","last_became_active":"2024-09-16T10:14:43.954475+0000","last_became_peered":"2024-09-16T10:14:43.954475+0000","last_unstale":"2024-09-16T10:18:13.281449+0000","last_undegraded":"2024-09-16T10:18:13.281449+0000","last_fullsized":"2024-09-16T10:18:13.281449+0000","mapping_epoch":81,"log_start":"133'9922","ondisk_log_start":"133'9922","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'6489","last_scrub_stamp":"2024-09-16T10:16:02.172829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:14:42.920859+0000","last_clean_scrub_stamp":"2024-09-16T10:16:02.172829+0000","objects_scrubbed":6427,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:34.148686+0000","scrub_duration":2056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14843,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":235,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207843+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:18:12.207843+0000","last_peered":"2024-09-16T10:18:12.207843+0000","last_clean":"2024-09-16T10:18:12.207843+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:18:12.207843+0000","last_undegraded":"2024-09-16T10:18:12.207843+0000","last_fullsized":"2024-09-16T10:18:12.207843+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.521661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":234,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.214593+0000","last_change":"2024-09-16T10:15:25.529229+0000","last_active":"2024-09-16T10:18:12.214593+0000","last_peered":"2024-09-16T10:18:12.214593+0000","last_clean":"2024-09-16T10:18:12.214593+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:18:12.214593+0000","last_undegraded":"2024-09-16T10:18:12.214593+0000","last_fullsized":"2024-09-16T10:18:12.214593+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:15:25.529178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:25.529178+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":286,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213038+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:18:12.213038+0000","last_peered":"2024-09-16T10:18:12.213038+0000","last_clean":"2024-09-16T10:18:12.213038+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:18:12.213038+0000","last_undegraded":"2024-09-16T10:18:12.213038+0000","last_fullsized":"2024-09-16T10:18:12.213038+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:16.912935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":218,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208014+0000","last_change":"2024-09-16T10:17:46.443359+0000","last_active":"2024-09-16T10:18:12.208014+0000","last_peered":"2024-09-16T10:18:12.208014+0000","last_clean":"2024-09-16T10:18:12.208014+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:18:12.208014+0000","last_undegraded":"2024-09-16T10:18:12.208014+0000","last_fullsized":"2024-09-16T10:18:12.208014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:46.443295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:46.443295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:59.510329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":220,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.206517+0000","last_change":"2024-09-16T10:15:55.157938+0000","last_active":"2024-09-16T10:18:12.206517+0000","last_peered":"2024-09-16T10:18:12.206517+0000","last_clean":"2024-09-16T10:18:12.206517+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:18:12.206517+0000","last_undegraded":"2024-09-16T10:18:12.206517+0000","last_fullsized":"2024-09-16T10:18:12.206517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:55.157860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:55.157860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:02.154094+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":242,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.216617+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:18:12.216617+0000","last_peered":"2024-09-16T10:18:12.216617+0000","last_clean":"2024-09-16T10:18:12.216617+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:18:12.216617+0000","last_undegraded":"2024-09-16T10:18:12.216617+0000","last_fullsized":"2024-09-16T10:18:12.216617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:47.968995+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":205,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208567+0000","last_change":"2024-09-16T10:13:41.113609+0000","last_active":"2024-09-16T10:18:12.208567+0000","last_peered":"2024-09-16T10:18:12.208567+0000","last_clean":"2024-09-16T10:18:12.208567+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:18:12.208567+0000","last_undegraded":"2024-09-16T10:18:12.208567+0000","last_fullsized":"2024-09-16T10:18:12.208567+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:52.407184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":209,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213898+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:18:12.213898+0000","last_peered":"2024-09-16T10:18:12.213898+0000","last_clean":"2024-09-16T10:18:12.213898+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:18:12.213898+0000","last_undegraded":"2024-09-16T10:18:12.213898+0000","last_fullsized":"2024-09-16T10:18:12.213898+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":240,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205863+0000","last_change":"2024-09-16T10:15:54.369062+0000","last_active":"2024-09-16T10:18:12.205863+0000","last_peered":"2024-09-16T10:18:12.205863+0000","last_clean":"2024-09-16T10:18:12.205863+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:18:12.205863+0000","last_undegraded":"2024-09-16T10:18:12.205863+0000","last_fullsized":"2024-09-16T10:18:12.205863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:54.368987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:54.368987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:17.060776+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213243+0000","last_change":"2024-09-16T10:15:28.537739+0000","last_active":"2024-09-16T10:18:12.213243+0000","last_peered":"2024-09-16T10:18:12.213243+0000","last_clean":"2024-09-16T10:18:12.213243+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:18:12.213243+0000","last_undegraded":"2024-09-16T10:18:12.213243+0000","last_fullsized":"2024-09-16T10:18:12.213243+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:28.537673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:28.537673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:53.681531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":213,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.213322+0000","last_change":"2024-09-16T10:16:37.237971+0000","last_active":"2024-09-16T10:18:12.213322+0000","last_peered":"2024-09-16T10:18:12.213322+0000","last_clean":"2024-09-16T10:18:12.213322+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:18:12.213322+0000","last_undegraded":"2024-09-16T10:18:12.213322+0000","last_fullsized":"2024-09-16T10:18:12.213322+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:37.237849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.237849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":201,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204103+0000","last_change":"2024-09-16T10:13:43.119551+0000","last_active":"2024-09-16T10:18:12.204103+0000","last_peered":"2024-09-16T10:18:12.204103+0000","last_clean":"2024-09-16T10:18:12.204103+0000","last_became_active":"2024-09-16T10:13:43.119240+0000","last_became_peered":"2024-09-16T10:13:43.119240+0000","last_unstale":"2024-09-16T10:18:12.204103+0000","last_undegraded":"2024-09-16T10:18:12.204103+0000","last_fullsized":"2024-09-16T10:18:12.204103+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:30.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":241,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208595+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:18:12.208595+0000","last_peered":"2024-09-16T10:18:12.208595+0000","last_clean":"2024-09-16T10:18:12.208595+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:18:12.208595+0000","last_undegraded":"2024-09-16T10:18:12.208595+0000","last_fullsized":"2024-09-16T10:18:12.208595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":228,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205129+0000","last_change":"2024-09-16T10:18:03.120551+0000","last_active":"2024-09-16T10:18:12.205129+0000","last_peered":"2024-09-16T10:18:12.205129+0000","last_clean":"2024-09-16T10:18:12.205129+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:18:12.205129+0000","last_undegraded":"2024-09-16T10:18:12.205129+0000","last_fullsized":"2024-09-16T10:18:12.205129+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:03.120474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:03.120474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:28.652145+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":285,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.209330+0000","last_change":"2024-09-16T10:17:06.052878+0000","last_active":"2024-09-16T10:18:12.209330+0000","last_peered":"2024-09-16T10:18:12.209330+0000","last_clean":"2024-09-16T10:18:12.209330+0000","last_became_active":"2024-09-16T10:13:41.119010+0000","last_became_peered":"2024-09-16T10:13:41.119010+0000","last_unstale":"2024-09-16T10:18:12.209330+0000","last_undegraded":"2024-09-16T10:18:12.209330+0000","last_fullsized":"2024-09-16T10:18:12.209330+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:06.050469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.050469+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:22.355575+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":216,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.191493+0000","last_change":"2024-09-16T10:17:45.915676+0000","last_active":"2024-09-16T10:18:12.191493+0000","last_peered":"2024-09-16T10:18:12.191493+0000","last_clean":"2024-09-16T10:18:12.191493+0000","last_became_active":"2024-09-16T10:13:43.119244+0000","last_became_peered":"2024-09-16T10:13:43.119244+0000","last_unstale":"2024-09-16T10:18:12.191493+0000","last_undegraded":"2024-09-16T10:18:12.191493+0000","last_fullsized":"2024-09-16T10:18:12.191493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.915613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.915613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:57.303412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":240,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.209629+0000","last_change":"2024-09-16T10:17:44.208606+0000","last_active":"2024-09-16T10:18:12.209629+0000","last_peered":"2024-09-16T10:18:12.209629+0000","last_clean":"2024-09-16T10:18:12.209629+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:18:12.209629+0000","last_undegraded":"2024-09-16T10:18:12.209629+0000","last_fullsized":"2024-09-16T10:18:12.209629+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:44.208496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.208496+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:02.974829+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.569311+0000","last_change":"2024-09-16T10:18:12.569311+0000","last_active":"2024-09-16T10:18:12.569311+0000","last_peered":"2024-09-16T10:18:12.569311+0000","last_clean":"2024-09-16T10:18:12.569311+0000","last_became_active":"2024-09-16T10:13:37.094278+0000","last_became_peered":"2024-09-16T10:13:37.094278+0000","last_unstale":"2024-09-16T10:18:12.569311+0000","last_undegraded":"2024-09-16T10:18:12.569311+0000","last_fullsized":"2024-09-16T10:18:12.569311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:20.630183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":285,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208429+0000","last_change":"2024-09-16T10:17:34.543214+0000","last_active":"2024-09-16T10:18:12.208429+0000","last_peered":"2024-09-16T10:18:12.208429+0000","last_clean":"2024-09-16T10:18:12.208429+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:18:12.208429+0000","last_undegraded":"2024-09-16T10:18:12.208429+0000","last_fullsized":"2024-09-16T10:18:12.208429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:34.543121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.543121+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:51.702679+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":216,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205554+0000","last_change":"2024-09-16T10:18:09.148508+0000","last_active":"2024-09-16T10:18:12.205554+0000","last_peered":"2024-09-16T10:18:12.205554+0000","last_clean":"2024-09-16T10:18:12.205554+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:18:12.205554+0000","last_undegraded":"2024-09-16T10:18:12.205554+0000","last_fullsized":"2024-09-16T10:18:12.205554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:09.148447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.148447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:24.182337+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":235,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.190658+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:18:12.190658+0000","last_peered":"2024-09-16T10:18:12.190658+0000","last_clean":"2024-09-16T10:18:12.190658+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:18:12.190658+0000","last_undegraded":"2024-09-16T10:18:12.190658+0000","last_fullsized":"2024-09-16T10:18:12.190658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:56.914977+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":228,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.191502+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:18:12.191502+0000","last_peered":"2024-09-16T10:18:12.191502+0000","last_clean":"2024-09-16T10:18:12.191502+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:18:12.191502+0000","last_undegraded":"2024-09-16T10:18:12.191502+0000","last_fullsized":"2024-09-16T10:18:12.191502+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:45.277710+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.201518+0000","last_change":"2024-09-16T10:17:14.106033+0000","last_active":"2024-09-16T10:18:12.201518+0000","last_peered":"2024-09-16T10:18:12.201518+0000","last_clean":"2024-09-16T10:18:12.201518+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:18:12.201518+0000","last_undegraded":"2024-09-16T10:18:12.201518+0000","last_fullsized":"2024-09-16T10:18:12.201518+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:14.105971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:14.105971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:27.805018+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":201,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212630+0000","last_change":"2024-09-16T10:13:43.124886+0000","last_active":"2024-09-16T10:18:12.212630+0000","last_peered":"2024-09-16T10:18:12.212630+0000","last_clean":"2024-09-16T10:18:12.212630+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:18:12.212630+0000","last_undegraded":"2024-09-16T10:18:12.212630+0000","last_fullsized":"2024-09-16T10:18:12.212630+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:13:42.098882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:58.157730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":233,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.201743+0000","last_change":"2024-09-16T10:15:15.010801+0000","last_active":"2024-09-16T10:18:12.201743+0000","last_peered":"2024-09-16T10:18:12.201743+0000","last_clean":"2024-09-16T10:18:12.201743+0000","last_became_active":"2024-09-16T10:13:39.097919+0000","last_became_peered":"2024-09-16T10:13:39.097919+0000","last_unstale":"2024-09-16T10:18:12.201743+0000","last_undegraded":"2024-09-16T10:18:12.201743+0000","last_fullsized":"2024-09-16T10:18:12.201743+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:15:15.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:15:15.010709+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:30.212036+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.214097+0000","last_change":"2024-09-16T10:17:42.811582+0000","last_active":"2024-09-16T10:18:12.214097+0000","last_peered":"2024-09-16T10:18:12.214097+0000","last_clean":"2024-09-16T10:18:12.214097+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:18:12.214097+0000","last_undegraded":"2024-09-16T10:18:12.214097+0000","last_fullsized":"2024-09-16T10:18:12.214097+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:42.811515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.811515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:45.586424+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":41,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.204203+0000","last_change":"2024-09-16T10:17:37.303380+0000","last_active":"2024-09-16T10:18:12.204203+0000","last_peered":"2024-09-16T10:18:12.204203+0000","last_clean":"2024-09-16T10:18:12.204203+0000","last_became_active":"2024-09-16T10:17:37.302939+0000","last_became_peered":"2024-09-16T10:17:37.302939+0000","last_unstale":"2024-09-16T10:18:12.204203+0000","last_undegraded":"2024-09-16T10:18:12.204203+0000","last_fullsized":"2024-09-16T10:18:12.204203+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:40.762324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":208,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.202747+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:18:12.202747+0000","last_peered":"2024-09-16T10:18:12.202747+0000","last_clean":"2024-09-16T10:18:12.202747+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:18:12.202747+0000","last_undegraded":"2024-09-16T10:18:12.202747+0000","last_fullsized":"2024-09-16T10:18:12.202747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:49.801010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":228,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212837+0000","last_change":"2024-09-16T10:13:39.105299+0000","last_active":"2024-09-16T10:18:12.212837+0000","last_peered":"2024-09-16T10:18:12.212837+0000","last_clean":"2024-09-16T10:18:12.212837+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:18:12.212837+0000","last_undegraded":"2024-09-16T10:18:12.212837+0000","last_fullsized":"2024-09-16T10:18:12.212837+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:55.190823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":229,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.208648+0000","last_change":"2024-09-16T10:17:36.539962+0000","last_active":"2024-09-16T10:18:12.208648+0000","last_peered":"2024-09-16T10:18:12.208648+0000","last_clean":"2024-09-16T10:18:12.208648+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:18:12.208648+0000","last_undegraded":"2024-09-16T10:18:12.208648+0000","last_fullsized":"2024-09-16T10:18:12.208648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:36.539894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:36.539894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:41.349797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212409+0000","last_change":"2024-09-16T10:17:49.749440+0000","last_active":"2024-09-16T10:18:12.212409+0000","last_peered":"2024-09-16T10:18:12.212409+0000","last_clean":"2024-09-16T10:18:12.212409+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:18:12.212409+0000","last_undegraded":"2024-09-16T10:18:12.212409+0000","last_fullsized":"2024-09-16T10:18:12.212409+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:49.749388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.749388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:55.694164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":217,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.210429+0000","last_change":"2024-09-16T10:18:03.059727+0000","last_active":"2024-09-16T10:18:12.210429+0000","last_peered":"2024-09-16T10:18:12.210429+0000","last_clean":"2024-09-16T10:18:12.210429+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:18:12.210429+0000","last_undegraded":"2024-09-16T10:18:12.210429+0000","last_fullsized":"2024-09-16T10:18:12.210429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:03.059604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:03.059604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:04.439879+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":236,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212125+0000","last_change":"2024-09-16T10:15:08.953214+0000","last_active":"2024-09-16T10:18:12.212125+0000","last_peered":"2024-09-16T10:18:12.212125+0000","last_clean":"2024-09-16T10:18:12.212125+0000","last_became_active":"2024-09-16T10:15:08.930505+0000","last_became_peered":"2024-09-16T10:15:08.930505+0000","last_unstale":"2024-09-16T10:18:12.212125+0000","last_undegraded":"2024-09-16T10:18:12.212125+0000","last_fullsized":"2024-09-16T10:18:12.212125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:23.117953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":226,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.203622+0000","last_change":"2024-09-16T10:16:39.271363+0000","last_active":"2024-09-16T10:18:12.203622+0000","last_peered":"2024-09-16T10:18:12.203622+0000","last_clean":"2024-09-16T10:18:12.203622+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:18:12.203622+0000","last_undegraded":"2024-09-16T10:18:12.203622+0000","last_fullsized":"2024-09-16T10:18:12.203622+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:16:39.271283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.271283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":228,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.243030+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:18:12.243030+0000","last_peered":"2024-09-16T10:18:12.243030+0000","last_clean":"2024-09-16T10:18:12.243030+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:18:12.243030+0000","last_undegraded":"2024-09-16T10:18:12.243030+0000","last_fullsized":"2024-09-16T10:18:12.243030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:21.740866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":102,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.210559+0000","last_change":"2024-09-16T10:17:39.193790+0000","last_active":"2024-09-16T10:18:12.210559+0000","last_peered":"2024-09-16T10:18:12.210559+0000","last_clean":"2024-09-16T10:18:12.210559+0000","last_became_active":"2024-09-16T10:16:14.000930+0000","last_became_peered":"2024-09-16T10:16:14.000930+0000","last_unstale":"2024-09-16T10:18:12.210559+0000","last_undegraded":"2024-09-16T10:18:12.210559+0000","last_fullsized":"2024-09-16T10:18:12.210559+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:39.193466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:39.193466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:47.428002+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":217,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207582+0000","last_change":"2024-09-16T10:16:32.919293+0000","last_active":"2024-09-16T10:18:12.207582+0000","last_peered":"2024-09-16T10:18:12.207582+0000","last_clean":"2024-09-16T10:18:12.207582+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:18:12.207582+0000","last_undegraded":"2024-09-16T10:18:12.207582+0000","last_fullsized":"2024-09-16T10:18:12.207582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:32.919094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.919094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:52.402423+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.212366+0000","last_change":"2024-09-16T10:15:24.663908+0000","last_active":"2024-09-16T10:18:12.212366+0000","last_peered":"2024-09-16T10:18:12.212366+0000","last_clean":"2024-09-16T10:18:12.212366+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:18:12.212366+0000","last_undegraded":"2024-09-16T10:18:12.212366+0000","last_fullsized":"2024-09-16T10:18:12.212366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:24.663832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:24.663832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:01.086625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":252,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.207518+0000","last_change":"2024-09-16T10:17:35.029288+0000","last_active":"2024-09-16T10:18:12.207518+0000","last_peered":"2024-09-16T10:18:12.207518+0000","last_clean":"2024-09-16T10:18:12.207518+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:18:12.207518+0000","last_undegraded":"2024-09-16T10:18:12.207518+0000","last_fullsized":"2024-09-16T10:18:12.207518+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:35.029210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.029210+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:18:38.152396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":221,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-16T10:18:12.205744+0000","last_change":"2024-09-16T10:16:36.222414+0000","last_active":"2024-09-16T10:18:12.205744+0000","last_peered":"2024-09-16T10:18:12.205744+0000","last_clean":"2024-09-16T10:18:12.205744+0000","last_became_active":"2024-09-16T10:13:41.112663+0000","last_became_peered":"2024-09-16T10:13:41.112663+0000","last_unstale":"2024-09-16T10:18:12.205744+0000","last_undegraded":"2024-09-16T10:18:12.205744+0000","last_fullsized":"2024-09-16T10:18:12.205744+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:36.222281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:36.222281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":296878104,"num_objects":4531,"num_object_clones":0,"num_object_copies":13593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234023,"num_write_kb":7633665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893267968,"data_stored":893255752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82705,"ondisk_log_size":82705,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234043,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218780,"kb_used_data":107080,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93526372,"statfs":{"total":95995035648,"available":95771004928,"internally_reserved":0,"allocated":109649920,"data_stored":109233670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528383,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":211912,"kb_used_data":112692,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93533240,"statfs":{"total":95995035648,"available":95778037760,"internally_reserved":0,"allocated":115396608,"data_stored":114999764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790021,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172040,"kb_used_data":90872,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93573112,"statfs":{"total":95995035648,"available":95818866688,"internally_reserved":0,"allocated":93052928,"data_stored":92647145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217056,"kb_used_data":105360,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93528096,"statfs":{"total":95995035648,"available":95772770304,"internally_reserved":0,"allocated":107888640,"data_stored":107464141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215183,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338868,"kb_used_data":177184,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93406284,"statfs":{"total":95995035648,"available":95648034816,"internally_reserved":0,"allocated":181436416,"data_stored":180997356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":165542344},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":254988,"kb_used_data":125240,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93490164,"statfs":{"total":95995035648,"available":95733927936,"internally_reserved":0,"allocated":128245760,"data_stored":127845611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":242624,"kb_used_data":125356,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93502528,"statfs":{"total":95995035648,"available":95746588672,"internally_reserved":0,"allocated":128364544,"data_stored":127912247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290792,"kb_used_data":151252,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93454360,"statfs":{"total":95995035648,"available":95697264640,"internally_reserved":0,"allocated":154882048,"data_stored":154454236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143593472,"data_stored":143589400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109383680,"data_stored":109379608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110493696,"data_stored":110493696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":163119104,"data_stored":163119104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98238464,"data_stored":98238464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75563008,"data_stored":75563008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":95621120,"data_stored":95617048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":97255424,"data_stored":97255424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:18:17.309 INFO:tasks.thrashosds.thrasher:Setting 6.2 to [3, 2, 5, 1, 6, 4] 2024-09-16T10:18:17.310 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.2', '3', '2', '5', '1', '6', '4'] 2024-09-16T10:18:17.310 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 6.2 3 2 5 1 6 4 2024-09-16T10:18:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:17.312 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:17.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:17.313 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:18:17.394 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[29993]: from='client.45101 v1:172.21.15.63:0/896548166' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:18:17.394 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/633739500' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:17.394 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[29993]: 3.f scrub starts 2024-09-16T10:18:17.394 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[29993]: 3.f scrub ok 2024-09-16T10:18:17.394 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1835134829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:17.394 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:18:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:18:17.395 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[39470]: from='client.45101 v1:172.21.15.63:0/896548166' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:18:17.395 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/633739500' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:17.395 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[39470]: 3.f scrub starts 2024-09-16T10:18:17.395 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[39470]: 3.f scrub ok 2024-09-16T10:18:17.395 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:17 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1835134829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:17.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:17.460 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:17.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:17 smithi165 ceph-mon[32696]: from='client.45101 v1:172.21.15.63:0/896548166' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:18:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:17 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/633739500' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:17 smithi165 ceph-mon[32696]: 3.f scrub starts 2024-09-16T10:18:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:17 smithi165 ceph-mon[32696]: 3.f scrub ok 2024-09-16T10:18:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:17 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1835134829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:17.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:17.613 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:17.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:17.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:17.720 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Removed 119276 objects 2024-09-16T10:18:17.720 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Clean up completed and total clean up time :117.612 2024-09-16T10:18:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:17.741 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:17.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:17.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:17.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:17.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:17.871 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:17.965 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:17.988 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:17.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:18.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:18.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:18.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:18.213 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:18.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:18.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:18.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:18.361 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:18.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:18.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:18.486 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:18.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:18 smithi165 ceph-mon[32696]: pgmap v214: 158 pgs: 158 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 671 op/s 2024-09-16T10:18:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:18 smithi165 ceph-mon[32696]: 4.16 scrub starts 2024-09-16T10:18:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:18 smithi165 ceph-mon[32696]: 4.16 scrub ok 2024-09-16T10:18:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:18 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1835134829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:18:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:18 smithi165 ceph-mon[32696]: osdmap e160: 8 total, 8 up, 8 in 2024-09-16T10:18:18.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:18:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:18:18.578 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[29993]: pgmap v214: 158 pgs: 158 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 671 op/s 2024-09-16T10:18:18.578 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[29993]: 4.16 scrub starts 2024-09-16T10:18:18.578 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[29993]: 4.16 scrub ok 2024-09-16T10:18:18.578 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:18.578 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1835134829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:18:18.578 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[29993]: osdmap e160: 8 total, 8 up, 8 in 2024-09-16T10:18:18.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[39470]: pgmap v214: 158 pgs: 158 active+clean; 284 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 671 op/s 2024-09-16T10:18:18.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[39470]: 4.16 scrub starts 2024-09-16T10:18:18.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[39470]: 4.16 scrub ok 2024-09-16T10:18:18.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:18.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1835134829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:18:18.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:18 smithi063 ceph-mon[39470]: osdmap e160: 8 total, 8 up, 8 in 2024-09-16T10:18:18.587 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:18.609 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:18.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:18.711 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:18.766 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:18.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:18.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:18.894 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:18.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:18.926 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:18:18.926 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:18:18.926 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 0 2024-09-16T10:18:18.926 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:18:18.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:19.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:19.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:19.124 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:19.147 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:19.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:19.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:19.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:19.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:19.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:19.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:19.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:19.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:19.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:19.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:19.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:19.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:19.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:19.787 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:19.889 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:19.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:18:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:19.918 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:19.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:20.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:20.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:20.144 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:20.154 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:18:20.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:20.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:20.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:20.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:20.347 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:20.349 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:20.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.423 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:20.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:20.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:20 smithi063 ceph-mon[29993]: pgmap v216: 158 pgs: 158 active+clean; 583 KiB data, 962 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-16T10:18:20.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:20 smithi063 ceph-mon[39470]: pgmap v216: 158 pgs: 158 active+clean; 583 KiB data, 962 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-16T10:18:20.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:20.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:20.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:20 smithi165 ceph-mon[32696]: pgmap v216: 158 pgs: 158 active+clean; 583 KiB data, 962 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-16T10:18:20.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:20.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:20.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:20.811 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:20.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:20.938 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:20.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:21.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.063 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:21.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:21.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.189 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:21.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:21.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:21.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:21.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:21.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:21.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:21.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:21.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:21.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:21.695 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:21.794 INFO:teuthology.orchestra.run.smithi063.stderr:set 6.2 pg_upmap_items mapping to [3->2,5->1,6->4] 2024-09-16T10:18:21.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:21.820 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:21.863 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:21.863 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:21.863 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:21.863 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:21.864 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:21.865 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:21.865 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:21.865 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:21.865 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:21.865 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:21 smithi063 ceph-mon[29993]: pgmap v217: 158 pgs: 158 active+clean; 583 KiB data, 962 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-16T10:18:21.865 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:21 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1715827914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:21.865 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:21 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1449378209' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.2", "id": [3, 2, 5, 1, 6, 4]} : dispatch 2024-09-16T10:18:21.866 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:21 smithi063 ceph-mon[39470]: pgmap v217: 158 pgs: 158 active+clean; 583 KiB data, 962 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-16T10:18:21.866 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:21 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1715827914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:21.866 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:21 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1449378209' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.2", "id": [3, 2, 5, 1, 6, 4]} : dispatch 2024-09-16T10:18:21.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:21.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:21.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:22.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:21 smithi165 ceph-mon[32696]: pgmap v217: 158 pgs: 158 active+clean; 583 KiB data, 962 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-16T10:18:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:21 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1715827914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:21 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1449378209' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.2", "id": [3, 2, 5, 1, 6, 4]} : dispatch 2024-09-16T10:18:22.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:22.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:22.205 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:22.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:22.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:22.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:18:22.433 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:22.457 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:22.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:22.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:22.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:22.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:22.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:22.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:22.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:22.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:22.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:22 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1449378209' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.2", "id": [3, 2, 5, 1, 6, 4]}]': finished 2024-09-16T10:18:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:22 smithi165 ceph-mon[32696]: osdmap e161: 8 total, 8 up, 8 in 2024-09-16T10:18:23.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:23.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:23.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:22 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1449378209' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.2", "id": [3, 2, 5, 1, 6, 4]}]': finished 2024-09-16T10:18:23.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:22 smithi063 ceph-mon[29993]: osdmap e161: 8 total, 8 up, 8 in 2024-09-16T10:18:23.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:22 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1449378209' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.2", "id": [3, 2, 5, 1, 6, 4]}]': finished 2024-09-16T10:18:23.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:22 smithi063 ceph-mon[39470]: osdmap e161: 8 total, 8 up, 8 in 2024-09-16T10:18:23.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:23.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:23.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.353 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:23.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:23.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:23.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:23.530 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T10:18:23.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T10:18:23.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:23.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:23.709 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.733 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:23.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:23.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:23.856 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:23.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:23.980 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:23.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:23 smithi165 ceph-mon[32696]: pgmap v219: 158 pgs: 158 active+clean; 583 KiB data, 948 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s 2024-09-16T10:18:24.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:24.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:24.117 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:24.117 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:23 smithi063 ceph-mon[29993]: pgmap v219: 158 pgs: 158 active+clean; 583 KiB data, 948 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s 2024-09-16T10:18:24.117 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:24.117 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:23 smithi063 ceph-mon[39470]: pgmap v219: 158 pgs: 158 active+clean; 583 KiB data, 948 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s 2024-09-16T10:18:24.175 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:24.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:24.210 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:24.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:24.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:24.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:24.464 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.488 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:24.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:24.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:24.611 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:24.611 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:24.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:24.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:24.739 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:24.840 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:24.862 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:24.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:24.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:24.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:25.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:25.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:25.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:25.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:25.246 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:25.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:25.247 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:25.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:25.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:25.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:25.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:25.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:25.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:25.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:18:25.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:25.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:25.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:25.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:25.759 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:25.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:25.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:18:25.795 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:18:25.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:25.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:25.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:25.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:26.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:26.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:26.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:26.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:26.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:26.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:26.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[39470]: pgmap v220: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1792713494' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[39470]: from='client.45128 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[39470]: from='client.45128 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[39470]: osdmap e162: 8 total, 8 up, 8 in 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[29993]: pgmap v220: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s 2024-09-16T10:18:26.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1792713494' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:26.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[29993]: from='client.45128 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:26.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:26.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[29993]: from='client.45128 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:26.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:26 smithi063 ceph-mon[29993]: osdmap e162: 8 total, 8 up, 8 in 2024-09-16T10:18:26.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:26.522 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:26.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:26 smithi165 ceph-mon[32696]: pgmap v220: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s 2024-09-16T10:18:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1792713494' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:26 smithi165 ceph-mon[32696]: from='client.45128 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:26 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:26 smithi165 ceph-mon[32696]: from='client.45128 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:26 smithi165 ceph-mon[32696]: osdmap e162: 8 total, 8 up, 8 in 2024-09-16T10:18:26.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:26.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:26.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:26.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:26.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:26.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:26.904 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:26.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:27.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:27.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:27.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:27.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:27.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:27.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:27.349 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:18:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:18:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:18:27.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:27.412 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:27.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:27.541 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:27.585 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:27.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.672 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:27.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:27.773 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:27.796 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:27.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:27.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:27.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:27.999 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:28.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:28.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:28.154 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:28.176 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_0' removed 2024-09-16T10:18:28.176 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:28.224 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:28 smithi063 ceph-mon[29993]: pgmap v222: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:18:28.224 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3961498780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:18:28.224 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:28 smithi063 ceph-mon[39470]: pgmap v222: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:18:28.224 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3961498780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:18:28.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:28.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:28.404 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.428 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:28.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:28.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:28 smithi165 ceph-mon[32696]: pgmap v222: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T10:18:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3961498780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:18:28.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:18:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:18:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:28.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:28.663 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:28.663 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:28.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:28.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.802 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:28.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:28.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:28.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:28.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:29.028 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:29.050 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:29.143 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:18:29.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:18:29.143 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T10:18:29.143 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-09-16T10:18:29.143 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-16T10:18:29.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:29.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:29.269 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:29.270 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:29.276 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:29.287 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:29.287 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:29.301 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:29.306 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:29.307 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:29.323 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:29.378 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:29.379 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:29.379 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:29.379 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:29.379 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:29.380 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:29.380 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:29.380 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:18:29.390 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:29.391 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:29.400 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:29 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3961498780' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:18:29.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:29 smithi063 ceph-mon[29993]: osdmap e163: 8 total, 8 up, 8 in 2024-09-16T10:18:29.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:29.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:29 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3961498780' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:18:29.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:29 smithi063 ceph-mon[39470]: osdmap e163: 8 total, 8 up, 8 in 2024-09-16T10:18:29.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:29.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:29.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:29.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:29.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:29.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:29 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3961498780' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:18:29.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:29 smithi165 ceph-mon[32696]: osdmap e163: 8 total, 8 up, 8 in 2024-09-16T10:18:29.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:29.649 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:29.672 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:29.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:29.796 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:29.898 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:29.921 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:29.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:30.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:30.025 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-16T10:18:30.025 INFO:tasks.radosbench:Beginning radosbench... 2024-09-16T10:18:30.026 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-16T10:18:30.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-16T10:18:30.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:30.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:30.169 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:30.271 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:30.293 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:30.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:30.417 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:30.449 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:30 smithi063 ceph-mon[29993]: pgmap v224: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:18:30.449 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1500081766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:30.450 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:30 smithi063 ceph-mon[39470]: pgmap v224: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:18:30.450 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1500081766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:30.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:30.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:30.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:30 smithi165 ceph-mon[32696]: pgmap v224: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:18:30.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1500081766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:30.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:30.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:30.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:30.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:30.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:30.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:30.926 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:18:30.966 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:30.966 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:30.967 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:31.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:31.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:31.143 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.167 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:31.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:31.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:31.291 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:31.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:31.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:31.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:31.540 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:31.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:31.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:31.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:31.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:31.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:31.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:31.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:32.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:32.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:32.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:31 smithi165 ceph-mon[32696]: osdmap e164: 8 total, 8 up, 8 in 2024-09-16T10:18:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:31 smithi165 ceph-mon[32696]: pgmap v226: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:18:32.071 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:31 smithi063 ceph-mon[29993]: osdmap e164: 8 total, 8 up, 8 in 2024-09-16T10:18:32.072 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:31 smithi063 ceph-mon[29993]: pgmap v226: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:18:32.072 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:31 smithi063 ceph-mon[39470]: osdmap e164: 8 total, 8 up, 8 in 2024-09-16T10:18:32.072 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:31 smithi063 ceph-mon[39470]: pgmap v226: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:18:32.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:32.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:32.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:32.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:32.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:32.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.408 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:32.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:32.449 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:32.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:18:32.508 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:32.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:32.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:32.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:32.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:32.799 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:32.808 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:18:32.809 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T10:18:32.812 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_1' created 2024-09-16T10:18:32.900 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:32.923 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:32.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:32.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:33.024 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:33.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/974973933' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:33.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:18:33.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2329135668' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/295951374' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='client.45158 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/974973933' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2329135668' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/295951374' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='client.45158 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-16T10:18:33.059 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:18:33.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:18:33.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:33.148 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.172 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:33.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:33.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:33.295 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/974973933' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2329135668' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/295951374' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='client.45158 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:18:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:33.388 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:18:33.388 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:18:33.388 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 0 2024-09-16T10:18:33.388 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:18:33.397 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.419 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:33.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:33.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:33.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:33.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:33.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:33.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:33.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:33.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:33.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:33.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/974973933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2329135668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: from='client.45158 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: osdmap e165: 8 total, 8 up, 8 in 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: 5.1d scrub starts 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: 5.1d scrub ok 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: pgmap v228: 158 pgs: 4 creating+peering, 12 unknown, 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T10:18:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: 6.14 scrub starts 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[29993]: 6.14 scrub ok 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/974973933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2329135668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: from='client.45158 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: osdmap e165: 8 total, 8 up, 8 in 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: 5.1d scrub starts 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: 5.1d scrub ok 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: pgmap v228: 158 pgs: 4 creating+peering, 12 unknown, 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T10:18:33.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: 6.14 scrub starts 2024-09-16T10:18:33.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:33 smithi063 ceph-mon[39470]: 6.14 scrub ok 2024-09-16T10:18:34.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:34.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:34.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:18:34.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.163 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:34.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:34.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:34.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/974973933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2329135668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: from='client.45158 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: osdmap e165: 8 total, 8 up, 8 in 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: 5.1d scrub starts 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: 5.1d scrub ok 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: pgmap v228: 158 pgs: 4 creating+peering, 12 unknown, 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: 6.14 scrub starts 2024-09-16T10:18:34.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:33 smithi165 ceph-mon[32696]: 6.14 scrub ok 2024-09-16T10:18:34.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:34.410 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:34.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:34.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:34.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:34.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:34.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:34.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:34.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:34.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:34.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:34.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: osdmap e166: 8 total, 8 up, 8 in 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 6.b scrub starts 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 6.b scrub ok 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:18:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: osdmap e166: 8 total, 8 up, 8 in 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 6.b scrub starts 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 6.b scrub ok 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:18:34.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:34 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:18:35.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:35.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:35.112 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:35.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:35.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:35.254 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:35.276 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:35.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: osdmap e166: 8 total, 8 up, 8 in 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 6.b scrub starts 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 6.b scrub ok 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:18:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:34 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:18:35.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:35.400 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:35.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:35.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:35.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-16T10:18:35.579 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:35.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:35.647 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:35.749 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:35.771 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:35.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:35.895 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:35.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:35.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:36.020 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:36.076 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[29993]: osdmap e167: 8 total, 8 up, 8 in 2024-09-16T10:18:36.076 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[29993]: 6.1b scrub starts 2024-09-16T10:18:36.076 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[29993]: 6.1b scrub ok 2024-09-16T10:18:36.076 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[29993]: pgmap v231: 158 pgs: 5 peering, 16 creating+peering, 137 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:18:36.076 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[39470]: osdmap e167: 8 total, 8 up, 8 in 2024-09-16T10:18:36.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[39470]: 6.1b scrub starts 2024-09-16T10:18:36.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[39470]: 6.1b scrub ok 2024-09-16T10:18:36.077 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:35 smithi063 ceph-mon[39470]: pgmap v231: 158 pgs: 5 peering, 16 creating+peering, 137 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:18:36.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.145 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:36.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:36.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:36.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:36.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:36.279 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:36.280 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:36.296 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:36.297 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:36.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:35 smithi165 ceph-mon[32696]: osdmap e167: 8 total, 8 up, 8 in 2024-09-16T10:18:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:35 smithi165 ceph-mon[32696]: 6.1b scrub starts 2024-09-16T10:18:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:35 smithi165 ceph-mon[32696]: 6.1b scrub ok 2024-09-16T10:18:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:35 smithi165 ceph-mon[32696]: pgmap v231: 158 pgs: 5 peering, 16 creating+peering, 137 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:18:36.312 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:36.312 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:36.328 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:36.329 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:36.345 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:36.346 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:36.364 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:36.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:36.379 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:36.379 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:36.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:36.396 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:36.396 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:36.491 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:36.492 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:36.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:36.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:36.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:36.642 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:36.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:18:36.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:36.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:36.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:36.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:36.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:36.992 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:37.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-16T10:18:37.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1198335579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:37.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[29993]: 5.e scrub starts 2024-09-16T10:18:37.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[29993]: 5.e scrub ok 2024-09-16T10:18:37.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-16T10:18:37.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1198335579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:37.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[39470]: 5.e scrub starts 2024-09-16T10:18:37.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:36 smithi063 ceph-mon[39470]: 5.e scrub ok 2024-09-16T10:18:37.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:37.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:37.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:37.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:37.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:36 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-16T10:18:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1198335579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:36 smithi165 ceph-mon[32696]: 5.e scrub starts 2024-09-16T10:18:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:36 smithi165 ceph-mon[32696]: 5.e scrub ok 2024-09-16T10:18:37.312 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:37.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:37.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:37.435 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:18:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:18:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:18:37.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:37.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:37.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:37.634 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:37.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:37.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:37.860 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:37.882 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:37.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:37.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.007 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:38.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:38.045 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:37 smithi063 ceph-mon[29993]: pgmap v232: 158 pgs: 5 peering, 16 creating+peering, 137 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:18:38.046 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:37 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:18:38.046 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:37 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:18:38.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:37 smithi063 ceph-mon[39470]: pgmap v232: 158 pgs: 5 peering, 16 creating+peering, 137 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:18:38.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:37 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:18:38.046 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:37 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:18:38.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:18:38.109 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:38.136 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:38.207 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:38.237 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:38.259 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:38.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:37 smithi165 ceph-mon[32696]: pgmap v232: 158 pgs: 5 peering, 16 creating+peering, 137 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:18:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:37 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:18:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:37 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:18:38.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:38.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:38.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:38.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:38.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:38.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:38.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:38.755 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:38.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:18:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:18:38.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:38.880 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:38.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:38.888 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-16T10:18:38.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:39.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:39.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:39.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:39.132 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:18:39.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:18:39.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:18:39.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:39.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:39.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2516206011' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:18:39.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:39.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:18:39.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:18:39.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:39.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:39.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2516206011' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:18:39.157 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:38 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:39.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:39.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:38 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:18:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:38 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:18:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2516206011' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:18:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:38 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:39.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:39.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:39.455 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:39.455 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:39.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:39.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:39.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:39.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:39.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:39.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:39.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:39.875 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:39.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:39.893 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:18:39.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:40.002 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2516206011' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[29993]: osdmap e168: 8 total, 8 up, 8 in 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3711589513' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[29993]: pgmap v234: 158 pgs: 158 active+clean; 583 KiB data, 935 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2516206011' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:18:40.019 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[39470]: osdmap e168: 8 total, 8 up, 8 in 2024-09-16T10:18:40.020 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:18:40.020 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:18:40.020 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3711589513' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:40.020 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:39 smithi063 ceph-mon[39470]: pgmap v234: 158 pgs: 158 active+clean; 583 KiB data, 935 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:18:40.021 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:18:40.061 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:18:40.062 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:18:40.062 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T10:18:40.062 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T10:18:40.062 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T10:18:40.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-16T10:18:40.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:40.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:40.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.253 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:40.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2516206011' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:18:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:39 smithi165 ceph-mon[32696]: osdmap e168: 8 total, 8 up, 8 in 2024-09-16T10:18:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:39 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:18:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:39 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:18:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3711589513' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:39 smithi165 ceph-mon[32696]: pgmap v234: 158 pgs: 158 active+clean; 583 KiB data, 935 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:18:40.316 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:40.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:40.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:40.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:40.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:40.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:40.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:40.637 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:40.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:40.748 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:40.825 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:18:40.825 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "" 2024-09-16T10:18:40.825 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:18:40.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:40.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:40.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:40.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:40.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:41.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:40 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-16T10:18:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:40 smithi165 ceph-mon[32696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:18:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:40 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:18:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3711589513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:40 smithi165 ceph-mon[32696]: osdmap e169: 8 total, 8 up, 8 in 2024-09-16T10:18:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:40 smithi165 ceph-mon[32696]: 5.19 scrub starts 2024-09-16T10:18:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:40 smithi165 ceph-mon[32696]: 5.19 scrub ok 2024-09-16T10:18:41.063 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-16T10:18:41.063 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[29993]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3711589513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[29993]: osdmap e169: 8 total, 8 up, 8 in 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[29993]: 5.19 scrub starts 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[29993]: 5.19 scrub ok 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[39470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3711589513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[39470]: osdmap e169: 8 total, 8 up, 8 in 2024-09-16T10:18:41.064 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[39470]: 5.19 scrub starts 2024-09-16T10:18:41.065 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:40 smithi063 ceph-mon[39470]: 5.19 scrub ok 2024-09-16T10:18:41.096 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:41.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:41.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:41.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:41.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:41.343 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:41.344 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:41.368 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:41.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:41.437 DEBUG:teuthology.orchestra.run.smithi063:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-16T10:18:41.439 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-16T10:18:41.439 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-16T10:18:41.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:41.490 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:hints = 1 2024-09-16T10:18:41.490 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-16T10:18:41.490 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object prefix: benchmark_data_smithi063_101290 2024-09-16T10:18:41.490 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:18:41.490 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:18:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:41.497 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:41.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:41.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:41.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:41.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:41.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:41.770 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:41.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:41.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:41.876 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:41.876 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:41.912 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:41.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:41.912 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:41 smithi063 ceph-mon[29993]: pgmap v236: 158 pgs: 158 active+clean; 583 KiB data, 935 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-09-16T10:18:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4271674369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:41.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:41 smithi063 ceph-mon[39470]: pgmap v236: 158 pgs: 158 active+clean; 583 KiB data, 935 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-09-16T10:18:41.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4271674369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:41.947 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:41.947 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:41.979 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:41.979 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:42.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:42.014 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:42.014 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:42.053 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:42.053 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:42.056 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:42.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:42.076 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:18:42.076 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:1805 8.b 8:def1b487:::benchmark_data_smithi063_101290_object1804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.065542+0000", 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0064263699999999998, 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0064592699999999996, 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:42.077 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1805 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065542+0000", 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.078 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065542+0000", 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065545+0000", 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.092e-06 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065628+0000", 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3747999999999998e-05 2024-09-16T10:18:42.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065651+0000", 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2342000000000001e-05 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065662+0000", 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1758e-05 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065844+0000", 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000181254 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065963+0000", 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119658 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:18:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.066111+0000", 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014820499999999999 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.067594+0000", 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001482941 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:42.082 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.067606+0000", 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1797000000000001e-05 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:1806 8.0 8:0d07f997:::benchmark_data_smithi063_101290_object1805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.065684+0000", 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0062846689999999997, 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0064163040000000003, 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.083 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1806 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065684+0000", 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065684+0000", 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065685+0000", 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8899999999999998e-07 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.085 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065718+0000", 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2809999999999999e-05 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065726+0000", 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5429999999999994e-06 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065733+0000", 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5760000000000002e-06 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065787+0000", 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4141000000000002e-05 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065940+0000", 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015331700000000001 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:18:42.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.066079+0000", 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013839500000000001 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.067362+0000", 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012831470000000001 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.067914+0000", 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00055243299999999996 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.088 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.089 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.089 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:1809 8.0 8:08eb29b3:::benchmark_data_smithi063_101290_object1808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.065749+0000", 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.006219708, 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0064450549999999999, 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1809 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.090 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065749+0000", 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065749+0000", 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065750+0000", 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2799999999999995e-07 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065774+0000", 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4494e-05 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065783+0000", 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2910000000000007e-06 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.065789+0000", 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5379999999999999e-06 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.066682+0000", 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00089263999999999999 2024-09-16T10:18:42.093 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.066794+0000", 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011179999999999999 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.066934+0000", 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014040799999999999 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.094 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.067367+0000", 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00043306799999999999 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.068678+0000", 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001310561 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.095 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:1816 8.b 8:d6bbb2d7:::benchmark_data_smithi063_101290_object1815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.069980+0000", 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0019884780000000001, 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002296822, 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1816 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.096 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.069980+0000", 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.069980+0000", 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.069982+0000", 2024-09-16T10:18:42.097 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.872e-06 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.070040+0000", 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7494000000000002e-05 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.098 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.100 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.070105+0000", 2024-09-16T10:18:42.100 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4967000000000003e-05 2024-09-16T10:18:42.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.070114+0000", 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0480000000000004e-06 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.070177+0000", 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3662000000000001e-05 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.070433+0000", 2024-09-16T10:18:42.101 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025588999999999999 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.070574+0000", 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014107799999999999 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.071572+0000", 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00099806099999999996 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:42.102 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.071658+0000", 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5179999999999994e-05 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:1820 8.b 8:da30abb6:::benchmark_data_smithi063_101290_object1819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.071722+0000", 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.00024674400000000002, 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000637552, 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.103 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1820 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.071722+0000", 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.104 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.071722+0000", 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.071724+0000", 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.621e-06 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.071748+0000", 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4266000000000001e-05 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.105 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.071763+0000", 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5165e-05 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072079+0000", 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031604700000000002 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072182+0000", 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010309700000000001 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.106 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:1821 8.b 8.5e20e2bb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.072100+0000", 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9998689, 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000321992, 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:18:42.107 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1821 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072100+0000", 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072100+0000", 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072101+0000", 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9199999999999999e-07 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072125+0000", 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4355999999999999e-05 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.109 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072149+0000", 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3441000000000001e-05 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072162+0000", 2024-09-16T10:18:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3719e-05 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:1823 8.0 8.6c142230 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.072176+0000", 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9997926, 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030023000000000002, 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:42.111 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 1823 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072176+0000", 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.112 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072176+0000", 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072177+0000", 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3099999999999996e-07 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.113 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072205+0000", 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8178999999999999e-05 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072244+0000", 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8727999999999999e-05 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072251+0000", 2024-09-16T10:18:42.114 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7009999999999999e-06 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.15333.0:1815 8.8 e169/165)", 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.072279+0000", 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9996901, 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00024661000000000003, 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:18:42.115 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072279+0000", 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072279+0000", 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.116 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072280+0000", 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5799999999999998e-07 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072318+0000", 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7994000000000002e-05 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072325+0000", 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7540000000000006e-06 2024-09-16T10:18:42.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072331+0000", 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9309999999999996e-06 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.15333.0:1806 8.0 e169/165)", 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.072355+0000", 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9996138, 2024-09-16T10:18:42.118 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000211676, 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072355+0000", 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072355+0000", 2024-09-16T10:18:42.119 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072356+0000", 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2200000000000003e-07 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072359+0000", 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4580000000000002e-06 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.120 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072366+0000", 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9800000000000001e-06 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072372+0000", 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5199999999999997e-06 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.121 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.15333.0:1809 8.0 e169/165)", 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.072380+0000", 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9995885, 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000290334, 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072380+0000", 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072380+0000", 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072381+0000", 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.39e-07 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.123 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072383+0000", 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5490000000000001e-06 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072388+0000", 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1159999999999998e-06 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072393+0000", 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3930000000000001e-06 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.124 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.15333.0:1805 8.b e169/165)", 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.072398+0000", 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9995708, 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031124599999999999, 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.125 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072398+0000", 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072398+0000", 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072398+0000", 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0199999999999998e-07 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072400+0000", 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1469999999999999e-06 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072405+0000", 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6419999999999998e-06 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072409+0000", 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.036e-06 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.127 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.15333.0:1813 8.7 e169/165 8:e5794636:::benchmark_data_smithi063_101290_object1812:head v 169'109, mlcod=169'107)", 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:18:42.072415+0000", 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9995542, 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00033782300000000001, 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:42.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072415+0000", 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072415+0000", 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072415+0000", 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7300000000000002e-07 2024-09-16T10:18:42.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072440+0000", 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5602e-05 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072447+0000", 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1680000000000001e-06 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:42.130 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072452+0000", 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0830000000000003e-06 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072540+0000", 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7975999999999999e-05 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:18:42.072560+0000", 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0363000000000001e-05 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:42.132 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:18:42.132 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 12 2024-09-16T10:18:42.132 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:18:42.132 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:42.133 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:42.159 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:42.203 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:42.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:42.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:41 smithi165 ceph-mon[32696]: pgmap v236: 158 pgs: 158 active+clean; 583 KiB data, 935 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-09-16T10:18:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4271674369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:42.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:42.346 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:42.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:42.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:42.489 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:42.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:42.490 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 1 16 3165 3149 196.802 196.812 0.00151288 0.0050647 2024-09-16T10:18:42.591 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:42.632 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:42.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:42.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:42.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:42.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:42.869 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:42.916 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:42.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:42.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:18:43.016 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[29993]: 3.16 scrub starts 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[29993]: 3.16 scrub ok 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[39470]: 3.16 scrub starts 2024-09-16T10:18:43.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[39470]: 3.16 scrub ok 2024-09-16T10:18:43.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:18:43.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:42 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:18:43.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:43.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:43.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:43.091 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:18:43.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:43.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:43.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:43.277 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:43.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:42 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:18:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:42 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:18:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:42 smithi165 ceph-mon[32696]: 3.16 scrub starts 2024-09-16T10:18:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:42 smithi165 ceph-mon[32696]: 3.16 scrub ok 2024-09-16T10:18:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:42 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:18:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:42 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:18:43.331 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:43.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:43.433 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:43.484 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:43.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:43.490 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 2 16 6301 6285 196.378 196 0.0083433 0.00507995 2024-09-16T10:18:43.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:43.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:43.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:43.627 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:43.664 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:43.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:43.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:43.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:43.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:43.894 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:43.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:43.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:44.031 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:44.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:44.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:18:44.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:18:44.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[29993]: osdmap e170: 8 total, 8 up, 8 in 2024-09-16T10:18:44.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[29993]: pgmap v238: 158 pgs: 158 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 13 MiB/s wr, 212 op/s 2024-09-16T10:18:44.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[29993]: 5.12 scrub starts 2024-09-16T10:18:44.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[29993]: 5.12 scrub ok 2024-09-16T10:18:44.068 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:18:44.068 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:18:44.068 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[39470]: osdmap e170: 8 total, 8 up, 8 in 2024-09-16T10:18:44.068 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[39470]: pgmap v238: 158 pgs: 158 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 13 MiB/s wr, 212 op/s 2024-09-16T10:18:44.068 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[39470]: 5.12 scrub starts 2024-09-16T10:18:44.068 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:43 smithi063 ceph-mon[39470]: 5.12 scrub ok 2024-09-16T10:18:44.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:44.174 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:44.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:44.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:43 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:18:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:43 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:18:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:43 smithi165 ceph-mon[32696]: osdmap e170: 8 total, 8 up, 8 in 2024-09-16T10:18:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:43 smithi165 ceph-mon[32696]: pgmap v238: 158 pgs: 158 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 13 MiB/s wr, 212 op/s 2024-09-16T10:18:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:43 smithi165 ceph-mon[32696]: 5.12 scrub starts 2024-09-16T10:18:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:43 smithi165 ceph-mon[32696]: 5.12 scrub ok 2024-09-16T10:18:44.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:44.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:44.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:44.461 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:44.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:44.491 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 3 16 9216 9200 191.642 182.188 0.00711276 0.00520528 2024-09-16T10:18:44.538 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:18:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:18:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:18:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:18:44.538 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:18:44.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:44.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:44.707 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:44.746 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:44.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:44.847 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:44.887 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:44.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:44.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:45.020 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:45.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:45.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:45.160 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:45.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:45.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:44 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:18:45.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:44 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:18:45.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:44 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:18:45.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:44 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:18:45.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:45.301 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:45.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:44 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:18:45.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:44 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:18:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:45.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:45.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:18:45.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:45.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:45.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:45.491 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 4 16 12080 12064 188.474 179 0.00419768 0.00529882 2024-09-16T10:18:45.538 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:45.582 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:45.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:45.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:18:45.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:45.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:45.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:45.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:45.835 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:45.864 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:45.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:45.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:45.986 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:18:46.022 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:46.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:46.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:46.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:46.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:46.254 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:46.298 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:46.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:46.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: osdmap e171: 8 total, 8 up, 8 in 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2780633323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: from='client.45194 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: pgmap v240: 158 pgs: 158 active+clean; 359 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:18:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:45 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:18:46.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:46.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:46.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:46.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: osdmap e171: 8 total, 8 up, 8 in 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2780633323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: from='client.45194 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: pgmap v240: 158 pgs: 158 active+clean; 359 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:18:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: osdmap e171: 8 total, 8 up, 8 in 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2780633323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: from='client.45194 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: pgmap v240: 158 pgs: 158 active+clean; 359 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:18:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:45 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:18:46.491 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 5 16 14698 14682 183.501 163.625 0.00695938 0.00544114 2024-09-16T10:18:46.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:46.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:46.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:46.576 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:18:46.576 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:18:46.576 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:18:46.576 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd out 2 2024-09-16T10:18:46.637 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:39764 7.6 e97/81 7:680231a8:::benchmark_data_smithi063_73086_object39763:head v 97'2523, mlcod=97'2521)", 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.247108+0000", 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.38512712799999, 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13401402800000001, 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.638 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247108+0000", 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247108+0000", 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247109+0000", 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.102e-06 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247127+0000", 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7719000000000001e-05 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247135+0000", 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8030000000000003e-06 2024-09-16T10:18:46.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247141+0000", 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1959999999999999e-06 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247196+0000", 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5050999999999999e-05 2024-09-16T10:18:46.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247209+0000", 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2826999999999999e-05 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381064+0000", 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13385559399999999 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381122+0000", 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7735999999999997e-05 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39762 7.a 7:5fb7c7c3:::benchmark_data_smithi063_73086_object39761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.384457446, 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.135043899, 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.643 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39762 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247778+0000", 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.16e-07 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247809+0000", 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0239999999999998e-05 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247817+0000", 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1280000000000008e-06 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247823+0000", 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1360000000000002e-06 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249271+0000", 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001448193 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249391+0000", 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011986500000000001 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,5", 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249517+0000", 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125967 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252815+0000", 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003298282 2024-09-16T10:18:46.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382711+0000", 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12989600700000001 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382722+0000", 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0957e-05 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382773+0000", 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1057999999999998e-05 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382821+0000", 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8250000000000001e-05 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:39765 7.c e97/81 7:361d2464:::benchmark_data_smithi063_73086_object39764:head v 97'2540, mlcod=97'2537)", 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250036+0000", 2024-09-16T10:18:46.649 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.38219852, 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13163891799999999, 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250036+0000", 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250036+0000", 2024-09-16T10:18:46.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250038+0000", 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.528e-06 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250074+0000", 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5825999999999999e-05 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250084+0000", 2024-09-16T10:18:46.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9830000000000008e-06 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250090+0000", 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3829999999999997e-06 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250230+0000", 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014003400000000001 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250248+0000", 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8094999999999999e-05 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254816+0000", 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045677249999999999 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381675+0000", 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12685934400000001 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.653 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:39767 7.c e97/81 7:3eeafd38:::benchmark_data_smithi063_73086_object39766:head v 97'2541, mlcod=97'2537)", 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250109+0000", 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.38212585400001, 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.131597251, 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250109+0000", 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250109+0000", 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250110+0000", 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6699999999999999e-07 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250137+0000", 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6987000000000001e-05 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250142+0000", 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9089999999999997e-06 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250147+0000", 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6299999999999997e-06 2024-09-16T10:18:46.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251533+0000", 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001386036 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251551+0000", 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7396000000000001e-05 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381684+0000", 2024-09-16T10:18:46.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13013393600000001 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381706+0000", 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1889999999999999e-05 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39769 7.3 7:c947093a:::benchmark_data_smithi063_73086_object39768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250737+0000", 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.38149783099999, 2024-09-16T10:18:46.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.134149398, 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39769 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250737+0000", 2024-09-16T10:18:46.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250737+0000", 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250739+0000", 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.995e-06 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250851+0000", 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112194 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250870+0000", 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8774999999999999e-05 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250877+0000", 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2099999999999996e-06 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251917+0000", 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001039704 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252030+0000", 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113205 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252158+0000", 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012766000000000001 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254112+0000", 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001953761 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254277+0000", 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000165625 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384796+0000", 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13051881200000001 2024-09-16T10:18:46.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384848+0000", 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2370999999999997e-05 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384887+0000", 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8086000000000001e-05 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39773 7.2 7:405c1126:::benchmark_data_smithi063_73086_object39772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250895+0000", 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.38134021799999, 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13210709400000001, 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39773 2024-09-16T10:18:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250895+0000", 2024-09-16T10:18:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250895+0000", 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250896+0000", 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6500000000000008e-07 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250920+0000", 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4595000000000001e-05 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250927+0000", 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2139999999999999e-06 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250934+0000", 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8800000000000002e-06 2024-09-16T10:18:46.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251195+0000", 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026043099999999999 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251316+0000", 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121551 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251444+0000", 2024-09-16T10:18:46.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012788900000000001 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382829+0000", 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13138502499999999 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382861+0000", 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1996000000000003e-05 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382925+0000", 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3392999999999994e-05 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382969+0000", 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4174000000000001e-05 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383002+0000", 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2981000000000002e-05 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.671 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:39770 7.7 e97/81 7:ec0b9b23:::benchmark_data_smithi063_73086_object39769:head v 97'2526, mlcod=97'2524)", 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.251237+0000", 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.38099823600001, 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13188161600000001, 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251237+0000", 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251237+0000", 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251237+0000", 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2300000000000001e-07 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251269+0000", 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1847000000000002e-05 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251276+0000", 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4810000000000003e-06 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251281+0000", 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5609999999999998e-06 2024-09-16T10:18:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252614+0000", 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001333019 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252634+0000", 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9491999999999999e-05 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383081+0000", 2024-09-16T10:18:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.130447015 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383118+0000", 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7577999999999998e-05 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39775 7.2 7:4e14f661:::benchmark_data_smithi063_73086_object39774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.252983+0000", 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.379251702, 2024-09-16T10:18:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.131708993, 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39775 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252983+0000", 2024-09-16T10:18:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252983+0000", 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252985+0000", 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.227e-06 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.253009+0000", 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3686e-05 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.253020+0000", 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1182000000000001e-05 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.253028+0000", 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1679999999999998e-06 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.253962+0000", 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000934672 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254094+0000", 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013175099999999999 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254240+0000", 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145649 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382894+0000", 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12865486300000001 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383161+0000", 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026648999999999998 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384603+0000", 2024-09-16T10:18:46.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001441882 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384653+0000", 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0621000000000002e-05 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384692+0000", 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8801999999999999e-05 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39776 7.3 7:c5f4aa9e:::benchmark_data_smithi063_73086_object39775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.254885+0000", 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.377350143, 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.131484774, 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39776 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254885+0000", 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254885+0000", 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254887+0000", 2024-09-16T10:18:46.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.796e-06 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254924+0000", 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7669000000000001e-05 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381178+0000", 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12625325800000001 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381192+0000", 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4129e-05 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381255+0000", 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3628999999999997e-05 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381406+0000", 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015117599999999999 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381559+0000", 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000152511 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384757+0000", 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003197802 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.385728+0000", 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000971071 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.386207+0000", 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00047888800000000001 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.386329+0000", 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012217200000000001 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.386370+0000", 2024-09-16T10:18:46.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0673000000000002e-05 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65064 7.2 7:47084aaf:::benchmark_data_smithi063_73086_object65063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.092433511, 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.140792484, 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.689 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65064 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539802+0000", 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1500000000000004e-07 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539887+0000", 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5189e-05 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539896+0000", 2024-09-16T10:18:46.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2810000000000008e-06 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539908+0000", 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2140999999999999e-05 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545096+0000", 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0051880040000000004 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545228+0000", 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013243200000000001 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545886+0000", 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00065781499999999996 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.554263+0000", 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0083769059999999999 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.554308+0000", 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4771000000000002e-05 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680481+0000", 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12617313499999999 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:46.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680544+0000", 2024-09-16T10:18:46.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3608000000000005e-05 2024-09-16T10:18:46.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680594+0000", 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9586999999999998e-05 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65067 7.2 7:48a7aec7:::benchmark_data_smithi063_73086_object65066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.09126390700001, 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.139936807, 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.697 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65067 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:18:46.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540972+0000", 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.282e-06 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540992+0000", 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9539999999999999e-05 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541001+0000", 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.704e-06 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541007+0000", 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4269999999999996e-06 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548200+0000", 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071930350000000004 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548316+0000", 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116468 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548442+0000", 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012523999999999999 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563514+0000", 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015071782000000001 2024-09-16T10:18:46.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680655+0000", 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.117141604 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680817+0000", 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016221599999999999 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680870+0000", 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2448999999999999e-05 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680908+0000", 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8059999999999998e-05 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65068 7.2 7:4e1ab327:::benchmark_data_smithi063_73086_object65067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:18:46.703 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.09121814100001, 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14002576999999999, 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65068 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6400000000000002e-07 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541038+0000", 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0228999999999999e-05 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541044+0000", 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3770000000000001e-06 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541050+0000", 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.485e-06 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.550940+0000", 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0098900169999999992 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.551033+0000", 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3523000000000006e-05 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.551110+0000", 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7131000000000004e-05 2024-09-16T10:18:46.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680659+0000", 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.129549215 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680696+0000", 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6424000000000002e-05 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680964+0000", 2024-09-16T10:18:46.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026829000000000002 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681008+0000", 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3399000000000002e-05 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681043+0000", 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5015999999999998e-05 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65075 7.2 7:4c929240:::benchmark_data_smithi063_73086_object65074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.08627819500001, 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13568592600000001, 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65075 2024-09-16T10:18:46.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545959+0000", 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8649999999999999e-06 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545980+0000", 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1696e-05 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545991+0000", 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0359e-05 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545999+0000", 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1370000000000002e-06 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559321+0000", 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013321932 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559528+0000", 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020748899999999999 2024-09-16T10:18:46.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559651+0000", 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123013 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680667+0000", 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12101582399999999 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681281+0000", 2024-09-16T10:18:46.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061402500000000003 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681562+0000", 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00028057400000000001 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681607+0000", 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5578000000000003e-05 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681643+0000", 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5434e-05 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65082 7.a 7:552f4fed:::benchmark_data_smithi063_73086_object65081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.079333749, 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.144873693, 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.716 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65082 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552904+0000", 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4499999999999998e-06 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552927+0000", 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3240000000000001e-05 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552940+0000", 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3579e-05 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552949+0000", 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1790000000000008e-06 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681082+0000", 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12813308800000001 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681222+0000", 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000140464 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,5", 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681372+0000", 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000149369 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.685517+0000", 2024-09-16T10:18:46.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041449700000000004 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.685528+0000", 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1440999999999999e-05 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697681+0000", 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012153051 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697731+0000", 2024-09-16T10:18:46.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0453e-05 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697775+0000", 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3408999999999997e-05 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65094 7.2 7:41461aa3:::benchmark_data_smithi063_73086_object65093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.562052+0000", 2024-09-16T10:18:46.722 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.07018284, 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13390104, 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65094 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562052+0000", 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562052+0000", 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562054+0000", 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.68e-06 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562079+0000", 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4989000000000001e-05 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562087+0000", 2024-09-16T10:18:46.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.473e-06 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562093+0000", 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1820000000000003e-06 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681672+0000", 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119578091 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.684384+0000", 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027128970000000001 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.684688+0000", 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030324399999999999 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692119+0000", 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0074318259999999999 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692659+0000", 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00053903700000000002 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692736+0000", 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7001000000000004e-05 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692816+0000", 2024-09-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0469000000000005e-05 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.695953+0000", 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031371509999999999 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65095 7.2 7:414dfc36:::benchmark_data_smithi063_73086_object65094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.068704731, 2024-09-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13620320899999999, 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65095 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563532+0000", 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5659999999999999e-06 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563555+0000", 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2747000000000001e-05 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680763+0000", 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.117208851 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680777+0000", 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3148e-05 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689111+0000", 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0083343150000000001 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689240+0000", 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012911199999999999 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689823+0000", 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058294800000000002 2024-09-16T10:18:46.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.695962+0000", 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0061393569999999998 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697651+0000", 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016887930000000001 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:46.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699637+0000", 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986032 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699691+0000", 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4326000000000001e-05 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699733+0000", 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2014000000000001e-05 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90038 7.6 e112/81 7:605b4ba6:::benchmark_data_smithi063_73086_object90037:head v 112'5732, mlcod=112'5730)", 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 177.13043508999999, 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14626494100000001, 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501801+0000", 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.47e-07 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501838+0000", 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7397000000000001e-05 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501846+0000", 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2619999999999998e-06 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501852+0000", 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4210000000000003e-06 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504465+0000", 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026126890000000001 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504484+0000", 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9619999999999998e-05 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646628+0000", 2024-09-16T10:18:46.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.142143565 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648065+0000", 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014370400000000001 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90048 7.6 e112/81 7:6327a166:::benchmark_data_smithi063_73086_object90047:head v 112'5733, mlcod=112'5730)", 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:18:46.741 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 177.12421689499999, 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.140110544, 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:18:46.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508020+0000", 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.815e-06 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508049+0000", 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8785999999999998e-05 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508060+0000", 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1155000000000001e-05 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508067+0000", 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.289e-06 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508275+0000", 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020751799999999999 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508293+0000", 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7975e-05 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648068+0000", 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13977563800000001 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648129+0000", 2024-09-16T10:18:46.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0368000000000002e-05 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90050 7.6 e112/81 7:60112b96:::benchmark_data_smithi063_73086_object90049:head v 112'5734, mlcod=112'5730)", 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 177.12239064100001, 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13835439699999999, 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509846+0000", 2024-09-16T10:18:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2079999999999999e-06 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509888+0000", 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.244e-05 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509897+0000", 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3610000000000007e-06 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509904+0000", 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5420000000000002e-06 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.511017+0000", 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0011129860000000001 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.511034+0000", 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6891000000000001e-05 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648132+0000", 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13709860700000001 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648199+0000", 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6361999999999999e-05 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90056 7.0 e112/81 7:081070f4:::benchmark_data_smithi063_73086_object90055:head v 112'5520, mlcod=112'5518)", 2024-09-16T10:18:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.514081+0000", 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 177.11815347000001, 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.132391548, 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514081+0000", 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:18:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514081+0000", 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514083+0000", 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.128e-06 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514166+0000", 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2911999999999995e-05 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514175+0000", 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0780000000000002e-06 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514179+0000", 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6589999999999998e-06 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514223+0000", 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3744e-05 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514242+0000", 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8726999999999999e-05 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646382+0000", 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13214073300000001 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646473+0000", 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0567000000000002e-05 2024-09-16T10:18:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.755 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:18:46.755 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:18:46.755 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:18:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:46.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:46.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:46.856 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:46.905 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:46.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:47.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:47.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:47.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:47.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:47.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:47.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:46 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:46 smithi165 ceph-mon[32696]: from='client.45194 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:18:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:46 smithi165 ceph-mon[32696]: osdmap e172: 8 total, 8 up, 8 in 2024-09-16T10:18:47.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:47.419 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:46 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:46 smithi063 ceph-mon[29993]: from='client.45194 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:18:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:46 smithi063 ceph-mon[29993]: osdmap e172: 8 total, 8 up, 8 in 2024-09-16T10:18:47.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:18:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:18:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:18:47.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:46 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:47.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:46 smithi063 ceph-mon[39470]: from='client.45194 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:18:47.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:46 smithi063 ceph-mon[39470]: osdmap e172: 8 total, 8 up, 8 in 2024-09-16T10:18:47.456 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:47.491 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 6 16 17252 17236 179.516 159.625 0.0134729 0.0055607 2024-09-16T10:18:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:47.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:47.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:47.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:47.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:47.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:47.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:47.956 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:47.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:48.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:48.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:48.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:48.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:48.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:48.276 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:47 smithi063 ceph-mon[29993]: pgmap v242: 158 pgs: 158 active+clean; 359 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-16T10:18:48.278 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:47 smithi063 ceph-mon[39470]: pgmap v242: 158 pgs: 158 active+clean; 359 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-16T10:18:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:47 smithi165 ceph-mon[32696]: pgmap v242: 158 pgs: 158 active+clean; 359 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-16T10:18:48.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:18:48.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:48.428 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:48.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:48.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 7 16 19847 19831 177.03 162.188 0.00583078 0.00564256 2024-09-16T10:18:48.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:48.569 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:48.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:48.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:48.704 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:48.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:48.724 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:48.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:18:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:18:48.806 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:48.844 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:48.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:48.946 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:48.983 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:48.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:49.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:49.117 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:49.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:49.166 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:48 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:18:49.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:48 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:18:49.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:48 smithi165 ceph-mon[32696]: osdmap e173: 8 total, 8 up, 8 in 2024-09-16T10:18:49.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:48 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:49.190 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:18:49.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:49.285 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:49.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:49.313 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:18:49.313 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:18:49.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.313 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.15333.0:21644 8.5 e173/165 8:a5df18bb:::benchmark_data_smithi063_101290_object21643:head v 173'1337, mlcod=173'1335)", 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:18:49.309727+0000", 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.001230337, 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012562350000000001, 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:49.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309727+0000", 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309727+0000", 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309728+0000", 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1699999999999997e-07 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309748+0000", 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9978000000000001e-05 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309756+0000", 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8839999999999994e-06 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309761+0000", 2024-09-16T10:18:49.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4659999999999998e-06 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309822+0000", 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0328000000000002e-05 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.309841+0000", 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9417000000000001e-05 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:49.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:18:49.310946+0000", 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001104856 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-16T10:18:49.318 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[29993]: osdmap e173: 8 total, 8 up, 8 in 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[39470]: osdmap e173: 8 total, 8 up, 8 in 2024-09-16T10:18:49.343 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:48 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:49.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:49.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:49.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:49.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 8 16 22020 22004 171.877 135.812 0.00367979 0.00581248 2024-09-16T10:18:49.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:49.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:49.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:49.598 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:49.598 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:49.598 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:49.598 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:49.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:49.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:49.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:49.787 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:49.826 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:49.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:49.928 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:49.966 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:49.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:50.038 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:50.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:50.116 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:50.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:50.217 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:50.258 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:50.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:50.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:50 smithi165 ceph-mon[32696]: pgmap v244: 158 pgs: 158 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-16T10:18:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:50 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3636805274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:50.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:50.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:50.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:50.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:50 smithi063 ceph-mon[29993]: pgmap v244: 158 pgs: 158 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-16T10:18:50.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:50 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3636805274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:50.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:50 smithi063 ceph-mon[39470]: pgmap v244: 158 pgs: 158 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-16T10:18:50.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:50 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3636805274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:50.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 9 16 24406 24390 169.347 149.125 0.00200511 0.00589829 2024-09-16T10:18:50.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:50.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:50.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:50.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:50.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:50.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:50.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:50.779 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:50.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:50.881 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:50.923 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:50.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:50.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:18:51.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:51.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:51.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:51.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:51.230 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:51.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:51.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:51.366 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:51.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:51.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:51.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 10 16 26480 26464 165.374 129.625 0.00453221 0.00604008 2024-09-16T10:18:51.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:51.599 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:51.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:51.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:51.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:18:51.578+0000 7f17f27f5640 -1 mon.a@0(leader).osd e173 definitely_dead 0 2024-09-16T10:18:51.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:51.813 INFO:teuthology.orchestra.run.smithi063.stderr:marked out osd.2. 2024-09-16T10:18:51.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:51.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:51.906 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:51.906 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:51.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:51.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:51 smithi165 ceph-mon[32696]: pgmap v245: 158 pgs: 158 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 831 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-16T10:18:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/764915671' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-16T10:18:52.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:51 smithi165 ceph-mon[32696]: from='client.25270 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-16T10:18:52.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:51 smithi165 ceph-mon[32696]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-16T10:18:52.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:52.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:52.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:52.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[29993]: pgmap v245: 158 pgs: 158 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 831 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-16T10:18:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/764915671' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-16T10:18:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[29993]: from='client.25270 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-16T10:18:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[29993]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-16T10:18:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[39470]: pgmap v245: 158 pgs: 158 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 831 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-16T10:18:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/764915671' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-16T10:18:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[39470]: from='client.25270 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-16T10:18:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:51 smithi063 ceph-mon[39470]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-16T10:18:52.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:52.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:52.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:52.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:52.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:52.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:52.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:52.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:52.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:52.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 11 16 27140 27124 154.09 41.25 0.00234607 0.00609717 2024-09-16T10:18:52.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:52.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:52.611 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:18:52.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:18:52.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:52.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:52.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:52.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:52.881 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:52.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:52.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:52 smithi063 ceph-mon[29993]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-16T10:18:52.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:52 smithi063 ceph-mon[29993]: osdmap e174: 8 total, 8 up, 7 in 2024-09-16T10:18:52.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:52 smithi063 ceph-mon[29993]: osdmap e175: 8 total, 8 up, 7 in 2024-09-16T10:18:52.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:52 smithi063 ceph-mon[39470]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-16T10:18:52.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:52 smithi063 ceph-mon[39470]: osdmap e174: 8 total, 8 up, 7 in 2024-09-16T10:18:52.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:52 smithi063 ceph-mon[39470]: osdmap e175: 8 total, 8 up, 7 in 2024-09-16T10:18:52.962 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:52.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.008 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:53.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:52 smithi165 ceph-mon[32696]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-16T10:18:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:52 smithi165 ceph-mon[32696]: osdmap e174: 8 total, 8 up, 7 in 2024-09-16T10:18:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:52 smithi165 ceph-mon[32696]: osdmap e175: 8 total, 8 up, 7 in 2024-09-16T10:18:53.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:53.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:53.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:53.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:53.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:53.381 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:53.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:53.486 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:53.486 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:53.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 12 16 27140 27124 141.249 0 - 0.00609717 2024-09-16T10:18:53.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:53.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:53.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:53.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:53.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:53.816 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:18:53.854 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:53.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:53.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:54.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:54.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:54.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:54.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:54.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:54.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: pgmap v248: 158 pgs: 2 remapped+peering, 156 active+clean; 1.3 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.60k op/s; 43 KiB/s, 2 objects/s recovering 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1649974073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: from='client.25276 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[29993]: osdmap e176: 8 total, 8 up, 7 in 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:54.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: pgmap v248: 158 pgs: 2 remapped+peering, 156 active+clean; 1.3 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.60k op/s; 43 KiB/s, 2 objects/s recovering 2024-09-16T10:18:54.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1649974073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:54.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: from='client.25276 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:54.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:54.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:54.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:53 smithi063 ceph-mon[39470]: osdmap e176: 8 total, 8 up, 7 in 2024-09-16T10:18:54.263 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:54.302 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:18:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:18:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: pgmap v248: 158 pgs: 2 remapped+peering, 156 active+clean; 1.3 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.60k op/s; 43 KiB/s, 2 objects/s recovering 2024-09-16T10:18:54.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1649974073' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:54.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: from='client.25276 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:18:54.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:18:54.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: from='client.25276 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:18:54.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:53 smithi165 ceph-mon[32696]: osdmap e176: 8 total, 8 up, 7 in 2024-09-16T10:18:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:54.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:54.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:54.436 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:54.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:54.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 13 16 27917 27901 134.119 24.2812 0.00665262 0.00744152 2024-09-16T10:18:54.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:54.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:54.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:54.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:54.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:54.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:54.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:54.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:54.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:18:54.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:55.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:55.102 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:55.147 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:55.249 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.287 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:55.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:55.320 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:55.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:55.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:55.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 14 16 29159 29143 130.084 77.625 0.0083623 0.00767724 2024-09-16T10:18:55.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:55.663 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:55.697 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:55.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:55.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:55.948 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:55.978 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:55.978 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:55.990 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:18:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:55.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:56.092 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:56.133 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:56.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:56.235 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:56.258 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:56.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:56.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:56.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:56.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:56.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:56 smithi063 ceph-mon[29993]: pgmap v250: 158 pgs: 6 remapped+peering, 152 active+clean; 1.7 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 81 MiB/s wr, 1.30k op/s; 43 KiB/s, 8 objects/s recovering 2024-09-16T10:18:56.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:56 smithi063 ceph-mon[29993]: osdmap e177: 8 total, 8 up, 7 in 2024-09-16T10:18:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:56 smithi063 ceph-mon[39470]: pgmap v250: 158 pgs: 6 remapped+peering, 152 active+clean; 1.7 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 81 MiB/s wr, 1.30k op/s; 43 KiB/s, 8 objects/s recovering 2024-09-16T10:18:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:56 smithi063 ceph-mon[39470]: osdmap e177: 8 total, 8 up, 7 in 2024-09-16T10:18:56.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 15 16 30542 30526 127.174 86.4375 0.00417266 0.00785492 2024-09-16T10:18:56.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:56.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:56 smithi165 ceph-mon[32696]: pgmap v250: 158 pgs: 6 remapped+peering, 152 active+clean; 1.7 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 81 MiB/s wr, 1.30k op/s; 43 KiB/s, 8 objects/s recovering 2024-09-16T10:18:56.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:56 smithi165 ceph-mon[32696]: osdmap e177: 8 total, 8 up, 7 in 2024-09-16T10:18:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:56.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:56.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:56.641 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:56.674 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:56.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:56.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:56.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:18:56.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:56.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:56.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:56.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:56.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:57.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:57.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:57.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:57.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:57.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:57.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:57.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:57.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:57.260 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:57.347 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:18:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:18:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:18:57.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:57.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:57.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:57.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 16 16 32027 32011 125.026 92.8125 0.0170814 0.00799155 2024-09-16T10:18:57.502 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:57.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:57.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:18:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:57.677 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:57.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:18:57.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:57.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:57.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:57.889 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:18:57.889 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:18:57.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:57.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:57.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:58.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:58.106 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:58.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:58.207 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:18:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:58.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:58.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:18:58.295 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[39470]: pgmap v252: 158 pgs: 6 remapped+peering, 152 active+clean; 1.7 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 770 B/s rd, 92 MiB/s wr, 1.47k op/s; 48 KiB/s, 10 objects/s recovering 2024-09-16T10:18:58.295 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:18:58.295 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:18:58.295 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[39470]: osdmap e178: 8 total, 8 up, 7 in 2024-09-16T10:18:58.295 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[29993]: pgmap v252: 158 pgs: 6 remapped+peering, 152 active+clean; 1.7 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 770 B/s rd, 92 MiB/s wr, 1.47k op/s; 48 KiB/s, 10 objects/s recovering 2024-09-16T10:18:58.295 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:18:58.296 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:18:58.296 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:58 smithi063 ceph-mon[29993]: osdmap e178: 8 total, 8 up, 7 in 2024-09-16T10:18:58.338 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:58 smithi165 ceph-mon[32696]: pgmap v252: 158 pgs: 6 remapped+peering, 152 active+clean; 1.7 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 770 B/s rd, 92 MiB/s wr, 1.47k op/s; 48 KiB/s, 10 objects/s recovering 2024-09-16T10:18:58.339 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:58 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:18:58.339 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:58 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:18:58.339 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:58 smithi165 ceph-mon[32696]: osdmap e178: 8 total, 8 up, 7 in 2024-09-16T10:18:58.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:58.384 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:18:58.385 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:18:58.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:58.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:58.418 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:18:58.419 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:18:58.452 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:18:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:18:58.487 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:18:58.487 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:18:58.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:18:58.492 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 17 15 33504 33489 123.105 92.375 0.00887073 0.00811712 2024-09-16T10:18:58.517 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:18:58.517 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:18:58.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:58.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:18:58.559 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:18:58.559 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:18:58.592 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:18:58.592 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:18:58.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:18:58.628 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:18:58.628 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:18:58.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:58.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:58.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:18:58.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:18:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:18:58.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:58.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:58.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:58.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:18:58.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:58.927 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:58.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:59.028 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:59.067 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:59.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:59.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:59.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:59.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:59.255 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:18:59.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:18:59.361 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:59.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:18:59.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:59.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:18:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/561932903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:59.402 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:59.402 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:18:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/561932903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:59.438 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:18:59.438 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:18:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/561932903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:18:59.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:18:59.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 18 16 34948 34932 121.276 90.1875 0.0135042 0.00823937 2024-09-16T10:18:59.528 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:18:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:59.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:18:59.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:18:59.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:59.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:18:59.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:59.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:18:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:18:59.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:18:59.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:18:59.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:13786 7.0 7:0f5c2eba:::benchmark_data_smithi063_73086_object13785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:14:55.131144+0000", 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 244.784236746, 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069275028000000002, 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.920 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 13786 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131144+0000", 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131144+0000", 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131147+0000", 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.605e-06 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131185+0000", 2024-09-16T10:18:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8260000000000003e-05 2024-09-16T10:18:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131198+0000", 2024-09-16T10:18:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2865e-05 2024-09-16T10:18:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131207+0000", 2024-09-16T10:18:59.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4829999999999999e-06 2024-09-16T10:18:59.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131281+0000", 2024-09-16T10:18:59.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4697999999999995e-05 2024-09-16T10:18:59.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131404+0000", 2024-09-16T10:18:59.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012222099999999999 2024-09-16T10:18:59.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.131565+0000", 2024-09-16T10:18:59.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016159599999999999 2024-09-16T10:18:59.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.132841+0000", 2024-09-16T10:18:59.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012758909999999999 2024-09-16T10:18:59.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.133189+0000", 2024-09-16T10:18:59.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034776700000000002 2024-09-16T10:18:59.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.200316+0000", 2024-09-16T10:18:59.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.067126913999999996 2024-09-16T10:18:59.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.200375+0000", 2024-09-16T10:18:59.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9675e-05 2024-09-16T10:18:59.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:14:55.200419+0000", 2024-09-16T10:18:59.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4053e-05 2024-09-16T10:18:59.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.931 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.932 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:37949 7.0 7:0b15b38d:::benchmark_data_smithi063_73086_object37948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T10:18:59.932 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:08.092153+0000", 2024-09-16T10:18:59.932 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 231.82322843700001, 2024-09-16T10:18:59.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.064236773999999996, 2024-09-16T10:18:59.933 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.933 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.933 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.933 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.933 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.933 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.934 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 37949 2024-09-16T10:18:59.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.934 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.092153+0000", 2024-09-16T10:18:59.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.092153+0000", 2024-09-16T10:18:59.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.092154+0000", 2024-09-16T10:18:59.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-16T10:18:59.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.092184+0000", 2024-09-16T10:18:59.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0431000000000001e-05 2024-09-16T10:18:59.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.092192+0000", 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5100000000000001e-06 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.092197+0000", 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9290000000000003e-06 2024-09-16T10:18:59.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.093840+0000", 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016426279999999999 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.094017+0000", 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017700999999999999 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.094179+0000", 2024-09-16T10:18:59.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016181999999999999 2024-09-16T10:18:59.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.096010+0000", 2024-09-16T10:18:59.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018314309999999999 2024-09-16T10:18:59.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.096039+0000", 2024-09-16T10:18:59.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8521999999999999e-05 2024-09-16T10:18:59.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.156295+0000", 2024-09-16T10:18:59.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060256452000000002 2024-09-16T10:18:59.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.156350+0000", 2024-09-16T10:18:59.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4706000000000001e-05 2024-09-16T10:18:59.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:08.156390+0000", 2024-09-16T10:18:59.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9517999999999997e-05 2024-09-16T10:18:59.944 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.945 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.945 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40104 7.0 7:08de039f:::benchmark_data_smithi063_73086_object40103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:18:59.945 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.549461+0000", 2024-09-16T10:18:59.946 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 230.36592006699999, 2024-09-16T10:18:59.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12534163400000001, 2024-09-16T10:18:59.946 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.946 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.947 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.948 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40104 2024-09-16T10:18:59.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.948 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549461+0000", 2024-09-16T10:18:59.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549461+0000", 2024-09-16T10:18:59.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549469+0000", 2024-09-16T10:18:59.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-16T10:18:59.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549516+0000", 2024-09-16T10:18:59.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7321999999999999e-05 2024-09-16T10:18:59.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549532+0000", 2024-09-16T10:18:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6410999999999999e-05 2024-09-16T10:18:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549540+0000", 2024-09-16T10:18:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5279999999999998e-06 2024-09-16T10:18:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549593+0000", 2024-09-16T10:18:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2848e-05 2024-09-16T10:18:59.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549666+0000", 2024-09-16T10:18:59.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3510000000000006e-05 2024-09-16T10:18:59.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549862+0000", 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019536799999999999 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551035+0000", 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011732509999999999 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.552075+0000", 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001040132 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.674699+0000", 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122624263 2024-09-16T10:18:59.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.674756+0000", 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7046000000000002e-05 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.674803+0000", 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6495000000000001e-05 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:43497 7.0 7:0cd41ee2:::benchmark_data_smithi063_73086_object43496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:11.754252+0000", 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 228.16112959200001, 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103530996, 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 43497 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754252+0000", 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754252+0000", 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754254+0000", 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3439999999999999e-06 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754296+0000", 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1912999999999997e-05 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754313+0000", 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6886000000000001e-05 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754321+0000", 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.371e-06 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754371+0000", 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0217999999999999e-05 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754498+0000", 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012694200000000001 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.754678+0000", 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017977999999999999 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.757258+0000", 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025800979999999999 2024-09-16T10:18:59.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.836854+0000", 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.079595945000000001 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.857718+0000", 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020864100999999999 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.857759+0000", 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1270999999999998e-05 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:11.857783+0000", 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3127000000000001e-05 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:54870 7.0 7:0fd441e8:::benchmark_data_smithi063_73086_object54869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:19.469886+0000", 2024-09-16T10:18:59.965 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 220.445494798, 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.068969994000000007, 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 54870 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.469886+0000", 2024-09-16T10:18:59.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.469886+0000", 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.469888+0000", 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.874e-06 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.469938+0000", 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9465000000000002e-05 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.469949+0000", 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1742e-05 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.469956+0000", 2024-09-16T10:18:59.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.973e-06 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470006+0000", 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9558000000000002e-05 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470116+0000", 2024-09-16T10:18:59.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110276 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470274+0000", 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015793200000000001 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.473825+0000", 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035507220000000001 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.473841+0000", 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5908000000000001e-05 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.538739+0000", 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.064898196000000005 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.538812+0000", 2024-09-16T10:18:59.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3004999999999996e-05 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.538856+0000", 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4342999999999999e-05 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:54873 7.0 7:015a3b7a:::benchmark_data_smithi063_73086_object54872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:19.470034+0000", 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 220.44534684300001, 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076222207, 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.973 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 54873 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470034+0000", 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470034+0000", 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470035+0000", 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2699999999999998e-07 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470105+0000", 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9771999999999995e-05 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470113+0000", 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1289999999999996e-06 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.470120+0000", 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5359999999999998e-06 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.472291+0000", 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021716090000000001 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.472400+0000", 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109213 2024-09-16T10:18:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.472540+0000", 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000139123 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.473933+0000", 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013931970000000001 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.474359+0000", 2024-09-16T10:18:59.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000426661 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.546132+0000", 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071772098000000006 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.546205+0000", 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3343999999999994e-05 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:19.546257+0000", 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1697999999999999e-05 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:63444 7.0 7:05604893:::benchmark_data_smithi063_73086_object63443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:26.207787+0000", 2024-09-16T10:18:59.980 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 213.707593941, 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.059402964000000003, 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 63444 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.207787+0000", 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.207787+0000", 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.207789+0000", 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6509999999999999e-06 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.207851+0000", 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1674999999999995e-05 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.207861+0000", 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9899999999999992e-06 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.207866+0000", 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4219999999999999e-06 2024-09-16T10:18:59.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.208387+0000", 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00052113099999999998 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.208475+0000", 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8009000000000004e-05 2024-09-16T10:18:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.208620+0000", 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144671 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.210236+0000", 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016162080000000001 2024-09-16T10:18:59.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.212430+0000", 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021938869999999998 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.267117+0000", 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054686881 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.267158+0000", 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0840000000000002e-05 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.267190+0000", 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2598999999999997e-05 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.987 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:63456 7.0 7:01d243b1:::benchmark_data_smithi063_73086_object63455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:26.214855+0000", 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 213.70052602999999, 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.059724669000000001, 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 63456 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.214855+0000", 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.214855+0000", 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.214857+0000", 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0190000000000001e-06 2024-09-16T10:18:59.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.214894+0000", 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7095000000000003e-05 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.214911+0000", 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.66e-05 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.214919+0000", 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1499999999999999e-06 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.214976+0000", 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6718999999999997e-05 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.215106+0000", 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129901 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.215252+0000", 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014656200000000001 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.217248+0000", 2024-09-16T10:18:59.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001995443 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.219770+0000", 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025228339999999998 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:18:59.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.274478+0000", 2024-09-16T10:18:59.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054707565 2024-09-16T10:18:59.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:18:59.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.274533+0000", 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4950000000000001e-05 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:26.274580+0000", 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6830999999999999e-05 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65346 7.0 7:05f67841:::benchmark_data_smithi063_73086_object65345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.845998+0000", 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.06938304600001, 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.061818250999999998, 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:18:59.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65346 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.845998+0000", 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.845998+0000", 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:18:59.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.845999+0000", 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2240000000000001e-06 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.846050+0000", 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0772000000000001e-05 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.846058+0000", 2024-09-16T10:18:59.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2379999999999997e-06 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.846065+0000", 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7619999999999998e-06 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.846161+0000", 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6088999999999994e-05 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:18:59.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.846288+0000", 2024-09-16T10:19:00.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000126849 2024-09-16T10:19:00.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.846440+0000", 2024-09-16T10:19:00.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015173399999999999 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.848330+0000", 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018903850000000001 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.848877+0000", 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054720099999999998 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.907712+0000", 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058834466000000002 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.907769+0000", 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6813e-05 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.907816+0000", 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7718e-05 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.002 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65349 7.0 7:08f8b084:::benchmark_data_smithi063_73086_object65348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.851332+0000", 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.06404949200001, 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.061711805000000002, 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.003 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65349 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851332+0000", 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851332+0000", 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851333+0000", 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.683e-06 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851384+0000", 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0782999999999998e-05 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851394+0000", 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.023e-05 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851400+0000", 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3059999999999997e-06 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851439+0000", 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8865000000000001e-05 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851527+0000", 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8143e-05 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.851618+0000", 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1513000000000003e-05 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.853038+0000", 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014195220000000001 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.853222+0000", 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018474 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.912969+0000", 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.059746292999999999 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.913005+0000", 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6272000000000001e-05 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.913044+0000", 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8454999999999997e-05 2024-09-16T10:19:00.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90056 7.0 7:081070f4:::benchmark_data_smithi063_73086_object90055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 190.40179289299999, 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13312953299999999, 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90056 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513590+0000", 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.531e-06 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513668+0000", 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8187000000000002e-05 2024-09-16T10:19:00.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513679+0000", 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0781999999999999e-05 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513685+0000", 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.93e-06 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513734+0000", 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9539000000000001e-05 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513851+0000", 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116749 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513939+0000", 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8208999999999995e-05 2024-09-16T10:19:00.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519242+0000", 2024-09-16T10:19:00.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053023430000000002 2024-09-16T10:19:00.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:00.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:00.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:00.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523299+0000", 2024-09-16T10:19:00.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040579250000000004 2024-09-16T10:19:00.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646650+0000", 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.123350551 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646686+0000", 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6316999999999998e-05 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646718+0000", 2024-09-16T10:19:00.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1470000000000002e-05 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90058 7.0 7:0216797c:::benchmark_data_smithi063_73086_object90057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 190.401594206, 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13450020500000001, 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.020 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90058 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513788+0000", 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1289999999999999e-06 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516073+0000", 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022849010000000002 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516088+0000", 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5031000000000001e-05 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516096+0000", 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9680000000000006e-06 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516691+0000", 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00059533899999999996 2024-09-16T10:19:00.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516812+0000", 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120388 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516932+0000", 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120689 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519245+0000", 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023127080000000001 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523333+0000", 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040881149999999998 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648185+0000", 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124851428 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648246+0000", 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1354999999999998e-05 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648287+0000", 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1153999999999999e-05 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.026 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90059 7.0 7:0d97e368:::benchmark_data_smithi063_73086_object90058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 190.399265485, 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.1397098, 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90059 2024-09-16T10:19:00.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:19:00.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516117+0000", 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5400000000000001e-06 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516153+0000", 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5546000000000002e-05 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516161+0000", 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8539999999999997e-06 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516166+0000", 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2310000000000004e-06 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519295+0000", 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031290430000000002 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519373+0000", 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8521000000000003e-05 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519458+0000", 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4351000000000006e-05 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521937+0000", 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024794130000000002 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523362+0000", 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001425228 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655714+0000", 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13235127199999999 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655770+0000", 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6622000000000001e-05 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655826+0000", 2024-09-16T10:19:00.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5179000000000001e-05 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:91535 7.0 7:0328edd5:::benchmark_data_smithi063_73086_object91534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:51.077127+0000", 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 188.83825390199999, 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.068400143999999996, 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 91535 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077127+0000", 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077127+0000", 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077130+0000", 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7819999999999998e-06 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077179+0000", 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8761000000000003e-05 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077194+0000", 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4949999999999999e-05 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077202+0000", 2024-09-16T10:19:00.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2190000000000004e-06 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077260+0000", 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8038000000000002e-05 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077384+0000", 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012364900000000001 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.077517+0000", 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013345699999999999 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.080095+0000", 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025781559999999999 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.082494+0000", 2024-09-16T10:19:00.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023990499999999998 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.145436+0000", 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.062941947999999998 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.145496+0000", 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0007000000000003e-05 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:51.145527+0000", 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1127000000000002e-05 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:102064 7.0 7:0979ba92:::benchmark_data_smithi063_73086_object102063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:16:02.315022+0000", 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 177.60035961, 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058629578000000002, 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.091 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 102064 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.315022+0000", 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.315022+0000", 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.315022+0000", 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0800000000000004e-07 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.315073+0000", 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0419e-05 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.315080+0000", 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8689999999999998e-06 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.315085+0000", 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4779999999999999e-06 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.318255+0000", 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031703349999999998 2024-09-16T10:19:00.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.318342+0000", 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7070999999999994e-05 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.318463+0000", 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120512 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.321674+0000", 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003211111 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.324988+0000", 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033141049999999999 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.373543+0000", 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048555156000000002 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.373600+0000", 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6481999999999999e-05 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:02.373651+0000", 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1332000000000003e-05 2024-09-16T10:19:00.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:104189 7.0 7:0372fb45:::benchmark_data_smithi063_73086_object104188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:16:04.431990+0000", 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 175.48339069100001, 2024-09-16T10:19:00.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.075904948, 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104189 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.431990+0000", 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.431990+0000", 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.431993+0000", 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.508e-06 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.432059+0000", 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5549000000000005e-05 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.432079+0000", 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0112999999999999e-05 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.432095+0000", 2024-09-16T10:19:00.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6127999999999999e-05 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.432162+0000", 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6816999999999994e-05 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.432284+0000", 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012284000000000001 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.432433+0000", 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000148858 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.435818+0000", 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003384232 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.436010+0000", 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019266199999999999 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.507775+0000", 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071764993999999999 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.507848+0000", 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2711000000000002e-05 2024-09-16T10:19:00.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.507895+0000", 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7536e-05 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:104201 7.0 7:086d93d7:::benchmark_data_smithi063_73086_object104200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:16:04.438937+0000", 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 175.47644463399999, 2024-09-16T10:19:00.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071403649999999999, 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104201 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.438937+0000", 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.438937+0000", 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.438939+0000", 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4480000000000001e-06 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.438986+0000", 2024-09-16T10:19:00.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7336000000000002e-05 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.439000+0000", 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3434e-05 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.439007+0000", 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.661e-06 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.439065+0000", 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8066000000000003e-05 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.439201+0000", 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013508500000000001 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.439358+0000", 2024-09-16T10:19:00.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015774699999999999 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.441959+0000", 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026008350000000001 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.445843+0000", 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038835639999999999 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.510195+0000", 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.064352099999999995 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.510285+0000", 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0506000000000004e-05 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:16:04.510340+0000", 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4867999999999997e-05 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.114 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:200800 7.0 7:0ca43226:::benchmark_data_smithi063_73086_object81521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.239946+0000", 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.675435461999996, 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.084887246, 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200800 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.239946+0000", 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.239946+0000", 2024-09-16T10:19:00.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.239947+0000", 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.172e-06 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.239951+0000", 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7110000000000001e-06 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.239959+0000", 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9209999999999992e-06 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.239964+0000", 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9790000000000001e-06 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240006+0000", 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2064000000000002e-05 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240129+0000", 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012316900000000001 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240264+0000", 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135208 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.243150+0000", 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028858429999999999 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.244175+0000", 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010252250000000001 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324751+0000", 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080575593000000001 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324785+0000", 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3893999999999998e-05 2024-09-16T10:19:00.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324833+0000", 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8467000000000002e-05 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:217881 7.0 7:039e5847:::benchmark_data_smithi063_73086_object98602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:57.651533+0000", 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.263848381000003, 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10541695, 2024-09-16T10:19:00.121 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 217881 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651533+0000", 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651533+0000", 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651535+0000", 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8390000000000001e-06 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651541+0000", 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1700000000000002e-06 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651557+0000", 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6531000000000001e-05 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651566+0000", 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1880000000000002e-06 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651625+0000", 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9868999999999999e-05 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651750+0000", 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000124389 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651879+0000", 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128985 2024-09-16T10:19:00.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.656872+0000", 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049929559999999998 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.659638+0000", 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00276618 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.756887+0000", 2024-09-16T10:19:00.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097249446000000003 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.756922+0000", 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5009999999999999e-05 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.756950+0000", 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7387000000000001e-05 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:217883 7.0 7:0d7b51e6:::benchmark_data_smithi063_73086_object98604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:57.651820+0000", 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.263561463999999, 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105218676, 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 217883 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.128 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651820+0000", 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651820+0000", 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651821+0000", 2024-09-16T10:19:00.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.437e-06 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651826+0000", 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7709999999999997e-06 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651837+0000", 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1112e-05 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.651844+0000", 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.917e-06 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.654134+0000", 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022900170000000001 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.654249+0000", 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011518000000000001 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.654361+0000", 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111396 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.657136+0000", 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027751669999999998 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.659806+0000", 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026705040000000002 2024-09-16T10:19:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.756983+0000", 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097176901999999996 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.757014+0000", 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0409999999999999e-05 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.757038+0000", 2024-09-16T10:19:00.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4862999999999999e-05 2024-09-16T10:19:00.134 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.134 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.134 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.134 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:19:00.134 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:19:00.134 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:19:00.134 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:00.180 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:00.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:00.237 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:00.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:00.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:00 smithi165 ceph-mon[32696]: pgmap v254: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.0 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 682 B/s rd, 123 MiB/s wr, 1.97k op/s; 8212/100173 objects misplaced (8.198%); 20 MiB/s, 323 objects/s recovering 2024-09-16T10:19:00.313 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:00.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:00.415 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:00.452 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:00.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:00.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 19 16 36181 36165 118.949 77.0625 0.00661308 0.00840139 2024-09-16T10:19:00.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:00.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:00.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:00.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:00 smithi063 ceph-mon[29993]: pgmap v254: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.0 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 682 B/s rd, 123 MiB/s wr, 1.97k op/s; 8212/100173 objects misplaced (8.198%); 20 MiB/s, 323 objects/s recovering 2024-09-16T10:19:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:00 smithi063 ceph-mon[39470]: pgmap v254: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.0 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 682 B/s rd, 123 MiB/s wr, 1.97k op/s; 8212/100173 objects misplaced (8.198%); 20 MiB/s, 323 objects/s recovering 2024-09-16T10:19:00.735 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:00.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:00.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:00.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:00.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:00.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:01.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:01.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:01.153 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:01.255 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:01.296 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:01.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:01.430 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:01.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:19:01.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:19:01.492283+0000 min lat: 0.00128827 max lat: 2.11649 avg lat: 0.00855615 2024-09-16T10:19:01.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:19:01.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 20 16 37385 37369 116.763 75.25 0.0119789 0.00855615 2024-09-16T10:19:01.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:01.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:01.669 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:01.690 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:01.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:01.792 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:01.848 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:01.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:01.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:01.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:02.047 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:02.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:01 smithi165 ceph-mon[32696]: osdmap e179: 8 total, 8 up, 7 in 2024-09-16T10:19:02.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:01 smithi165 ceph-mon[32696]: pgmap v256: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.0 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 8212/100173 objects misplaced (8.198%); 20 MiB/s, 317 objects/s recovering 2024-09-16T10:19:02.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:02.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:02.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:02.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:19:02.186 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:01 smithi063 ceph-mon[29993]: osdmap e179: 8 total, 8 up, 7 in 2024-09-16T10:19:02.186 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:01 smithi063 ceph-mon[29993]: pgmap v256: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.0 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 8212/100173 objects misplaced (8.198%); 20 MiB/s, 317 objects/s recovering 2024-09-16T10:19:02.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:01 smithi063 ceph-mon[39470]: osdmap e179: 8 total, 8 up, 7 in 2024-09-16T10:19:02.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:01 smithi063 ceph-mon[39470]: pgmap v256: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.0 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 8212/100173 objects misplaced (8.198%); 20 MiB/s, 317 objects/s recovering 2024-09-16T10:19:02.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:02.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:02.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:19:02.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T10:19:02.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:02.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:02.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:02.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 21 16 38597 38581 114.809 75.75 0.0103121 0.00870345 2024-09-16T10:19:02.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:02.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:02.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:02.605 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:19:02.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:02.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:02.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:02.701 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:19:02.701 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:19:02.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:19:02.739 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:19:02.771 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:19:02.771 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:19:02.785 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:02.787 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:02.814 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:19:02.814 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:19:02.829 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:19:02.846 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:02.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:02.856 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:19:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:19:02.894 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:19:02.895 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:19:02.929 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:19:02.929 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:19:02.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:02.961 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:19:02.961 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:19:02.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:02.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/287830991' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:02 smithi165 ceph-mon[32696]: from='client.25288 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2848785916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:03.082 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:03.119 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:03.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:03.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/287830991' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:03.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:02 smithi063 ceph-mon[39470]: from='client.25288 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:03.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2848785916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:03.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/287830991' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:03.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:02 smithi063 ceph-mon[29993]: from='client.25288 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:03.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2848785916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:03.220 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:03.248 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:03.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:03.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:03.390 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:03.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:03.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:03.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 22 16 39916 39900 113.338 82.4375 0.00745828 0.0088166 2024-09-16T10:19:03.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:03.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:03.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:03.667 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:03.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:03.769 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:03.808 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:03.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:03.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:03.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:03.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:03.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:19:04.050 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:04.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:03 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:03 smithi165 ceph-mon[32696]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:19:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:03 smithi165 ceph-mon[32696]: osdmap e180: 8 total, 8 up, 7 in 2024-09-16T10:19:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:03 smithi165 ceph-mon[32696]: pgmap v258: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.1 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7471/104865 objects misplaced (7.124%); 28 MiB/s, 440 objects/s recovering 2024-09-16T10:19:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:03 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:19:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:03 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:19:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:04.088 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:04.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:04.138 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[29993]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[29993]: osdmap e180: 8 total, 8 up, 7 in 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[29993]: pgmap v258: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.1 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7471/104865 objects misplaced (7.124%); 28 MiB/s, 440 objects/s recovering 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[39470]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:19:04.139 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[39470]: osdmap e180: 8 total, 8 up, 7 in 2024-09-16T10:19:04.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[39470]: pgmap v258: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.1 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 7471/104865 objects misplaced (7.124%); 28 MiB/s, 440 objects/s recovering 2024-09-16T10:19:04.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:19:04.140 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:03 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:19:04.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:04.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:04.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:04.335 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:04.372 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:04.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:04.447 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:19:04.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:04.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 23 16 41346 41330 112.296 89.375 0.0101913 0.00889846 2024-09-16T10:19:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:04.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:04.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:04.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:04.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:04.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:04.776 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:04.809 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:04.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:04.868 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T10:19:04.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:04.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:04.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:05.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:05.065 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:05.065 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:05.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:05.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:05.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:04 smithi063 ceph-mon[29993]: osdmap e181: 8 total, 8 up, 7 in 2024-09-16T10:19:05.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:04 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/174233587' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:19:05.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:04 smithi063 ceph-mon[39470]: osdmap e181: 8 total, 8 up, 7 in 2024-09-16T10:19:05.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:04 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/174233587' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:19:05.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:05.240 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:05.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:04 smithi165 ceph-mon[32696]: osdmap e181: 8 total, 8 up, 7 in 2024-09-16T10:19:05.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:04 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/174233587' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:19:05.342 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:05.380 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:05.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:05.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:05.493 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 24 16 42736 42720 111.236 86.875 0.0156817 0.00898328 2024-09-16T10:19:05.517 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:05.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:05.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:05.656 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:05.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:05.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:05.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:19:05.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:05.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:05.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:19:05.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:05.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T10:19:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:05.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:05.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:06.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:05 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/174233587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:19:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:05 smithi165 ceph-mon[32696]: osdmap e182: 8 total, 8 up, 7 in 2024-09-16T10:19:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:05 smithi165 ceph-mon[32696]: pgmap v261: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 5957/119508 objects misplaced (4.985%); 33 MiB/s, 521 objects/s recovering 2024-09-16T10:19:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:05 smithi165 ceph-mon[32696]: osdmap e183: 8 total, 8 up, 7 in 2024-09-16T10:19:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:06.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:06.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/174233587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:19:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[29993]: osdmap e182: 8 total, 8 up, 7 in 2024-09-16T10:19:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[29993]: pgmap v261: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 5957/119508 objects misplaced (4.985%); 33 MiB/s, 521 objects/s recovering 2024-09-16T10:19:06.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[29993]: osdmap e183: 8 total, 8 up, 7 in 2024-09-16T10:19:06.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/174233587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:19:06.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[39470]: osdmap e182: 8 total, 8 up, 7 in 2024-09-16T10:19:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[39470]: pgmap v261: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 5957/119508 objects misplaced (4.985%); 33 MiB/s, 521 objects/s recovering 2024-09-16T10:19:06.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:05 smithi063 ceph-mon[39470]: osdmap e183: 8 total, 8 up, 7 in 2024-09-16T10:19:06.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:06.259 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:19:06.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:06.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:06.346 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:06.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:06.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:06.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:06.494 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 25 16 44091 44075 110.17 84.6875 0.00984157 0.00907123 2024-09-16T10:19:06.498 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:06.534 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:06.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:06.636 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:06.688 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:06.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:06.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:06.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:06.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:06.885 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:06.885 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:06.918 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:06.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:06.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:06.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:07.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:07.111 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:07.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:07.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:07.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:07.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:07.335 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:07.378 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:19:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:19:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:19:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:07.384 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:07.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:07.471 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:19:07.485 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:07.494 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 26 16 45352 45336 108.964 78.8125 0.00807002 0.00917136 2024-09-16T10:19:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:07.556 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:07.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:07.658 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:07.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:07.806 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:07.863 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:07.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:07.965 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:07.974 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:08.017 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:08.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:08.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:08.205 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T10:19:08.205 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:19:08.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:08.280 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:08.382 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:08.420 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: pgmap v263: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 5957/119508 objects misplaced (4.985%); 33 MiB/s, 523 objects/s recovering 2024-09-16T10:19:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2645688930' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1786172353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:19:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='client.25306 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: pgmap v263: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 5957/119508 objects misplaced (4.985%); 33 MiB/s, 523 objects/s recovering 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2645688930' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1786172353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:19:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='client.25306 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:19:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-16T10:19:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-16T10:19:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]} : dispatch 2024-09-16T10:19:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"} : dispatch 2024-09-16T10:19:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]} : dispatch 2024-09-16T10:19:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"} : dispatch 2024-09-16T10:19:08.494 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 27 16 46562 46546 107.729 75.625 0.00495587 0.00927252 2024-09-16T10:19:08.522 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.549 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:08.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: pgmap v263: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 5957/119508 objects misplaced (4.985%); 33 MiB/s, 523 objects/s recovering 2024-09-16T10:19:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2645688930' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1786172353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='client.25306 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]} : dispatch 2024-09-16T10:19:08.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"} : dispatch 2024-09-16T10:19:08.554 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:19:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:19:08.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.690 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:08.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:08.701 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:19:08.701 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:19:08.723 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:19:08.724 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:19:08.753 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:19:08.753 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:19:08.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:08.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:19:08.799 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:19:08.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:08.833 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:19:08.833 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:19:08.870 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:19:08.870 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:19:08.909 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:19:08.910 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:19:08.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:08.939 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:19:08.939 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:19:08.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:08.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:09.055 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:09.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:09.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:09.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:09.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:09.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:09.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:09.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:09.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2645688930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]}]': finished 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"}]': finished 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: osdmap e184: 8 total, 8 up, 7 in 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2030458814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2645688930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]}]': finished 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"}]': finished 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: osdmap e184: 8 total, 8 up, 7 in 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2030458814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:09.464 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:09.494 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 28 15 47639 47624 106.288 67.375 0.00666658 0.00940234 2024-09-16T10:19:09.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:09.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:09.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:19:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:19:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2645688930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:19:09.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-16T10:19:09.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [4, 6]}]': finished 2024-09-16T10:19:09.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "6.12"}]': finished 2024-09-16T10:19:09.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: osdmap e184: 8 total, 8 up, 7 in 2024-09-16T10:19:09.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:09.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2030458814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:09.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:09.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:09.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:09.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:09.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:09.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:09.912 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:09.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:10.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:10.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:10.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:10.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:10.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:10.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:10.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:10.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:10.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:10.428 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:10.460 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:10.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:10.494 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 29 16 48911 48895 105.362 79.4375 0.0350199 0.00946826 2024-09-16T10:19:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:10 smithi165 ceph-mon[32696]: pgmap v265: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 969 B/s rd, 80 MiB/s wr, 1.27k op/s; 3352/139674 objects misplaced (2.400%); 31 MiB/s, 493 objects/s recovering 2024-09-16T10:19:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:10 smithi165 ceph-mon[32696]: osdmap e185: 8 total, 8 up, 7 in 2024-09-16T10:19:10.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:10.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:10.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:10 smithi063 ceph-mon[29993]: pgmap v265: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 969 B/s rd, 80 MiB/s wr, 1.27k op/s; 3352/139674 objects misplaced (2.400%); 31 MiB/s, 493 objects/s recovering 2024-09-16T10:19:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:10 smithi063 ceph-mon[29993]: osdmap e185: 8 total, 8 up, 7 in 2024-09-16T10:19:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:10 smithi063 ceph-mon[39470]: pgmap v265: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 969 B/s rd, 80 MiB/s wr, 1.27k op/s; 3352/139674 objects misplaced (2.400%); 31 MiB/s, 493 objects/s recovering 2024-09-16T10:19:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:10 smithi063 ceph-mon[39470]: osdmap e185: 8 total, 8 up, 7 in 2024-09-16T10:19:10.702 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:10.739 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:10.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:10.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:10.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:10.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:10.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:11.018 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:11.120 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:11.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:11.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:11.294 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:11.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:11.431 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:11.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 30 16 50065 50049 104.254 72.125 0.00629891 0.00957252 2024-09-16T10:19:11.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:11.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:11.665 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:11.700 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:11.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:11.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:11.948 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:11.978 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:11.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:12.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:12.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:12.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:12.221 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:12.258 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:12.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:12.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:12.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:12.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:12.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:12.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 31 16 51175 51159 103.128 69.375 0.00246965 0.00965234 2024-09-16T10:19:12.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:12.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:12.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:12 smithi165 ceph-mon[32696]: pgmap v267: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 3352/139674 objects misplaced (2.400%); 27 MiB/s, 433 objects/s recovering 2024-09-16T10:19:12.621 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:12.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:12.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:12 smithi063 ceph-mon[29993]: pgmap v267: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 3352/139674 objects misplaced (2.400%); 27 MiB/s, 433 objects/s recovering 2024-09-16T10:19:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:12 smithi063 ceph-mon[39470]: pgmap v267: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 3352/139674 objects misplaced (2.400%); 27 MiB/s, 433 objects/s recovering 2024-09-16T10:19:12.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:12.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:12.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:12.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:12.896 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:12.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:12.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:19:12.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:13.026 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:13.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:13.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:13.247 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:19:13.254 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.278 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:13.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:13.380 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:13.403 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:13.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 32 16 51266 51250 100.083 5.6875 0.00257942 0.00993987 2024-09-16T10:19:13.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:13.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:13 smithi165 ceph-mon[32696]: osdmap e186: 8 total, 8 up, 7 in 2024-09-16T10:19:13.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:13.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:13.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:13 smithi063 ceph-mon[29993]: osdmap e186: 8 total, 8 up, 7 in 2024-09-16T10:19:13.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:13 smithi063 ceph-mon[39470]: osdmap e186: 8 total, 8 up, 7 in 2024-09-16T10:19:13.754 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:13.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:13.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:13.902 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:13.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:14.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:14.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:14.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:14.082 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:19:14.083 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:19:14.083 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:19:14.083 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:19:14.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:14.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:14.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:14.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:14.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:14.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:14.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 33 16 51396 51380 97.2968 8.125 0.002299 0.0102376 2024-09-16T10:19:14.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:14.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:14 smithi165 ceph-mon[32696]: 4.11 scrub starts 2024-09-16T10:19:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:14 smithi165 ceph-mon[32696]: 4.11 scrub ok 2024-09-16T10:19:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:14 smithi165 ceph-mon[32696]: pgmap v269: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 3352/140670 objects misplaced (2.383%); 27 MiB/s, 434 objects/s recovering 2024-09-16T10:19:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:14 smithi165 ceph-mon[32696]: osdmap e187: 8 total, 8 up, 7 in 2024-09-16T10:19:14.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.645 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:14.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:14.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[39470]: 4.11 scrub starts 2024-09-16T10:19:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[39470]: 4.11 scrub ok 2024-09-16T10:19:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[39470]: pgmap v269: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 3352/140670 objects misplaced (2.383%); 27 MiB/s, 434 objects/s recovering 2024-09-16T10:19:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[39470]: osdmap e187: 8 total, 8 up, 7 in 2024-09-16T10:19:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[29993]: 4.11 scrub starts 2024-09-16T10:19:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[29993]: 4.11 scrub ok 2024-09-16T10:19:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[29993]: pgmap v269: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 3352/140670 objects misplaced (2.383%); 27 MiB/s, 434 objects/s recovering 2024-09-16T10:19:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:14 smithi063 ceph-mon[29993]: osdmap e187: 8 total, 8 up, 7 in 2024-09-16T10:19:14.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:19:14.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:14.796 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:14.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:14.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:14.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:14.987 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:19:15.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:15.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:15.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:15.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:15.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:15.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:15.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:15.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:15.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:15.438 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:15.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:15.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[29993]: osdmap e188: 8 total, 8 up, 7 in 2024-09-16T10:19:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[29993]: 4.1c scrub starts 2024-09-16T10:19:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[29993]: 4.1c scrub ok 2024-09-16T10:19:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[29993]: osdmap e189: 8 total, 8 up, 7 in 2024-09-16T10:19:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[39470]: osdmap e188: 8 total, 8 up, 7 in 2024-09-16T10:19:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[39470]: 4.1c scrub starts 2024-09-16T10:19:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[39470]: 4.1c scrub ok 2024-09-16T10:19:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:15 smithi063 ceph-mon[39470]: osdmap e189: 8 total, 8 up, 7 in 2024-09-16T10:19:15.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:15 smithi165 ceph-mon[32696]: osdmap e188: 8 total, 8 up, 7 in 2024-09-16T10:19:15.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:15 smithi165 ceph-mon[32696]: 4.1c scrub starts 2024-09-16T10:19:15.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:15 smithi165 ceph-mon[32696]: 4.1c scrub ok 2024-09-16T10:19:15.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:15 smithi165 ceph-mon[32696]: osdmap e189: 8 total, 8 up, 7 in 2024-09-16T10:19:15.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 34 16 51678 51662 94.9535 17.625 0.00696426 0.010526 2024-09-16T10:19:15.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:15.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:15.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:15.625 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:19:15.625 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:19:15.625 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:19:15.625 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:19:15.625 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:19:15.682 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:15.686 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T10:19:15.719 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:15.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:15.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:15.857 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:15.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:15.959 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:16.003 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:16.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:16.104 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:16.146 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:16.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:16.154 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:16.249 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:16.286 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:16.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:16.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:19:16.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:16.428 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:16.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:16.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 35 16 53381 53365 95.2814 106.438 0.0114705 0.01049 2024-09-16T10:19:16.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[29993]: pgmap v272: 158 pgs: 2 peering, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 872 B/s rd, 52 MiB/s wr, 831 op/s; 337/154305 objects misplaced (0.218%); 32 MiB/s, 514 objects/s recovering 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[29993]: 3.1b scrub starts 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[29993]: 3.1b scrub ok 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[29993]: 5.d scrub starts 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[29993]: 5.d scrub ok 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[39470]: pgmap v272: 158 pgs: 2 peering, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 872 B/s rd, 52 MiB/s wr, 831 op/s; 337/154305 objects misplaced (0.218%); 32 MiB/s, 514 objects/s recovering 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[39470]: 3.1b scrub starts 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[39470]: 3.1b scrub ok 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[39470]: 5.d scrub starts 2024-09-16T10:19:16.542 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:16 smithi063 ceph-mon[39470]: 5.d scrub ok 2024-09-16T10:19:16.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:16 smithi165 ceph-mon[32696]: pgmap v272: 158 pgs: 2 peering, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 872 B/s rd, 52 MiB/s wr, 831 op/s; 337/154305 objects misplaced (0.218%); 32 MiB/s, 514 objects/s recovering 2024-09-16T10:19:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:16 smithi165 ceph-mon[32696]: 3.1b scrub starts 2024-09-16T10:19:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:16 smithi165 ceph-mon[32696]: 3.1b scrub ok 2024-09-16T10:19:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:16 smithi165 ceph-mon[32696]: 5.d scrub starts 2024-09-16T10:19:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:16 smithi165 ceph-mon[32696]: 5.d scrub ok 2024-09-16T10:19:16.560 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:16.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:16.662 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:16.704 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:16.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:16.779 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:19:16.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:19:16.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:16.877 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:16.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:16.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:17.016 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:17.118 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.169 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:17.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:17.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:19:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:19:17.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:19:17.258 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:17.270 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:17.316 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:17.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:17.320 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T10:19:17.419 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:17.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:17.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 36 16 54787 54771 95.0754 87.875 0.00292934 0.0105009 2024-09-16T10:19:17.540 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 5.16 scrub starts 2024-09-16T10:19:17.541 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 5.16 scrub ok 2024-09-16T10:19:17.541 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 5.1 scrub starts 2024-09-16T10:19:17.544 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 5.1 scrub ok 2024-09-16T10:19:17.544 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 5.1e scrub starts 2024-09-16T10:19:17.544 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 5.1e scrub ok 2024-09-16T10:19:17.544 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 3.0 scrub starts 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: 3.0 scrub ok 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2183686209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[29993]: from='client.25318 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 5.16 scrub starts 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 5.16 scrub ok 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 5.1 scrub starts 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 5.1 scrub ok 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 5.1e scrub starts 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 5.1e scrub ok 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 3.0 scrub starts 2024-09-16T10:19:17.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: 3.0 scrub ok 2024-09-16T10:19:17.546 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2183686209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:19:17.546 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:17 smithi063 ceph-mon[39470]: from='client.25318 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 5.16 scrub starts 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 5.16 scrub ok 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 5.1 scrub starts 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 5.1 scrub ok 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 5.1e scrub starts 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 5.1e scrub ok 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 3.0 scrub starts 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: 3.0 scrub ok 2024-09-16T10:19:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2183686209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:19:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:17 smithi165 ceph-mon[32696]: from='client.25318 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:19:17.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:17.606 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:17.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:17.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:17.831 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:17.867 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:17.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:17.995 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:17.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:18.006 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:19:18.007 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:19:18.040 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:19:18.040 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:19:18.070 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:19:18.070 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:19:18.093 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:19:18.093 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:19:18.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:18.121 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:19:18.121 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:19:18.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:18.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:18.157 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:19:18.158 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:19:18.191 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:19:18.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:19:18.227 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:19:18.227 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:19:18.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:18.272 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:18.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:18.341 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:18.341 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:18.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:18.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:18.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:18.419 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 4.c scrub starts 2024-09-16T10:19:18.419 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 4.c scrub ok 2024-09-16T10:19:18.419 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 3.b scrub starts 2024-09-16T10:19:18.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 3.b scrub ok 2024-09-16T10:19:18.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 5.6 scrub starts 2024-09-16T10:19:18.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 5.6 scrub ok 2024-09-16T10:19:18.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: pgmap v274: 158 pgs: 2 peering, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 337/154305 objects misplaced (0.218%); 39 MiB/s, 621 objects/s recovering 2024-09-16T10:19:18.420 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 3.10 scrub starts 2024-09-16T10:19:18.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 3.10 scrub ok 2024-09-16T10:19:18.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:19:18.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:19:18.421 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:19:18.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: osdmap e190: 8 total, 8 up, 7 in 2024-09-16T10:19:18.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 3.e scrub starts 2024-09-16T10:19:18.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: 3.e scrub ok 2024-09-16T10:19:18.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2776926047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:18.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 4.c scrub starts 2024-09-16T10:19:18.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 4.c scrub ok 2024-09-16T10:19:18.424 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 3.b scrub starts 2024-09-16T10:19:18.424 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 3.b scrub ok 2024-09-16T10:19:18.424 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 5.6 scrub starts 2024-09-16T10:19:18.424 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 5.6 scrub ok 2024-09-16T10:19:18.424 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: pgmap v274: 158 pgs: 2 peering, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 337/154305 objects misplaced (0.218%); 39 MiB/s, 621 objects/s recovering 2024-09-16T10:19:18.424 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 3.10 scrub starts 2024-09-16T10:19:18.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 3.10 scrub ok 2024-09-16T10:19:18.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:19:18.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:19:18.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:19:18.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: osdmap e190: 8 total, 8 up, 7 in 2024-09-16T10:19:18.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 3.e scrub starts 2024-09-16T10:19:18.426 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: 3.e scrub ok 2024-09-16T10:19:18.426 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2776926047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:18.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 37 16 56179 56163 94.857 87 0.0135981 0.0105369 2024-09-16T10:19:18.509 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:18.544 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:18.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 4.c scrub starts 2024-09-16T10:19:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 4.c scrub ok 2024-09-16T10:19:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 3.b scrub starts 2024-09-16T10:19:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 3.b scrub ok 2024-09-16T10:19:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 5.6 scrub starts 2024-09-16T10:19:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 5.6 scrub ok 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: pgmap v274: 158 pgs: 2 peering, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 337/154305 objects misplaced (0.218%); 39 MiB/s, 621 objects/s recovering 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 3.10 scrub starts 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 3.10 scrub ok 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: osdmap e190: 8 total, 8 up, 7 in 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 3.e scrub starts 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: 3.e scrub ok 2024-09-16T10:19:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2776926047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:18.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:19:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:19:18.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:18.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:18.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:18.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:18.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:18.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:18.907 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:18.942 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:18.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:18.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:19.044 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:19.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:19:19.071 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:19.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:19.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:19.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:19.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:19.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:19.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:19.448 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:19.480 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:19.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:19.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 38 16 57657 57641 94.7914 92.375 0.00873982 0.0105425 2024-09-16T10:19:19.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:19.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:19.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:19.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[29993]: 6.11 scrub starts 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[29993]: 6.11 scrub ok 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[29993]: 4.12 scrub starts 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[29993]: 4.12 scrub ok 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[39470]: 6.11 scrub starts 2024-09-16T10:19:19.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[39470]: 6.11 scrub ok 2024-09-16T10:19:19.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:19:19.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:19:19.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:19.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[39470]: 4.12 scrub starts 2024-09-16T10:19:19.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:19 smithi063 ceph-mon[39470]: 4.12 scrub ok 2024-09-16T10:19:19.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:19.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:19 smithi165 ceph-mon[32696]: 6.11 scrub starts 2024-09-16T10:19:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:19 smithi165 ceph-mon[32696]: 6.11 scrub ok 2024-09-16T10:19:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:19 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:19:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:19 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:19:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:19 smithi165 ceph-mon[32696]: 4.12 scrub starts 2024-09-16T10:19:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:19 smithi165 ceph-mon[32696]: 4.12 scrub ok 2024-09-16T10:19:19.855 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:19.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:19.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:19.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:20.023 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:20.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.163 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:20.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:20.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:20.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:20.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:20.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 39 16 59058 59042 94.6059 87.5625 0.00889334 0.0105582 2024-09-16T10:19:20.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:20.560 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:20.662 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:20.699 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[29993]: 5.a scrub starts 2024-09-16T10:19:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[29993]: 5.a scrub ok 2024-09-16T10:19:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[29993]: 6.d scrub starts 2024-09-16T10:19:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[29993]: 6.d scrub ok 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[29993]: pgmap v276: 158 pgs: 158 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 874 B/s rd, 81 MiB/s wr, 1.30k op/s; 24 MiB/s, 387 objects/s recovering 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[39470]: 5.a scrub starts 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[39470]: 5.a scrub ok 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[39470]: 6.d scrub starts 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[39470]: 6.d scrub ok 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[39470]: pgmap v276: 158 pgs: 158 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 874 B/s rd, 81 MiB/s wr, 1.30k op/s; 24 MiB/s, 387 objects/s recovering 2024-09-16T10:19:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:19:20.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:20 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:19:20.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:20.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:20 smithi165 ceph-mon[32696]: 5.a scrub starts 2024-09-16T10:19:20.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:20 smithi165 ceph-mon[32696]: 5.a scrub ok 2024-09-16T10:19:20.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:20 smithi165 ceph-mon[32696]: 6.d scrub starts 2024-09-16T10:19:20.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:20 smithi165 ceph-mon[32696]: 6.d scrub ok 2024-09-16T10:19:20.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:20 smithi165 ceph-mon[32696]: pgmap v276: 158 pgs: 158 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 874 B/s rd, 81 MiB/s wr, 1.30k op/s; 24 MiB/s, 387 objects/s recovering 2024-09-16T10:19:20.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:20 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:19:20.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:20 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:19:20.830 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:20.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:20.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:20.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:21.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:21.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:21.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:21.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:21.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:21.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:21.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:21.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:21.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:21.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:21.495 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:19:21.495040+0000 min lat: 0.00128827 max lat: 2.11649 avg lat: 0.0105945 2024-09-16T10:19:21.496 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:19:21.496 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 40 16 60396 60380 94.3312 83.625 0.00418583 0.0105945 2024-09-16T10:19:21.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:21.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:21.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:21.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:21.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:21.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:21.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:21.869 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:21.895 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:21.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:21.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:22.033 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:22.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:22.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:22.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 6.1d scrub starts 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 6.1d scrub ok 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 3.18 scrub starts 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 3.18 scrub ok 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: osdmap e191: 8 total, 8 up, 7 in 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 3.9 scrub starts 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 3.9 scrub ok 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:19:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 3.a scrub starts 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: 3.a scrub ok 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[29993]: pgmap v278: 158 pgs: 158 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 778 op/s; 3.5 MiB/s, 56 objects/s recovering 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 6.1d scrub starts 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 6.1d scrub ok 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 3.18 scrub starts 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 3.18 scrub ok 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: osdmap e191: 8 total, 8 up, 7 in 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 3.9 scrub starts 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 3.9 scrub ok 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:19:22.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:19:22.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 3.a scrub starts 2024-09-16T10:19:22.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: 3.a scrub ok 2024-09-16T10:19:22.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:21 smithi063 ceph-mon[39470]: pgmap v278: 158 pgs: 158 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 778 op/s; 3.5 MiB/s, 56 objects/s recovering 2024-09-16T10:19:22.278 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 6.1d scrub starts 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 6.1d scrub ok 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 3.18 scrub starts 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 3.18 scrub ok 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: osdmap e191: 8 total, 8 up, 7 in 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 3.9 scrub starts 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 3.9 scrub ok 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:19:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 3.a scrub starts 2024-09-16T10:19:22.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: 3.a scrub ok 2024-09-16T10:19:22.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:21 smithi165 ceph-mon[32696]: pgmap v278: 158 pgs: 158 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 778 op/s; 3.5 MiB/s, 56 objects/s recovering 2024-09-16T10:19:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:22.315 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:22.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:22.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:22.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T10:19:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:22.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:22.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:22.496 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 41 16 61657 61641 93.9513 78.8125 0.00847997 0.0106385 2024-09-16T10:19:22.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:22.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:22.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:22.683 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:22.733 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:22.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:22.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:22.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:22.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:22.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:23.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:23.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:19:23.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:19:23.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[29993]: osdmap e192: 8 total, 8 up, 7 in 2024-09-16T10:19:23.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[29993]: 4.3 scrub starts 2024-09-16T10:19:23.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[29993]: 4.3 scrub ok 2024-09-16T10:19:23.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:19:23.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:19:23.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[39470]: osdmap e192: 8 total, 8 up, 7 in 2024-09-16T10:19:23.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[39470]: 4.3 scrub starts 2024-09-16T10:19:23.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:22 smithi063 ceph-mon[39470]: 4.3 scrub ok 2024-09-16T10:19:23.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:23.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:23.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:23.296 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:23.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:22 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:19:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:22 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:19:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:22 smithi165 ceph-mon[32696]: osdmap e192: 8 total, 8 up, 7 in 2024-09-16T10:19:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:22 smithi165 ceph-mon[32696]: 4.3 scrub starts 2024-09-16T10:19:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:22 smithi165 ceph-mon[32696]: 4.3 scrub ok 2024-09-16T10:19:23.398 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.439 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:23.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:23.496 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 42 16 63156 63140 93.9448 93.6875 0.0089003 0.0106394 2024-09-16T10:19:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:23.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:23.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:23.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:23.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:23.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:23.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:23.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:23.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:23.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:23.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:23.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:24.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:24.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:19:24.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:19:24.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[29993]: 3.1a scrub starts 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[29993]: 3.1a scrub ok 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[29993]: pgmap v280: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 53 MiB/s wr, 851 op/s; 3.5 MiB/s, 56 objects/s recovering 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[39470]: 3.1a scrub starts 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[39470]: 3.1a scrub ok 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:24.089 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:24.090 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:23 smithi063 ceph-mon[39470]: pgmap v280: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 53 MiB/s wr, 851 op/s; 3.5 MiB/s, 56 objects/s recovering 2024-09-16T10:19:24.139 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:24.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:24.167 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:24.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:24.279 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:24.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:23 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:19:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:23 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:19:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:23 smithi165 ceph-mon[32696]: 3.1a scrub starts 2024-09-16T10:19:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:23 smithi165 ceph-mon[32696]: 3.1a scrub ok 2024-09-16T10:19:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:23 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:23 smithi165 ceph-mon[32696]: pgmap v280: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 53 MiB/s wr, 851 op/s; 3.5 MiB/s, 56 objects/s recovering 2024-09-16T10:19:24.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:24.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:24.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:24.496 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 43 15 64667 64652 93.9574 94.5 0.00646012 0.0106375 2024-09-16T10:19:24.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:24.590 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:24.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:24.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:24.728 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:24.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:24.809 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:24.809 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:24.822 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:24.831 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:24.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:24.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:24.953 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:19:24.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:25.016 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:25.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 6.7 scrub starts 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 6.7 scrub ok 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 4.a scrub starts 2024-09-16T10:19:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: 4.a scrub ok 2024-09-16T10:19:25.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3458653841' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:25.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:24 smithi165 ceph-mon[32696]: from='client.25330 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:25.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:25.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:25.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:25.298 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:25.400 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:25.421 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:25.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:19:25.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:19:25.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 6.7 scrub starts 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 6.7 scrub ok 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 4.a scrub starts 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: 4.a scrub ok 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3458653841' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[29993]: from='client.25330 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:19:25.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:19:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:19:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 6.7 scrub starts 2024-09-16T10:19:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 6.7 scrub ok 2024-09-16T10:19:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 4.a scrub starts 2024-09-16T10:19:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: 4.a scrub ok 2024-09-16T10:19:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3458653841' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:24 smithi063 ceph-mon[39470]: from='client.25330 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:25.496 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 44 15 65985 65970 93.6941 82.375 0.0101639 0.0106677 2024-09-16T10:19:25.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:25.560 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:25.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:19:25.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:25.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:25.799 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:25.842 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:25.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:25.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:25.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:26.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:26.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:26.206 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.242 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:26.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: osdmap e193: 8 total, 8 up, 7 in 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:19:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: pgmap v282: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:26.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:25 smithi165 ceph-mon[32696]: osdmap e194: 8 total, 8 up, 7 in 2024-09-16T10:19:26.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: osdmap e193: 8 total, 8 up, 7 in 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:19:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: pgmap v282: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[29993]: osdmap e194: 8 total, 8 up, 7 in 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: osdmap e193: 8 total, 8 up, 7 in 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:19:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:19:26.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: pgmap v282: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:26.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:25 smithi063 ceph-mon[39470]: osdmap e194: 8 total, 8 up, 7 in 2024-09-16T10:19:26.381 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:26.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:26.497 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 45 16 67474 67458 93.6777 93 0.0121925 0.0106692 2024-09-16T10:19:26.523 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:26.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:26.659 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:26.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:26.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:26.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:26.933 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:26.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:27.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:27.080 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:27.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:27.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:27.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:27.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:27.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:27.330 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:19:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:19:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:19:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:27.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:27.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:27.377 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:19:27.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:27.498 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 46 16 68989 68973 93.6965 94.6875 0.0085474 0.0106678 2024-09-16T10:19:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:27.506 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:27.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:27.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:27.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:27.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:27.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:27.781 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:27.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:27.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:27.940 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:27.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:27.956 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:28.026 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:28.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:28.088 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:28.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:28.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:28.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:28.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:28.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:28.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:28.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:28.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:28 smithi063 ceph-mon[29993]: pgmap v284: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:28 smithi063 ceph-mon[39470]: pgmap v284: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:28.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:28.495 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:28.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:28.498 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 47 16 70355 70339 93.5193 85.375 0.0126184 0.0106881 2024-09-16T10:19:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:28 smithi165 ceph-mon[32696]: pgmap v284: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:28.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:19:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:19:28.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:28.634 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:28.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:28.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:28.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:28.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:28.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:19:28.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:28.949 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:28.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:29.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:29.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:29.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:29.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:29.207 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T10:19:29.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:29.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:29.227 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:19:29.227 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:19:29.263 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:19:29.263 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:19:29.300 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:19:29.300 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:19:29.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:29.333 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:19:29.334 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:19:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:29.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:29.373 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:19:29.374 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:19:29.411 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:19:29.411 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:19:29.411 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:29 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1837462539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:19:29.412 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:29 smithi063 ceph-mon[39470]: from='client.25336 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:19:29.412 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:29.412 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:29 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1837462539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:19:29.412 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:29 smithi063 ceph-mon[29993]: from='client.25336 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:19:29.412 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:29.445 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:19:29.445 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:19:29.465 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:29.482 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:19:29.482 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:19:29.488 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:29 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1837462539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:19:29.488 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:29 smithi165 ceph-mon[32696]: from='client.25336 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:19:29.488 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:29.498 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 48 16 71673 71657 93.287 82.375 0.0114562 0.0107146 2024-09-16T10:19:29.505 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:29.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:29.603 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:19:29.604 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:19:29.604 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:19:29.604 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:19:29.604 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:19:29.607 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:29.640 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:29.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:29.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:29.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:29.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:29.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:29.913 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:29.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:30.015 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:30.047 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:30.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:30.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:30.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:30.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:30.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:30.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:30.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:30.400 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:19:30.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:30.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:30.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:30.489 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[39470]: pgmap v285: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-16T10:19:30.489 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2073710446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:30.489 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[39470]: from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:19:30.489 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[39470]: osdmap e195: 8 total, 8 up, 7 in 2024-09-16T10:19:30.489 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[29993]: pgmap v285: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-16T10:19:30.490 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2073710446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:30.490 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[29993]: from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:19:30.490 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:30 smithi063 ceph-mon[29993]: osdmap e195: 8 total, 8 up, 7 in 2024-09-16T10:19:30.498 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 49 16 72968 72952 93.0349 80.9375 0.0058219 0.0107436 2024-09-16T10:19:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:30 smithi165 ceph-mon[32696]: pgmap v285: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-16T10:19:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2073710446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:30 smithi165 ceph-mon[32696]: from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:19:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:30 smithi165 ceph-mon[32696]: osdmap e195: 8 total, 8 up, 7 in 2024-09-16T10:19:30.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:30.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:30.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:30.709 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:30.741 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:30.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:30.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:30.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:30.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:30.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T10:19:30.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:31.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:31.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:31.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:31.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:31.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:31.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:31.258 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:31.298 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:31.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:31.400 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:31.434 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:31.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:31.499 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 50 16 74365 74349 92.9203 87.3125 0.00652426 0.0107568 2024-09-16T10:19:31.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:31.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:31.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:31.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:31.717 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:31.748 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:31.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:31.797 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:31.849 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:31.888 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:31.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:31.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:32.026 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:32.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:32.128 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:32.161 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:32.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:32.263 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:32.298 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:32.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:32.400 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:32.437 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:32.438 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:32.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:32.499 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 51 16 75777 75761 92.8284 88.25 0.0141185 0.0107681 2024-09-16T10:19:32.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:32.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:32 smithi165 ceph-mon[32696]: pgmap v287: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 827 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-16T10:19:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:32 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:19:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:32 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:19:32.584 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:32.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:32.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:32 smithi063 ceph-mon[29993]: pgmap v287: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 827 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-16T10:19:32.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:32 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:19:32.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:32 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:19:32.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:32 smithi063 ceph-mon[39470]: pgmap v287: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 827 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-16T10:19:32.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:32 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:19:32.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:32 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:19:32.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:32.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:32.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:32.827 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:32.869 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:32.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:32.967 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:19:32.967 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:19:32.967 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:19:32.967 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:19:32.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.967 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39764 7.6 7:680231a8:::benchmark_data_smithi063_73086_object39763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:19:32.967 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 263.71687681399999, 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13555234199999999, 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39764 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.968 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.969 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246057+0000", 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3999999999999999e-06 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246080+0000", 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2629999999999998e-05 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246091+0000", 2024-09-16T10:19:32.970 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0978999999999999e-05 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246097+0000", 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6579999999999996e-06 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246213+0000", 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115435 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246353+0000", 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014010199999999999 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246471+0000", 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117877 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:32.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249103+0000", 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026325020000000001 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249476+0000", 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037260100000000002 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381493+0000", 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13201689999999999 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381556+0000", 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3516000000000006e-05 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381608+0000", 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1742e-05 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39772 7.6 7:6310c551:::benchmark_data_smithi063_73086_object39771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 263.71242794599999, 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13122466799999999, 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39772 2024-09-16T10:19:32.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250506+0000", 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.897e-06 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250528+0000", 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1883999999999999e-05 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250537+0000", 2024-09-16T10:19:32.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.285e-06 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250545+0000", 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.08e-06 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250610+0000", 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5287e-05 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250734+0000", 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012402299999999999 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252617+0000", 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018828919999999999 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254471+0000", 2024-09-16T10:19:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001854138 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.255185+0000", 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00071378999999999995 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381651+0000", 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12646639400000001 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381696+0000", 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5364999999999997e-05 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381729+0000", 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2632999999999997e-05 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39783 7.f 7:fb2e33c9:::benchmark_data_smithi063_73086_object39782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 263.70532960999998, 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13116349199999999, 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:32.982 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:32.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:32.983 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:32.983 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:32.983 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39783 2024-09-16T10:19:32.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:32.984 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257604+0000", 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7710000000000001e-06 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257627+0000", 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2589999999999999e-05 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257636+0000", 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.302e-06 2024-09-16T10:19:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257641+0000", 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257700+0000", 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8791e-05 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257856+0000", 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015637100000000001 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.258009+0000", 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015277300000000001 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383671+0000", 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12566208600000001 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383705+0000", 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3278999999999998e-05 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388613+0000", 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049080310000000002 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388687+0000", 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4381000000000005e-05 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388766+0000", 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9198999999999998e-05 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:40097 7.f 7:f223a0ce:::benchmark_data_smithi063_73086_object40096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 263.41762074399998, 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.133293199, 2024-09-16T10:19:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 40097 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545313+0000", 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0470000000000001e-06 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545333+0000", 2024-09-16T10:19:32.991 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0778e-05 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545341+0000", 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0739999999999993e-06 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545348+0000", 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3049999999999998e-06 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.992 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549229+0000", 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038809500000000002 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549377+0000", 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014856200000000001 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549537+0000", 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015973699999999999 2024-09-16T10:19:32.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659510+0000", 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.109972575 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.663519+0000", 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040092649999999997 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678483+0000", 2024-09-16T10:19:32.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0149646 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678559+0000", 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5891999999999998e-05 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678605+0000", 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5414000000000003e-05 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.995 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90038 7.6 7:605b4ba6:::benchmark_data_smithi063_73086_object90037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 223.46489571800001, 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.151200375, 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90038 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.996 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:32.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498038+0000", 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3200000000000001e-06 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498061+0000", 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2646e-05 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498070+0000", 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.217e-06 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.998 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498076+0000", 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1759999999999998e-06 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498121+0000", 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4870000000000002e-05 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498227+0000", 2024-09-16T10:19:32.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106239 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498358+0000", 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013121899999999999 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501482+0000", 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031239000000000002 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.503423+0000", 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019405819999999999 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648604+0000", 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14518165099999999 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648977+0000", 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000372641 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649237+0000", 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025991399999999998 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90048 7.6 7:6327a166:::benchmark_data_smithi063_73086_object90047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 223.45868056500001, 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14568709799999999, 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.002 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90048 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.003 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504254+0000", 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1409999999999999e-06 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504278+0000", 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3711999999999999e-05 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504289+0000", 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1331e-05 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504296+0000", 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4599999999999997e-06 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504413+0000", 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011668299999999999 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504522+0000", 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010877 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504652+0000", 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013002399999999999 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509094+0000", 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044417069999999996 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514152+0000", 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0050583470000000004 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649421+0000", 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13526861600000001 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649711+0000", 2024-09-16T10:19:33.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00029075100000000001 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649939+0000", 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00022755599999999999 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90050 7.6 7:60112b96:::benchmark_data_smithi063_73086_object90049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:19:33.008 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 223.45372606399999, 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14140651900000001, 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90050 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.009 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509208+0000", 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.407e-06 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509230+0000", 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1861e-05 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509237+0000", 2024-09-16T10:19:33.011 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3159999999999999e-06 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509242+0000", 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5820000000000001e-06 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.012 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509424+0000", 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018168599999999999 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509616+0000", 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019154600000000001 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509757+0000", 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000141485 2024-09-16T10:19:33.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514172+0000", 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044148629999999998 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514209+0000", 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6706999999999999e-05 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650127+0000", 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.135917818 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650433+0000", 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030668399999999999 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650613+0000", 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000179564 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.015 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90064 7.6 7:67c07ba3:::benchmark_data_smithi063_73086_object90063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 223.44237671299999, 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.130695903, 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.016 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90064 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:19:33.017 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520558+0000", 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.984e-06 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520581+0000", 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3700999999999999e-05 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.018 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520593+0000", 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.128e-05 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520600+0000", 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520651+0000", 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1133e-05 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.019 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520775+0000", 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012390099999999999 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520940+0000", 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016520200000000001 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523864+0000", 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029241139999999998 2024-09-16T10:19:33.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524288+0000", 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00042372299999999998 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650781+0000", 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12649292200000001 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.651082+0000", 2024-09-16T10:19:33.021 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030106300000000001 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.651252+0000", 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016956999999999999 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90066 7.f 7:fafdc57c:::benchmark_data_smithi063_73086_object90065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:19:33.022 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 223.44108041000001, 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.132960727, 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90066 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521853+0000", 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4419999999999999e-06 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.024 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521876+0000", 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2430999999999999e-05 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521885+0000", 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6989999999999998e-06 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521892+0000", 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7909999999999999e-06 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.025 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521943+0000", 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0321999999999997e-05 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.522046+0000", 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010361600000000001 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.522161+0000", 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000114833 2024-09-16T10:19:33.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.641264+0000", 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11910251299999999 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.641301+0000", 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7638000000000001e-05 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.027 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.652401+0000", 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011099367000000001 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.654749+0000", 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023483110000000001 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.654813+0000", 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3763999999999995e-05 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.028 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90070 7.6 7:6e0aafb6:::benchmark_data_smithi063_73086_object90069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 223.43864761, 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13177412199999999, 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.029 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90070 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524286+0000", 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3489999999999999e-06 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524308+0000", 2024-09-16T10:19:33.031 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.228e-05 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524317+0000", 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5310000000000001e-06 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.032 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524323+0000", 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3010000000000003e-06 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524360+0000", 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6721000000000003e-05 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524466+0000", 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106162 2024-09-16T10:19:33.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640787+0000", 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116321087 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.645632+0000", 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0048448600000000003 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.647100+0000", 2024-09-16T10:19:33.034 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001468254 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.655959+0000", 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088585079999999993 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.656015+0000", 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6378000000000001e-05 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.035 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.656059+0000", 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3690999999999999e-05 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201820 7.9 7:907729a7:::benchmark_data_smithi063_73086_object82541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.350555+0000", 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.612377048, 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.148878706, 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.036 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201820 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350555+0000", 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350555+0000", 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350556+0000", 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0200000000000001e-07 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350559+0000", 2024-09-16T10:19:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9739999999999998e-06 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350565+0000", 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.19e-06 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350570+0000", 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0849999999999996e-06 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.367568+0000", 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016997412999999999 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.367703+0000", 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013530899999999999 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.367862+0000", 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015915300000000001 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.040 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.041 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377527+0000", 2024-09-16T10:19:33.041 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0096649560000000006 2024-09-16T10:19:33.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.041 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.041 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387412+0000", 2024-09-16T10:19:33.042 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.009884716 2024-09-16T10:19:33.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.499312+0000", 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11189982900000001 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.499380+0000", 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8617999999999999e-05 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.043 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.499434+0000", 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3661e-05 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201823 7.9 7:91c75bba:::benchmark_data_smithi063_73086_object82544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.357374+0000", 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.60555802, 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14531649399999999, 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.044 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201823 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357374+0000", 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357374+0000", 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357377+0000", 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2520000000000002e-06 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357382+0000", 2024-09-16T10:19:33.046 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.383e-06 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357393+0000", 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0865e-05 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357400+0000", 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5530000000000004e-06 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.370715+0000", 2024-09-16T10:19:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.01331473 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.370834+0000", 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011859299999999999 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.370930+0000", 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5977000000000006e-05 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.381075+0000", 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010145711999999999 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387417+0000", 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063419729999999999 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.502472+0000", 2024-09-16T10:19:33.049 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11505420800000001 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.502611+0000", 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013925600000000001 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.502691+0000", 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9992000000000003e-05 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201826 7.4 7:2bb12e82:::benchmark_data_smithi063_73086_object82547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.363029+0000", 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.59990283899999, 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13118088, 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.051 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201826 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363029+0000", 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363029+0000", 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.052 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363032+0000", 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6910000000000002e-06 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363038+0000", 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0399999999999998e-06 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363050+0000", 2024-09-16T10:19:33.053 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1824000000000001e-05 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363058+0000", 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.588e-06 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363115+0000", 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6997999999999999e-05 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.054 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363230+0000", 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011523600000000001 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363338+0000", 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108513 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.368289+0000", 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049509360000000004 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.055 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.371441+0000", 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031520110000000001 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.494133+0000", 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.122692074 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.494167+0000", 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3141e-05 2024-09-16T10:19:33.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.494210+0000", 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3828000000000001e-05 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201831 7.9 7:99be6e1c:::benchmark_data_smithi063_73086_object82552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.365317+0000", 2024-09-16T10:19:33.057 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.597614854, 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.141925095, 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201831 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.058 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365317+0000", 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365317+0000", 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365319+0000", 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.215e-06 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.059 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365322+0000", 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7979999999999999e-06 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365329+0000", 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5869999999999998e-06 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.060 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365334+0000", 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3770000000000004e-06 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.373135+0000", 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0078000819999999998 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.373243+0000", 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108107 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.373345+0000", 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102804 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.381109+0000", 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0077639320000000003 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387422+0000", 2024-09-16T10:19:33.062 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063129229999999998 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507118+0000", 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11969548200000001 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507184+0000", 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6002999999999998e-05 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.063 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507243+0000", 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8785e-05 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201833 7.9 7:92434f80:::benchmark_data_smithi063_73086_object82554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.365699+0000", 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 111.597233123, 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14236796500000001, 2024-09-16T10:19:33.064 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201833 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365699+0000", 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365699+0000", 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365700+0000", 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.24e-06 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.066 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365705+0000", 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5689999999999998e-06 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365713+0000", 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2549999999999997e-06 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365720+0000", 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6449999999999999e-06 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.067 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377556+0000", 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011835714000000001 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377656+0000", 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000100579 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377769+0000", 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112946 2024-09-16T10:19:33.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.384425+0000", 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0066563439999999998 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387428+0000", 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003002041 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.069 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507290+0000", 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119862684 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507367+0000", 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7121999999999998e-05 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.508067+0000", 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069982599999999996 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51238 8.e 8:7236be56:::benchmark_data_smithi063_101290_object51237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 19.621296895, 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96343729, 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51238 2024-09-16T10:19:33.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341637+0000", 2024-09-16T10:19:33.072 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.621e-06 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341663+0000", 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6026000000000001e-05 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341674+0000", 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0871e-05 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.073 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341682+0000", 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8390000000000007e-06 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341780+0000", 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8233999999999995e-05 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341791+0000", 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1426999999999999e-05 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.074 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296465+0000", 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95467394000000005 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296604+0000", 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013824400000000001 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296727+0000", 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012366699999999999 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.075 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304672+0000", 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079445339999999996 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304896+0000", 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000224117 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304941+0000", 2024-09-16T10:19:33.076 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5374999999999998e-05 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305028+0000", 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6447999999999996e-05 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305073+0000", 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4947000000000002e-05 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51239 8.e 8:7166e593:::benchmark_data_smithi063_101290_object51238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 19.620878718, 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96520916099999998, 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51239 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.078 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342055+0000", 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.051e-06 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342073+0000", 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8325000000000002e-05 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342083+0000", 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9850000000000001e-06 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342090+0000", 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2119999999999997e-06 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342196+0000", 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010593400000000001 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342244+0000", 2024-09-16T10:19:33.081 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8272000000000001e-05 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299084+0000", 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95683958300000005 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299216+0000", 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131595 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:19:33.082 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299311+0000", 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5445000000000005e-05 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305117+0000", 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058058169999999996 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305157+0000", 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9836999999999998e-05 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307167+0000", 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020101390000000002 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307221+0000", 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4317000000000002e-05 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307263+0000", 2024-09-16T10:19:33.084 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1649000000000001e-05 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51244 8.e 8:7a51d24a:::benchmark_data_smithi063_101290_object51243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 19.617657587, 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96214958799999994, 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.085 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.086 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.086 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.086 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:33.086 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51244 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345277+0000", 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9810000000000002e-06 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345305+0000", 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7858999999999999e-05 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.088 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345320+0000", 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5268e-05 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345328+0000", 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2900000000000002e-06 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345404+0000", 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6241000000000004e-05 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.089 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345415+0000", 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0672e-05 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300950+0000", 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95553457900000005 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301082+0000", 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013229299999999999 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301242+0000", 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015961099999999999 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307271+0000", 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060297889999999998 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307310+0000", 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8288999999999999e-05 2024-09-16T10:19:33.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307349+0000", 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9499999999999998e-05 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307391+0000", 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1779000000000001e-05 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.092 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307424+0000", 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3436999999999999e-05 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51256 8.e 8:745a54ba:::benchmark_data_smithi063_101290_object51255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 19.605819107999999, 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95048443400000004, 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.093 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.094 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:33.094 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:33.094 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51256 2024-09-16T10:19:33.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.094 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.095 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357116+0000", 2024-09-16T10:19:33.096 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8820000000000001e-06 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357147+0000", 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1263999999999997e-05 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357167+0000", 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9164000000000002e-05 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.097 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357176+0000", 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8309999999999996e-06 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357255+0000", 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8213999999999993e-05 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357267+0000", 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2729e-05 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.098 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303081+0000", 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94581367599999999 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303176+0000", 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5471999999999997e-05 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303272+0000", 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5874999999999999e-05 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307320+0000", 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040474079999999997 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307459+0000", 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013938500000000001 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307514+0000", 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4948999999999999e-05 2024-09-16T10:19:33.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307560+0000", 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5803999999999997e-05 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307598+0000", 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7781000000000003e-05 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51260 8.e 8:7df47e01:::benchmark_data_smithi063_101290_object51259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 19.600696670000001, 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95708149499999995, 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51260 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:33.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362238+0000", 2024-09-16T10:19:33.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4140000000000001e-06 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362265+0000", 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7183999999999999e-05 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362279+0000", 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3832000000000001e-05 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.104 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362287+0000", 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6990000000000003e-06 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362347+0000", 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0560000000000003e-05 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362356+0000", 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1990000000000006e-06 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305199+0000", 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94284296700000003 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305311+0000", 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011281900000000001 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305449+0000", 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000137708 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309902+0000", 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044530100000000003 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312979+0000", 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030772859999999998 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317533+0000", 2024-09-16T10:19:33.107 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045533479999999996 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319259+0000", 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017263129999999999 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319317+0000", 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8155999999999997e-05 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.108 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.109 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:33.109 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:33.109 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:19:33.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:33.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:33.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:33.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:33.243 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:19:33.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:33.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:33.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:33.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:33.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:33.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:33 smithi063 ceph-mon[29993]: osdmap e196: 8 total, 8 up, 7 in 2024-09-16T10:19:33.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2502297208' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:33.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:33 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:19:33.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:33 smithi063 ceph-mon[39470]: osdmap e196: 8 total, 8 up, 7 in 2024-09-16T10:19:33.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2502297208' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:33.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:33 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:19:33.496 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:33.504 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 52 16 77063 77047 92.588 80.375 0.0126037 0.0107955 2024-09-16T10:19:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:33.544 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:33.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:33 smithi165 ceph-mon[32696]: osdmap e196: 8 total, 8 up, 7 in 2024-09-16T10:19:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2502297208' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:33 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:19:33.645 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:33.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:33.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:33.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:19:33.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:33.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:33.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:33.923 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:33.962 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:33.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:34.063 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:34.102 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:34.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:34.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:34.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:34.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:34.258 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T10:19:34.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:34.382 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:34.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:34.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[39470]: pgmap v289: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2502297208' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[39470]: osdmap e197: 8 total, 8 up, 7 in 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1056138656' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[39470]: from='client.45269 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[29993]: pgmap v289: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2502297208' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[29993]: osdmap e197: 8 total, 8 up, 7 in 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1056138656' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:19:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:34 smithi063 ceph-mon[29993]: from='client.45269 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:19:34.484 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:34.499 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 53 16 78439 78423 92.4636 86 0.0032118 0.010809 2024-09-16T10:19:34.513 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:34.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:34 smithi165 ceph-mon[32696]: pgmap v289: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-16T10:19:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:34 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2502297208' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:34 smithi165 ceph-mon[32696]: osdmap e197: 8 total, 8 up, 7 in 2024-09-16T10:19:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1056138656' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:19:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:34 smithi165 ceph-mon[32696]: from='client.45269 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:19:34.616 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:34.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:34.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:19:34.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:34.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:34.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:34.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:34.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:34.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:35.030 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:35.067 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:35.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:35.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:35.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:35.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:35.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:35.340 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:35.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:35.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:35.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:35.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:35.500 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 54 16 79853 79837 92.3877 88.375 0.0161422 0.0108193 2024-09-16T10:19:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:35 smithi165 ceph-mon[32696]: from='client.45269 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:19:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:35 smithi165 ceph-mon[32696]: osdmap e198: 8 total, 8 up, 7 in 2024-09-16T10:19:35.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:35.587 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:35 smithi063 ceph-mon[29993]: from='client.45269 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:19:35.587 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:35 smithi063 ceph-mon[29993]: osdmap e198: 8 total, 8 up, 7 in 2024-09-16T10:19:35.588 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:35 smithi063 ceph-mon[39470]: from='client.45269 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:19:35.588 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:35 smithi063 ceph-mon[39470]: osdmap e198: 8 total, 8 up, 7 in 2024-09-16T10:19:35.611 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:35.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:35.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T10:19:35.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:35.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:35.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:35.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:35.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:35.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:35.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:36.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:36.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:36.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:36.143 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:36.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:36.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:36.232 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:36.236 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:36.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:36.313 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:36.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:36.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:36.455 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:36.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:36.500 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 55 16 81128 81112 92.1567 79.6875 0.0123801 0.0108459 2024-09-16T10:19:36.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:36 smithi063 ceph-mon[29993]: pgmap v292: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 862 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:19:36.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:36 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:36.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:36 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:36.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:36 smithi063 ceph-mon[39470]: pgmap v292: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 862 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:19:36.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:36 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:36.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:36 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:36 smithi165 ceph-mon[32696]: pgmap v292: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 862 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:19:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:36 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:36 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:36.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:36.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:36.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:36.696 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:36.741 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:36.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:36.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:36.883 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:36.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:36.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:37.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:37.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:37.147 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:19:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:19:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:37.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:37.265 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.301 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:37.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:37.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:37.431 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:19:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:19:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/961818572' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:37.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:19:37.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:19:37.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:37.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/961818572' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:37.500 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 56 16 82372 82356 91.8993 77.75 0.00737406 0.0108768 2024-09-16T10:19:37.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:37 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:19:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:37 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:19:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:37 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/961818572' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:37.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:37.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:37.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:37.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:37.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:37.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:37.992 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:37.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:38.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:38.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:19:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:38.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:38.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:38.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:38.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:38.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:38.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:38.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:38.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:38.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:38.502 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 57 16 83775 83759 91.8214 87.6875 0.0064857 0.0108853 2024-09-16T10:19:38.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:19:38.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:19:38.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:19:38.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:19:38.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:19:38.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:19:38.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:19:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:19:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:38.529 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:38.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:38.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:38.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:38.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:38 smithi063 ceph-mon[29993]: pgmap v293: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-16T10:19:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:38.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:38 smithi063 ceph-mon[39470]: pgmap v293: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-16T10:19:38.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:38.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:38.775 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:38 smithi165 ceph-mon[32696]: pgmap v293: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-16T10:19:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:38.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:19:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:19:38.806 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:38.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:38.908 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:38.946 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:38.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:39.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:39.083 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:39.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:39.185 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:39.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:39.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:39.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:39.366 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:39.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:39.377 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T10:19:39.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:39.503 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 58 16 85247 85231 91.8242 92 0.0132539 0.0108857 2024-09-16T10:19:39.505 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:39.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:39.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:39.526 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/139634194' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:19:39.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:39.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/139634194' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:19:39.606 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:39.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:39.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:39.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:39.784 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:39.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:39.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/139634194' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:19:39.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:39.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:39.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:40.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:40.056 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:40.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:40.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:40.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:40.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:40.295 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:40.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:40.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:40.448 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T10:19:40.467 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:40.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:40.503 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 59 16 86697 86681 91.8038 90.625 0.00529897 0.0108879 2024-09-16T10:19:40.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:40.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:40.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:40.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:40.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:40 smithi063 ceph-mon[29993]: pgmap v294: 158 pgs: 158 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.95k op/s 2024-09-16T10:19:40.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/139634194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:19:40.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:40 smithi063 ceph-mon[29993]: osdmap e199: 8 total, 8 up, 7 in 2024-09-16T10:19:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:40 smithi063 ceph-mon[39470]: pgmap v294: 158 pgs: 158 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.95k op/s 2024-09-16T10:19:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/139634194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:19:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:40 smithi063 ceph-mon[39470]: osdmap e199: 8 total, 8 up, 7 in 2024-09-16T10:19:40.707 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:40.763 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:40.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:40.774 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:40.774 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:40 smithi165 ceph-mon[32696]: pgmap v294: 158 pgs: 158 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.95k op/s 2024-09-16T10:19:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/139634194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:19:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:40 smithi165 ceph-mon[32696]: osdmap e199: 8 total, 8 up, 7 in 2024-09-16T10:19:40.865 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:40.905 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:40.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:41.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:41.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:41.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:41.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:41.180 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:41.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:41.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:41.316 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:19:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:41.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:41.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:41.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:41.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:41.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:41.503 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:19:41.502256+0000 min lat: 0.00128827 max lat: 2.11649 avg lat: 0.0108993 2024-09-16T10:19:41.503 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:19:41.503 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 60 16 88074 88058 91.7079 86.0625 0.0109843 0.0108993 2024-09-16T10:19:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:41.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:41.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:19:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:41.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:41.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:41.686 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:41.726 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:41.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:41.828 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:41.877 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:41.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:41.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:41 smithi063 ceph-mon[29993]: pgmap v296: 158 pgs: 158 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 648 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:19:41.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:41 smithi063 ceph-mon[39470]: pgmap v296: 158 pgs: 158 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 648 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:19:41.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:42.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:42.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:42.113 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:42.162 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:42.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:42.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:42.301 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:42.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:42.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:41 smithi165 ceph-mon[32696]: pgmap v296: 158 pgs: 158 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 648 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:19:42.338 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:42.404 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:42.440 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:42.459 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:42.464 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:42.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:42.503 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 61 16 89343 89327 91.5043 79.3125 0.00775442 0.0109239 2024-09-16T10:19:42.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:42.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:42.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:42.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:42.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:42.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:42.835 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:42.861 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:19:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:42.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:42.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:42.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:43.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:43.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.157 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:43.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1514191502' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:42 smithi063 ceph-mon[29993]: from='client.45287 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1514191502' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:42 smithi063 ceph-mon[39470]: from='client.45287 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:43.259 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:43.294 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1514191502' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:42 smithi165 ceph-mon[32696]: from='client.45287 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:19:43.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:43.427 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:43.503 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 62 16 90773 90757 91.4698 89.375 0.011013 0.0109279 2024-09-16T10:19:43.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:43.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:43.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:43.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:43.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:43.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:43.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:43.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[29993]: from='client.45287 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:19:43.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[29993]: osdmap e200: 8 total, 8 up, 7 in 2024-09-16T10:19:43.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[29993]: 3.8 scrub starts 2024-09-16T10:19:43.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[29993]: 3.8 scrub ok 2024-09-16T10:19:43.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[29993]: pgmap v298: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 888 op/s 2024-09-16T10:19:43.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:43.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[39470]: from='client.45287 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:19:43.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[39470]: osdmap e200: 8 total, 8 up, 7 in 2024-09-16T10:19:43.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[39470]: 3.8 scrub starts 2024-09-16T10:19:43.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[39470]: 3.8 scrub ok 2024-09-16T10:19:43.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:43 smithi063 ceph-mon[39470]: pgmap v298: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 888 op/s 2024-09-16T10:19:43.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:43.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:43.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:44.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:44.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:44.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:44.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:44.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:44.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:43 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:19:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:43 smithi165 ceph-mon[32696]: from='client.45287 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:19:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:43 smithi165 ceph-mon[32696]: osdmap e200: 8 total, 8 up, 7 in 2024-09-16T10:19:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:43 smithi165 ceph-mon[32696]: 3.8 scrub starts 2024-09-16T10:19:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:43 smithi165 ceph-mon[32696]: 3.8 scrub ok 2024-09-16T10:19:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:43 smithi165 ceph-mon[32696]: pgmap v298: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 888 op/s 2024-09-16T10:19:44.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:44.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:44.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:44.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:44.508 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 63 16 92189 92173 91.4157 88.5 0.0108888 0.0109335 2024-09-16T10:19:44.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:44.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:44.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:44.666 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:44.666 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:44.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:44.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:44.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:44.819 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:44.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:44.857 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T10:19:44.921 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:44.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:19:44.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:19:44.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2179247463' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:19:44.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[29993]: from='client.45290 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:19:44.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:19:44.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:19:44.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2179247463' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:19:44.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:44 smithi063 ceph-mon[39470]: from='client.45290 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:19:44.960 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:44.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:45.062 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:45.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:45.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:45.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:45.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:19:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:19:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:19:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:19:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:19:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:19:45.295 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:19:45.296 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:19:45.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:44 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:19:45.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:44 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:19:45.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:44 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2179247463' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:19:45.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:44 smithi165 ceph-mon[32696]: from='client.45290 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:19:45.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:45.376 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:45.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:45.478 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:45.508 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 64 15 93610 93595 91.3759 88.875 0.0124385 0.0109391 2024-09-16T10:19:45.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:45.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:45.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:45.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:45.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:45.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:45.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:19:45.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:45.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:45.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:45.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:45.938 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:45.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:46.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:46.075 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:46.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 6.2 scrub ok 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: from='client.45290 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: osdmap e201: 8 total, 8 up, 7 in 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/194009824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 4.14 scrub starts 2024-09-16T10:19:46.151 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 4.14 scrub ok 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 3.6 scrub starts 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: 3.6 scrub ok 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[29993]: pgmap v300: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 6.2 scrub ok 2024-09-16T10:19:46.152 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: from='client.45290 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:19:46.153 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: osdmap e201: 8 total, 8 up, 7 in 2024-09-16T10:19:46.153 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/194009824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:46.153 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 4.14 scrub starts 2024-09-16T10:19:46.153 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 4.14 scrub ok 2024-09-16T10:19:46.153 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 3.6 scrub starts 2024-09-16T10:19:46.153 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: 3.6 scrub ok 2024-09-16T10:19:46.153 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:45 smithi063 ceph-mon[39470]: pgmap v300: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:46.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:46.214 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:46.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:19:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:19:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:19:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 6.2 scrub ok 2024-09-16T10:19:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: from='client.45290 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:19:46.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: osdmap e201: 8 total, 8 up, 7 in 2024-09-16T10:19:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/194009824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 4.14 scrub starts 2024-09-16T10:19:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 4.14 scrub ok 2024-09-16T10:19:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 3.6 scrub starts 2024-09-16T10:19:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: 3.6 scrub ok 2024-09-16T10:19:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:45 smithi165 ceph-mon[32696]: pgmap v300: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-16T10:19:46.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:46.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:46.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:46.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:46.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:46.508 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 65 16 95031 95015 91.3354 88.75 0.0125431 0.010944 2024-09-16T10:19:46.597 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:46.620 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:46.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:46.652 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:19:46.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:46.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:46.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:46.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:46.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:46.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[29993]: 5.10 scrub starts 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[29993]: 5.10 scrub ok 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[29993]: 8.1 scrub starts 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[39470]: 5.10 scrub starts 2024-09-16T10:19:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[39470]: 5.10 scrub ok 2024-09-16T10:19:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:19:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:19:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[39470]: 8.1 scrub starts 2024-09-16T10:19:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:19:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:46 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:19:46.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:47.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:47.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:47.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:47.166 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:47.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:47.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:46 smithi165 ceph-mon[32696]: 5.10 scrub starts 2024-09-16T10:19:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:46 smithi165 ceph-mon[32696]: 5.10 scrub ok 2024-09-16T10:19:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:46 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:19:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:46 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:19:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:46 smithi165 ceph-mon[32696]: 8.1 scrub starts 2024-09-16T10:19:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:46 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:19:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:46 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:19:47.310 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:47.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:47.412 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:47.446 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:47.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:47.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:19:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:19:47.508 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 66 16 96240 96224 91.0964 75.5625 0.00658311 0.0109726 2024-09-16T10:19:47.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:47.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:47.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:47.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:47.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:47.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:47.784 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T10:19:47.784 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:19:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:47.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:47.918 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:47.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:48.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:48.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:48.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:19:48.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:48.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:19:48.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:19:48.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:19:48.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:19:48.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[39470]: pgmap v301: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 659 B/s rd, 56 MiB/s wr, 900 op/s 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[39470]: 6.1e scrub starts 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[39470]: 6.1e scrub ok 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[29993]: pgmap v301: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 659 B/s rd, 56 MiB/s wr, 900 op/s 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[29993]: 6.1e scrub starts 2024-09-16T10:19:48.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:47 smithi063 ceph-mon[29993]: 6.1e scrub ok 2024-09-16T10:19:48.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:48.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:48.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:47 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:19:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:47 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:19:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:47 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:19:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:47 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:19:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:47 smithi165 ceph-mon[32696]: pgmap v301: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 659 B/s rd, 56 MiB/s wr, 900 op/s 2024-09-16T10:19:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:47 smithi165 ceph-mon[32696]: 6.1e scrub starts 2024-09-16T10:19:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:47 smithi165 ceph-mon[32696]: 6.1e scrub ok 2024-09-16T10:19:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:48.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:48.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:48.425 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:48.458 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:48.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:48.508 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 67 16 97443 97427 90.8589 75.1875 0.00387688 0.011001 2024-09-16T10:19:48.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:48.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:48.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:48.698 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:48.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:48.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:19:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:19:48.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:48.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:48.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:48.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:48.976 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:49.005 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:49.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:49.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:49.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:19:49.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[29993]: 6.1f scrub starts 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[29993]: 6.1f scrub ok 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[29993]: 8.1 scrub ok 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[39470]: 6.1f scrub starts 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[39470]: 6.1f scrub ok 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:48 smithi063 ceph-mon[39470]: 8.1 scrub ok 2024-09-16T10:19:49.245 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:49.266 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:48 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:19:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:48 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:19:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:48 smithi165 ceph-mon[32696]: 6.1f scrub starts 2024-09-16T10:19:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:48 smithi165 ceph-mon[32696]: 6.1f scrub ok 2024-09-16T10:19:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:48 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:49.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:48 smithi165 ceph-mon[32696]: 8.1 scrub ok 2024-09-16T10:19:49.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:49.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:49.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:49.514 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 68 16 98629 98613 90.6127 74.125 0.00838987 0.0110316 2024-09-16T10:19:49.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:49.641 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:49.679 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:49.781 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.819 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:49.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:49.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:49.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:49.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:50.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:50.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:50.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:50.186 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:49 smithi063 ceph-mon[29993]: 4.1e scrub starts 2024-09-16T10:19:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:49 smithi063 ceph-mon[29993]: 4.1e scrub ok 2024-09-16T10:19:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:49 smithi063 ceph-mon[29993]: pgmap v302: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-16T10:19:50.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:49 smithi063 ceph-mon[39470]: 4.1e scrub starts 2024-09-16T10:19:50.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:49 smithi063 ceph-mon[39470]: 4.1e scrub ok 2024-09-16T10:19:50.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:49 smithi063 ceph-mon[39470]: pgmap v302: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-16T10:19:50.219 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:50.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:49 smithi165 ceph-mon[32696]: 4.1e scrub starts 2024-09-16T10:19:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:49 smithi165 ceph-mon[32696]: 4.1e scrub ok 2024-09-16T10:19:50.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:49 smithi165 ceph-mon[32696]: pgmap v302: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-16T10:19:50.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:50.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:50.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:50.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:50.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:50.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:50.509 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 69 16 100013 99997 90.553 86.5 0.00622697 0.0110387 2024-09-16T10:19:50.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:50.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:50.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:50.636 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:50.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:50.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:50.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:50.868 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:50.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:50.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:50.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:50.910 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:19:51.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:51.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:51.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:51.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:51.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:51.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:51.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[29993]: 6.0 scrub starts 2024-09-16T10:19:51.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[29993]: 6.0 scrub ok 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2575477497' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[39470]: 6.0 scrub starts 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[39470]: 6.0 scrub ok 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2575477497' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:19:51.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:50 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:19:51.294 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:50 smithi165 ceph-mon[32696]: 6.0 scrub starts 2024-09-16T10:19:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:50 smithi165 ceph-mon[32696]: 6.0 scrub ok 2024-09-16T10:19:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:50 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2575477497' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:19:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:50 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:19:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:50 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:19:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:51.335 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:51.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:51.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:51.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:51.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:51.509 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 70 16 101342 101326 90.4459 83.0625 0.0149091 0.0110519 2024-09-16T10:19:51.573 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:51.608 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:51.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:51.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:51.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:51.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:51.848 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:51.881 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:51.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:51.983 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:52.024 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:52.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:52.121 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:52.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:52.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:52.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:52.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:19:52.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2575477497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[29993]: osdmap e202: 8 total, 8 up, 7 in 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[29993]: 5.1b scrub starts 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[29993]: 5.1b scrub ok 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[29993]: pgmap v304: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2575477497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[39470]: osdmap e202: 8 total, 8 up, 7 in 2024-09-16T10:19:52.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[39470]: 5.1b scrub starts 2024-09-16T10:19:52.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[39470]: 5.1b scrub ok 2024-09-16T10:19:52.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:51 smithi063 ceph-mon[39470]: pgmap v304: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-16T10:19:52.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:51 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:19:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:51 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:19:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2575477497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:19:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:51 smithi165 ceph-mon[32696]: osdmap e202: 8 total, 8 up, 7 in 2024-09-16T10:19:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:51 smithi165 ceph-mon[32696]: 5.1b scrub starts 2024-09-16T10:19:52.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:51 smithi165 ceph-mon[32696]: 5.1b scrub ok 2024-09-16T10:19:52.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:51 smithi165 ceph-mon[32696]: pgmap v304: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-16T10:19:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:52.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:52.385 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50949 8.a 8:59c98f0f:::benchmark_data_smithi063_101290_object50948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.116306506000001, 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0382288049999999, 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.386 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50949 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264644+0000", 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.978e-06 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264667+0000", 2024-09-16T10:19:52.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3649999999999999e-05 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264681+0000", 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3423e-05 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264688+0000", 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1010000000000003e-06 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265626+0000", 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00093838999999999999 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265747+0000", 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120319 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.267052+0000", 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013057990000000001 2024-09-16T10:19:52.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.281048+0000", 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013995452 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283470+0000", 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024224540000000001 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283482+0000", 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1581e-05 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302818+0000", 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0193361350000001 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302871+0000", 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2522999999999998e-05 2024-09-16T10:19:52.392 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50954 8.a 8:57f268c8:::benchmark_data_smithi063_101290_object50953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.112946078999997, 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0425692, 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.393 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50954 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268004+0000", 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1050000000000002e-06 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268031+0000", 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6468000000000001e-05 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268041+0000", 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0532e-05 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268048+0000", 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7259999999999996e-06 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.270471+0000", 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024233789999999998 2024-09-16T10:19:52.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275200+0000", 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047289130000000004 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275360+0000", 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000159531 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284217+0000", 2024-09-16T10:19:52.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088568970000000007 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284226+0000", 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3559999999999999e-06 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.303128+0000", 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0189021700000001 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:19:52.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304057+0000", 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00092923599999999995 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310454+0000", 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063961260000000002 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310571+0000", 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117761 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.399 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.400 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50965 8.a 8:5e5d9df4:::benchmark_data_smithi063_101290_object50964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.400 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:19:52.400 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.102486343999999, 2024-09-16T10:19:52.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0323928659999999, 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50965 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278464+0000", 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.968e-06 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278491+0000", 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7061999999999999e-05 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278500+0000", 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4970000000000007e-06 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278507+0000", 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5139999999999998e-06 2024-09-16T10:19:52.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284249+0000", 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0057418360000000002 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284255+0000", 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4529999999999999e-06 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304143+0000", 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.019888039 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304299+0000", 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155651 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304463+0000", 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016370800000000001 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308275+0000", 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038120049999999998 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310595+0000", 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023207620000000001 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310754+0000", 2024-09-16T10:19:52.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015848500000000001 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310795+0000", 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1655e-05 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310855+0000", 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9231000000000003e-05 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.407 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50996 8.a 8:5adb42d1:::benchmark_data_smithi063_101290_object50995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.075856502999997, 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005949921, 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.408 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50996 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305093+0000", 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.779e-06 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305117+0000", 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3370000000000002e-05 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305127+0000", 2024-09-16T10:19:52.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0284e-05 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305134+0000", 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9659999999999999e-06 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305192+0000", 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7587000000000003e-05 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305201+0000", 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2590000000000006e-06 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306679+0000", 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0014777290000001 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306790+0000", 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111675 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306942+0000", 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015151300000000001 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310399+0000", 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034570529999999999 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310904+0000", 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00050482900000000004 2024-09-16T10:19:52.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310943+0000", 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9048000000000003e-05 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310974+0000", 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1164999999999997e-05 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311042+0000", 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7663999999999995e-05 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51001 8.a 8:59406931:::benchmark_data_smithi063_101290_object51000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.071739383999997, 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0021313270000001, 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.415 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51001 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309210+0000", 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2589999999999999e-06 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309227+0000", 2024-09-16T10:19:52.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6765999999999998e-05 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309235+0000", 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1969999999999996e-06 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309240+0000", 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6290000000000001e-06 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309279+0000", 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9199999999999997e-05 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309285+0000", 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8810000000000001e-06 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308323+0000", 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99903792999999996 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308464+0000", 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014168000000000001 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308635+0000", 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000170383 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310405+0000", 2024-09-16T10:19:52.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001770134 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311104+0000", 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069940900000000001 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311178+0000", 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3911000000000004e-05 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311237+0000", 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9113000000000001e-05 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311340+0000", 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102835 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.422 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51015 8.a 8:5f39a03a:::benchmark_data_smithi063_101290_object51014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.055433415000003, 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99112391399999999, 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51015 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.423 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325516+0000", 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-07 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325537+0000", 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1900999999999999e-05 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325554+0000", 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6937999999999999e-05 2024-09-16T10:19:52.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325562+0000", 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3830000000000003e-06 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325628+0000", 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6418e-05 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325640+0000", 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2157000000000001e-05 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311387+0000", 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98574625999999999 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311498+0000", 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111768 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311624+0000", 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012560199999999999 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316394+0000", 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047700279999999999 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316457+0000", 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2819999999999998e-05 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316471+0000", 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4236000000000001e-05 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316522+0000", 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1001000000000002e-05 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:52.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316639+0000", 2024-09-16T10:19:52.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116641 2024-09-16T10:19:52.431 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.431 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51028 8.a 8:504c9504:::benchmark_data_smithi063_101290_object51027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.047051418000002, 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98302855600000005, 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.432 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51028 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333898+0000", 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.375e-06 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333916+0000", 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8236e-05 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333925+0000", 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.585e-06 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333931+0000", 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1070000000000001e-06 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334057+0000", 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012610900000000001 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334083+0000", 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5327e-05 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.312988+0000", 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97890501699999999 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313110+0000", 2024-09-16T10:19:52.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012285099999999999 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313254+0000", 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014338900000000001 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316650+0000", 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033961529999999998 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316706+0000", 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5878000000000003e-05 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316760+0000", 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4502000000000001e-05 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316833+0000", 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2632000000000005e-05 2024-09-16T10:19:52.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316925+0000", 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2394999999999999e-05 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.439 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51060 8.a 8:500fc7dc:::benchmark_data_smithi063_101290_object51059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.024051284000002, 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96027958700000005, 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51060 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.440 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356899+0000", 2024-09-16T10:19:52.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.821e-06 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356917+0000", 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7972999999999999e-05 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356926+0000", 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9679999999999995e-06 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356931+0000", 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356997+0000", 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6079000000000002e-05 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.357005+0000", 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7569999999999993e-06 2024-09-16T10:19:52.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314700+0000", 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95769431000000005 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314827+0000", 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012752799999999999 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314995+0000", 2024-09-16T10:19:52.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016789100000000001 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316982+0000", 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986603 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316999+0000", 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7319000000000001e-05 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317035+0000", 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5655000000000003e-05 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317088+0000", 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3278e-05 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317177+0000", 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8486999999999995e-05 2024-09-16T10:19:52.446 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51073 8.a 8:53b2a9fa:::benchmark_data_smithi063_101290_object51072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.012921720999998, 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.957136019, 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.447 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51073 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368028+0000", 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5939999999999999e-06 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368053+0000", 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5130000000000002e-05 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368066+0000", 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2397e-05 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368074+0000", 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2789999999999998e-06 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368143+0000", 2024-09-16T10:19:52.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9558e-05 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368154+0000", 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0869999999999999e-05 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317228+0000", 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.949073273 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317346+0000", 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011878400000000001 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317477+0000", 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013062600000000001 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322458+0000", 2024-09-16T10:19:52.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049812980000000003 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322513+0000", 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4561999999999997e-05 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325080+0000", 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025674819999999998 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325125+0000", 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5065000000000003e-05 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325163+0000", 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7101000000000003e-05 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51087 8.a 8:51205303:::benchmark_data_smithi063_101290_object51086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.454 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 40.001964188000002, 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94635271099999996, 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51087 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378986+0000", 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9850000000000001e-06 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379013+0000", 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6911e-05 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379029+0000", 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5928999999999999e-05 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379037+0000", 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7100000000000007e-06 2024-09-16T10:19:52.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379115+0000", 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8634999999999995e-05 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379128+0000", 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2903e-05 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319060+0000", 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93993237200000002 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319187+0000", 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012621 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319358+0000", 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017167699999999999 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322553+0000", 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003194625 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325067+0000", 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025139170000000001 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325170+0000", 2024-09-16T10:19:52.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102948 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325265+0000", 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5570999999999997e-05 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325337+0000", 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1317999999999997e-05 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.461 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51105 8.a 8:558fcaa8:::benchmark_data_smithi063_101290_object51104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 39.988071933000001, 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93274633699999998, 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51105 2024-09-16T10:19:52.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392878+0000", 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.516e-06 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392900+0000", 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2575e-05 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392913+0000", 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2196e-05 2024-09-16T10:19:52.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392920+0000", 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7170000000000008e-06 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393546+0000", 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00062592699999999997 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393555+0000", 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7809999999999994e-06 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320780+0000", 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92722495100000002 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320895+0000", 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115375 2024-09-16T10:19:52.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.321051+0000", 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155482 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325345+0000", 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042937840000000001 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325384+0000", 2024-09-16T10:19:52.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9261999999999998e-05 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325522+0000", 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0001383 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325584+0000", 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-05 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325623+0000", 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8940999999999999e-05 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51170 8.a 8:55be5a3b:::benchmark_data_smithi063_101290_object51169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 39.932124051999999, 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87700012299999996, 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.469 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51170 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448826+0000", 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7749999999999999e-06 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448849+0000", 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3298999999999998e-05 2024-09-16T10:19:52.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448867+0000", 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7467999999999999e-05 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448874+0000", 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-06 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448936+0000", 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1383999999999994e-05 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448947+0000", 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1756000000000001e-05 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322695+0000", 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87374773699999997 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322825+0000", 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000129579 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322994+0000", 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016874899999999999 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325349+0000", 2024-09-16T10:19:52.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002355514 2024-09-16T10:19:52.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325668+0000", 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000319034 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325723+0000", 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5376e-05 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325775+0000", 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1924e-05 2024-09-16T10:19:52.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325824+0000", 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8918000000000002e-05 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51179 8.a 8:5e35a76f:::benchmark_data_smithi063_101290_object51178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 39.867828721000002, 2024-09-16T10:19:52.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82446452000000003, 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51179 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:19:52.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513122+0000", 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5579999999999999e-06 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513149+0000", 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7101e-05 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513166+0000", 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7017e-05 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513175+0000", 2024-09-16T10:19:52.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6209999999999993e-06 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513268+0000", 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3287000000000003e-05 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513279+0000", 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0889e-05 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.482 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325865+0000", 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.812586527 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325989+0000", 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123347 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.326103+0000", 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011419199999999999 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331852+0000", 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005749466 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331901+0000", 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8854999999999999e-05 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337506+0000", 2024-09-16T10:19:52.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005604876 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337547+0000", 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0961999999999998e-05 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337584+0000", 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6822000000000001e-05 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.485 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51188 8.a 8:5734c940:::benchmark_data_smithi063_101290_object51187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 39.857812578999997, 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.814521685, 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.486 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51188 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.487 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523137+0000", 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.804e-06 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523159+0000", 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1447e-05 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523171+0000", 2024-09-16T10:19:52.488 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2347000000000001e-05 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523179+0000", 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6310000000000003e-06 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523240+0000", 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1150999999999998e-05 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523248+0000", 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5190000000000003e-06 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.327963+0000", 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.80471533100000003 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328066+0000", 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103287 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328230+0000", 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163521 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331938+0000", 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037081639999999999 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331973+0000", 2024-09-16T10:19:52.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5355999999999998e-05 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337597+0000", 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056237809999999996 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337637+0000", 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.019e-05 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337657+0000", 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0156000000000001e-05 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51286 8.1 8:897e7f6e:::benchmark_data_smithi063_101290_object51285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 38.063921618999998, 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.983161858, 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.493 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51286 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.494 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317028+0000", 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.141e-06 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317054+0000", 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6186e-05 2024-09-16T10:19:52.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317065+0000", 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.08e-05 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317072+0000", 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3520000000000001e-06 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317688+0000", 2024-09-16T10:19:52.496 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061638600000000002 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317698+0000", 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.346e-06 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292346+0000", 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97464779999999995 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292464+0000", 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118582 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292563+0000", 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9047000000000002e-05 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296570+0000", 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040065379999999996 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296637+0000", 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7547999999999998e-05 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300071+0000", 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003433298 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300139+0000", 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8702999999999996e-05 2024-09-16T10:19:52.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300188+0000", 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9131000000000001e-05 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51312 8.1 8:8f097c26:::benchmark_data_smithi063_101290_object51311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 38.044887967999998, 2024-09-16T10:19:52.500 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96891611600000005, 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51312 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336063+0000", 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3949999999999999e-06 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336103+0000", 2024-09-16T10:19:52.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0754000000000003e-05 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336138+0000", 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4662000000000001e-05 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336160+0000", 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1752000000000001e-05 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336277+0000", 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117586 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336288+0000", 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0852999999999999e-05 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294143+0000", 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95785426600000001 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294236+0000", 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3010000000000006e-05 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294323+0000", 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7619000000000003e-05 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298376+0000", 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004053175 2024-09-16T10:19:52.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300258+0000", 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018817300000000001 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304863+0000", 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046049539999999996 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.506 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3490000000000004e-05 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304976+0000", 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9870000000000002e-05 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51316 8.1 8:858e6b14:::benchmark_data_smithi063_101290_object51315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:19:52.507 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 38.043567991000003, 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96772945600000004, 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51316 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.508 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337381+0000", 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.192e-06 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.509 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337441+0000", 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9988000000000002e-05 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337452+0000", 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0857e-05 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337460+0000", 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2649999999999999e-06 2024-09-16T10:19:52.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337520+0000", 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0377999999999997e-05 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337530+0000", 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0416e-05 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295200+0000", 2024-09-16T10:19:52.511 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95766991499999998 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295307+0000", 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010660199999999999 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295446+0000", 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013906699999999999 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300014+0000", 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045680249999999999 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300329+0000", 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031521199999999999 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305018+0000", 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046893569999999999 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.513 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305073+0000", 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4243000000000002e-05 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305110+0000", 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6939e-05 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.514 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51363 8.1 8:83ceb830:::benchmark_data_smithi063_101290_object51362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 37.942041965999998, 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.86632090799999995, 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51363 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.515 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438908+0000", 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.454e-06 2024-09-16T10:19:52.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438931+0000", 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3197000000000002e-05 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438945+0000", 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3703000000000001e-05 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438952+0000", 2024-09-16T10:19:52.517 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5259999999999996e-06 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439023+0000", 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0457000000000007e-05 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439034+0000", 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1141000000000001e-05 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.518 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:52.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:52.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:52.520 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296689+0000", 2024-09-16T10:19:52.520 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.85765570300000005 2024-09-16T10:19:52.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296811+0000", 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121229 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296957+0000", 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014621999999999999 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300288+0000", 2024-09-16T10:19:52.521 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033314 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300372+0000", 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3277999999999998e-05 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305147+0000", 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047752990000000002 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.522 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305194+0000", 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6789999999999998e-05 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305227+0000", 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3510999999999998e-05 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51388 8.1 8:8d4128ae:::benchmark_data_smithi063_101290_object51387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:19:52.523 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 37.917327131999997, 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.84172315399999997, 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51388 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.524 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463623+0000", 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:19:52.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463644+0000", 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0937999999999998e-05 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463654+0000", 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0390999999999999e-05 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.526 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463661+0000", 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2400000000000001e-06 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463714+0000", 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3059999999999997e-05 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463722+0000", 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4769999999999997e-06 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.527 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298409+0000", 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.83468682000000005 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298629+0000", 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220258 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298774+0000", 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145012 2024-09-16T10:19:52.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300340+0000", 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001566352 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302114+0000", 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001773518 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.529 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305261+0000", 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003147721 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305310+0000", 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9051000000000002e-05 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305344+0000", 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3751000000000003e-05 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.530 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51396 8.1 8:8a8d0cb2:::benchmark_data_smithi063_101290_object51395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 37.898535031000002, 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82304446200000003, 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:19:52.531 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51396 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:19:52.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482414+0000", 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2300000000000001e-06 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482433+0000", 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8876e-05 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482445+0000", 2024-09-16T10:19:52.533 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2170999999999999e-05 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482453+0000", 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8939999999999993e-06 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482504+0000", 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0698000000000001e-05 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:19:52.534 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482512+0000", 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1319999999999994e-06 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300411+0000", 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.81789900699999996 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300560+0000", 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014893799999999999 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.535 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300718+0000", 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015829700000000001 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302254+0000", 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015359429999999999 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302509+0000", 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025458499999999999 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.536 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305379+0000", 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028699329999999999 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305426+0000", 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7330000000000003e-05 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305458+0000", 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1427999999999998e-05 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.537 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:19:52.538 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:19:52.538 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:19:52.538 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 71 15 102766 102751 90.4263 89.0625 0.0146023 0.011054 2024-09-16T10:19:52.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:52.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:52.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:52.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:52.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:52.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:52.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:52.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:53.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:53.061 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:52 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:19:53.062 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:52 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:19:53.062 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:52 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:19:53.062 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:52 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:19:53.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:53.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:53.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:53.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:53.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:52 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:19:53.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:52 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:19:53.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:53.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:53.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:53.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:53.481 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:53.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:53.509 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 72 16 104185 104169 90.4012 88.625 0.0138648 0.0110569 2024-09-16T10:19:53.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:53.614 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:53.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:53.715 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:53.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:53.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:19:53.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:53.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:53.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:53.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:54.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:54.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:54.129 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:19:54.134 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:54.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:54.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:54.159 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[39470]: pgmap v305: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 858 op/s 2024-09-16T10:19:54.160 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[39470]: 8.5 scrub starts 2024-09-16T10:19:54.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:54.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:54.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[29993]: pgmap v305: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 858 op/s 2024-09-16T10:19:54.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:53 smithi063 ceph-mon[29993]: 8.5 scrub starts 2024-09-16T10:19:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:54.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:53 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:19:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:19:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:53 smithi165 ceph-mon[32696]: pgmap v305: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 858 op/s 2024-09-16T10:19:54.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:53 smithi165 ceph-mon[32696]: 8.5 scrub starts 2024-09-16T10:19:54.318 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:54.363 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:54.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:54.390 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:54.466 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:54.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:54.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:54.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 73 16 105401 105385 90.2031 76 0.0217589 0.0110808 2024-09-16T10:19:54.610 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:54.651 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:54.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:54.688 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:19:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:19:54.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:54.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:54.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:54.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:54.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:54.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:55.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:55.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:55.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:55.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:55.187 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:19:55.188 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:19:55.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:55.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:55.219 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:19:55.219 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:19:55.219 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:54 smithi063 ceph-mon[29993]: 5.15 scrub starts 2024-09-16T10:19:55.220 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:54 smithi063 ceph-mon[29993]: 5.15 scrub ok 2024-09-16T10:19:55.220 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:54 smithi063 ceph-mon[29993]: 8.9 scrub starts 2024-09-16T10:19:55.220 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:54 smithi063 ceph-mon[29993]: 4.2 scrub starts 2024-09-16T10:19:55.220 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:54 smithi063 ceph-mon[29993]: 4.2 scrub ok 2024-09-16T10:19:55.220 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:54 smithi063 ceph-mon[29993]: from='client.15468 v1:172.21.15.63:0/3263772159' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:19:55.220 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[39470]: 5.15 scrub starts 2024-09-16T10:19:55.220 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[39470]: 5.15 scrub ok 2024-09-16T10:19:55.221 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[39470]: 8.9 scrub starts 2024-09-16T10:19:55.221 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[39470]: 4.2 scrub starts 2024-09-16T10:19:55.221 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[39470]: 4.2 scrub ok 2024-09-16T10:19:55.221 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[39470]: from='client.15468 v1:172.21.15.63:0/3263772159' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:19:55.254 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:19:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:19:55.281 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:19:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:19:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:54 smithi165 ceph-mon[32696]: 5.15 scrub starts 2024-09-16T10:19:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:54 smithi165 ceph-mon[32696]: 5.15 scrub ok 2024-09-16T10:19:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:54 smithi165 ceph-mon[32696]: 8.9 scrub starts 2024-09-16T10:19:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:54 smithi165 ceph-mon[32696]: 4.2 scrub starts 2024-09-16T10:19:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:54 smithi165 ceph-mon[32696]: 4.2 scrub ok 2024-09-16T10:19:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:54 smithi165 ceph-mon[32696]: from='client.15468 v1:172.21.15.63:0/3263772159' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:19:55.314 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:19:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:19:55.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:55.351 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:55.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:55.352 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:19:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:19:55.392 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:19:55.393 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:19:55.424 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:19:55.424 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:19:55.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:55.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:55.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:55.509 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 74 16 106606 106590 90.0018 75.3125 0.00597761 0.0111063 2024-09-16T10:19:55.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:55.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:55.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:55.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:55.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:55.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:55.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:55.909 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:55.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:56.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:56.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:56.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:56.151 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:56.174 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:56.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:56.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[29993]: pgmap v306: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-16T10:19:56.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2167885737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[29993]: 8.5 scrub ok 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:55 smithi063 ceph-mon[29993]: 8.a scrub starts 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:56 smithi063 ceph-mon[39470]: pgmap v306: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2167885737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:56 smithi063 ceph-mon[39470]: 8.5 scrub ok 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:56 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:56 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:19:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:56 smithi063 ceph-mon[39470]: 8.a scrub starts 2024-09-16T10:19:56.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:56 smithi165 ceph-mon[32696]: pgmap v306: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-16T10:19:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2167885737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:19:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:56 smithi165 ceph-mon[32696]: 8.5 scrub ok 2024-09-16T10:19:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:56 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:19:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:56 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:19:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:56 smithi165 ceph-mon[32696]: 8.a scrub starts 2024-09-16T10:19:56.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:56.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:56.382 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-16T10:19:53.144792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6401038875,"num_objects":97862,"num_object_clones":0,"num_object_copies":293586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97862,"num_whiteouts":0,"num_read":1091,"num_read_kb":843,"num_write":98099,"num_write_kb":6250477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10188,"num_bytes_recovered":663357700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72808,"ondisk_log_size":72808,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":20289880,"kb_used_data":18865188,"kb_used_omap":10,"kb_used_meta":1424565,"kb_avail":635926184,"statfs":{"total":671965249536,"available":651188412416,"internally_reserved":0,"allocated":19317952512,"data_stored":19315137209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":1458754668},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,8,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":465829888,"num_objects":7108,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7108,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7108,"num_write_kb":454912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.292867"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":328,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859542+0000","last_change":"2024-09-16T10:18:52.830567+0000","last_active":"2024-09-16T10:19:44.859542+0000","last_peered":"2024-09-16T10:19:44.859542+0000","last_clean":"2024-09-16T10:19:44.859542+0000","last_became_active":"2024-09-16T10:18:52.830260+0000","last_became_peered":"2024-09-16T10:18:52.830260+0000","last_unstale":"2024-09-16T10:19:44.859542+0000","last_undegraded":"2024-09-16T10:19:44.859542+0000","last_fullsized":"2024-09-16T10:19:44.859542+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:07.119072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.119072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":369,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918296+0000","last_change":"2024-09-16T10:18:52.851145+0000","last_active":"2024-09-16T10:19:50.918296+0000","last_peered":"2024-09-16T10:19:50.918296+0000","last_clean":"2024-09-16T10:19:50.918296+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:19:50.918296+0000","last_undegraded":"2024-09-16T10:19:50.918296+0000","last_fullsized":"2024-09-16T10:19:50.918296+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:08.743259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.743259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.854197+0000","last_change":"2024-09-16T10:18:40.752790+0000","last_active":"2024-09-16T10:19:44.854197+0000","last_peered":"2024-09-16T10:19:44.854197+0000","last_clean":"2024-09-16T10:19:44.854197+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:19:44.854197+0000","last_undegraded":"2024-09-16T10:19:44.854197+0000","last_fullsized":"2024-09-16T10:19:44.854197+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:40.752739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:40.752739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:49.398509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":303,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923036+0000","last_change":"2024-09-16T10:19:23.784420+0000","last_active":"2024-09-16T10:19:50.923036+0000","last_peered":"2024-09-16T10:19:50.923036+0000","last_clean":"2024-09-16T10:19:50.923036+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:19:50.923036+0000","last_undegraded":"2024-09-16T10:19:50.923036+0000","last_fullsized":"2024-09-16T10:19:50.923036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.784347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.784347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.734381+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870563+0000","last_change":"2024-09-16T10:18:34.944971+0000","last_active":"2024-09-16T10:19:44.870563+0000","last_peered":"2024-09-16T10:19:44.870563+0000","last_clean":"2024-09-16T10:19:44.870563+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:19:44.870563+0000","last_undegraded":"2024-09-16T10:19:44.870563+0000","last_fullsized":"2024-09-16T10:19:44.870563+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:34.944915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.944915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:53.549928+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870012+0000","last_change":"2024-09-16T10:18:52.836944+0000","last_active":"2024-09-16T10:19:44.870012+0000","last_peered":"2024-09-16T10:19:44.870012+0000","last_clean":"2024-09-16T10:19:44.870012+0000","last_became_active":"2024-09-16T10:18:52.836507+0000","last_became_peered":"2024-09-16T10:18:52.836507+0000","last_unstale":"2024-09-16T10:19:44.870012+0000","last_undegraded":"2024-09-16T10:19:44.870012+0000","last_fullsized":"2024-09-16T10:19:44.870012+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:13.018412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.018412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.942085+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":358,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.869195+0000","last_change":"2024-09-16T10:18:52.855408+0000","last_active":"2024-09-16T10:19:44.869195+0000","last_peered":"2024-09-16T10:19:44.869195+0000","last_clean":"2024-09-16T10:19:44.869195+0000","last_became_active":"2024-09-16T10:18:52.835715+0000","last_became_peered":"2024-09-16T10:18:52.835715+0000","last_unstale":"2024-09-16T10:19:44.869195+0000","last_undegraded":"2024-09-16T10:19:44.869195+0000","last_fullsized":"2024-09-16T10:19:44.869195+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:16:31.217192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.217192+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.892839+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918392+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:19:50.918392+0000","last_peered":"2024-09-16T10:19:50.918392+0000","last_clean":"2024-09-16T10:19:50.918392+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:19:50.918392+0000","last_undegraded":"2024-09-16T10:19:50.918392+0000","last_fullsized":"2024-09-16T10:19:50.918392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.862015+0000","last_change":"2024-09-16T10:18:14.132736+0000","last_active":"2024-09-16T10:19:44.862015+0000","last_peered":"2024-09-16T10:19:44.862015+0000","last_clean":"2024-09-16T10:19:44.862015+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:19:44.862015+0000","last_undegraded":"2024-09-16T10:19:44.862015+0000","last_fullsized":"2024-09-16T10:19:44.862015+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:14.132682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:14.132682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":327,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923351+0000","last_change":"2024-09-16T10:16:32.833966+0000","last_active":"2024-09-16T10:19:50.923351+0000","last_peered":"2024-09-16T10:19:50.923351+0000","last_clean":"2024-09-16T10:19:50.923351+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:19:50.923351+0000","last_undegraded":"2024-09-16T10:19:50.923351+0000","last_fullsized":"2024-09-16T10:19:50.923351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:32.833895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.833895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.724646+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.862036+0000","last_change":"2024-09-16T10:18:12.133375+0000","last_active":"2024-09-16T10:19:44.862036+0000","last_peered":"2024-09-16T10:19:44.862036+0000","last_clean":"2024-09-16T10:19:44.862036+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:19:44.862036+0000","last_undegraded":"2024-09-16T10:19:44.862036+0000","last_fullsized":"2024-09-16T10:19:44.862036+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.133300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.133300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":301,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.857627+0000","last_change":"2024-09-16T10:17:12.089641+0000","last_active":"2024-09-16T10:19:44.857627+0000","last_peered":"2024-09-16T10:19:44.857627+0000","last_clean":"2024-09-16T10:19:44.857627+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:19:44.857627+0000","last_undegraded":"2024-09-16T10:19:44.857627+0000","last_fullsized":"2024-09-16T10:19:44.857627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.089577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.089577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.220934+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":298,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861490+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:19:44.861490+0000","last_peered":"2024-09-16T10:19:44.861490+0000","last_clean":"2024-09-16T10:19:44.861490+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:19:44.861490+0000","last_undegraded":"2024-09-16T10:19:44.861490+0000","last_fullsized":"2024-09-16T10:19:44.861490+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.984980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":321,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923560+0000","last_change":"2024-09-16T10:17:16.937447+0000","last_active":"2024-09-16T10:19:50.923560+0000","last_peered":"2024-09-16T10:19:50.923560+0000","last_clean":"2024-09-16T10:19:50.923560+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:19:50.923560+0000","last_undegraded":"2024-09-16T10:19:50.923560+0000","last_fullsized":"2024-09-16T10:19:50.923560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:17:16.937353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.937353+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.713148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917261+0000","last_change":"2024-09-16T10:19:23.390542+0000","last_active":"2024-09-16T10:19:50.917261+0000","last_peered":"2024-09-16T10:19:50.917261+0000","last_clean":"2024-09-16T10:19:50.917261+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:19:50.917261+0000","last_undegraded":"2024-09-16T10:19:50.917261+0000","last_fullsized":"2024-09-16T10:19:50.917261+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:49.356783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:46.109382+0000","last_change":"2024-09-16T10:19:46.109382+0000","last_active":"2024-09-16T10:19:46.109382+0000","last_peered":"2024-09-16T10:19:46.109382+0000","last_clean":"2024-09-16T10:19:46.109382+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:19:46.109382+0000","last_undegraded":"2024-09-16T10:19:46.109382+0000","last_fullsized":"2024-09-16T10:19:46.109382+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:46.109310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.109310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.462251+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.920936+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:19:50.920936+0000","last_peered":"2024-09-16T10:19:50.920936+0000","last_clean":"2024-09-16T10:19:50.920936+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:19:50.920936+0000","last_undegraded":"2024-09-16T10:19:50.920936+0000","last_fullsized":"2024-09-16T10:19:50.920936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.949799+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":305,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859060+0000","last_change":"2024-09-16T10:19:15.182388+0000","last_active":"2024-09-16T10:19:44.859060+0000","last_peered":"2024-09-16T10:19:44.859060+0000","last_clean":"2024-09-16T10:19:44.859060+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:19:44.859060+0000","last_undegraded":"2024-09-16T10:19:44.859060+0000","last_fullsized":"2024-09-16T10:19:44.859060+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.182335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.182335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.600446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":134,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.853175+0000","last_change":"2024-09-16T10:19:14.579717+0000","last_active":"2024-09-16T10:19:44.853175+0000","last_peered":"2024-09-16T10:19:44.853175+0000","last_clean":"2024-09-16T10:19:44.853175+0000","last_became_active":"2024-09-16T10:18:52.841218+0000","last_became_peered":"2024-09-16T10:18:52.841218+0000","last_unstale":"2024-09-16T10:19:44.853175+0000","last_undegraded":"2024-09-16T10:19:44.853175+0000","last_fullsized":"2024-09-16T10:19:44.853175+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:14.868398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.854846+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:19:44.854846+0000","last_peered":"2024-09-16T10:19:44.854846+0000","last_clean":"2024-09-16T10:19:44.854846+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:19:44.854846+0000","last_undegraded":"2024-09-16T10:19:44.854846+0000","last_fullsized":"2024-09-16T10:19:44.854846+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:57.868630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":259,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.110737+0000","last_change":"2024-09-16T10:19:48.110737+0000","last_active":"2024-09-16T10:19:48.110737+0000","last_peered":"2024-09-16T10:19:48.110737+0000","last_clean":"2024-09-16T10:19:48.110737+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:19:48.110737+0000","last_undegraded":"2024-09-16T10:19:48.110737+0000","last_fullsized":"2024-09-16T10:19:48.110737+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:48.110674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.110674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.967276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":314,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.924334+0000","last_change":"2024-09-16T10:19:22.803999+0000","last_active":"2024-09-16T10:19:50.924334+0000","last_peered":"2024-09-16T10:19:50.924334+0000","last_clean":"2024-09-16T10:19:50.924334+0000","last_became_active":"2024-09-16T10:18:52.834723+0000","last_became_peered":"2024-09-16T10:18:52.834723+0000","last_unstale":"2024-09-16T10:19:50.924334+0000","last_undegraded":"2024-09-16T10:19:50.924334+0000","last_fullsized":"2024-09-16T10:19:50.924334+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.204568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":360,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870287+0000","last_change":"2024-09-16T10:19:24.829364+0000","last_active":"2024-09-16T10:19:44.870287+0000","last_peered":"2024-09-16T10:19:44.870287+0000","last_clean":"2024-09-16T10:19:44.870287+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:19:44.870287+0000","last_undegraded":"2024-09-16T10:19:44.870287+0000","last_fullsized":"2024-09-16T10:19:44.870287+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:24.829255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.829255+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.222182+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":322,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918385+0000","last_change":"2024-09-16T10:18:52.839387+0000","last_active":"2024-09-16T10:19:50.918385+0000","last_peered":"2024-09-16T10:19:50.918385+0000","last_clean":"2024-09-16T10:19:50.918385+0000","last_became_active":"2024-09-16T10:18:52.838765+0000","last_became_peered":"2024-09-16T10:18:52.838765+0000","last_unstale":"2024-09-16T10:19:50.918385+0000","last_undegraded":"2024-09-16T10:19:50.918385+0000","last_fullsized":"2024-09-16T10:19:50.918385+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:05.795954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.795954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916891+0000","last_change":"2024-09-16T10:18:52.847158+0000","last_active":"2024-09-16T10:19:50.916891+0000","last_peered":"2024-09-16T10:19:50.916891+0000","last_clean":"2024-09-16T10:19:50.916891+0000","last_became_active":"2024-09-16T10:18:52.838789+0000","last_became_peered":"2024-09-16T10:18:52.838789+0000","last_unstale":"2024-09-16T10:19:50.916891+0000","last_undegraded":"2024-09-16T10:19:50.916891+0000","last_fullsized":"2024-09-16T10:19:50.916891+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.622143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":321,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.854646+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:19:44.854646+0000","last_peered":"2024-09-16T10:19:44.854646+0000","last_clean":"2024-09-16T10:19:44.854646+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:19:44.854646+0000","last_undegraded":"2024-09-16T10:19:44.854646+0000","last_fullsized":"2024-09-16T10:19:44.854646+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.407870+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":325,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.863919+0000","last_change":"2024-09-16T10:17:35.057981+0000","last_active":"2024-09-16T10:19:44.863919+0000","last_peered":"2024-09-16T10:19:44.863919+0000","last_clean":"2024-09-16T10:19:44.863919+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:19:44.863919+0000","last_undegraded":"2024-09-16T10:19:44.863919+0000","last_fullsized":"2024-09-16T10:19:44.863919+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:35.057906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.057906+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:48.180956+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":399,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.863211+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:19:44.863211+0000","last_peered":"2024-09-16T10:19:44.863211+0000","last_clean":"2024-09-16T10:19:44.863211+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:19:44.863211+0000","last_undegraded":"2024-09-16T10:19:44.863211+0000","last_fullsized":"2024-09-16T10:19:44.863211+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.301174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"201'6141","reported_seq":6224,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.350221+0000","last_change":"2024-09-16T10:18:33.825761+0000","last_active":"2024-09-16T10:19:48.350221+0000","last_peered":"2024-09-16T10:19:48.350221+0000","last_clean":"2024-09-16T10:19:48.350221+0000","last_became_active":"2024-09-16T10:18:33.825267+0000","last_became_peered":"2024-09-16T10:18:33.825267+0000","last_unstale":"2024-09-16T10:19:48.350221+0000","last_undegraded":"2024-09-16T10:19:48.350221+0000","last_fullsized":"2024-09-16T10:19:48.350221+0000","mapping_epoch":165,"log_start":"173'1700","ondisk_log_start":"173'1700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:59.402037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402456576,"num_objects":6141,"num_object_clones":0,"num_object_copies":18423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":115,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859747+0000","last_change":"2024-09-16T10:18:52.828048+0000","last_active":"2024-09-16T10:19:44.859747+0000","last_peered":"2024-09-16T10:19:44.859747+0000","last_clean":"2024-09-16T10:19:44.859747+0000","last_became_active":"2024-09-16T10:18:52.825707+0000","last_became_peered":"2024-09-16T10:18:52.825707+0000","last_unstale":"2024-09-16T10:19:44.859747+0000","last_undegraded":"2024-09-16T10:19:44.859747+0000","last_fullsized":"2024-09-16T10:19:44.859747+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.115716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.115716+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.054496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":335,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.855502+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:19:44.855502+0000","last_peered":"2024-09-16T10:19:44.855502+0000","last_clean":"2024-09-16T10:19:44.855502+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:19:44.855502+0000","last_undegraded":"2024-09-16T10:19:44.855502+0000","last_fullsized":"2024-09-16T10:19:44.855502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.449585+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.869793+0000","last_change":"2024-09-16T10:19:43.028508+0000","last_active":"2024-09-16T10:19:44.869793+0000","last_peered":"2024-09-16T10:19:44.869793+0000","last_clean":"2024-09-16T10:19:44.869793+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:19:44.869793+0000","last_undegraded":"2024-09-16T10:19:44.869793+0000","last_fullsized":"2024-09-16T10:19:44.869793+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:43.028436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:43.028436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.274639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":383,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.921060+0000","last_change":"2024-09-16T10:18:36.623362+0000","last_active":"2024-09-16T10:19:50.921060+0000","last_peered":"2024-09-16T10:19:50.921060+0000","last_clean":"2024-09-16T10:19:50.921060+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:19:50.921060+0000","last_undegraded":"2024-09-16T10:19:50.921060+0000","last_fullsized":"2024-09-16T10:19:50.921060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:36.623302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:36.623302+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861440+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:19:44.861440+0000","last_peered":"2024-09-16T10:19:44.861440+0000","last_clean":"2024-09-16T10:19:44.861440+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:19:44.861440+0000","last_undegraded":"2024-09-16T10:19:44.861440+0000","last_fullsized":"2024-09-16T10:19:44.861440+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.040726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"201'6096","reported_seq":6179,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.525758+0000","last_change":"2024-09-16T10:18:33.893627+0000","last_active":"2024-09-16T10:19:48.525758+0000","last_peered":"2024-09-16T10:19:48.525758+0000","last_clean":"2024-09-16T10:19:48.525758+0000","last_became_active":"2024-09-16T10:18:33.893281+0000","last_became_peered":"2024-09-16T10:18:33.893281+0000","last_unstale":"2024-09-16T10:19:48.525758+0000","last_undegraded":"2024-09-16T10:19:48.525758+0000","last_fullsized":"2024-09-16T10:19:48.525758+0000","mapping_epoch":165,"log_start":"173'1400","ondisk_log_start":"173'1400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399507456,"num_objects":6096,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6096,"num_write_kb":390144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":117,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.921284+0000","last_change":"2024-09-16T10:18:52.824806+0000","last_active":"2024-09-16T10:19:50.921284+0000","last_peered":"2024-09-16T10:19:50.921284+0000","last_clean":"2024-09-16T10:19:50.921284+0000","last_became_active":"2024-09-16T10:18:52.824449+0000","last_became_peered":"2024-09-16T10:18:52.824449+0000","last_unstale":"2024-09-16T10:19:50.921284+0000","last_undegraded":"2024-09-16T10:19:50.921284+0000","last_fullsized":"2024-09-16T10:19:50.921284+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.752706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.752706+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":343,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.869195+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:19:44.869195+0000","last_peered":"2024-09-16T10:19:44.869195+0000","last_clean":"2024-09-16T10:19:44.869195+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:19:44.869195+0000","last_undegraded":"2024-09-16T10:19:44.869195+0000","last_fullsized":"2024-09-16T10:19:44.869195+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":271,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.871779+0000","last_change":"2024-09-16T10:18:33.893122+0000","last_active":"2024-09-16T10:19:44.871779+0000","last_peered":"2024-09-16T10:19:44.871779+0000","last_clean":"2024-09-16T10:19:44.871779+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:19:44.871779+0000","last_undegraded":"2024-09-16T10:19:44.871779+0000","last_fullsized":"2024-09-16T10:19:44.871779+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:38.748219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.748219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:48.861249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":52,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859083+0000","last_change":"2024-09-16T10:19:16.147371+0000","last_active":"2024-09-16T10:19:44.859083+0000","last_peered":"2024-09-16T10:19:44.859083+0000","last_clean":"2024-09-16T10:19:44.859083+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:19:44.859083+0000","last_undegraded":"2024-09-16T10:19:44.859083+0000","last_fullsized":"2024-09-16T10:19:44.859083+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.147304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.147304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:22.719288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":326,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923274+0000","last_change":"2024-09-16T10:19:44.711933+0000","last_active":"2024-09-16T10:19:50.923274+0000","last_peered":"2024-09-16T10:19:50.923274+0000","last_clean":"2024-09-16T10:19:50.923274+0000","last_became_active":"2024-09-16T10:18:52.835843+0000","last_became_peered":"2024-09-16T10:18:52.835843+0000","last_unstale":"2024-09-16T10:19:50.923274+0000","last_undegraded":"2024-09-16T10:19:50.923274+0000","last_fullsized":"2024-09-16T10:19:50.923274+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.727533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"201'6144","reported_seq":9735,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.170484+0000","last_change":"2024-09-16T10:19:15.299139+0000","last_active":"2024-09-16T10:19:48.170484+0000","last_peered":"2024-09-16T10:19:48.170484+0000","last_clean":"2024-09-16T10:19:48.170484+0000","last_became_active":"2024-09-16T10:19:15.293347+0000","last_became_peered":"2024-09-16T10:19:15.293347+0000","last_unstale":"2024-09-16T10:19:48.170484+0000","last_undegraded":"2024-09-16T10:19:48.170484+0000","last_fullsized":"2024-09-16T10:19:48.170484+0000","mapping_epoch":188,"log_start":"173'1700","ondisk_log_start":"173'1700","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402653184,"num_objects":6144,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1748,"num_bytes_recovered":114556928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":113,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870682+0000","last_change":"2024-09-16T10:18:07.024979+0000","last_active":"2024-09-16T10:19:44.870682+0000","last_peered":"2024-09-16T10:19:44.870682+0000","last_clean":"2024-09-16T10:19:44.870682+0000","last_became_active":"2024-09-16T10:18:06.896380+0000","last_became_peered":"2024-09-16T10:18:06.896380+0000","last_unstale":"2024-09-16T10:19:44.870682+0000","last_undegraded":"2024-09-16T10:19:44.870682+0000","last_fullsized":"2024-09-16T10:19:44.870682+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.024848+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":349,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918599+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:19:50.918599+0000","last_peered":"2024-09-16T10:19:50.918599+0000","last_clean":"2024-09-16T10:19:50.918599+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:19:50.918599+0000","last_undegraded":"2024-09-16T10:19:50.918599+0000","last_fullsized":"2024-09-16T10:19:50.918599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:15.756451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:45.133742+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:19:45.133742+0000","last_peered":"2024-09-16T10:19:45.133742+0000","last_clean":"2024-09-16T10:19:45.133742+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:19:45.133742+0000","last_undegraded":"2024-09-16T10:19:45.133742+0000","last_fullsized":"2024-09-16T10:19:45.133742+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:14.162775+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":393,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.868943+0000","last_change":"2024-09-16T10:18:04.023115+0000","last_active":"2024-09-16T10:19:44.868943+0000","last_peered":"2024-09-16T10:19:44.868943+0000","last_clean":"2024-09-16T10:19:44.868943+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:19:44.868943+0000","last_undegraded":"2024-09-16T10:19:44.868943+0000","last_fullsized":"2024-09-16T10:19:44.868943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:04.022579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.022579+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":303,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917551+0000","last_change":"2024-09-16T10:18:52.824583+0000","last_active":"2024-09-16T10:19:50.917551+0000","last_peered":"2024-09-16T10:19:50.917551+0000","last_clean":"2024-09-16T10:19:50.917551+0000","last_became_active":"2024-09-16T10:18:52.824265+0000","last_became_peered":"2024-09-16T10:18:52.824265+0000","last_unstale":"2024-09-16T10:19:50.917551+0000","last_undegraded":"2024-09-16T10:19:50.917551+0000","last_fullsized":"2024-09-16T10:19:50.917551+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.491962+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"201'6016","reported_seq":9412,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.412876+0000","last_change":"2024-09-16T10:19:14.298863+0000","last_active":"2024-09-16T10:19:48.412876+0000","last_peered":"2024-09-16T10:19:48.412876+0000","last_clean":"2024-09-16T10:19:48.412876+0000","last_became_active":"2024-09-16T10:19:14.293461+0000","last_became_peered":"2024-09-16T10:19:14.293461+0000","last_unstale":"2024-09-16T10:19:48.412876+0000","last_undegraded":"2024-09-16T10:19:48.412876+0000","last_fullsized":"2024-09-16T10:19:48.412876+0000","mapping_epoch":187,"log_start":"173'1400","ondisk_log_start":"173'1400","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:52.985361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394264576,"num_objects":6016,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6016,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":184,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860392+0000","last_change":"2024-09-16T10:18:07.192860+0000","last_active":"2024-09-16T10:19:44.860392+0000","last_peered":"2024-09-16T10:19:44.860392+0000","last_clean":"2024-09-16T10:19:44.860392+0000","last_became_active":"2024-09-16T10:18:06.896951+0000","last_became_peered":"2024-09-16T10:18:06.896951+0000","last_unstale":"2024-09-16T10:19:44.860392+0000","last_undegraded":"2024-09-16T10:19:44.860392+0000","last_fullsized":"2024-09-16T10:19:44.860392+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.192777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.192777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.243306+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":333,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.855621+0000","last_change":"2024-09-16T10:17:44.394734+0000","last_active":"2024-09-16T10:19:44.855621+0000","last_peered":"2024-09-16T10:19:44.855621+0000","last_clean":"2024-09-16T10:19:44.855621+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:19:44.855621+0000","last_undegraded":"2024-09-16T10:19:44.855621+0000","last_fullsized":"2024-09-16T10:19:44.855621+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:17:44.392303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.392303+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:48.633590+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":321,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.862094+0000","last_change":"2024-09-16T10:18:52.836576+0000","last_active":"2024-09-16T10:19:44.862094+0000","last_peered":"2024-09-16T10:19:44.862094+0000","last_clean":"2024-09-16T10:19:44.862094+0000","last_became_active":"2024-09-16T10:18:52.836260+0000","last_became_peered":"2024-09-16T10:18:52.836260+0000","last_unstale":"2024-09-16T10:19:44.862094+0000","last_undegraded":"2024-09-16T10:19:44.862094+0000","last_fullsized":"2024-09-16T10:19:44.862094+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.187181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.187181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.017380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":380,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.858261+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:19:44.858261+0000","last_peered":"2024-09-16T10:19:44.858261+0000","last_clean":"2024-09-16T10:19:44.858261+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:19:44.858261+0000","last_undegraded":"2024-09-16T10:19:44.858261+0000","last_fullsized":"2024-09-16T10:19:44.858261+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.246556+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.857956+0000","last_change":"2024-09-16T10:18:52.844240+0000","last_active":"2024-09-16T10:19:44.857956+0000","last_peered":"2024-09-16T10:19:44.857956+0000","last_clean":"2024-09-16T10:19:44.857956+0000","last_became_active":"2024-09-16T10:18:52.842382+0000","last_became_peered":"2024-09-16T10:18:52.842382+0000","last_unstale":"2024-09-16T10:19:44.857956+0000","last_undegraded":"2024-09-16T10:19:44.857956+0000","last_fullsized":"2024-09-16T10:19:44.857956+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:06.111161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:06.111161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"201'6065","reported_seq":9478,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.441250+0000","last_change":"2024-09-16T10:19:14.298487+0000","last_active":"2024-09-16T10:19:48.441250+0000","last_peered":"2024-09-16T10:19:48.441250+0000","last_clean":"2024-09-16T10:19:48.441250+0000","last_became_active":"2024-09-16T10:19:14.292726+0000","last_became_peered":"2024-09-16T10:19:14.292726+0000","last_unstale":"2024-09-16T10:19:48.441250+0000","last_undegraded":"2024-09-16T10:19:48.441250+0000","last_fullsized":"2024-09-16T10:19:48.441250+0000","mapping_epoch":187,"log_start":"176'1800","ondisk_log_start":"176'1800","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:51.610313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397475840,"num_objects":6065,"num_object_clones":0,"num_object_copies":18195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":388160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":108724224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":184,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860727+0000","last_change":"2024-09-16T10:18:08.171074+0000","last_active":"2024-09-16T10:19:44.860727+0000","last_peered":"2024-09-16T10:19:44.860727+0000","last_clean":"2024-09-16T10:19:44.860727+0000","last_became_active":"2024-09-16T10:18:06.910799+0000","last_became_peered":"2024-09-16T10:18:06.910799+0000","last_unstale":"2024-09-16T10:19:44.860727+0000","last_undegraded":"2024-09-16T10:19:44.860727+0000","last_fullsized":"2024-09-16T10:19:44.860727+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.171021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.171021+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.281938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":353,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.858822+0000","last_change":"2024-09-16T10:19:22.293172+0000","last_active":"2024-09-16T10:19:44.858822+0000","last_peered":"2024-09-16T10:19:44.858822+0000","last_clean":"2024-09-16T10:19:44.858822+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:19:44.858822+0000","last_undegraded":"2024-09-16T10:19:44.858822+0000","last_fullsized":"2024-09-16T10:19:44.858822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:22.293076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.293076+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.645714+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":321,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.853489+0000","last_change":"2024-09-16T10:18:52.833335+0000","last_active":"2024-09-16T10:19:44.853489+0000","last_peered":"2024-09-16T10:19:44.853489+0000","last_clean":"2024-09-16T10:19:44.853489+0000","last_became_active":"2024-09-16T10:18:52.832749+0000","last_became_peered":"2024-09-16T10:18:52.832749+0000","last_unstale":"2024-09-16T10:19:44.853489+0000","last_undegraded":"2024-09-16T10:19:44.853489+0000","last_fullsized":"2024-09-16T10:19:44.853489+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:33.949798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.949798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.578873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":305,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860040+0000","last_change":"2024-09-16T10:18:34.084469+0000","last_active":"2024-09-16T10:19:44.860040+0000","last_peered":"2024-09-16T10:19:44.860040+0000","last_clean":"2024-09-16T10:19:44.860040+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:19:44.860040+0000","last_undegraded":"2024-09-16T10:19:44.860040+0000","last_fullsized":"2024-09-16T10:19:44.860040+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:34.084423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.084423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:54.128208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.854111+0000","last_change":"2024-09-16T10:18:52.827865+0000","last_active":"2024-09-16T10:19:44.854111+0000","last_peered":"2024-09-16T10:19:44.854111+0000","last_clean":"2024-09-16T10:19:44.854111+0000","last_became_active":"2024-09-16T10:18:52.827511+0000","last_became_peered":"2024-09-16T10:18:52.827511+0000","last_unstale":"2024-09-16T10:19:44.854111+0000","last_undegraded":"2024-09-16T10:19:44.854111+0000","last_fullsized":"2024-09-16T10:19:44.854111+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:28.543646+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"201'6272","reported_seq":6355,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.159820+0000","last_change":"2024-09-16T10:18:33.893596+0000","last_active":"2024-09-16T10:19:48.159820+0000","last_peered":"2024-09-16T10:19:48.159820+0000","last_clean":"2024-09-16T10:19:48.159820+0000","last_became_active":"2024-09-16T10:18:33.893254+0000","last_became_peered":"2024-09-16T10:18:33.893254+0000","last_unstale":"2024-09-16T10:19:48.159820+0000","last_undegraded":"2024-09-16T10:19:48.159820+0000","last_fullsized":"2024-09-16T10:19:48.159820+0000","mapping_epoch":165,"log_start":"176'1800","ondisk_log_start":"176'1800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411041792,"num_objects":6272,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":115,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859374+0000","last_change":"2024-09-16T10:18:52.831104+0000","last_active":"2024-09-16T10:19:44.859374+0000","last_peered":"2024-09-16T10:19:44.859374+0000","last_clean":"2024-09-16T10:19:44.859374+0000","last_became_active":"2024-09-16T10:18:52.830618+0000","last_became_peered":"2024-09-16T10:18:52.830618+0000","last_unstale":"2024-09-16T10:19:44.859374+0000","last_undegraded":"2024-09-16T10:19:44.859374+0000","last_fullsized":"2024-09-16T10:19:44.859374+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.131122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.131122+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:41.045857+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"175'29","reported_seq":478,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.853369+0000","last_change":"2024-09-16T10:18:52.860272+0000","last_active":"2024-09-16T10:19:44.853369+0000","last_peered":"2024-09-16T10:19:44.853369+0000","last_clean":"2024-09-16T10:19:44.853369+0000","last_became_active":"2024-09-16T10:18:52.841202+0000","last_became_peered":"2024-09-16T10:18:52.841202+0000","last_unstale":"2024-09-16T10:19:44.853369+0000","last_undegraded":"2024-09-16T10:19:44.853369+0000","last_fullsized":"2024-09-16T10:19:44.853369+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:18:45.654714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.654714+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:50.754866+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.925181+0000","last_change":"2024-09-16T10:17:35.928758+0000","last_active":"2024-09-16T10:19:50.925181+0000","last_peered":"2024-09-16T10:19:50.925181+0000","last_clean":"2024-09-16T10:19:50.925181+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:19:50.925181+0000","last_undegraded":"2024-09-16T10:19:50.925181+0000","last_fullsized":"2024-09-16T10:19:50.925181+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.928701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.928701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:41.782908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":325,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861865+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:19:44.861865+0000","last_peered":"2024-09-16T10:19:44.861865+0000","last_clean":"2024-09-16T10:19:44.861865+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:19:44.861865+0000","last_undegraded":"2024-09-16T10:19:44.861865+0000","last_fullsized":"2024-09-16T10:19:44.861865+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.056983+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.858622+0000","last_change":"2024-09-16T10:16:04.405318+0000","last_active":"2024-09-16T10:19:44.858622+0000","last_peered":"2024-09-16T10:19:44.858622+0000","last_clean":"2024-09-16T10:19:44.858622+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:19:44.858622+0000","last_undegraded":"2024-09-16T10:19:44.858622+0000","last_fullsized":"2024-09-16T10:19:44.858622+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:04.405243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.405243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.279683+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.869736+0000","last_change":"2024-09-16T10:18:41.823802+0000","last_active":"2024-09-16T10:19:44.869736+0000","last_peered":"2024-09-16T10:19:44.869736+0000","last_clean":"2024-09-16T10:19:44.869736+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:19:44.869736+0000","last_undegraded":"2024-09-16T10:19:44.869736+0000","last_fullsized":"2024-09-16T10:19:44.869736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:41.823721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:41.823721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:10.790493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"201'6177","reported_seq":6260,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.493411+0000","last_change":"2024-09-16T10:18:33.821147+0000","last_active":"2024-09-16T10:19:48.493411+0000","last_peered":"2024-09-16T10:19:48.493411+0000","last_clean":"2024-09-16T10:19:48.493411+0000","last_became_active":"2024-09-16T10:18:33.820857+0000","last_became_peered":"2024-09-16T10:18:33.820857+0000","last_unstale":"2024-09-16T10:19:48.493411+0000","last_undegraded":"2024-09-16T10:19:48.493411+0000","last_fullsized":"2024-09-16T10:19:48.493411+0000","mapping_epoch":165,"log_start":"177'1900","ondisk_log_start":"177'1900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:51.120957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404815872,"num_objects":6177,"num_object_clones":0,"num_object_copies":18531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":395328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":385,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:51.588520+0000","last_change":"2024-09-16T10:19:51.588520+0000","last_active":"2024-09-16T10:19:51.588520+0000","last_peered":"2024-09-16T10:19:51.588520+0000","last_clean":"2024-09-16T10:19:51.588520+0000","last_became_active":"2024-09-16T10:18:06.878175+0000","last_became_peered":"2024-09-16T10:18:06.878175+0000","last_unstale":"2024-09-16T10:19:51.588520+0000","last_undegraded":"2024-09-16T10:19:51.588520+0000","last_fullsized":"2024-09-16T10:19:51.588520+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:51.588464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.588464+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.008418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":329,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.855598+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:19:44.855598+0000","last_peered":"2024-09-16T10:19:44.855598+0000","last_clean":"2024-09-16T10:19:44.855598+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:19:44.855598+0000","last_undegraded":"2024-09-16T10:19:44.855598+0000","last_fullsized":"2024-09-16T10:19:44.855598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.443985+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.853332+0000","last_change":"2024-09-16T10:19:16.602946+0000","last_active":"2024-09-16T10:19:44.853332+0000","last_peered":"2024-09-16T10:19:44.853332+0000","last_clean":"2024-09-16T10:19:44.853332+0000","last_became_active":"2024-09-16T10:18:52.833059+0000","last_became_peered":"2024-09-16T10:18:52.833059+0000","last_unstale":"2024-09-16T10:19:44.853332+0000","last_undegraded":"2024-09-16T10:19:44.853332+0000","last_fullsized":"2024-09-16T10:19:44.853332+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.602895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.602895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:21.600988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":81,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.869310+0000","last_change":"2024-09-16T10:18:52.850312+0000","last_active":"2024-09-16T10:19:44.869310+0000","last_peered":"2024-09-16T10:19:44.869310+0000","last_clean":"2024-09-16T10:19:44.869310+0000","last_became_active":"2024-09-16T10:18:52.836346+0000","last_became_peered":"2024-09-16T10:18:52.836346+0000","last_unstale":"2024-09-16T10:19:44.869310+0000","last_undegraded":"2024-09-16T10:19:44.869310+0000","last_fullsized":"2024-09-16T10:19:44.869310+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:17:47.455142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:47.455142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":73,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859545+0000","last_change":"2024-09-16T10:19:16.940617+0000","last_active":"2024-09-16T10:19:44.859545+0000","last_peered":"2024-09-16T10:19:44.859545+0000","last_clean":"2024-09-16T10:19:44.859545+0000","last_became_active":"2024-09-16T10:18:52.827678+0000","last_became_peered":"2024-09-16T10:18:52.827678+0000","last_unstale":"2024-09-16T10:19:44.859545+0000","last_undegraded":"2024-09-16T10:19:44.859545+0000","last_fullsized":"2024-09-16T10:19:44.859545+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.940551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.940551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.979377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917207+0000","last_change":"2024-09-16T10:18:37.611634+0000","last_active":"2024-09-16T10:19:50.917207+0000","last_peered":"2024-09-16T10:19:50.917207+0000","last_clean":"2024-09-16T10:19:50.917207+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:19:50.917207+0000","last_undegraded":"2024-09-16T10:19:50.917207+0000","last_fullsized":"2024-09-16T10:19:50.917207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.611578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.611578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:01.244128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"201'6088","reported_seq":9645,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.317304+0000","last_change":"2024-09-16T10:19:13.324770+0000","last_active":"2024-09-16T10:19:48.317304+0000","last_peered":"2024-09-16T10:19:48.317304+0000","last_clean":"2024-09-16T10:19:48.317304+0000","last_became_active":"2024-09-16T10:19:13.300503+0000","last_became_peered":"2024-09-16T10:19:13.300503+0000","last_unstale":"2024-09-16T10:19:48.317304+0000","last_undegraded":"2024-09-16T10:19:48.317304+0000","last_fullsized":"2024-09-16T10:19:48.317304+0000","mapping_epoch":186,"log_start":"173'1700","ondisk_log_start":"173'1700","created":165,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398983168,"num_objects":6088,"num_object_clones":0,"num_object_copies":18264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":389632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1730,"num_bytes_recovered":113377280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":108,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.855222+0000","last_change":"2024-09-16T10:18:07.544995+0000","last_active":"2024-09-16T10:19:44.855222+0000","last_peered":"2024-09-16T10:19:44.855222+0000","last_clean":"2024-09-16T10:19:44.855222+0000","last_became_active":"2024-09-16T10:18:06.877808+0000","last_became_peered":"2024-09-16T10:18:06.877808+0000","last_unstale":"2024-09-16T10:19:44.855222+0000","last_undegraded":"2024-09-16T10:19:44.855222+0000","last_fullsized":"2024-09-16T10:19:44.855222+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.544697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.544697+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.369068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":364,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.864358+0000","last_change":"2024-09-16T10:19:19.231411+0000","last_active":"2024-09-16T10:19:44.864358+0000","last_peered":"2024-09-16T10:19:44.864358+0000","last_clean":"2024-09-16T10:19:44.864358+0000","last_became_active":"2024-09-16T10:18:52.837634+0000","last_became_peered":"2024-09-16T10:18:52.837634+0000","last_unstale":"2024-09-16T10:19:44.864358+0000","last_undegraded":"2024-09-16T10:19:44.864358+0000","last_fullsized":"2024-09-16T10:19:44.864358+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:19.231125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:19.231125+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.896769+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":329,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861483+0000","last_change":"2024-09-16T10:18:45.207140+0000","last_active":"2024-09-16T10:19:44.861483+0000","last_peered":"2024-09-16T10:19:44.861483+0000","last_clean":"2024-09-16T10:19:44.861483+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:19:44.861483+0000","last_undegraded":"2024-09-16T10:19:44.861483+0000","last_fullsized":"2024-09-16T10:19:44.861483+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:45.207071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.207071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:57.506006+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":698,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:51.491043+0000","last_change":"2024-09-16T10:16:05.559992+0000","last_active":"2024-09-16T10:19:51.491043+0000","last_peered":"2024-09-16T10:19:51.491043+0000","last_clean":"2024-09-16T10:19:51.491043+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:19:51.491043+0000","last_undegraded":"2024-09-16T10:19:51.491043+0000","last_fullsized":"2024-09-16T10:19:51.491043+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:16:05.559913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.559913+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:26.478610+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:47.107063+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:19:47.107063+0000","last_peered":"2024-09-16T10:19:47.107063+0000","last_clean":"2024-09-16T10:19:47.107063+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:19:47.107063+0000","last_undegraded":"2024-09-16T10:19:47.107063+0000","last_fullsized":"2024-09-16T10:19:47.107063+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.497294+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.852977+0000","last_change":"2024-09-16T10:18:44.647162+0000","last_active":"2024-09-16T10:19:44.852977+0000","last_peered":"2024-09-16T10:19:44.852977+0000","last_clean":"2024-09-16T10:19:44.852977+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:19:44.852977+0000","last_undegraded":"2024-09-16T10:19:44.852977+0000","last_fullsized":"2024-09-16T10:19:44.852977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:44.646771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:44.646771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:13.207875+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"201'5908","reported_seq":5991,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.104754+0000","last_change":"2024-09-16T10:18:33.825548+0000","last_active":"2024-09-16T10:19:48.104754+0000","last_peered":"2024-09-16T10:19:48.104754+0000","last_clean":"2024-09-16T10:19:48.104754+0000","last_became_active":"2024-09-16T10:18:33.825120+0000","last_became_peered":"2024-09-16T10:18:33.825120+0000","last_unstale":"2024-09-16T10:19:48.104754+0000","last_undegraded":"2024-09-16T10:19:48.104754+0000","last_fullsized":"2024-09-16T10:19:48.104754+0000","mapping_epoch":165,"log_start":"173'1200","ondisk_log_start":"173'1200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:51.308235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387186688,"num_objects":5908,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5908,"num_write_kb":378112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":398,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860443+0000","last_change":"2024-09-16T10:16:10.182532+0000","last_active":"2024-09-16T10:19:44.860443+0000","last_peered":"2024-09-16T10:19:44.860443+0000","last_clean":"2024-09-16T10:19:44.860443+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:19:44.860443+0000","last_undegraded":"2024-09-16T10:19:44.860443+0000","last_fullsized":"2024-09-16T10:19:44.860443+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:16:10.182468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:10.182468+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.211143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":287,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870398+0000","last_change":"2024-09-16T10:19:44.015480+0000","last_active":"2024-09-16T10:19:44.870398+0000","last_peered":"2024-09-16T10:19:44.870398+0000","last_clean":"2024-09-16T10:19:44.870398+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:19:44.870398+0000","last_undegraded":"2024-09-16T10:19:44.870398+0000","last_fullsized":"2024-09-16T10:19:44.870398+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.015430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.015430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.463886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":372,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916718+0000","last_change":"2024-09-16T10:18:05.620756+0000","last_active":"2024-09-16T10:19:50.916718+0000","last_peered":"2024-09-16T10:19:50.916718+0000","last_clean":"2024-09-16T10:19:50.916718+0000","last_became_active":"2024-09-16T10:18:04.861294+0000","last_became_peered":"2024-09-16T10:18:04.861294+0000","last_unstale":"2024-09-16T10:19:50.916718+0000","last_undegraded":"2024-09-16T10:19:50.916718+0000","last_fullsized":"2024-09-16T10:19:50.916718+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:05.620685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.620685+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917489+0000","last_change":"2024-09-16T10:18:52.827862+0000","last_active":"2024-09-16T10:19:50.917489+0000","last_peered":"2024-09-16T10:19:50.917489+0000","last_clean":"2024-09-16T10:19:50.917489+0000","last_became_active":"2024-09-16T10:18:52.827467+0000","last_became_peered":"2024-09-16T10:18:52.827467+0000","last_unstale":"2024-09-16T10:19:50.917489+0000","last_undegraded":"2024-09-16T10:19:50.917489+0000","last_fullsized":"2024-09-16T10:19:50.917489+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.575356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860095+0000","last_change":"2024-09-16T10:19:23.923755+0000","last_active":"2024-09-16T10:19:44.860095+0000","last_peered":"2024-09-16T10:19:44.860095+0000","last_clean":"2024-09-16T10:19:44.860095+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:19:44.860095+0000","last_undegraded":"2024-09-16T10:19:44.860095+0000","last_fullsized":"2024-09-16T10:19:44.860095+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.923690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.923690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.831083+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"202'6419","reported_seq":6504,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:53.102586+0000","last_change":"2024-09-16T10:18:33.825497+0000","last_active":"2024-09-16T10:19:53.102586+0000","last_peered":"2024-09-16T10:19:53.102586+0000","last_clean":"2024-09-16T10:19:53.102586+0000","last_became_active":"2024-09-16T10:18:33.824979+0000","last_became_peered":"2024-09-16T10:18:33.824979+0000","last_unstale":"2024-09-16T10:19:53.102586+0000","last_undegraded":"2024-09-16T10:19:53.102586+0000","last_fullsized":"2024-09-16T10:19:53.102586+0000","mapping_epoch":165,"log_start":"178'2000","ondisk_log_start":"178'2000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420675584,"num_objects":6419,"num_object_clones":0,"num_object_copies":19257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6419,"num_write_kb":410816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":349,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.921106+0000","last_change":"2024-09-16T10:19:09.316812+0000","last_active":"2024-09-16T10:19:50.921106+0000","last_peered":"2024-09-16T10:19:50.921106+0000","last_clean":"2024-09-16T10:19:50.921106+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:19:50.921106+0000","last_undegraded":"2024-09-16T10:19:50.921106+0000","last_fullsized":"2024-09-16T10:19:50.921106+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:06.903808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.903808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.730406+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":314,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919127+0000","last_change":"2024-09-16T10:19:20.834753+0000","last_active":"2024-09-16T10:19:50.919127+0000","last_peered":"2024-09-16T10:19:50.919127+0000","last_clean":"2024-09-16T10:19:50.919127+0000","last_became_active":"2024-09-16T10:18:52.835626+0000","last_became_peered":"2024-09-16T10:18:52.835626+0000","last_unstale":"2024-09-16T10:19:50.919127+0000","last_undegraded":"2024-09-16T10:19:50.919127+0000","last_fullsized":"2024-09-16T10:19:50.919127+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.834667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.834667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:23.948748+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860011+0000","last_change":"2024-09-16T10:19:18.014961+0000","last_active":"2024-09-16T10:19:44.860011+0000","last_peered":"2024-09-16T10:19:44.860011+0000","last_clean":"2024-09-16T10:19:44.860011+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:19:44.860011+0000","last_undegraded":"2024-09-16T10:19:44.860011+0000","last_fullsized":"2024-09-16T10:19:44.860011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.014882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.014882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.450402+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":259,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870153+0000","last_change":"2024-09-16T10:18:33.824358+0000","last_active":"2024-09-16T10:19:44.870153+0000","last_peered":"2024-09-16T10:19:44.870153+0000","last_clean":"2024-09-16T10:19:44.870153+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:19:44.870153+0000","last_undegraded":"2024-09-16T10:19:44.870153+0000","last_fullsized":"2024-09-16T10:19:44.870153+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:47.013173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:47.013173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"201'6090","reported_seq":9622,"reported_epoch":201,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:19:48.396092+0000","last_change":"2024-09-16T10:19:46.434468+0000","last_active":"2024-09-16T10:19:48.396092+0000","last_peered":"2024-09-16T10:19:48.396092+0000","last_clean":"2024-09-16T10:19:48.396092+0000","last_became_active":"2024-09-16T10:19:15.290152+0000","last_became_peered":"2024-09-16T10:19:15.290152+0000","last_unstale":"2024-09-16T10:19:48.396092+0000","last_undegraded":"2024-09-16T10:19:48.396092+0000","last_fullsized":"2024-09-16T10:19:48.396092+0000","mapping_epoch":188,"log_start":"176'1700","ondisk_log_start":"176'1700","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":6009,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399114240,"num_objects":6090,"num_object_clones":0,"num_object_copies":18270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6090,"num_write_kb":389760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":110428160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":126,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.864173+0000","last_change":"2024-09-16T10:18:52.847306+0000","last_active":"2024-09-16T10:19:44.864173+0000","last_peered":"2024-09-16T10:19:44.864173+0000","last_clean":"2024-09-16T10:19:44.864173+0000","last_became_active":"2024-09-16T10:18:52.842646+0000","last_became_peered":"2024-09-16T10:18:52.842646+0000","last_unstale":"2024-09-16T10:19:44.864173+0000","last_undegraded":"2024-09-16T10:19:44.864173+0000","last_fullsized":"2024-09-16T10:19:44.864173+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:09.128070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.128070+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":356,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918724+0000","last_change":"2024-09-16T10:18:52.845218+0000","last_active":"2024-09-16T10:19:50.918724+0000","last_peered":"2024-09-16T10:19:50.918724+0000","last_clean":"2024-09-16T10:19:50.918724+0000","last_became_active":"2024-09-16T10:18:52.826328+0000","last_became_peered":"2024-09-16T10:18:52.826328+0000","last_unstale":"2024-09-16T10:19:50.918724+0000","last_undegraded":"2024-09-16T10:19:50.918724+0000","last_fullsized":"2024-09-16T10:19:50.918724+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:18:34.648463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.648463+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:02.630392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860407+0000","last_change":"2024-09-16T10:19:21.116145+0000","last_active":"2024-09-16T10:19:44.860407+0000","last_peered":"2024-09-16T10:19:44.860407+0000","last_clean":"2024-09-16T10:19:44.860407+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:19:44.860407+0000","last_undegraded":"2024-09-16T10:19:44.860407+0000","last_fullsized":"2024-09-16T10:19:44.860407+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:21.116098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:21.116098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.939990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":298,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.853089+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:19:44.853089+0000","last_peered":"2024-09-16T10:19:44.853089+0000","last_clean":"2024-09-16T10:19:44.853089+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:19:44.853089+0000","last_undegraded":"2024-09-16T10:19:44.853089+0000","last_fullsized":"2024-09-16T10:19:44.853089+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:26.573061+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":66,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.871990+0000","last_change":"2024-09-16T10:18:52.838759+0000","last_active":"2024-09-16T10:19:44.871990+0000","last_peered":"2024-09-16T10:19:44.871990+0000","last_clean":"2024-09-16T10:19:44.871990+0000","last_became_active":"2024-09-16T10:18:52.838070+0000","last_became_peered":"2024-09-16T10:18:52.838070+0000","last_unstale":"2024-09-16T10:19:44.871990+0000","last_undegraded":"2024-09-16T10:19:44.871990+0000","last_fullsized":"2024-09-16T10:19:44.871990+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:50.491234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.491234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.059410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"201'6248","reported_seq":6330,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.483488+0000","last_change":"2024-09-16T10:18:33.823010+0000","last_active":"2024-09-16T10:19:48.483488+0000","last_peered":"2024-09-16T10:19:48.483488+0000","last_clean":"2024-09-16T10:19:48.483488+0000","last_became_active":"2024-09-16T10:18:33.822701+0000","last_became_peered":"2024-09-16T10:18:33.822701+0000","last_unstale":"2024-09-16T10:19:48.483488+0000","last_undegraded":"2024-09-16T10:19:48.483488+0000","last_fullsized":"2024-09-16T10:19:48.483488+0000","mapping_epoch":165,"log_start":"177'2000","ondisk_log_start":"177'2000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409468928,"num_objects":6248,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6248,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":384,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916833+0000","last_change":"2024-09-16T10:18:52.838356+0000","last_active":"2024-09-16T10:19:50.916833+0000","last_peered":"2024-09-16T10:19:50.916833+0000","last_clean":"2024-09-16T10:19:50.916833+0000","last_became_active":"2024-09-16T10:18:52.838011+0000","last_became_peered":"2024-09-16T10:18:52.838011+0000","last_unstale":"2024-09-16T10:19:50.916833+0000","last_undegraded":"2024-09-16T10:19:50.916833+0000","last_fullsized":"2024-09-16T10:19:50.916833+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.644158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.644158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":318,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918507+0000","last_change":"2024-09-16T10:19:16.768415+0000","last_active":"2024-09-16T10:19:50.918507+0000","last_peered":"2024-09-16T10:19:50.918507+0000","last_clean":"2024-09-16T10:19:50.918507+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:19:50.918507+0000","last_undegraded":"2024-09-16T10:19:50.918507+0000","last_fullsized":"2024-09-16T10:19:50.918507+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:16.768062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.768062+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.827070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.874137+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:19:44.874137+0000","last_peered":"2024-09-16T10:19:44.874137+0000","last_clean":"2024-09-16T10:19:44.874137+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:19:44.874137+0000","last_undegraded":"2024-09-16T10:19:44.874137+0000","last_fullsized":"2024-09-16T10:19:44.874137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.020575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"175'9","reported_seq":118,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917709+0000","last_change":"2024-09-16T10:19:15.342743+0000","last_active":"2024-09-16T10:19:50.917709+0000","last_peered":"2024-09-16T10:19:50.917709+0000","last_clean":"2024-09-16T10:19:50.917709+0000","last_became_active":"2024-09-16T10:18:52.825124+0000","last_became_peered":"2024-09-16T10:18:52.825124+0000","last_unstale":"2024-09-16T10:19:50.917709+0000","last_undegraded":"2024-09-16T10:19:50.917709+0000","last_fullsized":"2024-09-16T10:19:50.917709+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:19:15.342674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.342674+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.191331+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":310,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.920843+0000","last_change":"2024-09-16T10:18:52.834703+0000","last_active":"2024-09-16T10:19:50.920843+0000","last_peered":"2024-09-16T10:19:50.920843+0000","last_clean":"2024-09-16T10:19:50.920843+0000","last_became_active":"2024-09-16T10:18:52.834312+0000","last_became_peered":"2024-09-16T10:18:52.834312+0000","last_unstale":"2024-09-16T10:19:50.920843+0000","last_undegraded":"2024-09-16T10:19:50.920843+0000","last_fullsized":"2024-09-16T10:19:50.920843+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.963621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.963621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.773176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"201'6050","reported_seq":6131,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.382497+0000","last_change":"2024-09-16T10:18:33.823489+0000","last_active":"2024-09-16T10:19:48.382497+0000","last_peered":"2024-09-16T10:19:48.382497+0000","last_clean":"2024-09-16T10:19:48.382497+0000","last_became_active":"2024-09-16T10:18:33.823198+0000","last_became_peered":"2024-09-16T10:18:33.823198+0000","last_unstale":"2024-09-16T10:19:48.382497+0000","last_undegraded":"2024-09-16T10:19:48.382497+0000","last_fullsized":"2024-09-16T10:19:48.382497+0000","mapping_epoch":165,"log_start":"176'1700","ondisk_log_start":"176'1700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396492800,"num_objects":6050,"num_object_clones":0,"num_object_copies":18150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6050,"num_write_kb":387200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":325,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.858717+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:19:44.858717+0000","last_peered":"2024-09-16T10:19:44.858717+0000","last_clean":"2024-09-16T10:19:44.858717+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:19:44.858717+0000","last_undegraded":"2024-09-16T10:19:44.858717+0000","last_fullsized":"2024-09-16T10:19:44.858717+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.191940+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861625+0000","last_change":"2024-09-16T10:18:34.127253+0000","last_active":"2024-09-16T10:19:44.861625+0000","last_peered":"2024-09-16T10:19:44.861625+0000","last_clean":"2024-09-16T10:19:44.861625+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:19:44.861625+0000","last_undegraded":"2024-09-16T10:19:44.861625+0000","last_fullsized":"2024-09-16T10:19:44.861625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:34.127213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.127213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:02.731145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":75,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923115+0000","last_change":"2024-09-16T10:19:18.748438+0000","last_active":"2024-09-16T10:19:50.923115+0000","last_peered":"2024-09-16T10:19:50.923115+0000","last_clean":"2024-09-16T10:19:50.923115+0000","last_became_active":"2024-09-16T10:18:52.835884+0000","last_became_peered":"2024-09-16T10:18:52.835884+0000","last_unstale":"2024-09-16T10:19:50.923115+0000","last_undegraded":"2024-09-16T10:19:50.923115+0000","last_fullsized":"2024-09-16T10:19:50.923115+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.670612+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861430+0000","last_change":"2024-09-16T10:18:52.849022+0000","last_active":"2024-09-16T10:19:44.861430+0000","last_peered":"2024-09-16T10:19:44.861430+0000","last_clean":"2024-09-16T10:19:44.861430+0000","last_became_active":"2024-09-16T10:18:52.839091+0000","last_became_peered":"2024-09-16T10:18:52.839091+0000","last_unstale":"2024-09-16T10:19:44.861430+0000","last_undegraded":"2024-09-16T10:19:44.861430+0000","last_fullsized":"2024-09-16T10:19:44.861430+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"201'6036","reported_seq":6119,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.526193+0000","last_change":"2024-09-16T10:18:33.894422+0000","last_active":"2024-09-16T10:19:48.526193+0000","last_peered":"2024-09-16T10:19:48.526193+0000","last_clean":"2024-09-16T10:19:48.526193+0000","last_became_active":"2024-09-16T10:18:33.893961+0000","last_became_peered":"2024-09-16T10:18:33.893961+0000","last_unstale":"2024-09-16T10:19:48.526193+0000","last_undegraded":"2024-09-16T10:19:48.526193+0000","last_fullsized":"2024-09-16T10:19:48.526193+0000","mapping_epoch":165,"log_start":"173'1400","ondisk_log_start":"173'1400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:53.629687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395575296,"num_objects":6036,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6036,"num_write_kb":386304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":348,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.860132+0000","last_change":"2024-09-16T10:19:24.115166+0000","last_active":"2024-09-16T10:19:44.860132+0000","last_peered":"2024-09-16T10:19:44.860132+0000","last_clean":"2024-09-16T10:19:44.860132+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:19:44.860132+0000","last_undegraded":"2024-09-16T10:19:44.860132+0000","last_fullsized":"2024-09-16T10:19:44.860132+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:24.115075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.115075+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.363776+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917504+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:19:50.917504+0000","last_peered":"2024-09-16T10:19:50.917504+0000","last_clean":"2024-09-16T10:19:50.917504+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:19:50.917504+0000","last_undegraded":"2024-09-16T10:19:50.917504+0000","last_fullsized":"2024-09-16T10:19:50.917504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.428291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":73,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.858110+0000","last_change":"2024-09-16T10:19:17.184744+0000","last_active":"2024-09-16T10:19:44.858110+0000","last_peered":"2024-09-16T10:19:44.858110+0000","last_clean":"2024-09-16T10:19:44.858110+0000","last_became_active":"2024-09-16T10:18:52.831796+0000","last_became_peered":"2024-09-16T10:18:52.831796+0000","last_unstale":"2024-09-16T10:19:44.858110+0000","last_undegraded":"2024-09-16T10:19:44.858110+0000","last_fullsized":"2024-09-16T10:19:44.858110+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.184674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.184674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:20.369045+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":311,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916630+0000","last_change":"2024-09-16T10:18:52.841267+0000","last_active":"2024-09-16T10:19:50.916630+0000","last_peered":"2024-09-16T10:19:50.916630+0000","last_clean":"2024-09-16T10:19:50.916630+0000","last_became_active":"2024-09-16T10:18:52.840958+0000","last_became_peered":"2024-09-16T10:18:52.840958+0000","last_unstale":"2024-09-16T10:19:50.916630+0000","last_undegraded":"2024-09-16T10:19:50.916630+0000","last_fullsized":"2024-09-16T10:19:50.916630+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.495303+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"201'6076","reported_seq":6159,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.127876+0000","last_change":"2024-09-16T10:18:33.894540+0000","last_active":"2024-09-16T10:19:48.127876+0000","last_peered":"2024-09-16T10:19:48.127876+0000","last_clean":"2024-09-16T10:19:48.127876+0000","last_became_active":"2024-09-16T10:18:33.894026+0000","last_became_peered":"2024-09-16T10:18:33.894026+0000","last_unstale":"2024-09-16T10:19:48.127876+0000","last_undegraded":"2024-09-16T10:19:48.127876+0000","last_fullsized":"2024-09-16T10:19:48.127876+0000","mapping_epoch":165,"log_start":"173'1400","ondisk_log_start":"173'1400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:51.015074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398196736,"num_objects":6076,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":334,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918993+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:19:50.918993+0000","last_peered":"2024-09-16T10:19:50.918993+0000","last_clean":"2024-09-16T10:19:50.918993+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:19:50.918993+0000","last_undegraded":"2024-09-16T10:19:50.918993+0000","last_fullsized":"2024-09-16T10:19:50.918993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.799332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917294+0000","last_change":"2024-09-16T10:19:17.357917+0000","last_active":"2024-09-16T10:19:50.917294+0000","last_peered":"2024-09-16T10:19:50.917294+0000","last_clean":"2024-09-16T10:19:50.917294+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:19:50.917294+0000","last_undegraded":"2024-09-16T10:19:50.917294+0000","last_fullsized":"2024-09-16T10:19:50.917294+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.357866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.357866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.135568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":65,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.857605+0000","last_change":"2024-09-16T10:18:52.830267+0000","last_active":"2024-09-16T10:19:44.857605+0000","last_peered":"2024-09-16T10:19:44.857605+0000","last_clean":"2024-09-16T10:19:44.857605+0000","last_became_active":"2024-09-16T10:18:52.829845+0000","last_became_peered":"2024-09-16T10:18:52.829845+0000","last_unstale":"2024-09-16T10:19:44.857605+0000","last_undegraded":"2024-09-16T10:19:44.857605+0000","last_fullsized":"2024-09-16T10:19:44.857605+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:53.182365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870656+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:19:44.870656+0000","last_peered":"2024-09-16T10:19:44.870656+0000","last_clean":"2024-09-16T10:19:44.870656+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:19:44.870656+0000","last_undegraded":"2024-09-16T10:19:44.870656+0000","last_fullsized":"2024-09-16T10:19:44.870656+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"201'5837","reported_seq":9221,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:48.172833+0000","last_change":"2024-09-16T10:19:14.298104+0000","last_active":"2024-09-16T10:19:48.172833+0000","last_peered":"2024-09-16T10:19:48.172833+0000","last_clean":"2024-09-16T10:19:48.172833+0000","last_became_active":"2024-09-16T10:19:14.291692+0000","last_became_peered":"2024-09-16T10:19:14.291692+0000","last_unstale":"2024-09-16T10:19:48.172833+0000","last_undegraded":"2024-09-16T10:19:48.172833+0000","last_fullsized":"2024-09-16T10:19:48.172833+0000","mapping_epoch":187,"log_start":"173'1400","ondisk_log_start":"173'1400","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:54.528429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382533632,"num_objects":5837,"num_object_clones":0,"num_object_copies":17511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":373568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":319,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.864261+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:19:44.864261+0000","last_peered":"2024-09-16T10:19:44.864261+0000","last_clean":"2024-09-16T10:19:44.864261+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:19:44.864261+0000","last_undegraded":"2024-09-16T10:19:44.864261+0000","last_fullsized":"2024-09-16T10:19:44.864261+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":328,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918129+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:19:50.918129+0000","last_peered":"2024-09-16T10:19:50.918129+0000","last_clean":"2024-09-16T10:19:50.918129+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:19:50.918129+0000","last_undegraded":"2024-09-16T10:19:50.918129+0000","last_fullsized":"2024-09-16T10:19:50.918129+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":392,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916775+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:19:50.916775+0000","last_peered":"2024-09-16T10:19:50.916775+0000","last_clean":"2024-09-16T10:19:50.916775+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:19:50.916775+0000","last_undegraded":"2024-09-16T10:19:50.916775+0000","last_fullsized":"2024-09-16T10:19:50.916775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.564166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859968+0000","last_change":"2024-09-16T10:19:20.941840+0000","last_active":"2024-09-16T10:19:44.859968+0000","last_peered":"2024-09-16T10:19:44.859968+0000","last_clean":"2024-09-16T10:19:44.859968+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:19:44.859968+0000","last_undegraded":"2024-09-16T10:19:44.859968+0000","last_fullsized":"2024-09-16T10:19:44.859968+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.941753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.941753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.997567+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859660+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:19:44.859660+0000","last_peered":"2024-09-16T10:19:44.859660+0000","last_clean":"2024-09-16T10:19:44.859660+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:19:44.859660+0000","last_undegraded":"2024-09-16T10:19:44.859660+0000","last_fullsized":"2024-09-16T10:19:44.859660+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.667031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":327,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.871559+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:19:44.871559+0000","last_peered":"2024-09-16T10:19:44.871559+0000","last_clean":"2024-09-16T10:19:44.871559+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:19:44.871559+0000","last_undegraded":"2024-09-16T10:19:44.871559+0000","last_fullsized":"2024-09-16T10:19:44.871559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.155197+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":298,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859634+0000","last_change":"2024-09-16T10:19:15.972005+0000","last_active":"2024-09-16T10:19:44.859634+0000","last_peered":"2024-09-16T10:19:44.859634+0000","last_clean":"2024-09-16T10:19:44.859634+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:19:44.859634+0000","last_undegraded":"2024-09-16T10:19:44.859634+0000","last_fullsized":"2024-09-16T10:19:44.859634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.971951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.971951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:28.073056+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":296,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917113+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:19:50.917113+0000","last_peered":"2024-09-16T10:19:50.917113+0000","last_clean":"2024-09-16T10:19:50.917113+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:19:50.917113+0000","last_undegraded":"2024-09-16T10:19:50.917113+0000","last_fullsized":"2024-09-16T10:19:50.917113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.610870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":333,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.864262+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:19:44.864262+0000","last_peered":"2024-09-16T10:19:44.864262+0000","last_clean":"2024-09-16T10:19:44.864262+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:19:44.864262+0000","last_undegraded":"2024-09-16T10:19:44.864262+0000","last_fullsized":"2024-09-16T10:19:44.864262+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918226+0000","last_change":"2024-09-16T10:19:22.431194+0000","last_active":"2024-09-16T10:19:50.918226+0000","last_peered":"2024-09-16T10:19:50.918226+0000","last_clean":"2024-09-16T10:19:50.918226+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:19:50.918226+0000","last_undegraded":"2024-09-16T10:19:50.918226+0000","last_fullsized":"2024-09-16T10:19:50.918226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.431126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.431126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.492625+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":298,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.871375+0000","last_change":"2024-09-16T10:16:37.237971+0000","last_active":"2024-09-16T10:19:44.871375+0000","last_peered":"2024-09-16T10:19:44.871375+0000","last_clean":"2024-09-16T10:19:44.871375+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:19:44.871375+0000","last_undegraded":"2024-09-16T10:19:44.871375+0000","last_fullsized":"2024-09-16T10:19:44.871375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:37.237849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.237849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.908375+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":68,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918613+0000","last_change":"2024-09-16T10:18:52.823189+0000","last_active":"2024-09-16T10:19:50.918613+0000","last_peered":"2024-09-16T10:19:50.918613+0000","last_clean":"2024-09-16T10:19:50.918613+0000","last_became_active":"2024-09-16T10:18:52.822465+0000","last_became_peered":"2024-09-16T10:18:52.822465+0000","last_unstale":"2024-09-16T10:19:50.918613+0000","last_undegraded":"2024-09-16T10:19:50.918613+0000","last_fullsized":"2024-09-16T10:19:50.918613+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.568361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.568361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:55.849277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":326,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861922+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:19:44.861922+0000","last_peered":"2024-09-16T10:19:44.861922+0000","last_clean":"2024-09-16T10:19:44.861922+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:19:44.861922+0000","last_undegraded":"2024-09-16T10:19:44.861922+0000","last_fullsized":"2024-09-16T10:19:44.861922+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.022112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.858731+0000","last_change":"2024-09-16T10:18:03.120551+0000","last_active":"2024-09-16T10:19:44.858731+0000","last_peered":"2024-09-16T10:19:44.858731+0000","last_clean":"2024-09-16T10:19:44.858731+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:19:44.858731+0000","last_undegraded":"2024-09-16T10:19:44.858731+0000","last_fullsized":"2024-09-16T10:19:44.858731+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:03.120474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:03.120474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"175'9","reported_seq":408,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870366+0000","last_change":"2024-09-16T10:18:52.852723+0000","last_active":"2024-09-16T10:19:44.870366+0000","last_peered":"2024-09-16T10:19:44.870366+0000","last_clean":"2024-09-16T10:19:44.870366+0000","last_became_active":"2024-09-16T10:18:52.845153+0000","last_became_peered":"2024-09-16T10:18:52.845153+0000","last_unstale":"2024-09-16T10:19:44.870366+0000","last_undegraded":"2024-09-16T10:19:44.870366+0000","last_fullsized":"2024-09-16T10:19:44.870366+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:38.926860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:38.926860+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:06.735603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":310,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.921086+0000","last_change":"2024-09-16T10:18:52.821349+0000","last_active":"2024-09-16T10:19:50.921086+0000","last_peered":"2024-09-16T10:19:50.921086+0000","last_clean":"2024-09-16T10:19:50.921086+0000","last_became_active":"2024-09-16T10:18:52.821050+0000","last_became_peered":"2024-09-16T10:18:52.821050+0000","last_unstale":"2024-09-16T10:19:50.921086+0000","last_undegraded":"2024-09-16T10:19:50.921086+0000","last_fullsized":"2024-09-16T10:19:50.921086+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.915613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.915613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.778010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":333,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:45.045388+0000","last_change":"2024-09-16T10:19:45.045388+0000","last_active":"2024-09-16T10:19:45.045388+0000","last_peered":"2024-09-16T10:19:45.045388+0000","last_clean":"2024-09-16T10:19:45.045388+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:19:45.045388+0000","last_undegraded":"2024-09-16T10:19:45.045388+0000","last_fullsized":"2024-09-16T10:19:45.045388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:45.045164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.045164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.833432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":323,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917810+0000","last_change":"2024-09-16T10:18:52.820977+0000","last_active":"2024-09-16T10:19:50.917810+0000","last_peered":"2024-09-16T10:19:50.917810+0000","last_clean":"2024-09-16T10:19:50.917810+0000","last_became_active":"2024-09-16T10:18:52.820693+0000","last_became_peered":"2024-09-16T10:18:52.820693+0000","last_unstale":"2024-09-16T10:19:50.917810+0000","last_undegraded":"2024-09-16T10:19:50.917810+0000","last_fullsized":"2024-09-16T10:19:50.917810+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":389,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.861626+0000","last_change":"2024-09-16T10:17:34.543214+0000","last_active":"2024-09-16T10:19:44.861626+0000","last_peered":"2024-09-16T10:19:44.861626+0000","last_clean":"2024-09-16T10:19:44.861626+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:19:44.861626+0000","last_undegraded":"2024-09-16T10:19:44.861626+0000","last_fullsized":"2024-09-16T10:19:44.861626+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:34.543121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.543121+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:52.000048+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":301,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859416+0000","last_change":"2024-09-16T10:18:09.148508+0000","last_active":"2024-09-16T10:19:44.859416+0000","last_peered":"2024-09-16T10:19:44.859416+0000","last_clean":"2024-09-16T10:19:44.859416+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:19:44.859416+0000","last_undegraded":"2024-09-16T10:19:44.859416+0000","last_fullsized":"2024-09-16T10:19:44.859416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:09.148447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.148447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":322,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923647+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:19:50.923647+0000","last_peered":"2024-09-16T10:19:50.923647+0000","last_clean":"2024-09-16T10:19:50.923647+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:19:50.923647+0000","last_undegraded":"2024-09-16T10:19:50.923647+0000","last_fullsized":"2024-09-16T10:19:50.923647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.805984+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918739+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:19:50.918739+0000","last_peered":"2024-09-16T10:19:50.918739+0000","last_clean":"2024-09-16T10:19:50.918739+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:19:50.918739+0000","last_undegraded":"2024-09-16T10:19:50.918739+0000","last_fullsized":"2024-09-16T10:19:50.918739+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.756954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.855394+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:19:44.855394+0000","last_peered":"2024-09-16T10:19:44.855394+0000","last_clean":"2024-09-16T10:19:44.855394+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:19:44.855394+0000","last_undegraded":"2024-09-16T10:19:44.855394+0000","last_fullsized":"2024-09-16T10:19:44.855394+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:55.416985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.868569+0000","last_change":"2024-09-16T10:19:17.803486+0000","last_active":"2024-09-16T10:19:44.868569+0000","last_peered":"2024-09-16T10:19:44.868569+0000","last_clean":"2024-09-16T10:19:44.868569+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:19:44.868569+0000","last_undegraded":"2024-09-16T10:19:44.868569+0000","last_fullsized":"2024-09-16T10:19:44.868569+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.803412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.803412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.806543+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":343,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.855718+0000","last_change":"2024-09-16T10:19:18.621273+0000","last_active":"2024-09-16T10:19:44.855718+0000","last_peered":"2024-09-16T10:19:44.855718+0000","last_clean":"2024-09-16T10:19:44.855718+0000","last_became_active":"2024-09-16T10:18:52.832688+0000","last_became_peered":"2024-09-16T10:18:52.832688+0000","last_unstale":"2024-09-16T10:19:44.855718+0000","last_undegraded":"2024-09-16T10:19:44.855718+0000","last_fullsized":"2024-09-16T10:19:44.855718+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:18.620605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.620605+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:19.358298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917374+0000","last_change":"2024-09-16T10:19:47.608315+0000","last_active":"2024-09-16T10:19:50.917374+0000","last_peered":"2024-09-16T10:19:50.917374+0000","last_clean":"2024-09-16T10:19:50.917374+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:19:50.917374+0000","last_undegraded":"2024-09-16T10:19:50.917374+0000","last_fullsized":"2024-09-16T10:19:50.917374+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.608243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.608243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.564159+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":271,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.870747+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:19:44.870747+0000","last_peered":"2024-09-16T10:19:44.870747+0000","last_clean":"2024-09-16T10:19:44.870747+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:19:44.870747+0000","last_undegraded":"2024-09-16T10:19:44.870747+0000","last_fullsized":"2024-09-16T10:19:44.870747+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.046746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":293,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.857465+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:19:44.857465+0000","last_peered":"2024-09-16T10:19:44.857465+0000","last_clean":"2024-09-16T10:19:44.857465+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:19:44.857465+0000","last_undegraded":"2024-09-16T10:19:44.857465+0000","last_fullsized":"2024-09-16T10:19:44.857465+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:50.190863+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":321,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.868783+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:19:44.868783+0000","last_peered":"2024-09-16T10:19:44.868783+0000","last_clean":"2024-09-16T10:19:44.868783+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:19:44.868783+0000","last_undegraded":"2024-09-16T10:19:44.868783+0000","last_fullsized":"2024-09-16T10:19:44.868783+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:13.959478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859774+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:19:44.859774+0000","last_peered":"2024-09-16T10:19:44.859774+0000","last_clean":"2024-09-16T10:19:44.859774+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:19:44.859774+0000","last_undegraded":"2024-09-16T10:19:44.859774+0000","last_fullsized":"2024-09-16T10:19:44.859774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:59.132108+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916214+0000","last_change":"2024-09-16T10:19:45.534275+0000","last_active":"2024-09-16T10:19:50.916214+0000","last_peered":"2024-09-16T10:19:50.916214+0000","last_clean":"2024-09-16T10:19:50.916214+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:19:50.916214+0000","last_undegraded":"2024-09-16T10:19:50.916214+0000","last_fullsized":"2024-09-16T10:19:50.916214+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:45.533998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.533998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:47.342382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:47.129026+0000","last_change":"2024-09-16T10:19:47.129026+0000","last_active":"2024-09-16T10:19:47.129026+0000","last_peered":"2024-09-16T10:19:47.129026+0000","last_clean":"2024-09-16T10:19:47.129026+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:19:47.129026+0000","last_undegraded":"2024-09-16T10:19:47.129026+0000","last_fullsized":"2024-09-16T10:19:47.129026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.128964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.128964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.566335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":334,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.869100+0000","last_change":"2024-09-16T10:18:52.855399+0000","last_active":"2024-09-16T10:19:44.869100+0000","last_peered":"2024-09-16T10:19:44.869100+0000","last_clean":"2024-09-16T10:19:44.869100+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:19:44.869100+0000","last_undegraded":"2024-09-16T10:19:44.869100+0000","last_fullsized":"2024-09-16T10:19:44.869100+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:14:43.322335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:14:43.322335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:53.799430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":319,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859626+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:19:44.859626+0000","last_peered":"2024-09-16T10:19:44.859626+0000","last_clean":"2024-09-16T10:19:44.859626+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:19:44.859626+0000","last_undegraded":"2024-09-16T10:19:44.859626+0000","last_fullsized":"2024-09-16T10:19:44.859626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859408+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:19:44.859408+0000","last_peered":"2024-09-16T10:19:44.859408+0000","last_clean":"2024-09-16T10:19:44.859408+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:19:44.859408+0000","last_undegraded":"2024-09-16T10:19:44.859408+0000","last_fullsized":"2024-09-16T10:19:44.859408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.226339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":74,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917095+0000","last_change":"2024-09-16T10:19:44.487481+0000","last_active":"2024-09-16T10:19:50.917095+0000","last_peered":"2024-09-16T10:19:50.917095+0000","last_clean":"2024-09-16T10:19:50.917095+0000","last_became_active":"2024-09-16T10:18:52.838002+0000","last_became_peered":"2024-09-16T10:18:52.838002+0000","last_unstale":"2024-09-16T10:19:50.917095+0000","last_undegraded":"2024-09-16T10:19:50.917095+0000","last_fullsized":"2024-09-16T10:19:50.917095+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.584073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.854424+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:19:44.854424+0000","last_peered":"2024-09-16T10:19:44.854424+0000","last_clean":"2024-09-16T10:19:44.854424+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:19:44.854424+0000","last_undegraded":"2024-09-16T10:19:44.854424+0000","last_fullsized":"2024-09-16T10:19:44.854424+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:43.126929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.869714+0000","last_change":"2024-09-16T10:19:20.738852+0000","last_active":"2024-09-16T10:19:44.869714+0000","last_peered":"2024-09-16T10:19:44.869714+0000","last_clean":"2024-09-16T10:19:44.869714+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:19:44.869714+0000","last_undegraded":"2024-09-16T10:19:44.869714+0000","last_fullsized":"2024-09-16T10:19:44.869714+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.738779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.738779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.219018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":344,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.863275+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:19:44.863275+0000","last_peered":"2024-09-16T10:19:44.863275+0000","last_clean":"2024-09-16T10:19:44.863275+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:19:44.863275+0000","last_undegraded":"2024-09-16T10:19:44.863275+0000","last_fullsized":"2024-09-16T10:19:44.863275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:09.457028+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":321,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T10:19:44.859788+0000","last_change":"2024-09-16T10:19:16.217856+0000","last_active":"2024-09-16T10:19:44.859788+0000","last_peered":"2024-09-16T10:19:44.859788+0000","last_clean":"2024-09-16T10:19:44.859788+0000","last_became_active":"2024-09-16T10:18:52.834424+0000","last_became_peered":"2024-09-16T10:18:52.834424+0000","last_unstale":"2024-09-16T10:19:44.859788+0000","last_undegraded":"2024-09-16T10:19:44.859788+0000","last_fullsized":"2024-09-16T10:19:44.859788+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.131801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6400442368,"num_objects":97663,"num_object_clones":0,"num_object_copies":292989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97663,"num_write_kb":6250432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10118,"num_bytes_recovered":663093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19997458432,"data_stored":19997458432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71463,"ondisk_log_size":71463,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234063,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2664368,"kb_used_data":2480476,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91080784,"statfs":{"total":95995035648,"available":93266722816,"internally_reserved":0,"allocated":2540007424,"data_stored":2539614103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528402,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2843104,"kb_used_data":2667536,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":90902048,"statfs":{"total":95995035648,"available":93083697152,"internally_reserved":0,"allocated":2731556864,"data_stored":2731168889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179763654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790040,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3664036,"kb_used_data":3498196,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":90081116,"statfs":{"total":95995035648,"available":92243062784,"internally_reserved":0,"allocated":3582152704,"data_stored":3581761957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986270,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2641844,"kb_used_data":2457956,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91103308,"statfs":{"total":95995035648,"available":93289787392,"internally_reserved":0,"allocated":2516946944,"data_stored":2516545374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215202,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3009408,"kb_used_data":2743600,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90735744,"statfs":{"total":95995035648,"available":92913401856,"internally_reserved":0,"allocated":2809446400,"data_stored":2809027925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":272169416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411432,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2925632,"kb_used_data":2709808,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":90819520,"statfs":{"total":95995035648,"available":92999188480,"internally_reserved":0,"allocated":2774843392,"data_stored":2774423528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2541488,"kb_used_data":2307616,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91203664,"statfs":{"total":95995035648,"available":93392551936,"internally_reserved":0,"allocated":2362998784,"data_stored":2362595433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2374434816,"data_stored":2374434816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2807693312,"data_stored":2807693312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":562823168,"data_stored":562823168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2820014080,"data_stored":2820014080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2530213888,"data_stored":2530213888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3582984192,"data_stored":3582984192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2761162752,"data_stored":2761162752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2558132224,"data_stored":2558132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:19:56.390 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:19:56.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:56.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:56.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:56.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 75 16 107822 107806 89.8151 76 0.00486517 0.0111297 2024-09-16T10:19:56.554 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:56.599 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:56.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:56.701 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:56.752 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:56.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:56.854 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:56.876 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:19:56.876 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:19:56.886 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:56.912 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:56.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:57.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:57.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:57.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:57.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:57.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:57.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:57.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:19:57.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:19:57.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:19:57.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:19:57.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[29993]: 4.10 scrub starts 2024-09-16T10:19:57.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[29993]: 4.10 scrub ok 2024-09-16T10:19:57.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[29993]: 8.9 scrub ok 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:19:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[39470]: 4.10 scrub starts 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[39470]: 4.10 scrub ok 2024-09-16T10:19:57.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:57 smithi063 ceph-mon[39470]: 8.9 scrub ok 2024-09-16T10:19:57.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:57 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:19:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:57 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:19:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:57 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:19:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:57 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:19:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:57 smithi165 ceph-mon[32696]: 4.10 scrub starts 2024-09-16T10:19:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:57 smithi165 ceph-mon[32696]: 4.10 scrub ok 2024-09-16T10:19:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:57 smithi165 ceph-mon[32696]: 8.9 scrub ok 2024-09-16T10:19:57.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:57.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:57.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:57.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:57.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:57.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 76 16 109136 109120 89.7138 82.125 0.00470159 0.0111422 2024-09-16T10:19:57.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:57.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:57.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:57.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:57.684 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:57.685 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:57.729 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:19:57.742 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:57.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:57.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:57.898 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:57.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:58.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:58.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:58.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:58.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:58.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:58.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[29993]: pgmap v307: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[29993]: 6.c scrub starts 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[29993]: 6.c scrub ok 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[29993]: 8.a scrub ok 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[39470]: pgmap v307: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[39470]: 6.c scrub starts 2024-09-16T10:19:58.244 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[39470]: 6.c scrub ok 2024-09-16T10:19:58.245 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:19:58.245 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:19:58.245 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:58 smithi063 ceph-mon[39470]: 8.a scrub ok 2024-09-16T10:19:58.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:58 smithi165 ceph-mon[32696]: pgmap v307: 158 pgs: 158 active+clean; 6.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-16T10:19:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:58 smithi165 ceph-mon[32696]: 6.c scrub starts 2024-09-16T10:19:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:58 smithi165 ceph-mon[32696]: 6.c scrub ok 2024-09-16T10:19:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:58 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:19:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:58 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:19:58.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:58 smithi165 ceph-mon[32696]: 8.a scrub ok 2024-09-16T10:19:58.320 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:58.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:58.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:58.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:58.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:58.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 77 16 110446 110430 89.6119 81.875 0.0234982 0.0111542 2024-09-16T10:19:58.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:58.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:19:58.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:58.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:58.702 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:19:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:58.776 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:58.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:19:58.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:19:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:19:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:19:58.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:19:58.950 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:58.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:58.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:19:59.049 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:19:59.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:19:59.098 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:59.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:19:59.111 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:19:59.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[29993]: 2.1 scrub starts 2024-09-16T10:19:59.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[29993]: 2.1 scrub ok 2024-09-16T10:19:59.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:59.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[29993]: 8.7 scrub starts 2024-09-16T10:19:59.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/845281317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:59.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[39470]: 2.1 scrub starts 2024-09-16T10:19:59.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[39470]: 2.1 scrub ok 2024-09-16T10:19:59.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:59.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[39470]: 8.7 scrub starts 2024-09-16T10:19:59.112 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:19:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/845281317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:59.119 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:59 smithi165 ceph-mon[32696]: 2.1 scrub starts 2024-09-16T10:19:59.119 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:59 smithi165 ceph-mon[32696]: 2.1 scrub ok 2024-09-16T10:19:59.119 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:19:59.120 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:59 smithi165 ceph-mon[32696]: 8.7 scrub starts 2024-09-16T10:19:59.120 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:19:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/845281317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:19:59.120 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:19:59.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:19:59.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:59.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:59.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:19:59.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:19:59.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:59.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:19:59.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:59.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 78 16 111557 111541 89.3532 69.4375 0.00654892 0.0111867 2024-09-16T10:19:59.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:59.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:19:59.627 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:19:59.658 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:19:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:59.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:19:59.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:19:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:59.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:59.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:19:59.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:19:59.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:19:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:19:59.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:00.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:00.080 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:00.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:00.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:00.214 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:00.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:00.316 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:00.336 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:00 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:00.336 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/845281317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:20:00.336 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:00 smithi165 ceph-mon[32696]: osdmap e203: 8 total, 8 up, 7 in 2024-09-16T10:20:00.336 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:00 smithi165 ceph-mon[32696]: from='client.25396 v1:172.21.15.63:0/1590560546' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:00.336 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:00 smithi165 ceph-mon[32696]: pgmap v309: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-16T10:20:00.336 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:00 smithi165 ceph-mon[32696]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-16T10:20:00.337 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:00 smithi165 ceph-mon[32696]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-16T10:20:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:00.354 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:00.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/845281317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[29993]: osdmap e203: 8 total, 8 up, 7 in 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[29993]: from='client.25396 v1:172.21.15.63:0/1590560546' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[29993]: pgmap v309: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[29993]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[29993]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/845281317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:20:00.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[39470]: osdmap e203: 8 total, 8 up, 7 in 2024-09-16T10:20:00.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[39470]: from='client.25396 v1:172.21.15.63:0/1590560546' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:00.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[39470]: pgmap v309: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-16T10:20:00.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[39470]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-16T10:20:00.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:00 smithi063 ceph-mon[39470]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-16T10:20:00.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:00.481 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:00.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:00.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 79 16 112807 112791 89.211 78.125 0.00692617 0.0112052 2024-09-16T10:20:00.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:00.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:00.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:00.724 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:00.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:00.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:00.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:00.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:01.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:01.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:01.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:01.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:01.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:01.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:01.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:01.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:01.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:01.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:01.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:01.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:01.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:01.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:01.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:20:01.509595+0000 min lat: 0.00128827 max lat: 2.11649 avg lat: 0.0112263 2024-09-16T10:20:01.574 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:20:01.575 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 80 16 114015 113999 89.0396 75.5 0.00401831 0.0112263 2024-09-16T10:20:01.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:01.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:01.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:01.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:01.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:01.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:01.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:01.884 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:01.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:01.986 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:01 smithi165 ceph-mon[32696]: pgmap v310: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-16T10:20:02.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:02.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:02.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:01 smithi063 ceph-mon[29993]: pgmap v310: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-16T10:20:02.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:01 smithi063 ceph-mon[39470]: pgmap v310: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-16T10:20:02.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:02.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:02.413 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:02.441 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:02.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 81 16 115231 115215 88.8786 76 0.00318585 0.0112366 2024-09-16T10:20:02.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:02.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:02.679 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:02.713 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:02.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:02.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:02.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:02.981 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:02.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:03.048 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:03.049 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:03.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:03.119 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:03.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:03.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:02 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:03.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:02 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:03.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:02 smithi063 ceph-mon[29993]: osdmap e204: 8 total, 8 up, 7 in 2024-09-16T10:20:03.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:02 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:03.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:02 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:03.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:02 smithi063 ceph-mon[39470]: osdmap e204: 8 total, 8 up, 7 in 2024-09-16T10:20:03.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:03.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:03.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:03.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:02 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:03.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:02 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:03.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:02 smithi165 ceph-mon[32696]: osdmap e204: 8 total, 8 up, 7 in 2024-09-16T10:20:03.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:03.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:03.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:03.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:03.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 82 16 116559 116543 88.8068 83 0.00667473 0.0112559 2024-09-16T10:20:03.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:03.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:03.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:03.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:03.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:03.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:03.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:03.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:03.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:03.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:03.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:04.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:04.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:04.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:04.185 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:04.200 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:03 smithi063 ceph-mon[29993]: pgmap v312: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 833 op/s 2024-09-16T10:20:04.200 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:03 smithi063 ceph-mon[39470]: pgmap v312: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 833 op/s 2024-09-16T10:20:04.222 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:04.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:04.252 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-16T10:19:59.049523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7226464795,"num_objects":110457,"num_object_clones":0,"num_object_copies":331371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110457,"num_whiteouts":0,"num_read":1096,"num_read_kb":848,"num_write":110694,"num_write_kb":7056557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10188,"num_bytes_recovered":663357700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73103,"ondisk_log_size":73103,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22567440,"kb_used_data":20992800,"kb_used_omap":10,"kb_used_meta":1574517,"kb_avail":633648624,"statfs":{"total":671965249536,"available":648856190976,"internally_reserved":0,"allocated":21496627200,"data_stored":21493784737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":1612305516},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":852164608,"num_objects":13003,"num_object_clones":0,"num_object_copies":39009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13003,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13003,"num_write_kb":832192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.906244"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":331,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913525+0000","last_change":"2024-09-16T10:18:52.830567+0000","last_active":"2024-09-16T10:19:50.913525+0000","last_peered":"2024-09-16T10:19:50.913525+0000","last_clean":"2024-09-16T10:19:50.913525+0000","last_became_active":"2024-09-16T10:18:52.830260+0000","last_became_peered":"2024-09-16T10:18:52.830260+0000","last_unstale":"2024-09-16T10:19:50.913525+0000","last_undegraded":"2024-09-16T10:19:50.913525+0000","last_fullsized":"2024-09-16T10:19:50.913525+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:07.119072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.119072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":377,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:55.724523+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:19:55.724523+0000","last_peered":"2024-09-16T10:19:55.724523+0000","last_clean":"2024-09-16T10:19:55.724523+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:19:55.724523+0000","last_undegraded":"2024-09-16T10:19:55.724523+0000","last_fullsized":"2024-09-16T10:19:55.724523+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:25.723031+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911771+0000","last_change":"2024-09-16T10:18:40.752790+0000","last_active":"2024-09-16T10:19:50.911771+0000","last_peered":"2024-09-16T10:19:50.911771+0000","last_clean":"2024-09-16T10:19:50.911771+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:19:50.911771+0000","last_undegraded":"2024-09-16T10:19:50.911771+0000","last_fullsized":"2024-09-16T10:19:50.911771+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:40.752739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:40.752739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":303,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923036+0000","last_change":"2024-09-16T10:19:23.784420+0000","last_active":"2024-09-16T10:19:50.923036+0000","last_peered":"2024-09-16T10:19:50.923036+0000","last_clean":"2024-09-16T10:19:50.923036+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:19:50.923036+0000","last_undegraded":"2024-09-16T10:19:50.923036+0000","last_fullsized":"2024-09-16T10:19:50.923036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.784347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.784347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.734381+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":304,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918292+0000","last_change":"2024-09-16T10:18:34.944971+0000","last_active":"2024-09-16T10:19:50.918292+0000","last_peered":"2024-09-16T10:19:50.918292+0000","last_clean":"2024-09-16T10:19:50.918292+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:19:50.918292+0000","last_undegraded":"2024-09-16T10:19:50.918292+0000","last_fullsized":"2024-09-16T10:19:50.918292+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:34.944915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.944915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:53.549928+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918598+0000","last_change":"2024-09-16T10:18:52.836944+0000","last_active":"2024-09-16T10:19:50.918598+0000","last_peered":"2024-09-16T10:19:50.918598+0000","last_clean":"2024-09-16T10:19:50.918598+0000","last_became_active":"2024-09-16T10:18:52.836507+0000","last_became_peered":"2024-09-16T10:18:52.836507+0000","last_unstale":"2024-09-16T10:19:50.918598+0000","last_undegraded":"2024-09-16T10:19:50.918598+0000","last_fullsized":"2024-09-16T10:19:50.918598+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:13.018412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.018412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.942085+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":360,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917042+0000","last_change":"2024-09-16T10:18:52.855408+0000","last_active":"2024-09-16T10:19:50.917042+0000","last_peered":"2024-09-16T10:19:50.917042+0000","last_clean":"2024-09-16T10:19:50.917042+0000","last_became_active":"2024-09-16T10:18:52.835715+0000","last_became_peered":"2024-09-16T10:18:52.835715+0000","last_unstale":"2024-09-16T10:19:50.917042+0000","last_undegraded":"2024-09-16T10:19:50.917042+0000","last_fullsized":"2024-09-16T10:19:50.917042+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:16:31.217192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.217192+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.892839+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918392+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:19:50.918392+0000","last_peered":"2024-09-16T10:19:50.918392+0000","last_clean":"2024-09-16T10:19:50.918392+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:19:50.918392+0000","last_undegraded":"2024-09-16T10:19:50.918392+0000","last_fullsized":"2024-09-16T10:19:50.918392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:57.272422+0000","last_change":"2024-09-16T10:19:57.272422+0000","last_active":"2024-09-16T10:19:57.272422+0000","last_peered":"2024-09-16T10:19:57.272422+0000","last_clean":"2024-09-16T10:19:57.272422+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:19:57.272422+0000","last_undegraded":"2024-09-16T10:19:57.272422+0000","last_fullsized":"2024-09-16T10:19:57.272422+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.272362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.272362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:13.881596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":327,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923351+0000","last_change":"2024-09-16T10:16:32.833966+0000","last_active":"2024-09-16T10:19:50.923351+0000","last_peered":"2024-09-16T10:19:50.923351+0000","last_clean":"2024-09-16T10:19:50.923351+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:19:50.923351+0000","last_undegraded":"2024-09-16T10:19:50.923351+0000","last_fullsized":"2024-09-16T10:19:50.923351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:32.833895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.833895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.724646+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:51.099949+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:19:51.099949+0000","last_peered":"2024-09-16T10:19:51.099949+0000","last_clean":"2024-09-16T10:19:51.099949+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:19:51.099949+0000","last_undegraded":"2024-09-16T10:19:51.099949+0000","last_fullsized":"2024-09-16T10:19:51.099949+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.206970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":303,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.912070+0000","last_change":"2024-09-16T10:17:12.089641+0000","last_active":"2024-09-16T10:19:50.912070+0000","last_peered":"2024-09-16T10:19:50.912070+0000","last_clean":"2024-09-16T10:19:50.912070+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:19:50.912070+0000","last_undegraded":"2024-09-16T10:19:50.912070+0000","last_fullsized":"2024-09-16T10:19:50.912070+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.089577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.089577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.220934+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":300,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917133+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:19:50.917133+0000","last_peered":"2024-09-16T10:19:50.917133+0000","last_clean":"2024-09-16T10:19:50.917133+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:19:50.917133+0000","last_undegraded":"2024-09-16T10:19:50.917133+0000","last_fullsized":"2024-09-16T10:19:50.917133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.984980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":321,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923560+0000","last_change":"2024-09-16T10:17:16.937447+0000","last_active":"2024-09-16T10:19:50.923560+0000","last_peered":"2024-09-16T10:19:50.923560+0000","last_clean":"2024-09-16T10:19:50.923560+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:19:50.923560+0000","last_undegraded":"2024-09-16T10:19:50.923560+0000","last_fullsized":"2024-09-16T10:19:50.923560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:17:16.937353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.937353+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.713148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917261+0000","last_change":"2024-09-16T10:19:23.390542+0000","last_active":"2024-09-16T10:19:50.917261+0000","last_peered":"2024-09-16T10:19:50.917261+0000","last_clean":"2024-09-16T10:19:50.917261+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:19:50.917261+0000","last_undegraded":"2024-09-16T10:19:50.917261+0000","last_fullsized":"2024-09-16T10:19:50.917261+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:49.356783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916511+0000","last_change":"2024-09-16T10:19:46.109382+0000","last_active":"2024-09-16T10:19:50.916511+0000","last_peered":"2024-09-16T10:19:50.916511+0000","last_clean":"2024-09-16T10:19:50.916511+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:19:50.916511+0000","last_undegraded":"2024-09-16T10:19:50.916511+0000","last_fullsized":"2024-09-16T10:19:50.916511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:46.109310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.109310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.462251+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.920936+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:19:50.920936+0000","last_peered":"2024-09-16T10:19:50.920936+0000","last_clean":"2024-09-16T10:19:50.920936+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:19:50.920936+0000","last_undegraded":"2024-09-16T10:19:50.920936+0000","last_fullsized":"2024-09-16T10:19:50.920936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.949799+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":307,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914044+0000","last_change":"2024-09-16T10:19:15.182388+0000","last_active":"2024-09-16T10:19:50.914044+0000","last_peered":"2024-09-16T10:19:50.914044+0000","last_clean":"2024-09-16T10:19:50.914044+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:19:50.914044+0000","last_undegraded":"2024-09-16T10:19:50.914044+0000","last_fullsized":"2024-09-16T10:19:50.914044+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.182335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.182335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.600446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":136,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913145+0000","last_change":"2024-09-16T10:19:14.579717+0000","last_active":"2024-09-16T10:19:50.913145+0000","last_peered":"2024-09-16T10:19:50.913145+0000","last_clean":"2024-09-16T10:19:50.913145+0000","last_became_active":"2024-09-16T10:18:52.841218+0000","last_became_peered":"2024-09-16T10:18:52.841218+0000","last_unstale":"2024-09-16T10:19:50.913145+0000","last_undegraded":"2024-09-16T10:19:50.913145+0000","last_fullsized":"2024-09-16T10:19:50.913145+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:14.868398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911976+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:19:50.911976+0000","last_peered":"2024-09-16T10:19:50.911976+0000","last_clean":"2024-09-16T10:19:50.911976+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:19:50.911976+0000","last_undegraded":"2024-09-16T10:19:50.911976+0000","last_fullsized":"2024-09-16T10:19:50.911976+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:57.868630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":261,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919550+0000","last_change":"2024-09-16T10:19:48.110737+0000","last_active":"2024-09-16T10:19:50.919550+0000","last_peered":"2024-09-16T10:19:50.919550+0000","last_clean":"2024-09-16T10:19:50.919550+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:19:50.919550+0000","last_undegraded":"2024-09-16T10:19:50.919550+0000","last_fullsized":"2024-09-16T10:19:50.919550+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:48.110674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.110674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.967276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":314,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.924334+0000","last_change":"2024-09-16T10:19:22.803999+0000","last_active":"2024-09-16T10:19:50.924334+0000","last_peered":"2024-09-16T10:19:50.924334+0000","last_clean":"2024-09-16T10:19:50.924334+0000","last_became_active":"2024-09-16T10:18:52.834723+0000","last_became_peered":"2024-09-16T10:18:52.834723+0000","last_unstale":"2024-09-16T10:19:50.924334+0000","last_undegraded":"2024-09-16T10:19:50.924334+0000","last_fullsized":"2024-09-16T10:19:50.924334+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.204568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":362,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917470+0000","last_change":"2024-09-16T10:19:24.829364+0000","last_active":"2024-09-16T10:19:50.917470+0000","last_peered":"2024-09-16T10:19:50.917470+0000","last_clean":"2024-09-16T10:19:50.917470+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:19:50.917470+0000","last_undegraded":"2024-09-16T10:19:50.917470+0000","last_fullsized":"2024-09-16T10:19:50.917470+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:24.829255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.829255+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.222182+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":322,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918385+0000","last_change":"2024-09-16T10:18:52.839387+0000","last_active":"2024-09-16T10:19:50.918385+0000","last_peered":"2024-09-16T10:19:50.918385+0000","last_clean":"2024-09-16T10:19:50.918385+0000","last_became_active":"2024-09-16T10:18:52.838765+0000","last_became_peered":"2024-09-16T10:18:52.838765+0000","last_unstale":"2024-09-16T10:19:50.918385+0000","last_undegraded":"2024-09-16T10:19:50.918385+0000","last_fullsized":"2024-09-16T10:19:50.918385+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:05.795954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.795954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916891+0000","last_change":"2024-09-16T10:18:52.847158+0000","last_active":"2024-09-16T10:19:50.916891+0000","last_peered":"2024-09-16T10:19:50.916891+0000","last_clean":"2024-09-16T10:19:50.916891+0000","last_became_active":"2024-09-16T10:18:52.838789+0000","last_became_peered":"2024-09-16T10:18:52.838789+0000","last_unstale":"2024-09-16T10:19:50.916891+0000","last_undegraded":"2024-09-16T10:19:50.916891+0000","last_fullsized":"2024-09-16T10:19:50.916891+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.622143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":323,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911914+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:19:50.911914+0000","last_peered":"2024-09-16T10:19:50.911914+0000","last_clean":"2024-09-16T10:19:50.911914+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:19:50.911914+0000","last_undegraded":"2024-09-16T10:19:50.911914+0000","last_fullsized":"2024-09-16T10:19:50.911914+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.407870+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":335,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914268+0000","last_change":"2024-09-16T10:19:49.067027+0000","last_active":"2024-09-16T10:19:50.914268+0000","last_peered":"2024-09-16T10:19:50.914268+0000","last_clean":"2024-09-16T10:19:50.914268+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:19:50.914268+0000","last_undegraded":"2024-09-16T10:19:50.914268+0000","last_fullsized":"2024-09-16T10:19:50.914268+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:49.066953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:49.066953+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.060028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":404,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:57.235056+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:19:57.235056+0000","last_peered":"2024-09-16T10:19:57.235056+0000","last_clean":"2024-09-16T10:19:57.235056+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:19:57.235056+0000","last_undegraded":"2024-09-16T10:19:57.235056+0000","last_fullsized":"2024-09-16T10:19:57.235056+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.301174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"202'6939","reported_seq":7024,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.397090+0000","last_change":"2024-09-16T10:18:33.825761+0000","last_active":"2024-09-16T10:19:58.397090+0000","last_peered":"2024-09-16T10:19:58.397090+0000","last_clean":"2024-09-16T10:19:58.397090+0000","last_became_active":"2024-09-16T10:18:33.825267+0000","last_became_peered":"2024-09-16T10:18:33.825267+0000","last_unstale":"2024-09-16T10:19:58.397090+0000","last_undegraded":"2024-09-16T10:19:58.397090+0000","last_fullsized":"2024-09-16T10:19:58.397090+0000","mapping_epoch":165,"log_start":"179'2500","ondisk_log_start":"179'2500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:59.402037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454754304,"num_objects":6939,"num_object_clones":0,"num_object_copies":20817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6939,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":117,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914797+0000","last_change":"2024-09-16T10:18:52.828048+0000","last_active":"2024-09-16T10:19:50.914797+0000","last_peered":"2024-09-16T10:19:50.914797+0000","last_clean":"2024-09-16T10:19:50.914797+0000","last_became_active":"2024-09-16T10:18:52.825707+0000","last_became_peered":"2024-09-16T10:18:52.825707+0000","last_unstale":"2024-09-16T10:19:50.914797+0000","last_undegraded":"2024-09-16T10:19:50.914797+0000","last_fullsized":"2024-09-16T10:19:50.914797+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.115716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.115716+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.054496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":337,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914467+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:19:50.914467+0000","last_peered":"2024-09-16T10:19:50.914467+0000","last_clean":"2024-09-16T10:19:50.914467+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:19:50.914467+0000","last_undegraded":"2024-09-16T10:19:50.914467+0000","last_fullsized":"2024-09-16T10:19:50.914467+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.449585+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917138+0000","last_change":"2024-09-16T10:19:43.028508+0000","last_active":"2024-09-16T10:19:50.917138+0000","last_peered":"2024-09-16T10:19:50.917138+0000","last_clean":"2024-09-16T10:19:50.917138+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:19:50.917138+0000","last_undegraded":"2024-09-16T10:19:50.917138+0000","last_fullsized":"2024-09-16T10:19:50.917138+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:43.028436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:43.028436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.274639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":384,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:52.232875+0000","last_change":"2024-09-16T10:18:36.623362+0000","last_active":"2024-09-16T10:19:52.232875+0000","last_peered":"2024-09-16T10:19:52.232875+0000","last_clean":"2024-09-16T10:19:52.232875+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:19:52.232875+0000","last_undegraded":"2024-09-16T10:19:52.232875+0000","last_fullsized":"2024-09-16T10:19:52.232875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:36.623302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:36.623302+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":304,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.922393+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:19:50.922393+0000","last_peered":"2024-09-16T10:19:50.922393+0000","last_clean":"2024-09-16T10:19:50.922393+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:19:50.922393+0000","last_undegraded":"2024-09-16T10:19:50.922393+0000","last_fullsized":"2024-09-16T10:19:50.922393+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.040726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"202'6905","reported_seq":6990,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.537809+0000","last_change":"2024-09-16T10:18:33.893627+0000","last_active":"2024-09-16T10:19:58.537809+0000","last_peered":"2024-09-16T10:19:58.537809+0000","last_clean":"2024-09-16T10:19:58.537809+0000","last_became_active":"2024-09-16T10:18:33.893281+0000","last_became_peered":"2024-09-16T10:18:33.893281+0000","last_unstale":"2024-09-16T10:19:58.537809+0000","last_undegraded":"2024-09-16T10:19:58.537809+0000","last_fullsized":"2024-09-16T10:19:58.537809+0000","mapping_epoch":165,"log_start":"178'2200","ondisk_log_start":"178'2200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452526080,"num_objects":6905,"num_object_clones":0,"num_object_copies":20715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6905,"num_write_kb":441920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":117,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.921284+0000","last_change":"2024-09-16T10:18:52.824806+0000","last_active":"2024-09-16T10:19:50.921284+0000","last_peered":"2024-09-16T10:19:50.921284+0000","last_clean":"2024-09-16T10:19:50.921284+0000","last_became_active":"2024-09-16T10:18:52.824449+0000","last_became_peered":"2024-09-16T10:18:52.824449+0000","last_unstale":"2024-09-16T10:19:50.921284+0000","last_undegraded":"2024-09-16T10:19:50.921284+0000","last_fullsized":"2024-09-16T10:19:50.921284+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.752706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.752706+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":345,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916821+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:19:50.916821+0000","last_peered":"2024-09-16T10:19:50.916821+0000","last_clean":"2024-09-16T10:19:50.916821+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:19:50.916821+0000","last_undegraded":"2024-09-16T10:19:50.916821+0000","last_fullsized":"2024-09-16T10:19:50.916821+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":281,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919917+0000","last_change":"2024-09-16T10:19:50.114162+0000","last_active":"2024-09-16T10:19:50.919917+0000","last_peered":"2024-09-16T10:19:50.919917+0000","last_clean":"2024-09-16T10:19:50.919917+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:19:50.919917+0000","last_undegraded":"2024-09-16T10:19:50.919917+0000","last_fullsized":"2024-09-16T10:19:50.919917+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.114098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.114098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:16.082035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":54,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914675+0000","last_change":"2024-09-16T10:19:16.147371+0000","last_active":"2024-09-16T10:19:50.914675+0000","last_peered":"2024-09-16T10:19:50.914675+0000","last_clean":"2024-09-16T10:19:50.914675+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:19:50.914675+0000","last_undegraded":"2024-09-16T10:19:50.914675+0000","last_fullsized":"2024-09-16T10:19:50.914675+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.147304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.147304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:22.719288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":327,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923308+0000","last_change":"2024-09-16T10:19:44.711933+0000","last_active":"2024-09-16T10:19:50.923308+0000","last_peered":"2024-09-16T10:19:50.923308+0000","last_clean":"2024-09-16T10:19:50.923308+0000","last_became_active":"2024-09-16T10:18:52.835843+0000","last_became_peered":"2024-09-16T10:18:52.835843+0000","last_unstale":"2024-09-16T10:19:50.923308+0000","last_undegraded":"2024-09-16T10:19:50.923308+0000","last_fullsized":"2024-09-16T10:19:50.923308+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.727533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"202'6994","reported_seq":10587,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.218372+0000","last_change":"2024-09-16T10:19:15.299139+0000","last_active":"2024-09-16T10:19:58.218372+0000","last_peered":"2024-09-16T10:19:58.218372+0000","last_clean":"2024-09-16T10:19:58.218372+0000","last_became_active":"2024-09-16T10:19:15.293347+0000","last_became_peered":"2024-09-16T10:19:15.293347+0000","last_unstale":"2024-09-16T10:19:58.218372+0000","last_undegraded":"2024-09-16T10:19:58.218372+0000","last_fullsized":"2024-09-16T10:19:58.218372+0000","mapping_epoch":188,"log_start":"180'2500","ondisk_log_start":"180'2500","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458358784,"num_objects":6994,"num_object_clones":0,"num_object_copies":20982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6994,"num_write_kb":447616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1748,"num_bytes_recovered":114556928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":115,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917251+0000","last_change":"2024-09-16T10:18:07.024979+0000","last_active":"2024-09-16T10:19:50.917251+0000","last_peered":"2024-09-16T10:19:50.917251+0000","last_clean":"2024-09-16T10:19:50.917251+0000","last_became_active":"2024-09-16T10:18:06.896380+0000","last_became_peered":"2024-09-16T10:18:06.896380+0000","last_unstale":"2024-09-16T10:19:50.917251+0000","last_undegraded":"2024-09-16T10:19:50.917251+0000","last_fullsized":"2024-09-16T10:19:50.917251+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.024848+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":349,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918599+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:19:50.918599+0000","last_peered":"2024-09-16T10:19:50.918599+0000","last_clean":"2024-09-16T10:19:50.918599+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:19:50.918599+0000","last_undegraded":"2024-09-16T10:19:50.918599+0000","last_fullsized":"2024-09-16T10:19:50.918599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:15.756451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914415+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:19:50.914415+0000","last_peered":"2024-09-16T10:19:50.914415+0000","last_clean":"2024-09-16T10:19:50.914415+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:19:50.914415+0000","last_undegraded":"2024-09-16T10:19:50.914415+0000","last_fullsized":"2024-09-16T10:19:50.914415+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:14.162775+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":397,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:52.232757+0000","last_change":"2024-09-16T10:18:04.023115+0000","last_active":"2024-09-16T10:19:52.232757+0000","last_peered":"2024-09-16T10:19:52.232757+0000","last_clean":"2024-09-16T10:19:52.232757+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:19:52.232757+0000","last_undegraded":"2024-09-16T10:19:52.232757+0000","last_fullsized":"2024-09-16T10:19:52.232757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:04.022579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.022579+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":304,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917587+0000","last_change":"2024-09-16T10:18:52.824583+0000","last_active":"2024-09-16T10:19:50.917587+0000","last_peered":"2024-09-16T10:19:50.917587+0000","last_clean":"2024-09-16T10:19:50.917587+0000","last_became_active":"2024-09-16T10:18:52.824265+0000","last_became_peered":"2024-09-16T10:18:52.824265+0000","last_unstale":"2024-09-16T10:19:50.917587+0000","last_undegraded":"2024-09-16T10:19:50.917587+0000","last_fullsized":"2024-09-16T10:19:50.917587+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.491962+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"202'6798","reported_seq":10195,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.533600+0000","last_change":"2024-09-16T10:19:14.298863+0000","last_active":"2024-09-16T10:19:58.533600+0000","last_peered":"2024-09-16T10:19:58.533600+0000","last_clean":"2024-09-16T10:19:58.533600+0000","last_became_active":"2024-09-16T10:19:14.293461+0000","last_became_peered":"2024-09-16T10:19:14.293461+0000","last_unstale":"2024-09-16T10:19:58.533600+0000","last_undegraded":"2024-09-16T10:19:58.533600+0000","last_fullsized":"2024-09-16T10:19:58.533600+0000","mapping_epoch":187,"log_start":"178'2100","ondisk_log_start":"178'2100","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445513728,"num_objects":6798,"num_object_clones":0,"num_object_copies":20394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6798,"num_write_kb":435072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":186,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916019+0000","last_change":"2024-09-16T10:18:07.192860+0000","last_active":"2024-09-16T10:19:50.916019+0000","last_peered":"2024-09-16T10:19:50.916019+0000","last_clean":"2024-09-16T10:19:50.916019+0000","last_became_active":"2024-09-16T10:18:06.896951+0000","last_became_peered":"2024-09-16T10:18:06.896951+0000","last_unstale":"2024-09-16T10:19:50.916019+0000","last_undegraded":"2024-09-16T10:19:50.916019+0000","last_fullsized":"2024-09-16T10:19:50.916019+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.192777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.192777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.243306+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":344,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:54.487378+0000","last_change":"2024-09-16T10:19:54.487297+0000","last_active":"2024-09-16T10:19:54.487378+0000","last_peered":"2024-09-16T10:19:54.487378+0000","last_clean":"2024-09-16T10:19:54.487378+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:19:54.487378+0000","last_undegraded":"2024-09-16T10:19:54.487378+0000","last_fullsized":"2024-09-16T10:19:54.487378+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:54.487219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.487219+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.245307+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.922805+0000","last_change":"2024-09-16T10:18:52.836576+0000","last_active":"2024-09-16T10:19:50.922805+0000","last_peered":"2024-09-16T10:19:50.922805+0000","last_clean":"2024-09-16T10:19:50.922805+0000","last_became_active":"2024-09-16T10:18:52.836260+0000","last_became_peered":"2024-09-16T10:18:52.836260+0000","last_unstale":"2024-09-16T10:19:50.922805+0000","last_undegraded":"2024-09-16T10:19:50.922805+0000","last_fullsized":"2024-09-16T10:19:50.922805+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.187181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.187181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.017380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":385,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:57.234131+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:19:57.234131+0000","last_peered":"2024-09-16T10:19:57.234131+0000","last_clean":"2024-09-16T10:19:57.234131+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:19:57.234131+0000","last_undegraded":"2024-09-16T10:19:57.234131+0000","last_fullsized":"2024-09-16T10:19:57.234131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.246556+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":319,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.912265+0000","last_change":"2024-09-16T10:19:50.044357+0000","last_active":"2024-09-16T10:19:50.912265+0000","last_peered":"2024-09-16T10:19:50.912265+0000","last_clean":"2024-09-16T10:19:50.912265+0000","last_became_active":"2024-09-16T10:18:52.842382+0000","last_became_peered":"2024-09-16T10:18:52.842382+0000","last_unstale":"2024-09-16T10:19:50.912265+0000","last_undegraded":"2024-09-16T10:19:50.912265+0000","last_fullsized":"2024-09-16T10:19:50.912265+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.044297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.044297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.918339+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"202'6896","reported_seq":10311,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.453653+0000","last_change":"2024-09-16T10:19:14.298487+0000","last_active":"2024-09-16T10:19:58.453653+0000","last_peered":"2024-09-16T10:19:58.453653+0000","last_clean":"2024-09-16T10:19:58.453653+0000","last_became_active":"2024-09-16T10:19:14.292726+0000","last_became_peered":"2024-09-16T10:19:14.292726+0000","last_unstale":"2024-09-16T10:19:58.453653+0000","last_undegraded":"2024-09-16T10:19:58.453653+0000","last_fullsized":"2024-09-16T10:19:58.453653+0000","mapping_epoch":187,"log_start":"182'2600","ondisk_log_start":"182'2600","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451936256,"num_objects":6896,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6896,"num_write_kb":441344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":108724224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":186,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.915439+0000","last_change":"2024-09-16T10:18:08.171074+0000","last_active":"2024-09-16T10:19:50.915439+0000","last_peered":"2024-09-16T10:19:50.915439+0000","last_clean":"2024-09-16T10:19:50.915439+0000","last_became_active":"2024-09-16T10:18:06.910799+0000","last_became_peered":"2024-09-16T10:18:06.910799+0000","last_unstale":"2024-09-16T10:19:50.915439+0000","last_undegraded":"2024-09-16T10:19:50.915439+0000","last_fullsized":"2024-09-16T10:19:50.915439+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.171021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.171021+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.281938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":356,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913607+0000","last_change":"2024-09-16T10:19:22.293172+0000","last_active":"2024-09-16T10:19:50.913607+0000","last_peered":"2024-09-16T10:19:50.913607+0000","last_clean":"2024-09-16T10:19:50.913607+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:19:50.913607+0000","last_undegraded":"2024-09-16T10:19:50.913607+0000","last_fullsized":"2024-09-16T10:19:50.913607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:22.293076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.293076+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.645714+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913509+0000","last_change":"2024-09-16T10:18:52.833335+0000","last_active":"2024-09-16T10:19:50.913509+0000","last_peered":"2024-09-16T10:19:50.913509+0000","last_clean":"2024-09-16T10:19:50.913509+0000","last_became_active":"2024-09-16T10:18:52.832749+0000","last_became_peered":"2024-09-16T10:18:52.832749+0000","last_unstale":"2024-09-16T10:19:50.913509+0000","last_undegraded":"2024-09-16T10:19:50.913509+0000","last_fullsized":"2024-09-16T10:19:50.913509+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:33.949798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.949798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.578873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:55.433288+0000","last_change":"2024-09-16T10:19:55.433215+0000","last_active":"2024-09-16T10:19:55.433288+0000","last_peered":"2024-09-16T10:19:55.433288+0000","last_clean":"2024-09-16T10:19:55.433288+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:19:55.433288+0000","last_undegraded":"2024-09-16T10:19:55.433288+0000","last_fullsized":"2024-09-16T10:19:55.433288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:55.433149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.433149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.272672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911729+0000","last_change":"2024-09-16T10:18:52.827865+0000","last_active":"2024-09-16T10:19:50.911729+0000","last_peered":"2024-09-16T10:19:50.911729+0000","last_clean":"2024-09-16T10:19:50.911729+0000","last_became_active":"2024-09-16T10:18:52.827511+0000","last_became_peered":"2024-09-16T10:18:52.827511+0000","last_unstale":"2024-09-16T10:19:50.911729+0000","last_undegraded":"2024-09-16T10:19:50.911729+0000","last_fullsized":"2024-09-16T10:19:50.911729+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:28.543646+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"202'7036","reported_seq":7120,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.222173+0000","last_change":"2024-09-16T10:18:33.893596+0000","last_active":"2024-09-16T10:19:58.222173+0000","last_peered":"2024-09-16T10:19:58.222173+0000","last_clean":"2024-09-16T10:19:58.222173+0000","last_became_active":"2024-09-16T10:18:33.893254+0000","last_became_peered":"2024-09-16T10:18:33.893254+0000","last_unstale":"2024-09-16T10:19:58.222173+0000","last_undegraded":"2024-09-16T10:19:58.222173+0000","last_fullsized":"2024-09-16T10:19:58.222173+0000","mapping_epoch":165,"log_start":"181'2600","ondisk_log_start":"181'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461111296,"num_objects":7036,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7036,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":117,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916317+0000","last_change":"2024-09-16T10:18:52.831104+0000","last_active":"2024-09-16T10:19:50.916317+0000","last_peered":"2024-09-16T10:19:50.916317+0000","last_clean":"2024-09-16T10:19:50.916317+0000","last_became_active":"2024-09-16T10:18:52.830618+0000","last_became_peered":"2024-09-16T10:18:52.830618+0000","last_unstale":"2024-09-16T10:19:50.916317+0000","last_undegraded":"2024-09-16T10:19:50.916317+0000","last_fullsized":"2024-09-16T10:19:50.916317+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.131122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.131122+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:41.045857+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"175'29","reported_seq":483,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:57.233430+0000","last_change":"2024-09-16T10:18:52.860272+0000","last_active":"2024-09-16T10:19:57.233430+0000","last_peered":"2024-09-16T10:19:57.233430+0000","last_clean":"2024-09-16T10:19:57.233430+0000","last_became_active":"2024-09-16T10:18:52.841202+0000","last_became_peered":"2024-09-16T10:18:52.841202+0000","last_unstale":"2024-09-16T10:19:57.233430+0000","last_undegraded":"2024-09-16T10:19:57.233430+0000","last_fullsized":"2024-09-16T10:19:57.233430+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:18:45.654714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.654714+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.925181+0000","last_change":"2024-09-16T10:17:35.928758+0000","last_active":"2024-09-16T10:19:50.925181+0000","last_peered":"2024-09-16T10:19:50.925181+0000","last_clean":"2024-09-16T10:19:50.925181+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:19:50.925181+0000","last_undegraded":"2024-09-16T10:19:50.925181+0000","last_fullsized":"2024-09-16T10:19:50.925181+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.928701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.928701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:41.782908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":327,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.915437+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:19:50.915437+0000","last_peered":"2024-09-16T10:19:50.915437+0000","last_clean":"2024-09-16T10:19:50.915437+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:19:50.915437+0000","last_undegraded":"2024-09-16T10:19:50.915437+0000","last_fullsized":"2024-09-16T10:19:50.915437+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.056983+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":299,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.912869+0000","last_change":"2024-09-16T10:16:04.405318+0000","last_active":"2024-09-16T10:19:50.912869+0000","last_peered":"2024-09-16T10:19:50.912869+0000","last_clean":"2024-09-16T10:19:50.912869+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:19:50.912869+0000","last_undegraded":"2024-09-16T10:19:50.912869+0000","last_fullsized":"2024-09-16T10:19:50.912869+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:04.405243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.405243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.279683+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917915+0000","last_change":"2024-09-16T10:18:41.823802+0000","last_active":"2024-09-16T10:19:50.917915+0000","last_peered":"2024-09-16T10:19:50.917915+0000","last_clean":"2024-09-16T10:19:50.917915+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:19:50.917915+0000","last_undegraded":"2024-09-16T10:19:50.917915+0000","last_fullsized":"2024-09-16T10:19:50.917915+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:41.823721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:41.823721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:10.790493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"202'6963","reported_seq":7048,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.498859+0000","last_change":"2024-09-16T10:18:33.821147+0000","last_active":"2024-09-16T10:19:58.498859+0000","last_peered":"2024-09-16T10:19:58.498859+0000","last_clean":"2024-09-16T10:19:58.498859+0000","last_became_active":"2024-09-16T10:18:33.820857+0000","last_became_peered":"2024-09-16T10:18:33.820857+0000","last_unstale":"2024-09-16T10:19:58.498859+0000","last_undegraded":"2024-09-16T10:19:58.498859+0000","last_fullsized":"2024-09-16T10:19:58.498859+0000","mapping_epoch":165,"log_start":"182'2700","ondisk_log_start":"182'2700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456327168,"num_objects":6963,"num_object_clones":0,"num_object_copies":20889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6963,"num_write_kb":445632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":385,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:51.588520+0000","last_change":"2024-09-16T10:19:51.588520+0000","last_active":"2024-09-16T10:19:51.588520+0000","last_peered":"2024-09-16T10:19:51.588520+0000","last_clean":"2024-09-16T10:19:51.588520+0000","last_became_active":"2024-09-16T10:18:06.878175+0000","last_became_peered":"2024-09-16T10:18:06.878175+0000","last_unstale":"2024-09-16T10:19:51.588520+0000","last_undegraded":"2024-09-16T10:19:51.588520+0000","last_fullsized":"2024-09-16T10:19:51.588520+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:51.588464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.588464+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.008418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":331,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914559+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:19:50.914559+0000","last_peered":"2024-09-16T10:19:50.914559+0000","last_clean":"2024-09-16T10:19:50.914559+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:19:50.914559+0000","last_undegraded":"2024-09-16T10:19:50.914559+0000","last_fullsized":"2024-09-16T10:19:50.914559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.443985+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911742+0000","last_change":"2024-09-16T10:19:16.602946+0000","last_active":"2024-09-16T10:19:50.911742+0000","last_peered":"2024-09-16T10:19:50.911742+0000","last_clean":"2024-09-16T10:19:50.911742+0000","last_became_active":"2024-09-16T10:18:52.833059+0000","last_became_peered":"2024-09-16T10:18:52.833059+0000","last_unstale":"2024-09-16T10:19:50.911742+0000","last_undegraded":"2024-09-16T10:19:50.911742+0000","last_fullsized":"2024-09-16T10:19:50.911742+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.602895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.602895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:21.600988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":91,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.199140+0000","last_change":"2024-09-16T10:19:58.199140+0000","last_active":"2024-09-16T10:19:58.199140+0000","last_peered":"2024-09-16T10:19:58.199140+0000","last_clean":"2024-09-16T10:19:58.199140+0000","last_became_active":"2024-09-16T10:18:52.836346+0000","last_became_peered":"2024-09-16T10:18:52.836346+0000","last_unstale":"2024-09-16T10:19:58.199140+0000","last_undegraded":"2024-09-16T10:19:58.199140+0000","last_fullsized":"2024-09-16T10:19:58.199140+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:21.557184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":75,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.915294+0000","last_change":"2024-09-16T10:19:16.940617+0000","last_active":"2024-09-16T10:19:50.915294+0000","last_peered":"2024-09-16T10:19:50.915294+0000","last_clean":"2024-09-16T10:19:50.915294+0000","last_became_active":"2024-09-16T10:18:52.827678+0000","last_became_peered":"2024-09-16T10:18:52.827678+0000","last_unstale":"2024-09-16T10:19:50.915294+0000","last_undegraded":"2024-09-16T10:19:50.915294+0000","last_fullsized":"2024-09-16T10:19:50.915294+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.940551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.940551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.979377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":313,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917247+0000","last_change":"2024-09-16T10:18:37.611634+0000","last_active":"2024-09-16T10:19:50.917247+0000","last_peered":"2024-09-16T10:19:50.917247+0000","last_clean":"2024-09-16T10:19:50.917247+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:19:50.917247+0000","last_undegraded":"2024-09-16T10:19:50.917247+0000","last_fullsized":"2024-09-16T10:19:50.917247+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.611578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.611578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:01.244128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"202'6893","reported_seq":10527,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.401898+0000","last_change":"2024-09-16T10:19:57.299979+0000","last_active":"2024-09-16T10:19:58.401898+0000","last_peered":"2024-09-16T10:19:58.401898+0000","last_clean":"2024-09-16T10:19:58.401898+0000","last_became_active":"2024-09-16T10:19:13.300503+0000","last_became_peered":"2024-09-16T10:19:13.300503+0000","last_unstale":"2024-09-16T10:19:58.401898+0000","last_undegraded":"2024-09-16T10:19:58.401898+0000","last_fullsized":"2024-09-16T10:19:58.401898+0000","mapping_epoch":186,"log_start":"179'2500","ondisk_log_start":"179'2500","created":165,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6795","last_scrub_stamp":"2024-09-16T10:19:57.299850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.299850+0000","objects_scrubbed":6727,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:23.983417+0000","scrub_duration":1826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451739648,"num_objects":6893,"num_object_clones":0,"num_object_copies":20679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6893,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1730,"num_bytes_recovered":113377280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":110,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.912392+0000","last_change":"2024-09-16T10:18:07.544995+0000","last_active":"2024-09-16T10:19:50.912392+0000","last_peered":"2024-09-16T10:19:50.912392+0000","last_clean":"2024-09-16T10:19:50.912392+0000","last_became_active":"2024-09-16T10:18:06.877808+0000","last_became_peered":"2024-09-16T10:18:06.877808+0000","last_unstale":"2024-09-16T10:19:50.912392+0000","last_undegraded":"2024-09-16T10:19:50.912392+0000","last_fullsized":"2024-09-16T10:19:50.912392+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.544697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.544697+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.369068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":366,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914322+0000","last_change":"2024-09-16T10:19:19.231411+0000","last_active":"2024-09-16T10:19:50.914322+0000","last_peered":"2024-09-16T10:19:50.914322+0000","last_clean":"2024-09-16T10:19:50.914322+0000","last_became_active":"2024-09-16T10:18:52.837634+0000","last_became_peered":"2024-09-16T10:18:52.837634+0000","last_unstale":"2024-09-16T10:19:50.914322+0000","last_undegraded":"2024-09-16T10:19:50.914322+0000","last_fullsized":"2024-09-16T10:19:50.914322+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:19.231125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:19.231125+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.896769+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":332,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913766+0000","last_change":"2024-09-16T10:18:45.207140+0000","last_active":"2024-09-16T10:19:50.913766+0000","last_peered":"2024-09-16T10:19:50.913766+0000","last_clean":"2024-09-16T10:19:50.913766+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:19:50.913766+0000","last_undegraded":"2024-09-16T10:19:50.913766+0000","last_fullsized":"2024-09-16T10:19:50.913766+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:45.207071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.207071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:57.506006+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":703,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:56.500160+0000","last_change":"2024-09-16T10:16:05.559992+0000","last_active":"2024-09-16T10:19:56.500160+0000","last_peered":"2024-09-16T10:19:56.500160+0000","last_clean":"2024-09-16T10:19:56.500160+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:19:56.500160+0000","last_undegraded":"2024-09-16T10:19:56.500160+0000","last_fullsized":"2024-09-16T10:19:56.500160+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:16:05.559913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.559913+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:26.478610+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916873+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:19:50.916873+0000","last_peered":"2024-09-16T10:19:50.916873+0000","last_clean":"2024-09-16T10:19:50.916873+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:19:50.916873+0000","last_undegraded":"2024-09-16T10:19:50.916873+0000","last_fullsized":"2024-09-16T10:19:50.916873+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.497294+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":313,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.912983+0000","last_change":"2024-09-16T10:18:44.647162+0000","last_active":"2024-09-16T10:19:50.912983+0000","last_peered":"2024-09-16T10:19:50.912983+0000","last_clean":"2024-09-16T10:19:50.912983+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:19:50.912983+0000","last_undegraded":"2024-09-16T10:19:50.912983+0000","last_fullsized":"2024-09-16T10:19:50.912983+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:44.646771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:44.646771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:13.207875+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"202'6671","reported_seq":6828,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.169634+0000","last_change":"2024-09-16T10:19:56.367287+0000","last_active":"2024-09-16T10:19:58.169634+0000","last_peered":"2024-09-16T10:19:58.169634+0000","last_clean":"2024-09-16T10:19:58.169634+0000","last_became_active":"2024-09-16T10:18:33.825120+0000","last_became_peered":"2024-09-16T10:18:33.825120+0000","last_unstale":"2024-09-16T10:19:58.169634+0000","last_undegraded":"2024-09-16T10:19:58.169634+0000","last_fullsized":"2024-09-16T10:19:58.169634+0000","mapping_epoch":165,"log_start":"178'2000","ondisk_log_start":"178'2000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6534","last_scrub_stamp":"2024-09-16T10:19:56.367108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.367108+0000","objects_scrubbed":6469,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:20.198070+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437190656,"num_objects":6671,"num_object_clones":0,"num_object_copies":20013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":426944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":403,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:57.235043+0000","last_change":"2024-09-16T10:16:10.182532+0000","last_active":"2024-09-16T10:19:57.235043+0000","last_peered":"2024-09-16T10:19:57.235043+0000","last_clean":"2024-09-16T10:19:57.235043+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:19:57.235043+0000","last_undegraded":"2024-09-16T10:19:57.235043+0000","last_fullsized":"2024-09-16T10:19:57.235043+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:16:10.182468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:10.182468+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.211143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":289,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916976+0000","last_change":"2024-09-16T10:19:44.015480+0000","last_active":"2024-09-16T10:19:50.916976+0000","last_peered":"2024-09-16T10:19:50.916976+0000","last_clean":"2024-09-16T10:19:50.916976+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:19:50.916976+0000","last_undegraded":"2024-09-16T10:19:50.916976+0000","last_fullsized":"2024-09-16T10:19:50.916976+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.015430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.015430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.463886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":372,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916718+0000","last_change":"2024-09-16T10:18:05.620756+0000","last_active":"2024-09-16T10:19:50.916718+0000","last_peered":"2024-09-16T10:19:50.916718+0000","last_clean":"2024-09-16T10:19:50.916718+0000","last_became_active":"2024-09-16T10:18:04.861294+0000","last_became_peered":"2024-09-16T10:18:04.861294+0000","last_unstale":"2024-09-16T10:19:50.916718+0000","last_undegraded":"2024-09-16T10:19:50.916718+0000","last_fullsized":"2024-09-16T10:19:50.916718+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:05.620685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.620685+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917525+0000","last_change":"2024-09-16T10:18:52.827862+0000","last_active":"2024-09-16T10:19:50.917525+0000","last_peered":"2024-09-16T10:19:50.917525+0000","last_clean":"2024-09-16T10:19:50.917525+0000","last_became_active":"2024-09-16T10:18:52.827467+0000","last_became_peered":"2024-09-16T10:18:52.827467+0000","last_unstale":"2024-09-16T10:19:50.917525+0000","last_undegraded":"2024-09-16T10:19:50.917525+0000","last_fullsized":"2024-09-16T10:19:50.917525+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.575356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":304,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916670+0000","last_change":"2024-09-16T10:19:23.923755+0000","last_active":"2024-09-16T10:19:50.916670+0000","last_peered":"2024-09-16T10:19:50.916670+0000","last_clean":"2024-09-16T10:19:50.916670+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:19:50.916670+0000","last_undegraded":"2024-09-16T10:19:50.916670+0000","last_fullsized":"2024-09-16T10:19:50.916670+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.923690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.923690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.831083+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"202'6834","reported_seq":6919,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.113046+0000","last_change":"2024-09-16T10:18:33.825497+0000","last_active":"2024-09-16T10:19:58.113046+0000","last_peered":"2024-09-16T10:19:58.113046+0000","last_clean":"2024-09-16T10:19:58.113046+0000","last_became_active":"2024-09-16T10:18:33.824979+0000","last_became_peered":"2024-09-16T10:18:33.824979+0000","last_unstale":"2024-09-16T10:19:58.113046+0000","last_undegraded":"2024-09-16T10:19:58.113046+0000","last_fullsized":"2024-09-16T10:19:58.113046+0000","mapping_epoch":165,"log_start":"180'2400","ondisk_log_start":"180'2400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447873024,"num_objects":6834,"num_object_clones":0,"num_object_copies":20502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6834,"num_write_kb":437376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":349,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.921106+0000","last_change":"2024-09-16T10:19:09.316812+0000","last_active":"2024-09-16T10:19:50.921106+0000","last_peered":"2024-09-16T10:19:50.921106+0000","last_clean":"2024-09-16T10:19:50.921106+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:19:50.921106+0000","last_undegraded":"2024-09-16T10:19:50.921106+0000","last_fullsized":"2024-09-16T10:19:50.921106+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:06.903808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.903808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.730406+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":314,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919127+0000","last_change":"2024-09-16T10:19:20.834753+0000","last_active":"2024-09-16T10:19:50.919127+0000","last_peered":"2024-09-16T10:19:50.919127+0000","last_clean":"2024-09-16T10:19:50.919127+0000","last_became_active":"2024-09-16T10:18:52.835626+0000","last_became_peered":"2024-09-16T10:18:52.835626+0000","last_unstale":"2024-09-16T10:19:50.919127+0000","last_undegraded":"2024-09-16T10:19:50.919127+0000","last_fullsized":"2024-09-16T10:19:50.919127+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.834667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.834667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:23.948748+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916590+0000","last_change":"2024-09-16T10:19:18.014961+0000","last_active":"2024-09-16T10:19:50.916590+0000","last_peered":"2024-09-16T10:19:50.916590+0000","last_clean":"2024-09-16T10:19:50.916590+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:19:50.916590+0000","last_undegraded":"2024-09-16T10:19:50.916590+0000","last_fullsized":"2024-09-16T10:19:50.916590+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.014882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.014882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.450402+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":269,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:57.175078+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:19:57.175078+0000","last_peered":"2024-09-16T10:19:57.175078+0000","last_clean":"2024-09-16T10:19:57.175078+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:19:57.175078+0000","last_undegraded":"2024-09-16T10:19:57.175078+0000","last_fullsized":"2024-09-16T10:19:57.175078+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.607172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"202'6912","reported_seq":10450,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.366395+0000","last_change":"2024-09-16T10:19:48.403185+0000","last_active":"2024-09-16T10:19:58.366395+0000","last_peered":"2024-09-16T10:19:58.366395+0000","last_clean":"2024-09-16T10:19:58.366395+0000","last_became_active":"2024-09-16T10:19:15.290152+0000","last_became_peered":"2024-09-16T10:19:15.290152+0000","last_unstale":"2024-09-16T10:19:58.366395+0000","last_undegraded":"2024-09-16T10:19:58.366395+0000","last_fullsized":"2024-09-16T10:19:58.366395+0000","mapping_epoch":188,"log_start":"180'2500","ondisk_log_start":"180'2500","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'6091","last_scrub_stamp":"2024-09-16T10:19:48.403076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.403076+0000","objects_scrubbed":6009,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:13.478114+0000","scrub_duration":1968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452984832,"num_objects":6912,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":110428160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":128,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917785+0000","last_change":"2024-09-16T10:18:52.847306+0000","last_active":"2024-09-16T10:19:50.917785+0000","last_peered":"2024-09-16T10:19:50.917785+0000","last_clean":"2024-09-16T10:19:50.917785+0000","last_became_active":"2024-09-16T10:18:52.842646+0000","last_became_peered":"2024-09-16T10:18:52.842646+0000","last_unstale":"2024-09-16T10:19:50.917785+0000","last_undegraded":"2024-09-16T10:19:50.917785+0000","last_fullsized":"2024-09-16T10:19:50.917785+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:09.128070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.128070+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":356,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918724+0000","last_change":"2024-09-16T10:18:52.845218+0000","last_active":"2024-09-16T10:19:50.918724+0000","last_peered":"2024-09-16T10:19:50.918724+0000","last_clean":"2024-09-16T10:19:50.918724+0000","last_became_active":"2024-09-16T10:18:52.826328+0000","last_became_peered":"2024-09-16T10:18:52.826328+0000","last_unstale":"2024-09-16T10:19:50.918724+0000","last_undegraded":"2024-09-16T10:19:50.918724+0000","last_fullsized":"2024-09-16T10:19:50.918724+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:18:34.648463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.648463+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:02.630392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919341+0000","last_change":"2024-09-16T10:19:21.116145+0000","last_active":"2024-09-16T10:19:50.919341+0000","last_peered":"2024-09-16T10:19:50.919341+0000","last_clean":"2024-09-16T10:19:50.919341+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:19:50.919341+0000","last_undegraded":"2024-09-16T10:19:50.919341+0000","last_fullsized":"2024-09-16T10:19:50.919341+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:21.116098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:21.116098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.939990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":300,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913034+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:19:50.913034+0000","last_peered":"2024-09-16T10:19:50.913034+0000","last_clean":"2024-09-16T10:19:50.913034+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:19:50.913034+0000","last_undegraded":"2024-09-16T10:19:50.913034+0000","last_fullsized":"2024-09-16T10:19:50.913034+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:26.573061+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":68,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919424+0000","last_change":"2024-09-16T10:18:52.838759+0000","last_active":"2024-09-16T10:19:50.919424+0000","last_peered":"2024-09-16T10:19:50.919424+0000","last_clean":"2024-09-16T10:19:50.919424+0000","last_became_active":"2024-09-16T10:18:52.838070+0000","last_became_peered":"2024-09-16T10:18:52.838070+0000","last_unstale":"2024-09-16T10:19:50.919424+0000","last_undegraded":"2024-09-16T10:19:50.919424+0000","last_fullsized":"2024-09-16T10:19:50.919424+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:50.491234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.491234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.059410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"202'7123","reported_seq":7208,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.495247+0000","last_change":"2024-09-16T10:18:33.823010+0000","last_active":"2024-09-16T10:19:58.495247+0000","last_peered":"2024-09-16T10:19:58.495247+0000","last_clean":"2024-09-16T10:19:58.495247+0000","last_became_active":"2024-09-16T10:18:33.822701+0000","last_became_peered":"2024-09-16T10:18:33.822701+0000","last_unstale":"2024-09-16T10:19:58.495247+0000","last_undegraded":"2024-09-16T10:19:58.495247+0000","last_fullsized":"2024-09-16T10:19:58.495247+0000","mapping_epoch":165,"log_start":"182'2800","ondisk_log_start":"182'2800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466812928,"num_objects":7123,"num_object_clones":0,"num_object_copies":21369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":455872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":392,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:55.550754+0000","last_change":"2024-09-16T10:19:55.550754+0000","last_active":"2024-09-16T10:19:55.550754+0000","last_peered":"2024-09-16T10:19:55.550754+0000","last_clean":"2024-09-16T10:19:55.550754+0000","last_became_active":"2024-09-16T10:18:52.838011+0000","last_became_peered":"2024-09-16T10:18:52.838011+0000","last_unstale":"2024-09-16T10:19:55.550754+0000","last_undegraded":"2024-09-16T10:19:55.550754+0000","last_fullsized":"2024-09-16T10:19:55.550754+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:55.550670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.550670+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.441146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":318,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918507+0000","last_change":"2024-09-16T10:19:16.768415+0000","last_active":"2024-09-16T10:19:50.918507+0000","last_peered":"2024-09-16T10:19:50.918507+0000","last_clean":"2024-09-16T10:19:50.918507+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:19:50.918507+0000","last_undegraded":"2024-09-16T10:19:50.918507+0000","last_fullsized":"2024-09-16T10:19:50.918507+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:16.768062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.768062+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.827070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918667+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:19:50.918667+0000","last_peered":"2024-09-16T10:19:50.918667+0000","last_clean":"2024-09-16T10:19:50.918667+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:19:50.918667+0000","last_undegraded":"2024-09-16T10:19:50.918667+0000","last_fullsized":"2024-09-16T10:19:50.918667+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.020575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"175'9","reported_seq":119,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:52.234161+0000","last_change":"2024-09-16T10:19:15.342743+0000","last_active":"2024-09-16T10:19:52.234161+0000","last_peered":"2024-09-16T10:19:52.234161+0000","last_clean":"2024-09-16T10:19:52.234161+0000","last_became_active":"2024-09-16T10:18:52.825124+0000","last_became_peered":"2024-09-16T10:18:52.825124+0000","last_unstale":"2024-09-16T10:19:52.234161+0000","last_undegraded":"2024-09-16T10:19:52.234161+0000","last_fullsized":"2024-09-16T10:19:52.234161+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:19:15.342674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.342674+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.191331+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":310,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.920843+0000","last_change":"2024-09-16T10:18:52.834703+0000","last_active":"2024-09-16T10:19:50.920843+0000","last_peered":"2024-09-16T10:19:50.920843+0000","last_clean":"2024-09-16T10:19:50.920843+0000","last_became_active":"2024-09-16T10:18:52.834312+0000","last_became_peered":"2024-09-16T10:18:52.834312+0000","last_unstale":"2024-09-16T10:19:50.920843+0000","last_undegraded":"2024-09-16T10:19:50.920843+0000","last_fullsized":"2024-09-16T10:19:50.920843+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.963621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.963621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.773176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"202'6870","reported_seq":6955,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.393225+0000","last_change":"2024-09-16T10:18:33.823489+0000","last_active":"2024-09-16T10:19:58.393225+0000","last_peered":"2024-09-16T10:19:58.393225+0000","last_clean":"2024-09-16T10:19:58.393225+0000","last_became_active":"2024-09-16T10:18:33.823198+0000","last_became_peered":"2024-09-16T10:18:33.823198+0000","last_unstale":"2024-09-16T10:19:58.393225+0000","last_undegraded":"2024-09-16T10:19:58.393225+0000","last_fullsized":"2024-09-16T10:19:58.393225+0000","mapping_epoch":165,"log_start":"181'2500","ondisk_log_start":"181'2500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450232320,"num_objects":6870,"num_object_clones":0,"num_object_copies":20610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6870,"num_write_kb":439680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":327,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913490+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:19:50.913490+0000","last_peered":"2024-09-16T10:19:50.913490+0000","last_clean":"2024-09-16T10:19:50.913490+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:19:50.913490+0000","last_undegraded":"2024-09-16T10:19:50.913490+0000","last_fullsized":"2024-09-16T10:19:50.913490+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.191940+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.920635+0000","last_change":"2024-09-16T10:18:34.127253+0000","last_active":"2024-09-16T10:19:50.920635+0000","last_peered":"2024-09-16T10:19:50.920635+0000","last_clean":"2024-09-16T10:19:50.920635+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:19:50.920635+0000","last_undegraded":"2024-09-16T10:19:50.920635+0000","last_fullsized":"2024-09-16T10:19:50.920635+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:34.127213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.127213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:02.731145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":75,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923115+0000","last_change":"2024-09-16T10:19:18.748438+0000","last_active":"2024-09-16T10:19:50.923115+0000","last_peered":"2024-09-16T10:19:50.923115+0000","last_clean":"2024-09-16T10:19:50.923115+0000","last_became_active":"2024-09-16T10:18:52.835884+0000","last_became_peered":"2024-09-16T10:18:52.835884+0000","last_unstale":"2024-09-16T10:19:50.923115+0000","last_undegraded":"2024-09-16T10:19:50.923115+0000","last_fullsized":"2024-09-16T10:19:50.923115+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.670612+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913417+0000","last_change":"2024-09-16T10:18:52.849022+0000","last_active":"2024-09-16T10:19:50.913417+0000","last_peered":"2024-09-16T10:19:50.913417+0000","last_clean":"2024-09-16T10:19:50.913417+0000","last_became_active":"2024-09-16T10:18:52.839091+0000","last_became_peered":"2024-09-16T10:18:52.839091+0000","last_unstale":"2024-09-16T10:19:50.913417+0000","last_undegraded":"2024-09-16T10:19:50.913417+0000","last_fullsized":"2024-09-16T10:19:50.913417+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"202'6897","reported_seq":6981,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.552604+0000","last_change":"2024-09-16T10:18:33.894422+0000","last_active":"2024-09-16T10:19:58.552604+0000","last_peered":"2024-09-16T10:19:58.552604+0000","last_clean":"2024-09-16T10:19:58.552604+0000","last_became_active":"2024-09-16T10:18:33.893961+0000","last_became_peered":"2024-09-16T10:18:33.893961+0000","last_unstale":"2024-09-16T10:19:58.552604+0000","last_undegraded":"2024-09-16T10:19:58.552604+0000","last_fullsized":"2024-09-16T10:19:58.552604+0000","mapping_epoch":165,"log_start":"178'2200","ondisk_log_start":"178'2200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452001792,"num_objects":6897,"num_object_clones":0,"num_object_copies":20691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":441408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":350,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919094+0000","last_change":"2024-09-16T10:19:24.115166+0000","last_active":"2024-09-16T10:19:50.919094+0000","last_peered":"2024-09-16T10:19:50.919094+0000","last_clean":"2024-09-16T10:19:50.919094+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:19:50.919094+0000","last_undegraded":"2024-09-16T10:19:50.919094+0000","last_fullsized":"2024-09-16T10:19:50.919094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:24.115075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.115075+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.363776+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":308,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917504+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:19:50.917504+0000","last_peered":"2024-09-16T10:19:50.917504+0000","last_clean":"2024-09-16T10:19:50.917504+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:19:50.917504+0000","last_undegraded":"2024-09-16T10:19:50.917504+0000","last_fullsized":"2024-09-16T10:19:50.917504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.428291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":75,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913289+0000","last_change":"2024-09-16T10:19:17.184744+0000","last_active":"2024-09-16T10:19:50.913289+0000","last_peered":"2024-09-16T10:19:50.913289+0000","last_clean":"2024-09-16T10:19:50.913289+0000","last_became_active":"2024-09-16T10:18:52.831796+0000","last_became_peered":"2024-09-16T10:18:52.831796+0000","last_unstale":"2024-09-16T10:19:50.913289+0000","last_undegraded":"2024-09-16T10:19:50.913289+0000","last_fullsized":"2024-09-16T10:19:50.913289+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.184674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.184674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:20.369045+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":311,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916630+0000","last_change":"2024-09-16T10:18:52.841267+0000","last_active":"2024-09-16T10:19:50.916630+0000","last_peered":"2024-09-16T10:19:50.916630+0000","last_clean":"2024-09-16T10:19:50.916630+0000","last_became_active":"2024-09-16T10:18:52.840958+0000","last_became_peered":"2024-09-16T10:18:52.840958+0000","last_unstale":"2024-09-16T10:19:50.916630+0000","last_undegraded":"2024-09-16T10:19:50.916630+0000","last_fullsized":"2024-09-16T10:19:50.916630+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.495303+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"202'6886","reported_seq":7045,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.182335+0000","last_change":"2024-09-16T10:19:55.377394+0000","last_active":"2024-09-16T10:19:58.182335+0000","last_peered":"2024-09-16T10:19:58.182335+0000","last_clean":"2024-09-16T10:19:58.182335+0000","last_became_active":"2024-09-16T10:18:33.894026+0000","last_became_peered":"2024-09-16T10:18:33.894026+0000","last_unstale":"2024-09-16T10:19:58.182335+0000","last_undegraded":"2024-09-16T10:19:58.182335+0000","last_fullsized":"2024-09-16T10:19:58.182335+0000","mapping_epoch":165,"log_start":"178'2200","ondisk_log_start":"178'2200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6662","last_scrub_stamp":"2024-09-16T10:19:55.377245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.377245+0000","objects_scrubbed":6589,"log_size":4686,"log_dups_size":0,"ondisk_log_size":4686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.117653+0000","scrub_duration":2000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451280896,"num_objects":6886,"num_object_clones":0,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":440704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":334,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918993+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:19:50.918993+0000","last_peered":"2024-09-16T10:19:50.918993+0000","last_clean":"2024-09-16T10:19:50.918993+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:19:50.918993+0000","last_undegraded":"2024-09-16T10:19:50.918993+0000","last_fullsized":"2024-09-16T10:19:50.918993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.799332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917294+0000","last_change":"2024-09-16T10:19:17.357917+0000","last_active":"2024-09-16T10:19:50.917294+0000","last_peered":"2024-09-16T10:19:50.917294+0000","last_clean":"2024-09-16T10:19:50.917294+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:19:50.917294+0000","last_undegraded":"2024-09-16T10:19:50.917294+0000","last_fullsized":"2024-09-16T10:19:50.917294+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.357866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.357866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.135568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":67,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911992+0000","last_change":"2024-09-16T10:18:52.830267+0000","last_active":"2024-09-16T10:19:50.911992+0000","last_peered":"2024-09-16T10:19:50.911992+0000","last_clean":"2024-09-16T10:19:50.911992+0000","last_became_active":"2024-09-16T10:18:52.829845+0000","last_became_peered":"2024-09-16T10:18:52.829845+0000","last_unstale":"2024-09-16T10:19:50.911992+0000","last_undegraded":"2024-09-16T10:19:50.911992+0000","last_fullsized":"2024-09-16T10:19:50.911992+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:53.182365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918385+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:19:50.918385+0000","last_peered":"2024-09-16T10:19:50.918385+0000","last_clean":"2024-09-16T10:19:50.918385+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:19:50.918385+0000","last_undegraded":"2024-09-16T10:19:50.918385+0000","last_fullsized":"2024-09-16T10:19:50.918385+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"202'6641","reported_seq":10028,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:58.207670+0000","last_change":"2024-09-16T10:19:14.298104+0000","last_active":"2024-09-16T10:19:58.207670+0000","last_peered":"2024-09-16T10:19:58.207670+0000","last_clean":"2024-09-16T10:19:58.207670+0000","last_became_active":"2024-09-16T10:19:14.291692+0000","last_became_peered":"2024-09-16T10:19:14.291692+0000","last_unstale":"2024-09-16T10:19:58.207670+0000","last_undegraded":"2024-09-16T10:19:58.207670+0000","last_fullsized":"2024-09-16T10:19:58.207670+0000","mapping_epoch":187,"log_start":"178'2200","ondisk_log_start":"178'2200","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435224576,"num_objects":6641,"num_object_clones":0,"num_object_copies":19923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6641,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":321,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917951+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:19:50.917951+0000","last_peered":"2024-09-16T10:19:50.917951+0000","last_clean":"2024-09-16T10:19:50.917951+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:19:50.917951+0000","last_undegraded":"2024-09-16T10:19:50.917951+0000","last_fullsized":"2024-09-16T10:19:50.917951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":328,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918129+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:19:50.918129+0000","last_peered":"2024-09-16T10:19:50.918129+0000","last_clean":"2024-09-16T10:19:50.918129+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:19:50.918129+0000","last_undegraded":"2024-09-16T10:19:50.918129+0000","last_fullsized":"2024-09-16T10:19:50.918129+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":393,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:52.233482+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:19:52.233482+0000","last_peered":"2024-09-16T10:19:52.233482+0000","last_clean":"2024-09-16T10:19:52.233482+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:19:52.233482+0000","last_undegraded":"2024-09-16T10:19:52.233482+0000","last_fullsized":"2024-09-16T10:19:52.233482+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.564166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.915068+0000","last_change":"2024-09-16T10:19:20.941840+0000","last_active":"2024-09-16T10:19:50.915068+0000","last_peered":"2024-09-16T10:19:50.915068+0000","last_clean":"2024-09-16T10:19:50.915068+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:19:50.915068+0000","last_undegraded":"2024-09-16T10:19:50.915068+0000","last_fullsized":"2024-09-16T10:19:50.915068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.941753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.941753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.997567+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.915071+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:19:50.915071+0000","last_peered":"2024-09-16T10:19:50.915071+0000","last_clean":"2024-09-16T10:19:50.915071+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:19:50.915071+0000","last_undegraded":"2024-09-16T10:19:50.915071+0000","last_fullsized":"2024-09-16T10:19:50.915071+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.667031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":329,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919767+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:19:50.919767+0000","last_peered":"2024-09-16T10:19:50.919767+0000","last_clean":"2024-09-16T10:19:50.919767+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:19:50.919767+0000","last_undegraded":"2024-09-16T10:19:50.919767+0000","last_fullsized":"2024-09-16T10:19:50.919767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.155197+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":300,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917085+0000","last_change":"2024-09-16T10:19:15.972005+0000","last_active":"2024-09-16T10:19:50.917085+0000","last_peered":"2024-09-16T10:19:50.917085+0000","last_clean":"2024-09-16T10:19:50.917085+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:19:50.917085+0000","last_undegraded":"2024-09-16T10:19:50.917085+0000","last_fullsized":"2024-09-16T10:19:50.917085+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.971951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.971951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:28.073056+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":296,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917113+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:19:50.917113+0000","last_peered":"2024-09-16T10:19:50.917113+0000","last_clean":"2024-09-16T10:19:50.917113+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:19:50.917113+0000","last_undegraded":"2024-09-16T10:19:50.917113+0000","last_fullsized":"2024-09-16T10:19:50.917113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.610870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":335,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914355+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:19:50.914355+0000","last_peered":"2024-09-16T10:19:50.914355+0000","last_clean":"2024-09-16T10:19:50.914355+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:19:50.914355+0000","last_undegraded":"2024-09-16T10:19:50.914355+0000","last_fullsized":"2024-09-16T10:19:50.914355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918226+0000","last_change":"2024-09-16T10:19:22.431194+0000","last_active":"2024-09-16T10:19:50.918226+0000","last_peered":"2024-09-16T10:19:50.918226+0000","last_clean":"2024-09-16T10:19:50.918226+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:19:50.918226+0000","last_undegraded":"2024-09-16T10:19:50.918226+0000","last_fullsized":"2024-09-16T10:19:50.918226+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.431126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.431126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.492625+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":300,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.919642+0000","last_change":"2024-09-16T10:16:37.237971+0000","last_active":"2024-09-16T10:19:50.919642+0000","last_peered":"2024-09-16T10:19:50.919642+0000","last_clean":"2024-09-16T10:19:50.919642+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:19:50.919642+0000","last_undegraded":"2024-09-16T10:19:50.919642+0000","last_fullsized":"2024-09-16T10:19:50.919642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:37.237849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.237849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.908375+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":68,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918613+0000","last_change":"2024-09-16T10:18:52.823189+0000","last_active":"2024-09-16T10:19:50.918613+0000","last_peered":"2024-09-16T10:19:50.918613+0000","last_clean":"2024-09-16T10:19:50.918613+0000","last_became_active":"2024-09-16T10:18:52.822465+0000","last_became_peered":"2024-09-16T10:18:52.822465+0000","last_unstale":"2024-09-16T10:19:50.918613+0000","last_undegraded":"2024-09-16T10:19:50.918613+0000","last_fullsized":"2024-09-16T10:19:50.918613+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.568361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.568361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:55.849277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":328,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.922676+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:19:50.922676+0000","last_peered":"2024-09-16T10:19:50.922676+0000","last_clean":"2024-09-16T10:19:50.922676+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:19:50.922676+0000","last_undegraded":"2024-09-16T10:19:50.922676+0000","last_fullsized":"2024-09-16T10:19:50.922676+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.022112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913006+0000","last_change":"2024-09-16T10:18:03.120551+0000","last_active":"2024-09-16T10:19:50.913006+0000","last_peered":"2024-09-16T10:19:50.913006+0000","last_clean":"2024-09-16T10:19:50.913006+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:19:50.913006+0000","last_undegraded":"2024-09-16T10:19:50.913006+0000","last_fullsized":"2024-09-16T10:19:50.913006+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:03.120474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:03.120474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"175'9","reported_seq":412,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:52.232542+0000","last_change":"2024-09-16T10:18:52.852723+0000","last_active":"2024-09-16T10:19:52.232542+0000","last_peered":"2024-09-16T10:19:52.232542+0000","last_clean":"2024-09-16T10:19:52.232542+0000","last_became_active":"2024-09-16T10:18:52.845153+0000","last_became_peered":"2024-09-16T10:18:52.845153+0000","last_unstale":"2024-09-16T10:19:52.232542+0000","last_undegraded":"2024-09-16T10:19:52.232542+0000","last_fullsized":"2024-09-16T10:19:52.232542+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:38.926860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:38.926860+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:06.735603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":310,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.921086+0000","last_change":"2024-09-16T10:18:52.821349+0000","last_active":"2024-09-16T10:19:50.921086+0000","last_peered":"2024-09-16T10:19:50.921086+0000","last_clean":"2024-09-16T10:19:50.921086+0000","last_became_active":"2024-09-16T10:18:52.821050+0000","last_became_peered":"2024-09-16T10:18:52.821050+0000","last_unstale":"2024-09-16T10:19:50.921086+0000","last_undegraded":"2024-09-16T10:19:50.921086+0000","last_fullsized":"2024-09-16T10:19:50.921086+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.915613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.915613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.778010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":335,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917182+0000","last_change":"2024-09-16T10:19:45.045388+0000","last_active":"2024-09-16T10:19:50.917182+0000","last_peered":"2024-09-16T10:19:50.917182+0000","last_clean":"2024-09-16T10:19:50.917182+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:19:50.917182+0000","last_undegraded":"2024-09-16T10:19:50.917182+0000","last_fullsized":"2024-09-16T10:19:50.917182+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:45.045164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.045164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.833432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":323,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917810+0000","last_change":"2024-09-16T10:18:52.820977+0000","last_active":"2024-09-16T10:19:50.917810+0000","last_peered":"2024-09-16T10:19:50.917810+0000","last_clean":"2024-09-16T10:19:50.917810+0000","last_became_active":"2024-09-16T10:18:52.820693+0000","last_became_peered":"2024-09-16T10:18:52.820693+0000","last_unstale":"2024-09-16T10:19:50.917810+0000","last_undegraded":"2024-09-16T10:19:50.917810+0000","last_fullsized":"2024-09-16T10:19:50.917810+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":401,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:54.181149+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:19:54.181149+0000","last_peered":"2024-09-16T10:19:54.181149+0000","last_clean":"2024-09-16T10:19:54.181149+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:19:54.181149+0000","last_undegraded":"2024-09-16T10:19:54.181149+0000","last_fullsized":"2024-09-16T10:19:54.181149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.555917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":303,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913512+0000","last_change":"2024-09-16T10:18:09.148508+0000","last_active":"2024-09-16T10:19:50.913512+0000","last_peered":"2024-09-16T10:19:50.913512+0000","last_clean":"2024-09-16T10:19:50.913512+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:19:50.913512+0000","last_undegraded":"2024-09-16T10:19:50.913512+0000","last_fullsized":"2024-09-16T10:19:50.913512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:09.148447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.148447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":322,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.923647+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:19:50.923647+0000","last_peered":"2024-09-16T10:19:50.923647+0000","last_clean":"2024-09-16T10:19:50.923647+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:19:50.923647+0000","last_undegraded":"2024-09-16T10:19:50.923647+0000","last_fullsized":"2024-09-16T10:19:50.923647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.805984+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918739+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:19:50.918739+0000","last_peered":"2024-09-16T10:19:50.918739+0000","last_clean":"2024-09-16T10:19:50.918739+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:19:50.918739+0000","last_undegraded":"2024-09-16T10:19:50.918739+0000","last_fullsized":"2024-09-16T10:19:50.918739+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.756954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.912221+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:19:50.912221+0000","last_peered":"2024-09-16T10:19:50.912221+0000","last_clean":"2024-09-16T10:19:50.912221+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:19:50.912221+0000","last_undegraded":"2024-09-16T10:19:50.912221+0000","last_fullsized":"2024-09-16T10:19:50.912221+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:55.416985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":296,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916542+0000","last_change":"2024-09-16T10:19:17.803486+0000","last_active":"2024-09-16T10:19:50.916542+0000","last_peered":"2024-09-16T10:19:50.916542+0000","last_clean":"2024-09-16T10:19:50.916542+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:19:50.916542+0000","last_undegraded":"2024-09-16T10:19:50.916542+0000","last_fullsized":"2024-09-16T10:19:50.916542+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.803412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.803412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.806543+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":345,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913250+0000","last_change":"2024-09-16T10:19:18.621273+0000","last_active":"2024-09-16T10:19:50.913250+0000","last_peered":"2024-09-16T10:19:50.913250+0000","last_clean":"2024-09-16T10:19:50.913250+0000","last_became_active":"2024-09-16T10:18:52.832688+0000","last_became_peered":"2024-09-16T10:18:52.832688+0000","last_unstale":"2024-09-16T10:19:50.913250+0000","last_undegraded":"2024-09-16T10:19:50.913250+0000","last_fullsized":"2024-09-16T10:19:50.913250+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:18.620605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.620605+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:19.358298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917374+0000","last_change":"2024-09-16T10:19:47.608315+0000","last_active":"2024-09-16T10:19:50.917374+0000","last_peered":"2024-09-16T10:19:50.917374+0000","last_clean":"2024-09-16T10:19:50.917374+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:19:50.917374+0000","last_undegraded":"2024-09-16T10:19:50.917374+0000","last_fullsized":"2024-09-16T10:19:50.917374+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.608243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.608243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.564159+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":273,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918464+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:19:50.918464+0000","last_peered":"2024-09-16T10:19:50.918464+0000","last_clean":"2024-09-16T10:19:50.918464+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:19:50.918464+0000","last_undegraded":"2024-09-16T10:19:50.918464+0000","last_fullsized":"2024-09-16T10:19:50.918464+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.046746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":295,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911898+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:19:50.911898+0000","last_peered":"2024-09-16T10:19:50.911898+0000","last_clean":"2024-09-16T10:19:50.911898+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:19:50.911898+0000","last_undegraded":"2024-09-16T10:19:50.911898+0000","last_fullsized":"2024-09-16T10:19:50.911898+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":323,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916696+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:19:50.916696+0000","last_peered":"2024-09-16T10:19:50.916696+0000","last_clean":"2024-09-16T10:19:50.916696+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:19:50.916696+0000","last_undegraded":"2024-09-16T10:19:50.916696+0000","last_fullsized":"2024-09-16T10:19:50.916696+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:13.959478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":324,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917550+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:19:50.917550+0000","last_peered":"2024-09-16T10:19:50.917550+0000","last_clean":"2024-09-16T10:19:50.917550+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:19:50.917550+0000","last_undegraded":"2024-09-16T10:19:50.917550+0000","last_fullsized":"2024-09-16T10:19:50.917550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:19:59.132108+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916214+0000","last_change":"2024-09-16T10:19:45.534275+0000","last_active":"2024-09-16T10:19:50.916214+0000","last_peered":"2024-09-16T10:19:50.916214+0000","last_clean":"2024-09-16T10:19:50.916214+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:19:50.916214+0000","last_undegraded":"2024-09-16T10:19:50.916214+0000","last_fullsized":"2024-09-16T10:19:50.916214+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:45.533998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.533998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:47.342382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.918592+0000","last_change":"2024-09-16T10:19:47.129026+0000","last_active":"2024-09-16T10:19:50.918592+0000","last_peered":"2024-09-16T10:19:50.918592+0000","last_clean":"2024-09-16T10:19:50.918592+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:19:50.918592+0000","last_undegraded":"2024-09-16T10:19:50.918592+0000","last_fullsized":"2024-09-16T10:19:50.918592+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.128964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.128964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.566335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":344,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:56.200787+0000","last_change":"2024-09-16T10:19:56.200787+0000","last_active":"2024-09-16T10:19:56.200787+0000","last_peered":"2024-09-16T10:19:56.200787+0000","last_clean":"2024-09-16T10:19:56.200787+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:19:56.200787+0000","last_undegraded":"2024-09-16T10:19:56.200787+0000","last_fullsized":"2024-09-16T10:19:56.200787+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:19:56.200704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.200704+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.350145+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":321,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.913550+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:19:50.913550+0000","last_peered":"2024-09-16T10:19:50.913550+0000","last_clean":"2024-09-16T10:19:50.913550+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:19:50.913550+0000","last_undegraded":"2024-09-16T10:19:50.913550+0000","last_fullsized":"2024-09-16T10:19:50.913550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":315,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914975+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:19:50.914975+0000","last_peered":"2024-09-16T10:19:50.914975+0000","last_clean":"2024-09-16T10:19:50.914975+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:19:50.914975+0000","last_undegraded":"2024-09-16T10:19:50.914975+0000","last_fullsized":"2024-09-16T10:19:50.914975+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.226339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":74,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917095+0000","last_change":"2024-09-16T10:19:44.487481+0000","last_active":"2024-09-16T10:19:50.917095+0000","last_peered":"2024-09-16T10:19:50.917095+0000","last_clean":"2024-09-16T10:19:50.917095+0000","last_became_active":"2024-09-16T10:18:52.838002+0000","last_became_peered":"2024-09-16T10:18:52.838002+0000","last_unstale":"2024-09-16T10:19:50.917095+0000","last_undegraded":"2024-09-16T10:19:50.917095+0000","last_fullsized":"2024-09-16T10:19:50.917095+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.584073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":312,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.911374+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:19:50.911374+0000","last_peered":"2024-09-16T10:19:50.911374+0000","last_clean":"2024-09-16T10:19:50.911374+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:19:50.911374+0000","last_undegraded":"2024-09-16T10:19:50.911374+0000","last_fullsized":"2024-09-16T10:19:50.911374+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:43.126929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":316,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.917049+0000","last_change":"2024-09-16T10:19:20.738852+0000","last_active":"2024-09-16T10:19:50.917049+0000","last_peered":"2024-09-16T10:19:50.917049+0000","last_clean":"2024-09-16T10:19:50.917049+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:19:50.917049+0000","last_undegraded":"2024-09-16T10:19:50.917049+0000","last_fullsized":"2024-09-16T10:19:50.917049+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.738779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.738779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.219018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":347,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.916327+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:19:50.916327+0000","last_peered":"2024-09-16T10:19:50.916327+0000","last_clean":"2024-09-16T10:19:50.916327+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:19:50.916327+0000","last_undegraded":"2024-09-16T10:19:50.916327+0000","last_fullsized":"2024-09-16T10:19:50.916327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:09.457028+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":323,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-16T10:19:50.914180+0000","last_change":"2024-09-16T10:19:16.217856+0000","last_active":"2024-09-16T10:19:50.914180+0000","last_peered":"2024-09-16T10:19:50.914180+0000","last_clean":"2024-09-16T10:19:50.914180+0000","last_became_active":"2024-09-16T10:18:52.834424+0000","last_became_peered":"2024-09-16T10:18:52.834424+0000","last_unstale":"2024-09-16T10:19:50.914180+0000","last_undegraded":"2024-09-16T10:19:50.914180+0000","last_fullsized":"2024-09-16T10:19:50.914180+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.131801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7225868288,"num_objects":110258,"num_object_clones":0,"num_object_copies":330774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110258,"num_write_kb":7056512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10118,"num_bytes_recovered":663093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22223716352,"data_stored":22223716352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71758,"ondisk_log_size":71758,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234064,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2804204,"kb_used_data":2628636,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":90940948,"statfs":{"total":95995035648,"available":93123530752,"internally_reserved":0,"allocated":2691723264,"data_stored":2691329943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":179763658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528404,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3285164,"kb_used_data":3033244,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90459988,"statfs":{"total":95995035648,"available":92631027712,"internally_reserved":0,"allocated":3106041856,"data_stored":3105648449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790042,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4132912,"kb_used_data":3921248,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":89612240,"statfs":{"total":95995035648,"available":91762933760,"internally_reserved":0,"allocated":4015357952,"data_stored":4014961773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986271,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2785716,"kb_used_data":2610148,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":90959436,"statfs":{"total":95995035648,"available":93142462464,"internally_reserved":0,"allocated":2672791552,"data_stored":2672389982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215204,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3378764,"kb_used_data":3108796,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90366388,"statfs":{"total":95995035648,"available":92535181312,"internally_reserved":0,"allocated":3183407104,"data_stored":3182983197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":276429256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3332428,"kb_used_data":3076348,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":90412724,"statfs":{"total":95995035648,"available":92582629376,"internally_reserved":0,"allocated":3150180352,"data_stored":3149755056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2848252,"kb_used_data":2614380,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90896900,"statfs":{"total":95995035648,"available":93078425600,"internally_reserved":0,"allocated":2677125120,"data_stored":2676716337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2693857280,"data_stored":2693857280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3183476736,"data_stored":3183476736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":562823168,"data_stored":562823168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3193307136,"data_stored":3193307136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2681470976,"data_stored":2681470976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4061855744,"data_stored":4061855744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3134128128,"data_stored":3134128128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2712797184,"data_stored":2712797184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:20:04.259 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T10:20:04.259 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:20:04.260 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T10:20:04.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:20:04.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:03 smithi165 ceph-mon[32696]: pgmap v312: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 833 op/s 2024-09-16T10:20:04.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:04.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:04.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:04.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:04.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:04.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:04.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 83 16 117987 117971 88.8121 89.25 0.0146644 0.0112549 2024-09-16T10:20:04.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:04.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:04.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:04.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:04.773 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:04.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:04.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:20:04.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:04.879 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:04.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:04.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:04.927 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:05.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:05.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:05.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:05.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:05.189 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:05.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:05.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:05.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:05.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:05.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:05.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:05.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:05.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 84 16 119345 119329 88.7651 84.875 0.0108165 0.0112612 2024-09-16T10:20:05.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:05.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:05.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:05.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:05.778 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:05.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:05.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:05.808 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:20:05.880 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:05.937 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:05.938 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:05.943 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:05.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:05.976 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:05.976 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:06.009 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:06.009 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:06.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:06.053 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:06.054 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:06.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:06.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:06.101 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:06.101 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:06.102 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:06.102 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:06.102 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[29993]: osdmap e205: 8 total, 8 up, 7 in 2024-09-16T10:20:06.102 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[29993]: pgmap v314: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 839 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-16T10:20:06.102 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:06.102 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:06.103 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[39470]: osdmap e205: 8 total, 8 up, 7 in 2024-09-16T10:20:06.103 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:05 smithi063 ceph-mon[39470]: pgmap v314: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 839 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-16T10:20:06.142 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:06.142 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:06.172 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:06.172 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:06.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:06.211 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:06.212 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:06.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:06.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:05 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:05 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:05 smithi165 ceph-mon[32696]: osdmap e205: 8 total, 8 up, 7 in 2024-09-16T10:20:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:05 smithi165 ceph-mon[32696]: pgmap v314: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 839 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-16T10:20:06.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:06.364 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:06.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:06.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:06.483 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:06.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:06.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:06.510 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 85 16 120599 120583 88.6428 78.375 0.0129861 0.0112762 2024-09-16T10:20:06.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:20:06.569 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:20:06.594 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:06.634 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:06.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:06.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:06.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:06.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:06.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:06.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:06.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:07.004 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:07.040 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:07.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:07.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:07.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:07.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:07.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:20:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:20:07.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3418440755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:07.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:06 smithi063 ceph-mon[29993]: from='client.15492 v1:172.21.15.63:0/4178429721' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:07.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3418440755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:07.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:06 smithi063 ceph-mon[39470]: from='client.15492 v1:172.21.15.63:0/4178429721' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:07.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:07.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3418440755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:06 smithi165 ceph-mon[32696]: from='client.15492 v1:172.21.15.63:0/4178429721' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:07.325 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:07.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:07.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:07.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:07.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:07.511 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 86 16 122003 121987 88.6324 87.75 0.00649312 0.0112782 2024-09-16T10:20:07.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:07.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:07.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:07.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:07.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:07.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:07.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:07.874 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:07.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:07.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:08.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:08.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:08.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:08.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[29993]: osdmap e206: 8 total, 8 up, 7 in 2024-09-16T10:20:08.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[29993]: pgmap v316: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-16T10:20:08.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:08.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:08.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[39470]: osdmap e206: 8 total, 8 up, 7 in 2024-09-16T10:20:08.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:07 smithi063 ceph-mon[39470]: pgmap v316: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-16T10:20:08.245 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:08.278 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:07 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:07 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:07 smithi165 ceph-mon[32696]: osdmap e206: 8 total, 8 up, 7 in 2024-09-16T10:20:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:07 smithi165 ceph-mon[32696]: pgmap v316: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-16T10:20:08.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:08.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:08.511 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 87 16 123371 123355 88.5963 85.5 0.00482531 0.0112823 2024-09-16T10:20:08.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:08.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:08.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:08.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:08.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:08.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:20:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:20:08.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:08.872 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-16T10:20:05.148968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7626496539,"num_objects":116561,"num_object_clones":0,"num_object_copies":349683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116561,"num_whiteouts":0,"num_read":1101,"num_read_kb":853,"num_write":116798,"num_write_kb":7447213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10188,"num_bytes_recovered":663357700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72907,"ondisk_log_size":72907,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23750028,"kb_used_data":22161480,"kb_used_omap":10,"kb_used_meta":1588405,"kb_avail":632466036,"statfs":{"total":671965249536,"available":647645220864,"internally_reserved":0,"allocated":22693355520,"data_stored":22690437359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":1626526828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":400031744,"num_objects":6104,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6104,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.099444"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":335,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864149+0000","last_change":"2024-09-16T10:18:52.830567+0000","last_active":"2024-09-16T10:20:01.864149+0000","last_peered":"2024-09-16T10:20:01.864149+0000","last_clean":"2024-09-16T10:20:01.864149+0000","last_became_active":"2024-09-16T10:18:52.830260+0000","last_became_peered":"2024-09-16T10:18:52.830260+0000","last_unstale":"2024-09-16T10:20:01.864149+0000","last_undegraded":"2024-09-16T10:20:01.864149+0000","last_fullsized":"2024-09-16T10:20:01.864149+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:07.119072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.119072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":381,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.858910+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:20:01.858910+0000","last_peered":"2024-09-16T10:20:01.858910+0000","last_clean":"2024-09-16T10:20:01.858910+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:20:01.858910+0000","last_undegraded":"2024-09-16T10:20:01.858910+0000","last_fullsized":"2024-09-16T10:20:01.858910+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:25.723031+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856769+0000","last_change":"2024-09-16T10:18:40.752790+0000","last_active":"2024-09-16T10:20:01.856769+0000","last_peered":"2024-09-16T10:20:01.856769+0000","last_clean":"2024-09-16T10:20:01.856769+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:20:01.856769+0000","last_undegraded":"2024-09-16T10:20:01.856769+0000","last_fullsized":"2024-09-16T10:20:01.856769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:40.752739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:40.752739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.494552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":307,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859208+0000","last_change":"2024-09-16T10:19:23.784420+0000","last_active":"2024-09-16T10:20:01.859208+0000","last_peered":"2024-09-16T10:20:01.859208+0000","last_clean":"2024-09-16T10:20:01.859208+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:20:01.859208+0000","last_undegraded":"2024-09-16T10:20:01.859208+0000","last_fullsized":"2024-09-16T10:20:01.859208+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.784347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.784347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.734381+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":308,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854685+0000","last_change":"2024-09-16T10:18:34.944971+0000","last_active":"2024-09-16T10:20:01.854685+0000","last_peered":"2024-09-16T10:20:01.854685+0000","last_clean":"2024-09-16T10:20:01.854685+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:20:01.854685+0000","last_undegraded":"2024-09-16T10:20:01.854685+0000","last_fullsized":"2024-09-16T10:20:01.854685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:34.944915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.944915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":328,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855000+0000","last_change":"2024-09-16T10:18:52.836944+0000","last_active":"2024-09-16T10:20:01.855000+0000","last_peered":"2024-09-16T10:20:01.855000+0000","last_clean":"2024-09-16T10:20:01.855000+0000","last_became_active":"2024-09-16T10:18:52.836507+0000","last_became_peered":"2024-09-16T10:18:52.836507+0000","last_unstale":"2024-09-16T10:20:01.855000+0000","last_undegraded":"2024-09-16T10:20:01.855000+0000","last_fullsized":"2024-09-16T10:20:01.855000+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:13.018412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.018412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.942085+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":364,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855871+0000","last_change":"2024-09-16T10:18:52.855408+0000","last_active":"2024-09-16T10:20:01.855871+0000","last_peered":"2024-09-16T10:20:01.855871+0000","last_clean":"2024-09-16T10:20:01.855871+0000","last_became_active":"2024-09-16T10:18:52.835715+0000","last_became_peered":"2024-09-16T10:18:52.835715+0000","last_unstale":"2024-09-16T10:20:01.855871+0000","last_undegraded":"2024-09-16T10:20:01.855871+0000","last_fullsized":"2024-09-16T10:20:01.855871+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:16:31.217192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:31.217192+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.892839+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.858788+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:20:01.858788+0000","last_peered":"2024-09-16T10:20:01.858788+0000","last_clean":"2024-09-16T10:20:01.858788+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:20:01.858788+0000","last_undegraded":"2024-09-16T10:20:01.858788+0000","last_fullsized":"2024-09-16T10:20:01.858788+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":328,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.858047+0000","last_change":"2024-09-16T10:19:57.272422+0000","last_active":"2024-09-16T10:20:01.858047+0000","last_peered":"2024-09-16T10:20:01.858047+0000","last_clean":"2024-09-16T10:20:01.858047+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:20:01.858047+0000","last_undegraded":"2024-09-16T10:20:01.858047+0000","last_fullsized":"2024-09-16T10:20:01.858047+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.272362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.272362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:13.881596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859628+0000","last_change":"2024-09-16T10:16:32.833966+0000","last_active":"2024-09-16T10:20:01.859628+0000","last_peered":"2024-09-16T10:20:01.859628+0000","last_clean":"2024-09-16T10:20:01.859628+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:20:01.859628+0000","last_undegraded":"2024-09-16T10:20:01.859628+0000","last_fullsized":"2024-09-16T10:20:01.859628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:32.833895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:32.833895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.724646+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856289+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:20:01.856289+0000","last_peered":"2024-09-16T10:20:01.856289+0000","last_clean":"2024-09-16T10:20:01.856289+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:20:01.856289+0000","last_undegraded":"2024-09-16T10:20:01.856289+0000","last_fullsized":"2024-09-16T10:20:01.856289+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.206970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":307,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862720+0000","last_change":"2024-09-16T10:17:12.089641+0000","last_active":"2024-09-16T10:20:01.862720+0000","last_peered":"2024-09-16T10:20:01.862720+0000","last_clean":"2024-09-16T10:20:01.862720+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:20:01.862720+0000","last_undegraded":"2024-09-16T10:20:01.862720+0000","last_fullsized":"2024-09-16T10:20:01.862720+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:12.089577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:12.089577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.220934+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857231+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:20:01.857231+0000","last_peered":"2024-09-16T10:20:01.857231+0000","last_clean":"2024-09-16T10:20:01.857231+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:20:01.857231+0000","last_undegraded":"2024-09-16T10:20:01.857231+0000","last_fullsized":"2024-09-16T10:20:01.857231+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.984980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":325,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860400+0000","last_change":"2024-09-16T10:17:16.937447+0000","last_active":"2024-09-16T10:20:01.860400+0000","last_peered":"2024-09-16T10:20:01.860400+0000","last_clean":"2024-09-16T10:20:01.860400+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:20:01.860400+0000","last_undegraded":"2024-09-16T10:20:01.860400+0000","last_fullsized":"2024-09-16T10:20:01.860400+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:17:16.937353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:16.937353+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.713148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863520+0000","last_change":"2024-09-16T10:19:23.390542+0000","last_active":"2024-09-16T10:20:01.863520+0000","last_peered":"2024-09-16T10:20:01.863520+0000","last_clean":"2024-09-16T10:20:01.863520+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:20:01.863520+0000","last_undegraded":"2024-09-16T10:20:01.863520+0000","last_fullsized":"2024-09-16T10:20:01.863520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:49.356783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860416+0000","last_change":"2024-09-16T10:19:46.109382+0000","last_active":"2024-09-16T10:20:01.860416+0000","last_peered":"2024-09-16T10:20:01.860416+0000","last_clean":"2024-09-16T10:20:01.860416+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:20:01.860416+0000","last_undegraded":"2024-09-16T10:20:01.860416+0000","last_fullsized":"2024-09-16T10:20:01.860416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:46.109310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.109310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.462251+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860601+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:20:01.860601+0000","last_peered":"2024-09-16T10:20:01.860601+0000","last_clean":"2024-09-16T10:20:01.860601+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:20:01.860601+0000","last_undegraded":"2024-09-16T10:20:01.860601+0000","last_fullsized":"2024-09-16T10:20:01.860601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.949799+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":311,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863578+0000","last_change":"2024-09-16T10:19:15.182388+0000","last_active":"2024-09-16T10:20:01.863578+0000","last_peered":"2024-09-16T10:20:01.863578+0000","last_clean":"2024-09-16T10:20:01.863578+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:20:01.863578+0000","last_undegraded":"2024-09-16T10:20:01.863578+0000","last_fullsized":"2024-09-16T10:20:01.863578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.182335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.182335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.600446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":140,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856850+0000","last_change":"2024-09-16T10:19:14.579717+0000","last_active":"2024-09-16T10:20:01.856850+0000","last_peered":"2024-09-16T10:20:01.856850+0000","last_clean":"2024-09-16T10:20:01.856850+0000","last_became_active":"2024-09-16T10:18:52.841218+0000","last_became_peered":"2024-09-16T10:18:52.841218+0000","last_unstale":"2024-09-16T10:20:01.856850+0000","last_undegraded":"2024-09-16T10:20:01.856850+0000","last_fullsized":"2024-09-16T10:20:01.856850+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:14.868398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856673+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:20:01.856673+0000","last_peered":"2024-09-16T10:20:01.856673+0000","last_clean":"2024-09-16T10:20:01.856673+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:20:01.856673+0000","last_undegraded":"2024-09-16T10:20:01.856673+0000","last_fullsized":"2024-09-16T10:20:01.856673+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":265,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854500+0000","last_change":"2024-09-16T10:19:48.110737+0000","last_active":"2024-09-16T10:20:01.854500+0000","last_peered":"2024-09-16T10:20:01.854500+0000","last_clean":"2024-09-16T10:20:01.854500+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:20:01.854500+0000","last_undegraded":"2024-09-16T10:20:01.854500+0000","last_fullsized":"2024-09-16T10:20:01.854500+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:48.110674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.110674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.967276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859770+0000","last_change":"2024-09-16T10:19:22.803999+0000","last_active":"2024-09-16T10:20:01.859770+0000","last_peered":"2024-09-16T10:20:01.859770+0000","last_clean":"2024-09-16T10:20:01.859770+0000","last_became_active":"2024-09-16T10:18:52.834723+0000","last_became_peered":"2024-09-16T10:18:52.834723+0000","last_unstale":"2024-09-16T10:20:01.859770+0000","last_undegraded":"2024-09-16T10:20:01.859770+0000","last_fullsized":"2024-09-16T10:20:01.859770+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.204568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":366,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854442+0000","last_change":"2024-09-16T10:19:24.829364+0000","last_active":"2024-09-16T10:20:01.854442+0000","last_peered":"2024-09-16T10:20:01.854442+0000","last_clean":"2024-09-16T10:20:01.854442+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:20:01.854442+0000","last_undegraded":"2024-09-16T10:20:01.854442+0000","last_fullsized":"2024-09-16T10:20:01.854442+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:24.829255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.829255+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.222182+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":326,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860034+0000","last_change":"2024-09-16T10:18:52.839387+0000","last_active":"2024-09-16T10:20:01.860034+0000","last_peered":"2024-09-16T10:20:01.860034+0000","last_clean":"2024-09-16T10:20:01.860034+0000","last_became_active":"2024-09-16T10:18:52.838765+0000","last_became_peered":"2024-09-16T10:18:52.838765+0000","last_unstale":"2024-09-16T10:20:01.860034+0000","last_undegraded":"2024-09-16T10:20:01.860034+0000","last_fullsized":"2024-09-16T10:20:01.860034+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:05.795954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.795954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864312+0000","last_change":"2024-09-16T10:18:52.847158+0000","last_active":"2024-09-16T10:20:01.864312+0000","last_peered":"2024-09-16T10:20:01.864312+0000","last_clean":"2024-09-16T10:20:01.864312+0000","last_became_active":"2024-09-16T10:18:52.838789+0000","last_became_peered":"2024-09-16T10:18:52.838789+0000","last_unstale":"2024-09-16T10:20:01.864312+0000","last_undegraded":"2024-09-16T10:20:01.864312+0000","last_fullsized":"2024-09-16T10:20:01.864312+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.622143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":327,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856866+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:20:01.856866+0000","last_peered":"2024-09-16T10:20:01.856866+0000","last_clean":"2024-09-16T10:20:01.856866+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:20:01.856866+0000","last_undegraded":"2024-09-16T10:20:01.856866+0000","last_fullsized":"2024-09-16T10:20:01.856866+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.407870+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":339,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865061+0000","last_change":"2024-09-16T10:19:49.067027+0000","last_active":"2024-09-16T10:20:01.865061+0000","last_peered":"2024-09-16T10:20:01.865061+0000","last_clean":"2024-09-16T10:20:01.865061+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:20:01.865061+0000","last_undegraded":"2024-09-16T10:20:01.865061+0000","last_fullsized":"2024-09-16T10:20:01.865061+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:49.066953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:49.066953+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.060028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":409,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.867236+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:20:01.867236+0000","last_peered":"2024-09-16T10:20:01.867236+0000","last_clean":"2024-09-16T10:20:01.867236+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:20:01.867236+0000","last_undegraded":"2024-09-16T10:20:01.867236+0000","last_fullsized":"2024-09-16T10:20:01.867236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.301174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"204'7325","reported_seq":7414,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.397547+0000","last_change":"2024-09-16T10:18:33.825761+0000","last_active":"2024-09-16T10:20:03.397547+0000","last_peered":"2024-09-16T10:20:03.397547+0000","last_clean":"2024-09-16T10:20:03.397547+0000","last_became_active":"2024-09-16T10:18:33.825267+0000","last_became_peered":"2024-09-16T10:18:33.825267+0000","last_unstale":"2024-09-16T10:20:03.397547+0000","last_undegraded":"2024-09-16T10:20:03.397547+0000","last_fullsized":"2024-09-16T10:20:03.397547+0000","mapping_epoch":165,"log_start":"183'2900","ondisk_log_start":"183'2900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480051200,"num_objects":7325,"num_object_clones":0,"num_object_copies":21975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":468800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":121,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860058+0000","last_change":"2024-09-16T10:18:52.828048+0000","last_active":"2024-09-16T10:20:01.860058+0000","last_peered":"2024-09-16T10:20:01.860058+0000","last_clean":"2024-09-16T10:20:01.860058+0000","last_became_active":"2024-09-16T10:18:52.825707+0000","last_became_peered":"2024-09-16T10:18:52.825707+0000","last_unstale":"2024-09-16T10:20:01.860058+0000","last_undegraded":"2024-09-16T10:20:01.860058+0000","last_fullsized":"2024-09-16T10:20:01.860058+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.115716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.115716+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.054496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":341,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857617+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:20:01.857617+0000","last_peered":"2024-09-16T10:20:01.857617+0000","last_clean":"2024-09-16T10:20:01.857617+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:20:01.857617+0000","last_undegraded":"2024-09-16T10:20:01.857617+0000","last_fullsized":"2024-09-16T10:20:01.857617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.449585+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854262+0000","last_change":"2024-09-16T10:19:43.028508+0000","last_active":"2024-09-16T10:20:01.854262+0000","last_peered":"2024-09-16T10:20:01.854262+0000","last_clean":"2024-09-16T10:20:01.854262+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:20:01.854262+0000","last_undegraded":"2024-09-16T10:20:01.854262+0000","last_fullsized":"2024-09-16T10:20:01.854262+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:43.028436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:43.028436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.274639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":389,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860714+0000","last_change":"2024-09-16T10:18:36.623362+0000","last_active":"2024-09-16T10:20:01.860714+0000","last_peered":"2024-09-16T10:20:01.860714+0000","last_clean":"2024-09-16T10:20:01.860714+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:20:01.860714+0000","last_undegraded":"2024-09-16T10:20:01.860714+0000","last_fullsized":"2024-09-16T10:20:01.860714+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:36.623302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:36.623302+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":308,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857587+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:20:01.857587+0000","last_peered":"2024-09-16T10:20:01.857587+0000","last_clean":"2024-09-16T10:20:01.857587+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:20:01.857587+0000","last_undegraded":"2024-09-16T10:20:01.857587+0000","last_fullsized":"2024-09-16T10:20:01.857587+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.040726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"204'7266","reported_seq":7355,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.535253+0000","last_change":"2024-09-16T10:18:33.893627+0000","last_active":"2024-09-16T10:20:03.535253+0000","last_peered":"2024-09-16T10:20:03.535253+0000","last_clean":"2024-09-16T10:20:03.535253+0000","last_became_active":"2024-09-16T10:18:33.893281+0000","last_became_peered":"2024-09-16T10:18:33.893281+0000","last_unstale":"2024-09-16T10:20:03.535253+0000","last_undegraded":"2024-09-16T10:20:03.535253+0000","last_fullsized":"2024-09-16T10:20:03.535253+0000","mapping_epoch":165,"log_start":"180'2600","ondisk_log_start":"180'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476184576,"num_objects":7266,"num_object_clones":0,"num_object_copies":21798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":121,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864025+0000","last_change":"2024-09-16T10:18:52.824806+0000","last_active":"2024-09-16T10:20:01.864025+0000","last_peered":"2024-09-16T10:20:01.864025+0000","last_clean":"2024-09-16T10:20:01.864025+0000","last_became_active":"2024-09-16T10:18:52.824449+0000","last_became_peered":"2024-09-16T10:18:52.824449+0000","last_unstale":"2024-09-16T10:20:01.864025+0000","last_undegraded":"2024-09-16T10:20:01.864025+0000","last_fullsized":"2024-09-16T10:20:01.864025+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.752706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.752706+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":349,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854939+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:20:01.854939+0000","last_peered":"2024-09-16T10:20:01.854939+0000","last_clean":"2024-09-16T10:20:01.854939+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:20:01.854939+0000","last_undegraded":"2024-09-16T10:20:01.854939+0000","last_fullsized":"2024-09-16T10:20:01.854939+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":285,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854850+0000","last_change":"2024-09-16T10:19:50.114162+0000","last_active":"2024-09-16T10:20:01.854850+0000","last_peered":"2024-09-16T10:20:01.854850+0000","last_clean":"2024-09-16T10:20:01.854850+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:20:01.854850+0000","last_undegraded":"2024-09-16T10:20:01.854850+0000","last_fullsized":"2024-09-16T10:20:01.854850+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.114098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.114098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:16.082035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":58,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865848+0000","last_change":"2024-09-16T10:19:16.147371+0000","last_active":"2024-09-16T10:20:01.865848+0000","last_peered":"2024-09-16T10:20:01.865848+0000","last_clean":"2024-09-16T10:20:01.865848+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:20:01.865848+0000","last_undegraded":"2024-09-16T10:20:01.865848+0000","last_fullsized":"2024-09-16T10:20:01.865848+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.147304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.147304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:22.719288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859529+0000","last_change":"2024-09-16T10:19:44.711933+0000","last_active":"2024-09-16T10:20:01.859529+0000","last_peered":"2024-09-16T10:20:01.859529+0000","last_clean":"2024-09-16T10:20:01.859529+0000","last_became_active":"2024-09-16T10:18:52.835843+0000","last_became_peered":"2024-09-16T10:18:52.835843+0000","last_unstale":"2024-09-16T10:20:01.859529+0000","last_undegraded":"2024-09-16T10:20:01.859529+0000","last_fullsized":"2024-09-16T10:20:01.859529+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.727533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"204'7372","reported_seq":10969,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.204966+0000","last_change":"2024-09-16T10:19:15.299139+0000","last_active":"2024-09-16T10:20:03.204966+0000","last_peered":"2024-09-16T10:20:03.204966+0000","last_clean":"2024-09-16T10:20:03.204966+0000","last_became_active":"2024-09-16T10:19:15.293347+0000","last_became_peered":"2024-09-16T10:19:15.293347+0000","last_unstale":"2024-09-16T10:20:03.204966+0000","last_undegraded":"2024-09-16T10:20:03.204966+0000","last_fullsized":"2024-09-16T10:20:03.204966+0000","mapping_epoch":188,"log_start":"184'2900","ondisk_log_start":"184'2900","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483131392,"num_objects":7372,"num_object_clones":0,"num_object_copies":22116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7372,"num_write_kb":471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1748,"num_bytes_recovered":114556928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":119,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856075+0000","last_change":"2024-09-16T10:18:07.024979+0000","last_active":"2024-09-16T10:20:01.856075+0000","last_peered":"2024-09-16T10:20:01.856075+0000","last_clean":"2024-09-16T10:20:01.856075+0000","last_became_active":"2024-09-16T10:18:06.896380+0000","last_became_peered":"2024-09-16T10:18:06.896380+0000","last_unstale":"2024-09-16T10:20:01.856075+0000","last_undegraded":"2024-09-16T10:20:01.856075+0000","last_fullsized":"2024-09-16T10:20:01.856075+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.024848+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:01.162165+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":353,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859129+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:20:01.859129+0000","last_peered":"2024-09-16T10:20:01.859129+0000","last_clean":"2024-09-16T10:20:01.859129+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:20:01.859129+0000","last_undegraded":"2024-09-16T10:20:01.859129+0000","last_fullsized":"2024-09-16T10:20:01.859129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:15.756451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":328,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865304+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:20:01.865304+0000","last_peered":"2024-09-16T10:20:01.865304+0000","last_clean":"2024-09-16T10:20:01.865304+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:20:01.865304+0000","last_undegraded":"2024-09-16T10:20:01.865304+0000","last_fullsized":"2024-09-16T10:20:01.865304+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:14.162775+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":402,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854030+0000","last_change":"2024-09-16T10:18:04.023115+0000","last_active":"2024-09-16T10:20:01.854030+0000","last_peered":"2024-09-16T10:20:01.854030+0000","last_clean":"2024-09-16T10:20:01.854030+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:20:01.854030+0000","last_undegraded":"2024-09-16T10:20:01.854030+0000","last_fullsized":"2024-09-16T10:20:01.854030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:04.022579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.022579+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:00.209183+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":308,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862990+0000","last_change":"2024-09-16T10:18:52.824583+0000","last_active":"2024-09-16T10:20:01.862990+0000","last_peered":"2024-09-16T10:20:01.862990+0000","last_clean":"2024-09-16T10:20:01.862990+0000","last_became_active":"2024-09-16T10:18:52.824265+0000","last_became_peered":"2024-09-16T10:18:52.824265+0000","last_unstale":"2024-09-16T10:20:01.862990+0000","last_undegraded":"2024-09-16T10:20:01.862990+0000","last_fullsized":"2024-09-16T10:20:01.862990+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.491962+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"204'7172","reported_seq":10574,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.542865+0000","last_change":"2024-09-16T10:19:14.298863+0000","last_active":"2024-09-16T10:20:03.542865+0000","last_peered":"2024-09-16T10:20:03.542865+0000","last_clean":"2024-09-16T10:20:03.542865+0000","last_became_active":"2024-09-16T10:19:14.293461+0000","last_became_peered":"2024-09-16T10:19:14.293461+0000","last_unstale":"2024-09-16T10:20:03.542865+0000","last_undegraded":"2024-09-16T10:20:03.542865+0000","last_fullsized":"2024-09-16T10:20:03.542865+0000","mapping_epoch":187,"log_start":"181'2500","ondisk_log_start":"181'2500","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470024192,"num_objects":7172,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":459008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":190,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.866811+0000","last_change":"2024-09-16T10:18:07.192860+0000","last_active":"2024-09-16T10:20:01.866811+0000","last_peered":"2024-09-16T10:20:01.866811+0000","last_clean":"2024-09-16T10:20:01.866811+0000","last_became_active":"2024-09-16T10:18:06.896951+0000","last_became_peered":"2024-09-16T10:18:06.896951+0000","last_unstale":"2024-09-16T10:20:01.866811+0000","last_undegraded":"2024-09-16T10:20:01.866811+0000","last_fullsized":"2024-09-16T10:20:01.866811+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.192777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.192777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.243306+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":348,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856228+0000","last_change":"2024-09-16T10:19:54.487297+0000","last_active":"2024-09-16T10:20:01.856228+0000","last_peered":"2024-09-16T10:20:01.856228+0000","last_clean":"2024-09-16T10:20:01.856228+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:20:01.856228+0000","last_undegraded":"2024-09-16T10:20:01.856228+0000","last_fullsized":"2024-09-16T10:20:01.856228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:54.487219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.487219+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.245307+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":328,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.858121+0000","last_change":"2024-09-16T10:18:52.836576+0000","last_active":"2024-09-16T10:20:01.858121+0000","last_peered":"2024-09-16T10:20:01.858121+0000","last_clean":"2024-09-16T10:20:01.858121+0000","last_became_active":"2024-09-16T10:18:52.836260+0000","last_became_peered":"2024-09-16T10:18:52.836260+0000","last_unstale":"2024-09-16T10:20:01.858121+0000","last_undegraded":"2024-09-16T10:20:01.858121+0000","last_fullsized":"2024-09-16T10:20:01.858121+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.187181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.187181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.017380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":390,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863235+0000","last_change":"2024-09-16T10:15:14.055005+0000","last_active":"2024-09-16T10:20:01.863235+0000","last_peered":"2024-09-16T10:20:01.863235+0000","last_clean":"2024-09-16T10:20:01.863235+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:20:01.863235+0000","last_undegraded":"2024-09-16T10:20:01.863235+0000","last_fullsized":"2024-09-16T10:20:01.863235+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:15:14.054920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:14.054920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.246556+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":323,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862918+0000","last_change":"2024-09-16T10:19:50.044357+0000","last_active":"2024-09-16T10:20:01.862918+0000","last_peered":"2024-09-16T10:20:01.862918+0000","last_clean":"2024-09-16T10:20:01.862918+0000","last_became_active":"2024-09-16T10:18:52.842382+0000","last_became_peered":"2024-09-16T10:18:52.842382+0000","last_unstale":"2024-09-16T10:20:01.862918+0000","last_undegraded":"2024-09-16T10:20:01.862918+0000","last_fullsized":"2024-09-16T10:20:01.862918+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.044297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.044297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.918339+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"204'7274","reported_seq":10693,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.491271+0000","last_change":"2024-09-16T10:19:14.298487+0000","last_active":"2024-09-16T10:20:03.491271+0000","last_peered":"2024-09-16T10:20:03.491271+0000","last_clean":"2024-09-16T10:20:03.491271+0000","last_became_active":"2024-09-16T10:19:14.292726+0000","last_became_peered":"2024-09-16T10:19:14.292726+0000","last_unstale":"2024-09-16T10:20:03.491271+0000","last_undegraded":"2024-09-16T10:20:03.491271+0000","last_fullsized":"2024-09-16T10:20:03.491271+0000","mapping_epoch":187,"log_start":"185'3000","ondisk_log_start":"185'3000","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4274,"log_dups_size":0,"ondisk_log_size":4274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476708864,"num_objects":7274,"num_object_clones":0,"num_object_copies":21822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":465536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":108724224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":190,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.866321+0000","last_change":"2024-09-16T10:18:08.171074+0000","last_active":"2024-09-16T10:20:01.866321+0000","last_peered":"2024-09-16T10:20:01.866321+0000","last_clean":"2024-09-16T10:20:01.866321+0000","last_became_active":"2024-09-16T10:18:06.910799+0000","last_became_peered":"2024-09-16T10:18:06.910799+0000","last_unstale":"2024-09-16T10:20:01.866321+0000","last_undegraded":"2024-09-16T10:20:01.866321+0000","last_fullsized":"2024-09-16T10:20:01.866321+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:08.171021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:08.171021+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.281938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":360,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863351+0000","last_change":"2024-09-16T10:19:22.293172+0000","last_active":"2024-09-16T10:20:01.863351+0000","last_peered":"2024-09-16T10:20:01.863351+0000","last_clean":"2024-09-16T10:20:01.863351+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:20:01.863351+0000","last_undegraded":"2024-09-16T10:20:01.863351+0000","last_fullsized":"2024-09-16T10:20:01.863351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:22.293076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.293076+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.645714+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":328,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857157+0000","last_change":"2024-09-16T10:18:52.833335+0000","last_active":"2024-09-16T10:20:01.857157+0000","last_peered":"2024-09-16T10:20:01.857157+0000","last_clean":"2024-09-16T10:20:01.857157+0000","last_became_active":"2024-09-16T10:18:52.832749+0000","last_became_peered":"2024-09-16T10:18:52.832749+0000","last_unstale":"2024-09-16T10:20:01.857157+0000","last_undegraded":"2024-09-16T10:20:01.857157+0000","last_fullsized":"2024-09-16T10:20:01.857157+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:33.949798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.949798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.578873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865591+0000","last_change":"2024-09-16T10:19:55.433215+0000","last_active":"2024-09-16T10:20:01.865591+0000","last_peered":"2024-09-16T10:20:01.865591+0000","last_clean":"2024-09-16T10:20:01.865591+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:20:01.865591+0000","last_undegraded":"2024-09-16T10:20:01.865591+0000","last_fullsized":"2024-09-16T10:20:01.865591+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:55.433149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.433149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.272672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856675+0000","last_change":"2024-09-16T10:18:52.827865+0000","last_active":"2024-09-16T10:20:01.856675+0000","last_peered":"2024-09-16T10:20:01.856675+0000","last_clean":"2024-09-16T10:20:01.856675+0000","last_became_active":"2024-09-16T10:18:52.827511+0000","last_became_peered":"2024-09-16T10:18:52.827511+0000","last_unstale":"2024-09-16T10:20:01.856675+0000","last_undegraded":"2024-09-16T10:20:01.856675+0000","last_fullsized":"2024-09-16T10:20:01.856675+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:28.543646+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"204'7407","reported_seq":7496,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.232753+0000","last_change":"2024-09-16T10:18:33.893596+0000","last_active":"2024-09-16T10:20:03.232753+0000","last_peered":"2024-09-16T10:20:03.232753+0000","last_clean":"2024-09-16T10:20:03.232753+0000","last_became_active":"2024-09-16T10:18:33.893254+0000","last_became_peered":"2024-09-16T10:18:33.893254+0000","last_unstale":"2024-09-16T10:20:03.232753+0000","last_undegraded":"2024-09-16T10:20:03.232753+0000","last_fullsized":"2024-09-16T10:20:03.232753+0000","mapping_epoch":165,"log_start":"183'2900","ondisk_log_start":"183'2900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4507,"log_dups_size":0,"ondisk_log_size":4507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485425152,"num_objects":7407,"num_object_clones":0,"num_object_copies":22221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":121,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860185+0000","last_change":"2024-09-16T10:18:52.831104+0000","last_active":"2024-09-16T10:20:01.860185+0000","last_peered":"2024-09-16T10:20:01.860185+0000","last_clean":"2024-09-16T10:20:01.860185+0000","last_became_active":"2024-09-16T10:18:52.830618+0000","last_became_peered":"2024-09-16T10:18:52.830618+0000","last_unstale":"2024-09-16T10:20:01.860185+0000","last_undegraded":"2024-09-16T10:20:01.860185+0000","last_fullsized":"2024-09-16T10:20:01.860185+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.131122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.131122+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:41.045857+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"175'29","reported_seq":488,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857066+0000","last_change":"2024-09-16T10:18:52.860272+0000","last_active":"2024-09-16T10:20:01.857066+0000","last_peered":"2024-09-16T10:20:01.857066+0000","last_clean":"2024-09-16T10:20:01.857066+0000","last_became_active":"2024-09-16T10:18:52.841202+0000","last_became_peered":"2024-09-16T10:18:52.841202+0000","last_unstale":"2024-09-16T10:20:01.857066+0000","last_undegraded":"2024-09-16T10:20:01.857066+0000","last_fullsized":"2024-09-16T10:20:01.857066+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:18:45.654714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.654714+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:01.512318+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860864+0000","last_change":"2024-09-16T10:17:35.928758+0000","last_active":"2024-09-16T10:20:01.860864+0000","last_peered":"2024-09-16T10:20:01.860864+0000","last_clean":"2024-09-16T10:20:01.860864+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:20:01.860864+0000","last_undegraded":"2024-09-16T10:20:01.860864+0000","last_fullsized":"2024-09-16T10:20:01.860864+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.928701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.928701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:41.782908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860700+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:20:01.860700+0000","last_peered":"2024-09-16T10:20:01.860700+0000","last_clean":"2024-09-16T10:20:01.860700+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:20:01.860700+0000","last_undegraded":"2024-09-16T10:20:01.860700+0000","last_fullsized":"2024-09-16T10:20:01.860700+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.056983+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863981+0000","last_change":"2024-09-16T10:16:04.405318+0000","last_active":"2024-09-16T10:20:01.863981+0000","last_peered":"2024-09-16T10:20:01.863981+0000","last_clean":"2024-09-16T10:20:01.863981+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:20:01.863981+0000","last_undegraded":"2024-09-16T10:20:01.863981+0000","last_fullsized":"2024-09-16T10:20:01.863981+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:04.405243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:04.405243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.279683+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854753+0000","last_change":"2024-09-16T10:18:41.823802+0000","last_active":"2024-09-16T10:20:01.854753+0000","last_peered":"2024-09-16T10:20:01.854753+0000","last_clean":"2024-09-16T10:20:01.854753+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:20:01.854753+0000","last_undegraded":"2024-09-16T10:20:01.854753+0000","last_fullsized":"2024-09-16T10:20:01.854753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:41.823721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:41.823721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:10.790493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"204'7333","reported_seq":7422,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.499169+0000","last_change":"2024-09-16T10:18:33.821147+0000","last_active":"2024-09-16T10:20:03.499169+0000","last_peered":"2024-09-16T10:20:03.499169+0000","last_clean":"2024-09-16T10:20:03.499169+0000","last_became_active":"2024-09-16T10:18:33.820857+0000","last_became_peered":"2024-09-16T10:18:33.820857+0000","last_unstale":"2024-09-16T10:20:03.499169+0000","last_undegraded":"2024-09-16T10:20:03.499169+0000","last_fullsized":"2024-09-16T10:20:03.499169+0000","mapping_epoch":165,"log_start":"185'3100","ondisk_log_start":"185'3100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480575488,"num_objects":7333,"num_object_clones":0,"num_object_copies":21999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7333,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":389,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863420+0000","last_change":"2024-09-16T10:19:51.588520+0000","last_active":"2024-09-16T10:20:01.863420+0000","last_peered":"2024-09-16T10:20:01.863420+0000","last_clean":"2024-09-16T10:20:01.863420+0000","last_became_active":"2024-09-16T10:18:06.878175+0000","last_became_peered":"2024-09-16T10:18:06.878175+0000","last_unstale":"2024-09-16T10:20:01.863420+0000","last_undegraded":"2024-09-16T10:20:01.863420+0000","last_fullsized":"2024-09-16T10:20:01.863420+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:51.588464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.588464+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.008418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":335,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857707+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:20:01.857707+0000","last_peered":"2024-09-16T10:20:01.857707+0000","last_clean":"2024-09-16T10:20:01.857707+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:20:01.857707+0000","last_undegraded":"2024-09-16T10:20:01.857707+0000","last_fullsized":"2024-09-16T10:20:01.857707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.443985+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855656+0000","last_change":"2024-09-16T10:19:16.602946+0000","last_active":"2024-09-16T10:20:01.855656+0000","last_peered":"2024-09-16T10:20:01.855656+0000","last_clean":"2024-09-16T10:20:01.855656+0000","last_became_active":"2024-09-16T10:18:52.833059+0000","last_became_peered":"2024-09-16T10:18:52.833059+0000","last_unstale":"2024-09-16T10:20:01.855656+0000","last_undegraded":"2024-09-16T10:20:01.855656+0000","last_fullsized":"2024-09-16T10:20:01.855656+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.602895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.602895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:21.600988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":95,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855966+0000","last_change":"2024-09-16T10:19:58.199140+0000","last_active":"2024-09-16T10:20:01.855966+0000","last_peered":"2024-09-16T10:20:01.855966+0000","last_clean":"2024-09-16T10:20:01.855966+0000","last_became_active":"2024-09-16T10:18:52.836346+0000","last_became_peered":"2024-09-16T10:18:52.836346+0000","last_unstale":"2024-09-16T10:20:01.855966+0000","last_undegraded":"2024-09-16T10:20:01.855966+0000","last_fullsized":"2024-09-16T10:20:01.855966+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:21.557184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":79,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856537+0000","last_change":"2024-09-16T10:19:16.940617+0000","last_active":"2024-09-16T10:20:01.856537+0000","last_peered":"2024-09-16T10:20:01.856537+0000","last_clean":"2024-09-16T10:20:01.856537+0000","last_became_active":"2024-09-16T10:18:52.827678+0000","last_became_peered":"2024-09-16T10:18:52.827678+0000","last_unstale":"2024-09-16T10:20:01.856537+0000","last_undegraded":"2024-09-16T10:20:01.856537+0000","last_fullsized":"2024-09-16T10:20:01.856537+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.940551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.940551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.979377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.866646+0000","last_change":"2024-09-16T10:18:37.611634+0000","last_active":"2024-09-16T10:20:01.866646+0000","last_peered":"2024-09-16T10:20:01.866646+0000","last_clean":"2024-09-16T10:20:01.866646+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:20:01.866646+0000","last_undegraded":"2024-09-16T10:20:01.866646+0000","last_fullsized":"2024-09-16T10:20:01.866646+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.611578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.611578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"204'7294","reported_seq":10932,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.407050+0000","last_change":"2024-09-16T10:19:57.299979+0000","last_active":"2024-09-16T10:20:03.407050+0000","last_peered":"2024-09-16T10:20:03.407050+0000","last_clean":"2024-09-16T10:20:03.407050+0000","last_became_active":"2024-09-16T10:19:13.300503+0000","last_became_peered":"2024-09-16T10:19:13.300503+0000","last_unstale":"2024-09-16T10:20:03.407050+0000","last_undegraded":"2024-09-16T10:20:03.407050+0000","last_fullsized":"2024-09-16T10:20:03.407050+0000","mapping_epoch":186,"log_start":"183'2900","ondisk_log_start":"183'2900","created":165,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6795","last_scrub_stamp":"2024-09-16T10:19:57.299850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.299850+0000","objects_scrubbed":6727,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:23.983417+0000","scrub_duration":1826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478019584,"num_objects":7294,"num_object_clones":0,"num_object_copies":21882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7294,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1730,"num_bytes_recovered":113377280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":114,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857661+0000","last_change":"2024-09-16T10:18:07.544995+0000","last_active":"2024-09-16T10:20:01.857661+0000","last_peered":"2024-09-16T10:20:01.857661+0000","last_clean":"2024-09-16T10:20:01.857661+0000","last_became_active":"2024-09-16T10:18:06.877808+0000","last_became_peered":"2024-09-16T10:18:06.877808+0000","last_unstale":"2024-09-16T10:20:01.857661+0000","last_undegraded":"2024-09-16T10:20:01.857661+0000","last_fullsized":"2024-09-16T10:20:01.857661+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.544697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.544697+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.369068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":370,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865144+0000","last_change":"2024-09-16T10:19:19.231411+0000","last_active":"2024-09-16T10:20:01.865144+0000","last_peered":"2024-09-16T10:20:01.865144+0000","last_clean":"2024-09-16T10:20:01.865144+0000","last_became_active":"2024-09-16T10:18:52.837634+0000","last_became_peered":"2024-09-16T10:18:52.837634+0000","last_unstale":"2024-09-16T10:20:01.865144+0000","last_undegraded":"2024-09-16T10:20:01.865144+0000","last_fullsized":"2024-09-16T10:20:01.865144+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:19.231125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:19.231125+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.896769+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":336,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863067+0000","last_change":"2024-09-16T10:18:45.207140+0000","last_active":"2024-09-16T10:20:01.863067+0000","last_peered":"2024-09-16T10:20:01.863067+0000","last_clean":"2024-09-16T10:20:01.863067+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:20:01.863067+0000","last_undegraded":"2024-09-16T10:20:01.863067+0000","last_fullsized":"2024-09-16T10:20:01.863067+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:45.207071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.207071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":712,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862776+0000","last_change":"2024-09-16T10:16:05.559992+0000","last_active":"2024-09-16T10:20:01.862776+0000","last_peered":"2024-09-16T10:20:01.862776+0000","last_clean":"2024-09-16T10:20:01.862776+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:20:01.862776+0000","last_undegraded":"2024-09-16T10:20:01.862776+0000","last_fullsized":"2024-09-16T10:20:01.862776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:16:05.559913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:16:05.559913+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:26.478610+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857238+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:20:01.857238+0000","last_peered":"2024-09-16T10:20:01.857238+0000","last_clean":"2024-09-16T10:20:01.857238+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:20:01.857238+0000","last_undegraded":"2024-09-16T10:20:01.857238+0000","last_fullsized":"2024-09-16T10:20:01.857238+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.497294+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856653+0000","last_change":"2024-09-16T10:18:44.647162+0000","last_active":"2024-09-16T10:20:01.856653+0000","last_peered":"2024-09-16T10:20:01.856653+0000","last_clean":"2024-09-16T10:20:01.856653+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:20:01.856653+0000","last_undegraded":"2024-09-16T10:20:01.856653+0000","last_fullsized":"2024-09-16T10:20:01.856653+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:44.646771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:44.646771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:13.207875+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"204'7037","reported_seq":7198,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.189290+0000","last_change":"2024-09-16T10:19:56.367287+0000","last_active":"2024-09-16T10:20:03.189290+0000","last_peered":"2024-09-16T10:20:03.189290+0000","last_clean":"2024-09-16T10:20:03.189290+0000","last_became_active":"2024-09-16T10:18:33.825120+0000","last_became_peered":"2024-09-16T10:18:33.825120+0000","last_unstale":"2024-09-16T10:20:03.189290+0000","last_undegraded":"2024-09-16T10:20:03.189290+0000","last_fullsized":"2024-09-16T10:20:03.189290+0000","mapping_epoch":165,"log_start":"179'2400","ondisk_log_start":"179'2400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6534","last_scrub_stamp":"2024-09-16T10:19:56.367108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.367108+0000","objects_scrubbed":6469,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:20.198070+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461176832,"num_objects":7037,"num_object_clones":0,"num_object_copies":21111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":408,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.867272+0000","last_change":"2024-09-16T10:16:10.182532+0000","last_active":"2024-09-16T10:20:01.867272+0000","last_peered":"2024-09-16T10:20:01.867272+0000","last_clean":"2024-09-16T10:20:01.867272+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:20:01.867272+0000","last_undegraded":"2024-09-16T10:20:01.867272+0000","last_fullsized":"2024-09-16T10:20:01.867272+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:16:10.182468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:10.182468+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.211143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":293,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854595+0000","last_change":"2024-09-16T10:19:44.015480+0000","last_active":"2024-09-16T10:20:01.854595+0000","last_peered":"2024-09-16T10:20:01.854595+0000","last_clean":"2024-09-16T10:20:01.854595+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:20:01.854595+0000","last_undegraded":"2024-09-16T10:20:01.854595+0000","last_fullsized":"2024-09-16T10:20:01.854595+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.015430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.015430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.463886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":376,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862904+0000","last_change":"2024-09-16T10:18:05.620756+0000","last_active":"2024-09-16T10:20:01.862904+0000","last_peered":"2024-09-16T10:20:01.862904+0000","last_clean":"2024-09-16T10:20:01.862904+0000","last_became_active":"2024-09-16T10:18:04.861294+0000","last_became_peered":"2024-09-16T10:18:04.861294+0000","last_unstale":"2024-09-16T10:20:01.862904+0000","last_undegraded":"2024-09-16T10:20:01.862904+0000","last_fullsized":"2024-09-16T10:20:01.862904+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:05.620685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.620685+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864525+0000","last_change":"2024-09-16T10:18:52.827862+0000","last_active":"2024-09-16T10:20:01.864525+0000","last_peered":"2024-09-16T10:20:01.864525+0000","last_clean":"2024-09-16T10:20:01.864525+0000","last_became_active":"2024-09-16T10:18:52.827467+0000","last_became_peered":"2024-09-16T10:18:52.827467+0000","last_unstale":"2024-09-16T10:20:01.864525+0000","last_undegraded":"2024-09-16T10:20:01.864525+0000","last_fullsized":"2024-09-16T10:20:01.864525+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.575356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":308,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857044+0000","last_change":"2024-09-16T10:19:23.923755+0000","last_active":"2024-09-16T10:20:01.857044+0000","last_peered":"2024-09-16T10:20:01.857044+0000","last_clean":"2024-09-16T10:20:01.857044+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:20:01.857044+0000","last_undegraded":"2024-09-16T10:20:01.857044+0000","last_fullsized":"2024-09-16T10:20:01.857044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.923690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.923690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.831083+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"204'7214","reported_seq":7303,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.121590+0000","last_change":"2024-09-16T10:18:33.825497+0000","last_active":"2024-09-16T10:20:03.121590+0000","last_peered":"2024-09-16T10:20:03.121590+0000","last_clean":"2024-09-16T10:20:03.121590+0000","last_became_active":"2024-09-16T10:18:33.824979+0000","last_became_peered":"2024-09-16T10:18:33.824979+0000","last_unstale":"2024-09-16T10:20:03.121590+0000","last_undegraded":"2024-09-16T10:20:03.121590+0000","last_fullsized":"2024-09-16T10:20:03.121590+0000","mapping_epoch":165,"log_start":"183'2800","ondisk_log_start":"183'2800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472776704,"num_objects":7214,"num_object_clones":0,"num_object_copies":21642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":461696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":353,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860760+0000","last_change":"2024-09-16T10:19:09.316812+0000","last_active":"2024-09-16T10:20:01.860760+0000","last_peered":"2024-09-16T10:20:01.860760+0000","last_clean":"2024-09-16T10:20:01.860760+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:20:01.860760+0000","last_undegraded":"2024-09-16T10:20:01.860760+0000","last_fullsized":"2024-09-16T10:20:01.860760+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:06.903808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:06.903808+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.730406+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859530+0000","last_change":"2024-09-16T10:19:20.834753+0000","last_active":"2024-09-16T10:20:01.859530+0000","last_peered":"2024-09-16T10:20:01.859530+0000","last_clean":"2024-09-16T10:20:01.859530+0000","last_became_active":"2024-09-16T10:18:52.835626+0000","last_became_peered":"2024-09-16T10:18:52.835626+0000","last_unstale":"2024-09-16T10:20:01.859530+0000","last_undegraded":"2024-09-16T10:20:01.859530+0000","last_fullsized":"2024-09-16T10:20:01.859530+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.834667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.834667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:23.948748+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856967+0000","last_change":"2024-09-16T10:19:18.014961+0000","last_active":"2024-09-16T10:20:01.856967+0000","last_peered":"2024-09-16T10:20:01.856967+0000","last_clean":"2024-09-16T10:20:01.856967+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:20:01.856967+0000","last_undegraded":"2024-09-16T10:20:01.856967+0000","last_fullsized":"2024-09-16T10:20:01.856967+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.014882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.014882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.450402+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":273,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856171+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:20:01.856171+0000","last_peered":"2024-09-16T10:20:01.856171+0000","last_clean":"2024-09-16T10:20:01.856171+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:20:01.856171+0000","last_undegraded":"2024-09-16T10:20:01.856171+0000","last_fullsized":"2024-09-16T10:20:01.856171+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.607172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"204'7281","reported_seq":10823,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.400159+0000","last_change":"2024-09-16T10:19:48.403185+0000","last_active":"2024-09-16T10:20:03.400159+0000","last_peered":"2024-09-16T10:20:03.400159+0000","last_clean":"2024-09-16T10:20:03.400159+0000","last_became_active":"2024-09-16T10:19:15.290152+0000","last_became_peered":"2024-09-16T10:19:15.290152+0000","last_unstale":"2024-09-16T10:20:03.400159+0000","last_undegraded":"2024-09-16T10:20:03.400159+0000","last_fullsized":"2024-09-16T10:20:03.400159+0000","mapping_epoch":188,"log_start":"184'2900","ondisk_log_start":"184'2900","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'6091","last_scrub_stamp":"2024-09-16T10:19:48.403076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.403076+0000","objects_scrubbed":6009,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:13.478114+0000","scrub_duration":1968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477167616,"num_objects":7281,"num_object_clones":0,"num_object_copies":21843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7281,"num_write_kb":465984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":110428160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":132,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856470+0000","last_change":"2024-09-16T10:18:52.847306+0000","last_active":"2024-09-16T10:20:01.856470+0000","last_peered":"2024-09-16T10:20:01.856470+0000","last_clean":"2024-09-16T10:20:01.856470+0000","last_became_active":"2024-09-16T10:18:52.842646+0000","last_became_peered":"2024-09-16T10:18:52.842646+0000","last_unstale":"2024-09-16T10:20:01.856470+0000","last_undegraded":"2024-09-16T10:20:01.856470+0000","last_fullsized":"2024-09-16T10:20:01.856470+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:09.128070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.128070+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.268836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":360,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863788+0000","last_change":"2024-09-16T10:18:52.845218+0000","last_active":"2024-09-16T10:20:01.863788+0000","last_peered":"2024-09-16T10:20:01.863788+0000","last_clean":"2024-09-16T10:20:01.863788+0000","last_became_active":"2024-09-16T10:18:52.826328+0000","last_became_peered":"2024-09-16T10:18:52.826328+0000","last_unstale":"2024-09-16T10:20:01.863788+0000","last_undegraded":"2024-09-16T10:20:01.863788+0000","last_fullsized":"2024-09-16T10:20:01.863788+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:18:34.648463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.648463+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:02.630392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865783+0000","last_change":"2024-09-16T10:19:21.116145+0000","last_active":"2024-09-16T10:20:01.865783+0000","last_peered":"2024-09-16T10:20:01.865783+0000","last_clean":"2024-09-16T10:20:01.865783+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:20:01.865783+0000","last_undegraded":"2024-09-16T10:20:01.865783+0000","last_fullsized":"2024-09-16T10:20:01.865783+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:21.116098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:21.116098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.939990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856747+0000","last_change":"2024-09-16T10:15:56.092136+0000","last_active":"2024-09-16T10:20:01.856747+0000","last_peered":"2024-09-16T10:20:01.856747+0000","last_clean":"2024-09-16T10:20:01.856747+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:20:01.856747+0000","last_undegraded":"2024-09-16T10:20:01.856747+0000","last_fullsized":"2024-09-16T10:20:01.856747+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:56.091748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:15:56.091748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:26.573061+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":72,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854406+0000","last_change":"2024-09-16T10:18:52.838759+0000","last_active":"2024-09-16T10:20:01.854406+0000","last_peered":"2024-09-16T10:20:01.854406+0000","last_clean":"2024-09-16T10:20:01.854406+0000","last_became_active":"2024-09-16T10:18:52.838070+0000","last_became_peered":"2024-09-16T10:18:52.838070+0000","last_unstale":"2024-09-16T10:20:01.854406+0000","last_undegraded":"2024-09-16T10:20:01.854406+0000","last_fullsized":"2024-09-16T10:20:01.854406+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:50.491234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.491234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.059410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"204'7518","reported_seq":7607,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.509019+0000","last_change":"2024-09-16T10:18:33.823010+0000","last_active":"2024-09-16T10:20:03.509019+0000","last_peered":"2024-09-16T10:20:03.509019+0000","last_clean":"2024-09-16T10:20:03.509019+0000","last_became_active":"2024-09-16T10:18:33.822701+0000","last_became_peered":"2024-09-16T10:18:33.822701+0000","last_unstale":"2024-09-16T10:20:03.509019+0000","last_undegraded":"2024-09-16T10:20:03.509019+0000","last_fullsized":"2024-09-16T10:20:03.509019+0000","mapping_epoch":165,"log_start":"185'3200","ondisk_log_start":"185'3200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4318,"log_dups_size":0,"ondisk_log_size":4318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492699648,"num_objects":7518,"num_object_clones":0,"num_object_copies":22554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7518,"num_write_kb":481152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":396,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862991+0000","last_change":"2024-09-16T10:19:55.550754+0000","last_active":"2024-09-16T10:20:01.862991+0000","last_peered":"2024-09-16T10:20:01.862991+0000","last_clean":"2024-09-16T10:20:01.862991+0000","last_became_active":"2024-09-16T10:18:52.838011+0000","last_became_peered":"2024-09-16T10:18:52.838011+0000","last_unstale":"2024-09-16T10:20:01.862991+0000","last_undegraded":"2024-09-16T10:20:01.862991+0000","last_fullsized":"2024-09-16T10:20:01.862991+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:55.550670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.550670+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.441146+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":322,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860113+0000","last_change":"2024-09-16T10:19:16.768415+0000","last_active":"2024-09-16T10:20:01.860113+0000","last_peered":"2024-09-16T10:20:01.860113+0000","last_clean":"2024-09-16T10:20:01.860113+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:20:01.860113+0000","last_undegraded":"2024-09-16T10:20:01.860113+0000","last_fullsized":"2024-09-16T10:20:01.860113+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:16.768062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.768062+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.827070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855405+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:20:01.855405+0000","last_peered":"2024-09-16T10:20:01.855405+0000","last_clean":"2024-09-16T10:20:01.855405+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:20:01.855405+0000","last_undegraded":"2024-09-16T10:20:01.855405+0000","last_fullsized":"2024-09-16T10:20:01.855405+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.020575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"175'9","reported_seq":124,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.867950+0000","last_change":"2024-09-16T10:19:15.342743+0000","last_active":"2024-09-16T10:20:01.867950+0000","last_peered":"2024-09-16T10:20:01.867950+0000","last_clean":"2024-09-16T10:20:01.867950+0000","last_became_active":"2024-09-16T10:18:52.825124+0000","last_became_peered":"2024-09-16T10:18:52.825124+0000","last_unstale":"2024-09-16T10:20:01.867950+0000","last_undegraded":"2024-09-16T10:20:01.867950+0000","last_fullsized":"2024-09-16T10:20:01.867950+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:19:15.342674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.342674+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:30.191331+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":314,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860522+0000","last_change":"2024-09-16T10:18:52.834703+0000","last_active":"2024-09-16T10:20:01.860522+0000","last_peered":"2024-09-16T10:20:01.860522+0000","last_clean":"2024-09-16T10:20:01.860522+0000","last_became_active":"2024-09-16T10:18:52.834312+0000","last_became_peered":"2024-09-16T10:18:52.834312+0000","last_unstale":"2024-09-16T10:20:01.860522+0000","last_undegraded":"2024-09-16T10:20:01.860522+0000","last_fullsized":"2024-09-16T10:20:01.860522+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.963621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.963621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.773176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"204'7307","reported_seq":7413,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.378185+0000","last_change":"2024-09-16T10:19:59.077503+0000","last_active":"2024-09-16T10:20:03.378185+0000","last_peered":"2024-09-16T10:20:03.378185+0000","last_clean":"2024-09-16T10:20:03.378185+0000","last_became_active":"2024-09-16T10:18:33.823198+0000","last_became_peered":"2024-09-16T10:18:33.823198+0000","last_unstale":"2024-09-16T10:20:03.378185+0000","last_undegraded":"2024-09-16T10:20:03.378185+0000","last_fullsized":"2024-09-16T10:20:03.378185+0000","mapping_epoch":165,"log_start":"184'2900","ondisk_log_start":"184'2900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":1399,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.069307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478871552,"num_objects":7307,"num_object_clones":0,"num_object_copies":21921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":467648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863281+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:20:01.863281+0000","last_peered":"2024-09-16T10:20:01.863281+0000","last_clean":"2024-09-16T10:20:01.863281+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:20:01.863281+0000","last_undegraded":"2024-09-16T10:20:01.863281+0000","last_fullsized":"2024-09-16T10:20:01.863281+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.191940+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857347+0000","last_change":"2024-09-16T10:18:34.127253+0000","last_active":"2024-09-16T10:20:01.857347+0000","last_peered":"2024-09-16T10:20:01.857347+0000","last_clean":"2024-09-16T10:20:01.857347+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:20:01.857347+0000","last_undegraded":"2024-09-16T10:20:01.857347+0000","last_fullsized":"2024-09-16T10:20:01.857347+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:34.127213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:34.127213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:02.731145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":79,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859353+0000","last_change":"2024-09-16T10:19:18.748438+0000","last_active":"2024-09-16T10:20:01.859353+0000","last_peered":"2024-09-16T10:20:01.859353+0000","last_clean":"2024-09-16T10:20:01.859353+0000","last_became_active":"2024-09-16T10:18:52.835884+0000","last_became_peered":"2024-09-16T10:18:52.835884+0000","last_unstale":"2024-09-16T10:20:01.859353+0000","last_undegraded":"2024-09-16T10:20:01.859353+0000","last_fullsized":"2024-09-16T10:20:01.859353+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:27.670612+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862956+0000","last_change":"2024-09-16T10:18:52.849022+0000","last_active":"2024-09-16T10:20:01.862956+0000","last_peered":"2024-09-16T10:20:01.862956+0000","last_clean":"2024-09-16T10:20:01.862956+0000","last_became_active":"2024-09-16T10:18:52.839091+0000","last_became_peered":"2024-09-16T10:18:52.839091+0000","last_unstale":"2024-09-16T10:20:01.862956+0000","last_undegraded":"2024-09-16T10:20:01.862956+0000","last_fullsized":"2024-09-16T10:20:01.862956+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"204'7272","reported_seq":7360,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.542513+0000","last_change":"2024-09-16T10:18:33.894422+0000","last_active":"2024-09-16T10:20:03.542513+0000","last_peered":"2024-09-16T10:20:03.542513+0000","last_clean":"2024-09-16T10:20:03.542513+0000","last_became_active":"2024-09-16T10:18:33.893961+0000","last_became_peered":"2024-09-16T10:18:33.893961+0000","last_unstale":"2024-09-16T10:20:03.542513+0000","last_undegraded":"2024-09-16T10:20:03.542513+0000","last_fullsized":"2024-09-16T10:20:03.542513+0000","mapping_epoch":165,"log_start":"181'2600","ondisk_log_start":"181'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476577792,"num_objects":7272,"num_object_clones":0,"num_object_copies":21816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":354,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865631+0000","last_change":"2024-09-16T10:19:24.115166+0000","last_active":"2024-09-16T10:20:01.865631+0000","last_peered":"2024-09-16T10:20:01.865631+0000","last_clean":"2024-09-16T10:20:01.865631+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:20:01.865631+0000","last_undegraded":"2024-09-16T10:20:01.865631+0000","last_fullsized":"2024-09-16T10:20:01.865631+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:24.115075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.115075+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.363776+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.868190+0000","last_change":"2024-09-16T10:15:19.633493+0000","last_active":"2024-09-16T10:20:01.868190+0000","last_peered":"2024-09-16T10:20:01.868190+0000","last_clean":"2024-09-16T10:20:01.868190+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:20:01.868190+0000","last_undegraded":"2024-09-16T10:20:01.868190+0000","last_fullsized":"2024-09-16T10:20:01.868190+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:15:19.633446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:15:19.633446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.428291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":79,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863126+0000","last_change":"2024-09-16T10:19:17.184744+0000","last_active":"2024-09-16T10:20:01.863126+0000","last_peered":"2024-09-16T10:20:01.863126+0000","last_clean":"2024-09-16T10:20:01.863126+0000","last_became_active":"2024-09-16T10:18:52.831796+0000","last_became_peered":"2024-09-16T10:18:52.831796+0000","last_unstale":"2024-09-16T10:20:01.863126+0000","last_undegraded":"2024-09-16T10:20:01.863126+0000","last_fullsized":"2024-09-16T10:20:01.863126+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.184674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.184674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:20.369045+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":315,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862825+0000","last_change":"2024-09-16T10:18:52.841267+0000","last_active":"2024-09-16T10:20:01.862825+0000","last_peered":"2024-09-16T10:20:01.862825+0000","last_clean":"2024-09-16T10:20:01.862825+0000","last_became_active":"2024-09-16T10:18:52.840958+0000","last_became_peered":"2024-09-16T10:18:52.840958+0000","last_unstale":"2024-09-16T10:20:01.862825+0000","last_undegraded":"2024-09-16T10:20:01.862825+0000","last_fullsized":"2024-09-16T10:20:01.862825+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.495303+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"204'7262","reported_seq":7425,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.156677+0000","last_change":"2024-09-16T10:19:55.377394+0000","last_active":"2024-09-16T10:20:03.156677+0000","last_peered":"2024-09-16T10:20:03.156677+0000","last_clean":"2024-09-16T10:20:03.156677+0000","last_became_active":"2024-09-16T10:18:33.894026+0000","last_became_peered":"2024-09-16T10:18:33.894026+0000","last_unstale":"2024-09-16T10:20:03.156677+0000","last_undegraded":"2024-09-16T10:20:03.156677+0000","last_fullsized":"2024-09-16T10:20:03.156677+0000","mapping_epoch":165,"log_start":"181'2600","ondisk_log_start":"181'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6662","last_scrub_stamp":"2024-09-16T10:19:55.377245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.377245+0000","objects_scrubbed":6589,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.117653+0000","scrub_duration":2000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475922432,"num_objects":7262,"num_object_clones":0,"num_object_copies":21786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7262,"num_write_kb":464768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":338,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.859382+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:20:01.859382+0000","last_peered":"2024-09-16T10:20:01.859382+0000","last_clean":"2024-09-16T10:20:01.859382+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:20:01.859382+0000","last_undegraded":"2024-09-16T10:20:01.859382+0000","last_fullsized":"2024-09-16T10:20:01.859382+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.799332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863452+0000","last_change":"2024-09-16T10:19:17.357917+0000","last_active":"2024-09-16T10:20:01.863452+0000","last_peered":"2024-09-16T10:20:01.863452+0000","last_clean":"2024-09-16T10:20:01.863452+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:20:01.863452+0000","last_undegraded":"2024-09-16T10:20:01.863452+0000","last_fullsized":"2024-09-16T10:20:01.863452+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.357866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.357866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.135568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":71,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862642+0000","last_change":"2024-09-16T10:18:52.830267+0000","last_active":"2024-09-16T10:20:01.862642+0000","last_peered":"2024-09-16T10:20:01.862642+0000","last_clean":"2024-09-16T10:20:01.862642+0000","last_became_active":"2024-09-16T10:18:52.829845+0000","last_became_peered":"2024-09-16T10:18:52.829845+0000","last_unstale":"2024-09-16T10:20:01.862642+0000","last_undegraded":"2024-09-16T10:20:01.862642+0000","last_fullsized":"2024-09-16T10:20:01.862642+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.955280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854731+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:20:01.854731+0000","last_peered":"2024-09-16T10:20:01.854731+0000","last_clean":"2024-09-16T10:20:01.854731+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:20:01.854731+0000","last_undegraded":"2024-09-16T10:20:01.854731+0000","last_fullsized":"2024-09-16T10:20:01.854731+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"204'7028","reported_seq":10419,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:03.216691+0000","last_change":"2024-09-16T10:19:14.298104+0000","last_active":"2024-09-16T10:20:03.216691+0000","last_peered":"2024-09-16T10:20:03.216691+0000","last_clean":"2024-09-16T10:20:03.216691+0000","last_became_active":"2024-09-16T10:19:14.291692+0000","last_became_peered":"2024-09-16T10:19:14.291692+0000","last_unstale":"2024-09-16T10:20:03.216691+0000","last_undegraded":"2024-09-16T10:20:03.216691+0000","last_fullsized":"2024-09-16T10:20:03.216691+0000","mapping_epoch":187,"log_start":"182'2600","ondisk_log_start":"182'2600","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460587008,"num_objects":7028,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7028,"num_write_kb":449792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":325,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856627+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:20:01.856627+0000","last_peered":"2024-09-16T10:20:01.856627+0000","last_clean":"2024-09-16T10:20:01.856627+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:20:01.856627+0000","last_undegraded":"2024-09-16T10:20:01.856627+0000","last_fullsized":"2024-09-16T10:20:01.856627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:29.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":332,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864006+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:20:01.864006+0000","last_peered":"2024-09-16T10:20:01.864006+0000","last_clean":"2024-09-16T10:20:01.864006+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:20:01.864006+0000","last_undegraded":"2024-09-16T10:20:01.864006+0000","last_fullsized":"2024-09-16T10:20:01.864006+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":398,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862935+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:20:01.862935+0000","last_peered":"2024-09-16T10:20:01.862935+0000","last_clean":"2024-09-16T10:20:01.862935+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:20:01.862935+0000","last_undegraded":"2024-09-16T10:20:01.862935+0000","last_fullsized":"2024-09-16T10:20:01.862935+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.564166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860379+0000","last_change":"2024-09-16T10:19:20.941840+0000","last_active":"2024-09-16T10:20:01.860379+0000","last_peered":"2024-09-16T10:20:01.860379+0000","last_clean":"2024-09-16T10:20:01.860379+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:20:01.860379+0000","last_undegraded":"2024-09-16T10:20:01.860379+0000","last_fullsized":"2024-09-16T10:20:01.860379+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.941753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.941753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.997567+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865986+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:20:01.865986+0000","last_peered":"2024-09-16T10:20:01.865986+0000","last_clean":"2024-09-16T10:20:01.865986+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:20:01.865986+0000","last_undegraded":"2024-09-16T10:20:01.865986+0000","last_fullsized":"2024-09-16T10:20:01.865986+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:32.667031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":333,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854702+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:20:01.854702+0000","last_peered":"2024-09-16T10:20:01.854702+0000","last_clean":"2024-09-16T10:20:01.854702+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:20:01.854702+0000","last_undegraded":"2024-09-16T10:20:01.854702+0000","last_fullsized":"2024-09-16T10:20:01.854702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.155197+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856627+0000","last_change":"2024-09-16T10:19:15.972005+0000","last_active":"2024-09-16T10:20:01.856627+0000","last_peered":"2024-09-16T10:20:01.856627+0000","last_clean":"2024-09-16T10:20:01.856627+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:20:01.856627+0000","last_undegraded":"2024-09-16T10:20:01.856627+0000","last_fullsized":"2024-09-16T10:20:01.856627+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.971951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.971951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:28.073056+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":300,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.866409+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:20:01.866409+0000","last_peered":"2024-09-16T10:20:01.866409+0000","last_clean":"2024-09-16T10:20:01.866409+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:20:01.866409+0000","last_undegraded":"2024-09-16T10:20:01.866409+0000","last_fullsized":"2024-09-16T10:20:01.866409+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.610870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":339,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.865226+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:20:01.865226+0000","last_peered":"2024-09-16T10:20:01.865226+0000","last_clean":"2024-09-16T10:20:01.865226+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:20:01.865226+0000","last_undegraded":"2024-09-16T10:20:01.865226+0000","last_fullsized":"2024-09-16T10:20:01.865226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863212+0000","last_change":"2024-09-16T10:19:22.431194+0000","last_active":"2024-09-16T10:20:01.863212+0000","last_peered":"2024-09-16T10:20:01.863212+0000","last_clean":"2024-09-16T10:20:01.863212+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:20:01.863212+0000","last_undegraded":"2024-09-16T10:20:01.863212+0000","last_fullsized":"2024-09-16T10:20:01.863212+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.431126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.431126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.492625+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854517+0000","last_change":"2024-09-16T10:16:37.237971+0000","last_active":"2024-09-16T10:20:01.854517+0000","last_peered":"2024-09-16T10:20:01.854517+0000","last_clean":"2024-09-16T10:20:01.854517+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:20:01.854517+0000","last_undegraded":"2024-09-16T10:20:01.854517+0000","last_fullsized":"2024-09-16T10:20:01.854517+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:37.237849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.237849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:31.908375+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":72,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860199+0000","last_change":"2024-09-16T10:18:52.823189+0000","last_active":"2024-09-16T10:20:01.860199+0000","last_peered":"2024-09-16T10:20:01.860199+0000","last_clean":"2024-09-16T10:20:01.860199+0000","last_became_active":"2024-09-16T10:18:52.822465+0000","last_became_peered":"2024-09-16T10:18:52.822465+0000","last_unstale":"2024-09-16T10:20:01.860199+0000","last_undegraded":"2024-09-16T10:20:01.860199+0000","last_fullsized":"2024-09-16T10:20:01.860199+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.568361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.568361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":332,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857949+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:20:01.857949+0000","last_peered":"2024-09-16T10:20:01.857949+0000","last_clean":"2024-09-16T10:20:01.857949+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:20:01.857949+0000","last_undegraded":"2024-09-16T10:20:01.857949+0000","last_fullsized":"2024-09-16T10:20:01.857949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.022112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864606+0000","last_change":"2024-09-16T10:18:03.120551+0000","last_active":"2024-09-16T10:20:01.864606+0000","last_peered":"2024-09-16T10:20:01.864606+0000","last_clean":"2024-09-16T10:20:01.864606+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:20:01.864606+0000","last_undegraded":"2024-09-16T10:20:01.864606+0000","last_fullsized":"2024-09-16T10:20:01.864606+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:03.120474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:03.120474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"175'9","reported_seq":417,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857001+0000","last_change":"2024-09-16T10:18:52.852723+0000","last_active":"2024-09-16T10:20:01.857001+0000","last_peered":"2024-09-16T10:20:01.857001+0000","last_clean":"2024-09-16T10:20:01.857001+0000","last_became_active":"2024-09-16T10:18:52.845153+0000","last_became_peered":"2024-09-16T10:18:52.845153+0000","last_unstale":"2024-09-16T10:20:01.857001+0000","last_undegraded":"2024-09-16T10:20:01.857001+0000","last_fullsized":"2024-09-16T10:20:01.857001+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:38.926860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:38.926860+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:06.735603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":314,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.861681+0000","last_change":"2024-09-16T10:18:52.821349+0000","last_active":"2024-09-16T10:20:01.861681+0000","last_peered":"2024-09-16T10:20:01.861681+0000","last_clean":"2024-09-16T10:20:01.861681+0000","last_became_active":"2024-09-16T10:18:52.821050+0000","last_became_peered":"2024-09-16T10:18:52.821050+0000","last_unstale":"2024-09-16T10:20:01.861681+0000","last_undegraded":"2024-09-16T10:20:01.861681+0000","last_fullsized":"2024-09-16T10:20:01.861681+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.915613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.915613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.778010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":339,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856618+0000","last_change":"2024-09-16T10:19:45.045388+0000","last_active":"2024-09-16T10:20:01.856618+0000","last_peered":"2024-09-16T10:20:01.856618+0000","last_clean":"2024-09-16T10:20:01.856618+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:20:01.856618+0000","last_undegraded":"2024-09-16T10:20:01.856618+0000","last_fullsized":"2024-09-16T10:20:01.856618+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:45.045164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.045164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.833432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":327,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863287+0000","last_change":"2024-09-16T10:18:52.820977+0000","last_active":"2024-09-16T10:20:01.863287+0000","last_peered":"2024-09-16T10:20:01.863287+0000","last_clean":"2024-09-16T10:20:01.863287+0000","last_became_active":"2024-09-16T10:18:52.820693+0000","last_became_peered":"2024-09-16T10:18:52.820693+0000","last_unstale":"2024-09-16T10:20:01.863287+0000","last_undegraded":"2024-09-16T10:20:01.863287+0000","last_fullsized":"2024-09-16T10:20:01.863287+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":406,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.857732+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:20:01.857732+0000","last_peered":"2024-09-16T10:20:01.857732+0000","last_clean":"2024-09-16T10:20:01.857732+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:20:01.857732+0000","last_undegraded":"2024-09-16T10:20:01.857732+0000","last_fullsized":"2024-09-16T10:20:01.857732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.555917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":307,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863430+0000","last_change":"2024-09-16T10:18:09.148508+0000","last_active":"2024-09-16T10:20:01.863430+0000","last_peered":"2024-09-16T10:20:01.863430+0000","last_clean":"2024-09-16T10:20:01.863430+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:20:01.863430+0000","last_undegraded":"2024-09-16T10:20:01.863430+0000","last_fullsized":"2024-09-16T10:20:01.863430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:09.148447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.148447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":326,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860487+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:20:01.860487+0000","last_peered":"2024-09-16T10:20:01.860487+0000","last_clean":"2024-09-16T10:20:01.860487+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:20:01.860487+0000","last_undegraded":"2024-09-16T10:20:01.860487+0000","last_fullsized":"2024-09-16T10:20:01.860487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.805984+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.860330+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:20:01.860330+0000","last_peered":"2024-09-16T10:20:01.860330+0000","last_clean":"2024-09-16T10:20:01.860330+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:20:01.860330+0000","last_undegraded":"2024-09-16T10:20:01.860330+0000","last_fullsized":"2024-09-16T10:20:01.860330+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:34.756954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856081+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:20:01.856081+0000","last_peered":"2024-09-16T10:20:01.856081+0000","last_clean":"2024-09-16T10:20:01.856081+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:20:01.856081+0000","last_undegraded":"2024-09-16T10:20:01.856081+0000","last_fullsized":"2024-09-16T10:20:01.856081+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":300,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855370+0000","last_change":"2024-09-16T10:19:17.803486+0000","last_active":"2024-09-16T10:20:01.855370+0000","last_peered":"2024-09-16T10:20:01.855370+0000","last_clean":"2024-09-16T10:20:01.855370+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:20:01.855370+0000","last_undegraded":"2024-09-16T10:20:01.855370+0000","last_fullsized":"2024-09-16T10:20:01.855370+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.803412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.803412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:25.806543+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":349,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856323+0000","last_change":"2024-09-16T10:19:18.621273+0000","last_active":"2024-09-16T10:20:01.856323+0000","last_peered":"2024-09-16T10:20:01.856323+0000","last_clean":"2024-09-16T10:20:01.856323+0000","last_became_active":"2024-09-16T10:18:52.832688+0000","last_became_peered":"2024-09-16T10:18:52.832688+0000","last_unstale":"2024-09-16T10:20:01.856323+0000","last_undegraded":"2024-09-16T10:20:01.856323+0000","last_fullsized":"2024-09-16T10:20:01.856323+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:18.620605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.620605+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:19.358298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":328,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.868036+0000","last_change":"2024-09-16T10:19:47.608315+0000","last_active":"2024-09-16T10:20:01.868036+0000","last_peered":"2024-09-16T10:20:01.868036+0000","last_clean":"2024-09-16T10:20:01.868036+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:20:01.868036+0000","last_undegraded":"2024-09-16T10:20:01.868036+0000","last_fullsized":"2024-09-16T10:20:01.868036+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.608243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.608243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.564159+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":277,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854789+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:20:01.854789+0000","last_peered":"2024-09-16T10:20:01.854789+0000","last_clean":"2024-09-16T10:20:01.854789+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:20:01.854789+0000","last_undegraded":"2024-09-16T10:20:01.854789+0000","last_fullsized":"2024-09-16T10:20:01.854789+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.046746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":299,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862511+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:20:01.862511+0000","last_peered":"2024-09-16T10:20:01.862511+0000","last_clean":"2024-09-16T10:20:01.862511+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:20:01.862511+0000","last_undegraded":"2024-09-16T10:20:01.862511+0000","last_fullsized":"2024-09-16T10:20:01.862511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:00.958267+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":327,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855576+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:20:01.855576+0000","last_peered":"2024-09-16T10:20:01.855576+0000","last_clean":"2024-09-16T10:20:01.855576+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:20:01.855576+0000","last_undegraded":"2024-09-16T10:20:01.855576+0000","last_fullsized":"2024-09-16T10:20:01.855576+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:13.959478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":328,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856761+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:20:01.856761+0000","last_peered":"2024-09-16T10:20:01.856761+0000","last_clean":"2024-09-16T10:20:01.856761+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:20:01.856761+0000","last_undegraded":"2024-09-16T10:20:01.856761+0000","last_fullsized":"2024-09-16T10:20:01.856761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.862460+0000","last_change":"2024-09-16T10:19:45.534275+0000","last_active":"2024-09-16T10:20:01.862460+0000","last_peered":"2024-09-16T10:20:01.862460+0000","last_clean":"2024-09-16T10:20:01.862460+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:20:01.862460+0000","last_undegraded":"2024-09-16T10:20:01.862460+0000","last_fullsized":"2024-09-16T10:20:01.862460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:45.533998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.533998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:47.342382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.855399+0000","last_change":"2024-09-16T10:19:47.129026+0000","last_active":"2024-09-16T10:20:01.855399+0000","last_peered":"2024-09-16T10:20:01.855399+0000","last_clean":"2024-09-16T10:20:01.855399+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:20:01.855399+0000","last_undegraded":"2024-09-16T10:20:01.855399+0000","last_fullsized":"2024-09-16T10:20:01.855399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.128964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.128964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.566335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":348,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854007+0000","last_change":"2024-09-16T10:19:56.200787+0000","last_active":"2024-09-16T10:20:01.854007+0000","last_peered":"2024-09-16T10:20:01.854007+0000","last_clean":"2024-09-16T10:20:01.854007+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:20:01.854007+0000","last_undegraded":"2024-09-16T10:20:01.854007+0000","last_fullsized":"2024-09-16T10:20:01.854007+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:19:56.200704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.200704+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.350145+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":325,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864224+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:20:01.864224+0000","last_peered":"2024-09-16T10:20:01.864224+0000","last_clean":"2024-09-16T10:20:01.864224+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:20:01.864224+0000","last_undegraded":"2024-09-16T10:20:01.864224+0000","last_fullsized":"2024-09-16T10:20:01.864224+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.866073+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:20:01.866073+0000","last_peered":"2024-09-16T10:20:01.866073+0000","last_clean":"2024-09-16T10:20:01.866073+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:20:01.866073+0000","last_undegraded":"2024-09-16T10:20:01.866073+0000","last_fullsized":"2024-09-16T10:20:01.866073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:33.226339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":78,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.863318+0000","last_change":"2024-09-16T10:19:44.487481+0000","last_active":"2024-09-16T10:20:01.863318+0000","last_peered":"2024-09-16T10:20:01.863318+0000","last_clean":"2024-09-16T10:20:01.863318+0000","last_became_active":"2024-09-16T10:18:52.838002+0000","last_became_peered":"2024-09-16T10:18:52.838002+0000","last_unstale":"2024-09-16T10:20:01.863318+0000","last_undegraded":"2024-09-16T10:20:01.863318+0000","last_fullsized":"2024-09-16T10:20:01.863318+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.584073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.856417+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:20:01.856417+0000","last_peered":"2024-09-16T10:20:01.856417+0000","last_clean":"2024-09-16T10:20:01.856417+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:20:01.856417+0000","last_undegraded":"2024-09-16T10:20:01.856417+0000","last_fullsized":"2024-09-16T10:20:01.856417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:43.126929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.854172+0000","last_change":"2024-09-16T10:19:20.738852+0000","last_active":"2024-09-16T10:20:01.854172+0000","last_peered":"2024-09-16T10:20:01.854172+0000","last_clean":"2024-09-16T10:20:01.854172+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:20:01.854172+0000","last_undegraded":"2024-09-16T10:20:01.854172+0000","last_fullsized":"2024-09-16T10:20:01.854172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.738779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.738779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:35.219018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":351,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.867216+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:20:01.867216+0000","last_peered":"2024-09-16T10:20:01.867216+0000","last_clean":"2024-09-16T10:20:01.867216+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:20:01.867216+0000","last_undegraded":"2024-09-16T10:20:01.867216+0000","last_fullsized":"2024-09-16T10:20:01.867216+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:09.457028+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":327,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-16T10:20:01.864973+0000","last_change":"2024-09-16T10:19:16.217856+0000","last_active":"2024-09-16T10:20:01.864973+0000","last_peered":"2024-09-16T10:20:01.864973+0000","last_clean":"2024-09-16T10:20:01.864973+0000","last_became_active":"2024-09-16T10:18:52.834424+0000","last_became_peered":"2024-09-16T10:18:52.834424+0000","last_unstale":"2024-09-16T10:20:01.864973+0000","last_undegraded":"2024-09-16T10:20:01.864973+0000","last_fullsized":"2024-09-16T10:20:01.864973+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.131801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7625900032,"num_objects":116362,"num_object_clones":0,"num_object_copies":349086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116362,"num_write_kb":7447168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10118,"num_bytes_recovered":663093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23422631936,"data_stored":23422631936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71562,"ondisk_log_size":71562,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234065,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2955400,"kb_used_data":2771508,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":90789752,"statfs":{"total":95995035648,"available":92968706048,"internally_reserved":0,"allocated":2838024192,"data_stored":2837620057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528405,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3452616,"kb_used_data":3209012,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":90292536,"statfs":{"total":95995035648,"available":92459556864,"internally_reserved":0,"allocated":3286028288,"data_stored":3285624067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":249428422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790043,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4363848,"kb_used_data":4143864,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":89381304,"statfs":{"total":95995035648,"available":91526455296,"internally_reserved":0,"allocated":4243316736,"data_stored":4242909743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2937936,"kb_used_data":2754044,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":90807216,"statfs":{"total":95995035648,"available":92986589184,"internally_reserved":0,"allocated":2820141056,"data_stored":2819728672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215205,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3544676,"kb_used_data":3278868,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90200476,"statfs":{"total":95995035648,"available":92365287424,"internally_reserved":0,"allocated":3357560832,"data_stored":3357126111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":272169416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3499112,"kb_used_data":3251348,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":90246040,"statfs":{"total":95995035648,"available":92411944960,"internally_reserved":0,"allocated":3329380352,"data_stored":3328944242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2996440,"kb_used_data":2752836,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":90748712,"statfs":{"total":95995035648,"available":92926681088,"internally_reserved":0,"allocated":2818904064,"data_stored":2818484467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2839805952,"data_stored":2839805952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3362324480,"data_stored":3362324480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":562823168,"data_stored":562823168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3371368448,"data_stored":3371368448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2828009472,"data_stored":2828009472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4287758336,"data_stored":4287758336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3311796224,"data_stored":3311796224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2858745856,"data_stored":2858745856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:20:08.878 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T10:20:08.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-16T10:20:08.943 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:08.976 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:08.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:09.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:09.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:09.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:09.112 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]} : dispatch 2024-09-16T10:20:09.113 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]} : dispatch 2024-09-16T10:20:09.113 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:09.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:09.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:09.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]} : dispatch 2024-09-16T10:20:09.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]} : dispatch 2024-09-16T10:20:09.113 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:09.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:09.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:09.124 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:09.217 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:09.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]} : dispatch 2024-09-16T10:20:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]} : dispatch 2024-09-16T10:20:09.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:09.361 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:09.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:20:09.396 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:09.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:09.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:09.511 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 88 16 124666 124650 88.509 80.9375 0.0174786 0.0112939 2024-09-16T10:20:09.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:09.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:09.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:09.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:09.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:09.746 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:20:09.746 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:20:09.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:20:09.746 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:20:09.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.746 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:40100 7.e 7:7326c3b0:::benchmark_data_smithi063_73086_object40099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:20:09.746 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 300.194020889, 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.113082665, 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 40100 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546900+0000", 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2330000000000001e-06 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546927+0000", 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6721999999999999e-05 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546939+0000", 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1793999999999999e-05 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546947+0000", 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.704e-06 2024-09-16T10:20:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547026+0000", 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8479999999999994e-05 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547144+0000", 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011862999999999999 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547278+0000", 2024-09-16T10:20:09.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013377800000000001 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.548993+0000", 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001715261 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549008+0000", 2024-09-16T10:20:09.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4158000000000001e-05 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659763+0000", 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.110755881 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659886+0000", 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012276600000000001 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659980+0000", 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4258000000000005e-05 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:43495 7.e 7:7a740846:::benchmark_data_smithi063_73086_object43494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 297.98802328300002, 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115699248, 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.753 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 43495 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752898+0000", 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1710000000000001e-06 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752922+0000", 2024-09-16T10:20:09.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4593e-05 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752933+0000", 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1083999999999999e-05 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752941+0000", 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6920000000000002e-06 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753001+0000", 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0109000000000003e-05 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753158+0000", 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000156668 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753289+0000", 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013114999999999999 2024-09-16T10:20:09.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.755266+0000", 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00197679 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.755404+0000", 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013795999999999999 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868497+0000", 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11309365 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868554+0000", 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6641999999999997e-05 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868595+0000", 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0738999999999997e-05 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.759 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65096 7.e 7:7bb27b3e:::benchmark_data_smithi063_73086_object65095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 282.17701502900002, 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.125924595, 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.760 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65096 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563905+0000", 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.606e-06 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563926+0000", 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1009000000000002e-05 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563936+0000", 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6460000000000005e-06 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563942+0000", 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1589999999999998e-06 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563991+0000", 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9279e-05 2024-09-16T10:20:09.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564082+0000", 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0890000000000006e-05 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564170+0000", 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7435000000000006e-05 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.565824+0000", 2024-09-16T10:20:09.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001654345 2024-09-16T10:20:09.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.566103+0000", 2024-09-16T10:20:09.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027876700000000002 2024-09-16T10:20:09.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689664+0000", 2024-09-16T10:20:09.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123561491 2024-09-16T10:20:09.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689778+0000", 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113677 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689828+0000", 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0290999999999997e-05 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70300 7.8 7:10588a9d:::benchmark_data_smithi063_73086_object70299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 277.83782559299999, 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12214414, 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70300 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.770 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903095+0000", 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9099999999999999e-06 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903119+0000", 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3856999999999999e-05 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903132+0000", 2024-09-16T10:20:09.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2873e-05 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903139+0000", 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9739999999999996e-06 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903195+0000", 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6121e-05 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903309+0000", 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113962 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903448+0000", 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013888799999999999 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905996+0000", 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025485479999999999 2024-09-16T10:20:09.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909244+0000", 2024-09-16T10:20:09.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032479420000000002 2024-09-16T10:20:09.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.024965+0000", 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115720986 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025185+0000", 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220177 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025237+0000", 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1901999999999999e-05 2024-09-16T10:20:09.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70303 7.8 7:176924a2:::benchmark_data_smithi063_73086_object70302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 277.83552513799998, 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120210123, 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.777 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70303 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:20:09.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.779 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905396+0000", 2024-09-16T10:20:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5900000000000002e-06 2024-09-16T10:20:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905422+0000", 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6364999999999999e-05 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905438+0000", 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5795e-05 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905448+0000", 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8020000000000003e-06 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906033+0000", 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058453800000000001 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906161+0000", 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012883799999999999 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906300+0000", 2024-09-16T10:20:09.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013873499999999999 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909282+0000", 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029820070000000001 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.910327+0000", 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001044332 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025288+0000", 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11496116100000001 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025559+0000", 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027147 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025604+0000", 2024-09-16T10:20:09.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4490000000000003e-05 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72689 7.e 7:7f5699d3:::benchmark_data_smithi063_73086_object72688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 275.87143136700001, 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12073505599999999, 2024-09-16T10:20:09.785 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72689 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869489+0000", 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.237e-06 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869521+0000", 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2728999999999998e-05 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869531+0000", 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8989999999999996e-06 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869539+0000", 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2209999999999997e-06 2024-09-16T10:20:09.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872936+0000", 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003396296 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873090+0000", 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015414000000000001 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873225+0000", 2024-09-16T10:20:09.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013536000000000001 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878402+0000", 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0051764419999999998 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878413+0000", 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1810999999999999e-05 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.989923+0000", 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.111509897 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990168+0000", 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00024499400000000001 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990222+0000", 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4030000000000003e-05 2024-09-16T10:20:09.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72696 7.e 7:7eb0d49d:::benchmark_data_smithi063_73086_object72695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 275.86813198200002, 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120396989, 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72696 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872789+0000", 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.979e-06 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872820+0000", 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1522000000000001e-05 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872830+0000", 2024-09-16T10:20:09.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7550000000000006e-06 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872837+0000", 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7399999999999998e-06 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875523+0000", 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026860759999999999 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875632+0000", 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000109343 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875757+0000", 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012532799999999999 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878452+0000", 2024-09-16T10:20:09.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026941249999999999 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.880841+0000", 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023892340000000001 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993090+0000", 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.112248941 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993145+0000", 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5043999999999997e-05 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993184+0000", 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8902000000000001e-05 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72702 7.e 7:720829b1:::benchmark_data_smithi063_73086_object72701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:20:09.798 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 275.86286819600002, 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119080255, 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72702 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878052+0000", 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3689999999999999e-06 2024-09-16T10:20:09.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878078+0000", 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6140999999999999e-05 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878088+0000", 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0147e-05 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878096+0000", 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5089999999999996e-06 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878485+0000", 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00038902000000000001 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878652+0000", 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016732800000000001 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878733+0000", 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0873999999999999e-05 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.883480+0000", 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047467120000000002 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884355+0000", 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087525899999999998 2024-09-16T10:20:09.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997027+0000", 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11267184399999999 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997084+0000", 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.766e-05 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997131+0000", 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6391999999999999e-05 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72710 7.e 7:792abf92:::benchmark_data_smithi063_73086_object72709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 275.85666954200002, 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11643550799999999, 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.805 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72710 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884251+0000", 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8929999999999999e-06 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884276+0000", 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4837999999999998e-05 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884287+0000", 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1464000000000001e-05 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884295+0000", 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2300000000000002e-06 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884389+0000", 2024-09-16T10:20:09.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4073000000000005e-05 2024-09-16T10:20:09.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884555+0000", 2024-09-16T10:20:09.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000166782 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884698+0000", 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000142917 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888931+0000", 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042330529999999996 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984052+0000", 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.095120447999999996 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000566+0000", 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016514217000000001 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000630+0000", 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4459000000000001e-05 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000685+0000", 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4134e-05 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72712 7.e 7:7d21703e:::benchmark_data_smithi063_73086_object72711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 275.85634808499998, 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.118747798, 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72712 2024-09-16T10:20:09.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884572+0000", 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4270000000000001e-06 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884607+0000", 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4535000000000001e-05 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884617+0000", 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0774000000000001e-05 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884625+0000", 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.644e-06 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.886978+0000", 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00235257 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887080+0000", 2024-09-16T10:20:09.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010283 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887167+0000", 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6706999999999995e-05 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984059+0000", 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096891692000000001 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984099+0000", 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0516000000000003e-05 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003069+0000", 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.018970035999999999 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003200+0000", 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000130961 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003318+0000", 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118106 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72715 7.e 7:70bb1898:::benchmark_data_smithi063_73086_object72714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 275.85271239600002, 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115258741, 2024-09-16T10:20:09.819 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72715 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888208+0000", 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0990000000000001e-06 2024-09-16T10:20:09.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888233+0000", 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4961999999999999e-05 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888243+0000", 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3680000000000008e-06 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888248+0000", 2024-09-16T10:20:09.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1479999999999997e-06 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888960+0000", 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00071208700000000003 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889036+0000", 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6175999999999997e-05 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981822+0000", 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.092785459000000001 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984260+0000", 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002438127 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.986355+0000", 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020951099999999999 2024-09-16T10:20:09.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003378+0000", 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.017022926000000001 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003430+0000", 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1847e-05 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003465+0000", 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5432000000000002e-05 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:72718 7.2 e107/81 7:450c0e00:::benchmark_data_smithi063_73086_object72717:head v 107'4483, mlcod=107'4481)", 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 275.85154824400001, 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.094911379000000004, 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.826 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889372+0000", 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3349999999999999e-06 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889396+0000", 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3961999999999999e-05 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981880+0000", 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.092484449999999996 2024-09-16T10:20:09.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981892+0000", 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1661000000000001e-05 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981956+0000", 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4168e-05 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981973+0000", 2024-09-16T10:20:09.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7357999999999999e-05 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984187+0000", 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022133159999999999 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984282+0000", 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5129000000000003e-05 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.830 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90069 7.e 7:7df0bbc7:::benchmark_data_smithi063_73086_object90068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 260.21720968300002, 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123224003, 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.831 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90069 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523711+0000", 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6309999999999999e-06 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523734+0000", 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3002000000000001e-05 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523745+0000", 2024-09-16T10:20:09.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1173e-05 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523752+0000", 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2579999999999998e-06 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523836+0000", 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3758999999999995e-05 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523945+0000", 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010883999999999999 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524079+0000", 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013397600000000001 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640421+0000", 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116342404 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640488+0000", 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6561000000000002e-05 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646786+0000", 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0062988599999999999 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646881+0000", 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4291999999999998e-05 2024-09-16T10:20:09.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646933+0000", 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2247000000000003e-05 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:90069 7.e e112/81 ondisk, result = 0)", 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.526896+0000", 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 260.21402232499997, 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11359791399999999, 2024-09-16T10:20:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526896+0000", 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526896+0000", 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526897+0000", 2024-09-16T10:20:09.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8899999999999998e-07 2024-09-16T10:20:09.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.839 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:09.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526901+0000", 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5999999999999998e-06 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640420+0000", 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.113519355 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640428+0000", 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8830000000000006e-06 2024-09-16T10:20:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640475+0000", 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6498000000000001e-05 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640483+0000", 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0369999999999995e-06 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640494+0000", 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1652e-05 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:106093 7.e 7:79e3a9e4:::benchmark_data_smithi063_73086_object106092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:16:06.441330+0000", 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 243.29958831299999, 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.101189283, 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.843 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 106093 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441330+0000", 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441330+0000", 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441333+0000", 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7760000000000002e-06 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441358+0000", 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4862e-05 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441372+0000", 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4235e-05 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441381+0000", 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4149999999999999e-06 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441443+0000", 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2061000000000001e-05 2024-09-16T10:20:09.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441608+0000", 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016502500000000001 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441739+0000", 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013140299999999999 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.445301+0000", 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003561678 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.446064+0000", 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00076295200000000001 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.542395+0000", 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096331128000000002 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.542463+0000", 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8294999999999996e-05 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.542520+0000", 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6453000000000003e-05 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:106096 7.e 7:7328d7e9:::benchmark_data_smithi063_73086_object106095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:16:06.447115+0000", 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 243.29380385600001, 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.099332932999999998, 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 106096 2024-09-16T10:20:09.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447115+0000", 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447115+0000", 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447117+0000", 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2809999999999998e-06 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447141+0000", 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4355e-05 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447206+0000", 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4389999999999998e-05 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447214+0000", 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5830000000000007e-06 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447332+0000", 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117148 2024-09-16T10:20:09.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447469+0000", 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013717699999999999 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447611+0000", 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014180200000000001 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.455600+0000", 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079892939999999992 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.455800+0000", 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000200514 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546278+0000", 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.090477345000000001 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546399+0000", 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121357 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546448+0000", 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8687000000000003e-05 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:106097 7.e 7:77c251f7:::benchmark_data_smithi063_73086_object106096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:16:06.447290+0000", 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 243.29362877700001, 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.100309696, 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 106097 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447290+0000", 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447290+0000", 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447293+0000", 2024-09-16T10:20:09.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5480000000000002e-06 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447317+0000", 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3870999999999999e-05 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447346+0000", 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8588000000000001e-05 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447358+0000", 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2554e-05 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.452036+0000", 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046777880000000004 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.452168+0000", 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131891 2024-09-16T10:20:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.452302+0000", 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013436700000000001 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.456017+0000", 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037144750000000001 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.458856+0000", 2024-09-16T10:20:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028388659999999998 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546498+0000", 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.087642128999999999 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.547428+0000", 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00092988800000000003 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.862 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.547600+0000", 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000171731 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:200781 7.e 7:72d3e70a:::benchmark_data_smithi063_73086_object81502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:40.222619+0000", 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 149.518299783, 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.109932639, 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.863 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 200781 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222619+0000", 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222619+0000", 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222620+0000", 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.201e-06 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222625+0000", 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5000000000000001e-06 2024-09-16T10:20:09.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222634+0000", 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.577e-06 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222641+0000", 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4790000000000001e-06 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.227085+0000", 2024-09-16T10:20:09.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044442830000000003 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.227204+0000", 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118829 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.227971+0000", 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00076681399999999999 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.234790+0000", 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0068190100000000003 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.235401+0000", 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061134600000000002 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332436+0000", 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.097035225000000003 2024-09-16T10:20:09.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332505+0000", 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9275000000000003e-05 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332552+0000", 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6100000000000002e-05 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:200797 7.e 7:7d66ac12:::benchmark_data_smithi063_73086_object81518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:40.236303+0000", 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 149.50461520600001, 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096365922000000007, 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 200797 2024-09-16T10:20:09.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236303+0000", 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236303+0000", 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236305+0000", 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7829999999999999e-06 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236310+0000", 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9799999999999998e-06 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236321+0000", 2024-09-16T10:20:09.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0311e-05 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236328+0000", 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4549999999999998e-06 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236384+0000", 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6155e-05 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236489+0000", 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000104453 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236592+0000", 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103695 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.245646+0000", 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0090532879999999996 2024-09-16T10:20:09.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.245736+0000", 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0842999999999997e-05 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332593+0000", 2024-09-16T10:20:09.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.086856218999999998 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332636+0000", 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3544000000000002e-05 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332669+0000", 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3195999999999999e-05 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.876 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201853 7.8 7:1d3a7237:::benchmark_data_smithi063_73086_object82574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.386689+0000", 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 148.35422980300001, 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10598900999999999, 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:09.877 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201853 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386689+0000", 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386689+0000", 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386691+0000", 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.829e-06 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386696+0000", 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6239999999999999e-06 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386708+0000", 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1989e-05 2024-09-16T10:20:09.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386716+0000", 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8699999999999992e-06 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386776+0000", 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0087999999999998e-05 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386908+0000", 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131321 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387012+0000", 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010425999999999999 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.390595+0000", 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0035829210000000002 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.390683+0000", 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8437999999999996e-05 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.492600+0000", 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10191681900000001 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.492643+0000", 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2914000000000003e-05 2024-09-16T10:20:09.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.492678+0000", 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4937000000000001e-05 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:09.883 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:20:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:09.884 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:09.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:09.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:10.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:10.051 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:20:10.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:10.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:10.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:10.296 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:10.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:10 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]': finished 2024-09-16T10:20:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:10 smithi165 ceph-mon[32696]: osdmap e207: 8 total, 8 up, 7 in 2024-09-16T10:20:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:10 smithi165 ceph-mon[32696]: pgmap v318: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-16T10:20:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1194605771' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:10 smithi165 ceph-mon[32696]: from='client.25414 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:10.398 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:10.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]': finished 2024-09-16T10:20:10.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[29993]: osdmap e207: 8 total, 8 up, 7 in 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[29993]: pgmap v318: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1194605771' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[29993]: from='client.25414 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]': finished 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[39470]: osdmap e207: 8 total, 8 up, 7 in 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[39470]: pgmap v318: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1194605771' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:10.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:10 smithi063 ceph-mon[39470]: from='client.25414 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:10.438 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:10.511 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 89 16 125935 125919 88.4056 79.3125 0.00932656 0.0113072 2024-09-16T10:20:10.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:10.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:10.679 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:10.710 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:10.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:10.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:10.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:10.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:10.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:11.094 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:11.129 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:11.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:11.231 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:11.266 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:11.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:11.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:11 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:11 smithi165 ceph-mon[32696]: from='client.25414 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:11 smithi165 ceph-mon[32696]: osdmap e208: 8 total, 8 up, 7 in 2024-09-16T10:20:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:11 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:11 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:11 smithi165 ceph-mon[32696]: osdmap e209: 8 total, 8 up, 7 in 2024-09-16T10:20:11.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:11.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[29993]: from='client.25414 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[29993]: osdmap e208: 8 total, 8 up, 7 in 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[29993]: osdmap e209: 8 total, 8 up, 7 in 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[39470]: from='client.25414 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[39470]: osdmap e208: 8 total, 8 up, 7 in 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:11.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:11 smithi063 ceph-mon[39470]: osdmap e209: 8 total, 8 up, 7 in 2024-09-16T10:20:11.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:20:11.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:11.425 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:11.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total time run: 90.0054 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total writes made: 127317 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Write size: 65536 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object size: 65536 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Bandwidth (MB/sec): 88.4093 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Bandwidth: 34.0394 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max bandwidth (MB/sec): 196.812 2024-09-16T10:20:11.512 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min bandwidth (MB/sec): 0 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average IOPS: 1414 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev IOPS: 544.641 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max IOPS: 3149 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min IOPS: 0 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average Latency(s): 0.0113071 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Latency(s): 0.0305354 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max latency(s): 2.11649 2024-09-16T10:20:11.513 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min latency(s): 0.00128827 2024-09-16T10:20:11.515 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Cleaning up (deleting benchmark objects) 2024-09-16T10:20:11.528 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:11.580 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:11.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:11.681 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:11.715 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:11.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:11.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:11.850 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:11.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:11.952 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:12.007 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:12.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:12.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:12.214 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:12 smithi063 ceph-mon[29993]: pgmap v321: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-16T10:20:12.214 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:12 smithi063 ceph-mon[39470]: pgmap v321: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-16T10:20:12.249 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.285 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:12.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:12.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:12 smithi165 ceph-mon[32696]: pgmap v321: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-16T10:20:12.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:12.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:12.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:12.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:12.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:12.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:12.799 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:12.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:12.936 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:12.965 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:12.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:13.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:13.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:13.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:13.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:13.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:13.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:13.281 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 1 pg_num to 21 2024-09-16T10:20:13.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2425313169' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:20:13.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:13.361 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:13.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:13.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2425313169' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:20:13.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2425313169' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:20:13.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:13.492 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:13.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:13.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:13.619 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:13.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:13.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:13.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:13.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:13.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:13.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:13.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:13.985 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:14.017 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:20:14.021 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:14.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:14.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:14.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:14.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:14.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:14.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:14.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2425313169' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T10:20:14.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:14 smithi063 ceph-mon[29993]: pgmap v322: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.21k op/s 2024-09-16T10:20:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:14 smithi063 ceph-mon[29993]: osdmap e210: 8 total, 8 up, 7 in 2024-09-16T10:20:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2425313169' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T10:20:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:14 smithi063 ceph-mon[39470]: pgmap v322: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.21k op/s 2024-09-16T10:20:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:14 smithi063 ceph-mon[39470]: osdmap e210: 8 total, 8 up, 7 in 2024-09-16T10:20:14.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:14.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:14.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:14.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2425313169' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T10:20:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:14 smithi165 ceph-mon[32696]: pgmap v322: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.21k op/s 2024-09-16T10:20:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:14 smithi165 ceph-mon[32696]: osdmap e210: 8 total, 8 up, 7 in 2024-09-16T10:20:14.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:14.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:14.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:14.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:14.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:14.948 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:14.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:15.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:15.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:15.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:15.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:15.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:15.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:15.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:15.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:15.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:15.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:15.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:15 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:15 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:15 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:15 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:15.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:15.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:15 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:15 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:15.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:15.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:15.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:15.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:15.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:15.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:15.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:15.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:15.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:15.998 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:16.034 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:16.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:16.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:16.173 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:16.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:16.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:16.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:16.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:16.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:16.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[29993]: pgmap v324: 158 pgs: 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 43 MiB/s wr, 1.11k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:16.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T10:20:16.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[29993]: osdmap e211: 8 total, 8 up, 7 in 2024-09-16T10:20:16.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[29993]: osdmap e212: 8 total, 8 up, 7 in 2024-09-16T10:20:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[39470]: pgmap v324: 158 pgs: 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 43 MiB/s wr, 1.11k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:16.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T10:20:16.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[39470]: osdmap e211: 8 total, 8 up, 7 in 2024-09-16T10:20:16.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:16 smithi063 ceph-mon[39470]: osdmap e212: 8 total, 8 up, 7 in 2024-09-16T10:20:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:16.454 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:16.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:16 smithi165 ceph-mon[32696]: pgmap v324: 158 pgs: 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 43 MiB/s wr, 1.11k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:16 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T10:20:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:16 smithi165 ceph-mon[32696]: osdmap e211: 8 total, 8 up, 7 in 2024-09-16T10:20:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:16 smithi165 ceph-mon[32696]: osdmap e212: 8 total, 8 up, 7 in 2024-09-16T10:20:16.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:16.574 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:16.602 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:16.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:16.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:16.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:16.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:16.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:16.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:16.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:16.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:17.023 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:17.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:17.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:17.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:17.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:17.263 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:17.282 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:20:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:20:17.282 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:20:17.282 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:17.282 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:17.282 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: osdmap e213: 8 total, 8 up, 7 in 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:17.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:20:17.284 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:20:17.284 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:20:17.284 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:20:17.284 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:20:17.284 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: osdmap e213: 8 total, 8 up, 7 in 2024-09-16T10:20:17.284 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:17.284 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:17 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:17.302 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:17.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:17.403 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:17.448 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:17.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:17.551 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:20:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:20:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:20:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:20:17.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:20:17.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:20:17.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: osdmap e213: 8 total, 8 up, 7 in 2024-09-16T10:20:17.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:17.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:17 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:20:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:17.596 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:17.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:17.696 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:17.696 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:17.697 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:17.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:17.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:17.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:17.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:17.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:17.739 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:17.740 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:17.740 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:17.740 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:17.740 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:17.740 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:17.741 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:17.741 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:17.741 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:17.741 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:17.741 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:17.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:17.742 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:17.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:17.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:17.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:17.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:17.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:18.029 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:18.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:18.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:18.272 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:18.308 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:18.409 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: pgmap v328: 168 pgs: 10 unknown, 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 60 MiB/s wr, 1.57k op/s; 3 B/s, 0 objects/s recovering 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/780601367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: osdmap e214: 8 total, 8 up, 7 in 2024-09-16T10:20:18.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3303240966' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:18.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: pgmap v328: 168 pgs: 10 unknown, 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 60 MiB/s wr, 1.57k op/s; 3 B/s, 0 objects/s recovering 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/780601367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: osdmap e214: 8 total, 8 up, 7 in 2024-09-16T10:20:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3303240966' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: pgmap v328: 168 pgs: 10 unknown, 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 60 MiB/s wr, 1.57k op/s; 3 B/s, 0 objects/s recovering 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/780601367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: osdmap e214: 8 total, 8 up, 7 in 2024-09-16T10:20:18.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3303240966' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:18.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:18.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:18.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:18.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:18.720 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:18.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:20:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:20:18.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:18.855 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:18.957 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:18.994 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:18.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:19.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:19.134 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:19.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:19.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:19.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:19.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:19.366 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:20:19.373 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:19.450 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:19.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3303240966' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[29993]: osdmap e215: 8 total, 8 up, 7 in 2024-09-16T10:20:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3303240966' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:20:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:19 smithi063 ceph-mon[39470]: osdmap e215: 8 total, 8 up, 7 in 2024-09-16T10:20:19.459 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:20:19.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:20:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:20:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:20:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:20:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:20:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:20:19.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:19.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3303240966' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:20:19.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:19 smithi165 ceph-mon[32696]: osdmap e215: 8 total, 8 up, 7 in 2024-09-16T10:20:19.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:19.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:19.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:19.726 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:19.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:19.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:19.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:19.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:19.963 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:19.998 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:19.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:20.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:20.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:20.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:20.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:20.239 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:20.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:20.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:20.373 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:20:20.373 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.e scrub ok 2024-09-16T10:20:20.373 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:20:20.373 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.11 scrub ok 2024-09-16T10:20:20.373 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: pgmap v331: 168 pgs: 10 peering, 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 13 KiB/s rd, 87 KiB/s wr, 1.76k op/s; 31 KiB/s, 0 objects/s recovering 2024-09-16T10:20:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.c scrub starts 2024-09-16T10:20:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.c scrub ok 2024-09-16T10:20:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:20:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:20:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:20:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.12 scrub ok 2024-09-16T10:20:20.375 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:20:20.375 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:20:20.375 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:20:20.375 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: 1.b scrub ok 2024-09-16T10:20:20.375 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-09-16T10:20:20.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:20.377 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:20:20.377 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.e scrub ok 2024-09-16T10:20:20.378 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:20:20.378 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.11 scrub ok 2024-09-16T10:20:20.378 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: pgmap v331: 168 pgs: 10 peering, 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 13 KiB/s rd, 87 KiB/s wr, 1.76k op/s; 31 KiB/s, 0 objects/s recovering 2024-09-16T10:20:20.378 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.c scrub starts 2024-09-16T10:20:20.378 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.c scrub ok 2024-09-16T10:20:20.378 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:20:20.379 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:20:20.379 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:20:20.379 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.12 scrub ok 2024-09-16T10:20:20.379 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:20:20.379 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:20:20.380 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:20:20.380 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: 1.b scrub ok 2024-09-16T10:20:20.380 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:20 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-09-16T10:20:20.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:20.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:20.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:20.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:20:20.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:20.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.e scrub ok 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.11 scrub ok 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: pgmap v331: 168 pgs: 10 peering, 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 13 KiB/s rd, 87 KiB/s wr, 1.76k op/s; 31 KiB/s, 0 objects/s recovering 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.c scrub starts 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.c scrub ok 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:20:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:20:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:20:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.12 scrub ok 2024-09-16T10:20:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:20:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:20:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:20:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: 1.b scrub ok 2024-09-16T10:20:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:20 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-09-16T10:20:20.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:20.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:20.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:20.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:20.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:20.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:20.895 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:20.895 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:20.915 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:20.950 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:20.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:21.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:21.092 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:21.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:21.177 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 6.14 scrub starts 2024-09-16T10:20:21.194 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:21.229 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:21.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:21.267 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:20:21.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:20:21.267 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:20:21.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:20:21.331 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:21.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:21.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 6.14 scrub starts 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 6.14 scrub ok 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 8.b scrub starts 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.d scrub ok 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.10 scrub ok 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.f scrub ok 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:20:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: 1.13 scrub ok 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[29993]: osdmap e216: 8 total, 8 up, 7 in 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 6.14 scrub ok 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 8.b scrub starts 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.d scrub ok 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.10 scrub ok 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.f scrub ok 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: 1.13 scrub ok 2024-09-16T10:20:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:21 smithi063 ceph-mon[39470]: osdmap e216: 8 total, 8 up, 7 in 2024-09-16T10:20:21.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:21.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 6.14 scrub starts 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 6.14 scrub ok 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 8.b scrub starts 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.d scrub ok 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.10 scrub ok 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:20:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.f scrub ok 2024-09-16T10:20:21.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:20:21.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: 1.13 scrub ok 2024-09-16T10:20:21.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:21 smithi165 ceph-mon[32696]: osdmap e216: 8 total, 8 up, 7 in 2024-09-16T10:20:21.599 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:21.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:21.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:21.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:21.781 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:21.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:21.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:21.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:21.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:22.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:22.066 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:22.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:22.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:20:22.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:22.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:22.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:22.305 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:22.343 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:22.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:22.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:22.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:22 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:20:22.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:22 smithi063 ceph-mon[29993]: 1.14 scrub ok 2024-09-16T10:20:22.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:22 smithi063 ceph-mon[29993]: pgmap v333: 168 pgs: 10 peering, 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 12 KiB/s rd, 80 KiB/s wr, 1.63k op/s; 28 KiB/s, 0 objects/s recovering 2024-09-16T10:20:22.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:22 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:20:22.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:22 smithi063 ceph-mon[39470]: 1.14 scrub ok 2024-09-16T10:20:22.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:22 smithi063 ceph-mon[39470]: pgmap v333: 168 pgs: 10 peering, 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 12 KiB/s rd, 80 KiB/s wr, 1.63k op/s; 28 KiB/s, 0 objects/s recovering 2024-09-16T10:20:22.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:22.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:22 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:20:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:22 smithi165 ceph-mon[32696]: 1.14 scrub ok 2024-09-16T10:20:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:22 smithi165 ceph-mon[32696]: pgmap v333: 168 pgs: 10 peering, 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 12 KiB/s rd, 80 KiB/s wr, 1.63k op/s; 28 KiB/s, 0 objects/s recovering 2024-09-16T10:20:22.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:22.597 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:22.598 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:20:22.650 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:22.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:22.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:22.796 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:22.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:22.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:22.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:22.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:23.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:23.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:23.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:23.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:23.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:23.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:23.269 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:23.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:23.346 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:23.347 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:23.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:23.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:23.381 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:23.381 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:23.407 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[39470]: 8.b scrub ok 2024-09-16T10:20:23.408 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[39470]: 8.f scrub starts 2024-09-16T10:20:23.408 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:23.408 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:23.408 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[29993]: 8.b scrub ok 2024-09-16T10:20:23.408 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[29993]: 8.f scrub starts 2024-09-16T10:20:23.408 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:23.408 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:23.409 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:23.441 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:23.442 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:23.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:23.475 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:23.475 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:23.510 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:23.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:23.546 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:23.547 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:23.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:23 smithi165 ceph-mon[32696]: 8.b scrub ok 2024-09-16T10:20:23.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:23 smithi165 ceph-mon[32696]: 8.f scrub starts 2024-09-16T10:20:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:23 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:23.580 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:23.580 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:23.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:23.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:23.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:23.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:23.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:23.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:23.889 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:23.931 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:23.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:24.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:24.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:24.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:24.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:24.210 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:24.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:24.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:24.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:24.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:24.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[29993]: pgmap v334: 168 pgs: 7 peering, 1 active+clean+scrubbing, 160 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 6.7 KiB/s rd, 37 KiB/s wr, 1.24k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3845807673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[29993]: 5.1 scrub starts 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[29993]: 5.1 scrub ok 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:20:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:20:24.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[39470]: pgmap v334: 168 pgs: 7 peering, 1 active+clean+scrubbing, 160 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 6.7 KiB/s rd, 37 KiB/s wr, 1.24k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-16T10:20:24.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3845807673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:24.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[39470]: 5.1 scrub starts 2024-09-16T10:20:24.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:24 smithi063 ceph-mon[39470]: 5.1 scrub ok 2024-09-16T10:20:24.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:24.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:24.502 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 216, 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:20:20.823283+0000", 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:18:51.797277+0000", 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:20:24.503 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 8, 2024-09-16T10:20:24.504 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:20:24.505 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:20:24.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 21, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 21, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 21, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 21, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 21, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:20:24.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "214", 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "211", 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:20:24.508 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:20:24.509 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:20:24.510 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.8999999761581421, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.8999999761581421, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-16T10:20:24.511 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:20:24.512 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:20:24.513 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:20:24.514 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:20:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:20:24.516 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6700000762939453, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6700000762939453, 2024-09-16T10:20:24.517 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.75, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:20:24.518 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:20:24.519 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:20:24.520 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:20:24.520 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:20:24.520 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:20:24.520 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:20:24.520 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:20:24.520 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:20:24.520 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:20:24.521 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:20:24.522 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:20:24.523 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:20:24.524 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.4900000095367432, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.4900000095367432, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.309999942779541, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.309999942779541, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:20:24.525 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:20:24.526 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:20:24.527 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:20:24.528 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:20:24.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:20:24.530 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.2400000095367432, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.2400000095367432, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:20:24.532 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:20:24.533 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:20:24.534 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:20:24.535 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:20:24.536 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.4900000095367432, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.4900000095367432, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.309999942779541, 2024-09-16T10:20:24.537 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.309999942779541, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:20:24.538 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:20:24.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:20:24.540 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:20:24.541 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:20:24.542 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.9900000095367432, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.9900000095367432, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 8, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_1", 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:18:32.502087+0000", 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:20:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:20:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "168", 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:20:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:20:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:20:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.9900000095367432, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.9900000095367432, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:20:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 214, 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:20:24.554 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.555 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.555 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:20:24.555 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:20:24.555 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:20:24.556 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:24 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:20:24.556 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:24 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:20:24.556 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:24 smithi165 ceph-mon[32696]: pgmap v334: 168 pgs: 7 peering, 1 active+clean+scrubbing, 160 active+clean; 7.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 6.7 KiB/s rd, 37 KiB/s wr, 1.24k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-16T10:20:24.556 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3845807673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:24.556 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:24 smithi165 ceph-mon[32696]: 5.1 scrub starts 2024-09-16T10:20:24.556 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:24 smithi165 ceph-mon[32696]: 5.1 scrub ok 2024-09-16T10:20:24.556 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:20:24.556 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.556 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.557 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 214, 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.558 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.559 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:20:24.561 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 0, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.562 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.563 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.565 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 214, 2024-09-16T10:20:24.566 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:20:24.568 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:20:24.569 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:20:24.570 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 214, 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.571 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:20:24.574 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 214, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:20:24.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.578 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.579 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 214, 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.580 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:20:24.581 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:20:24.582 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:20:24.583 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 214, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:20:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:20:24.585 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:20:24.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:20:24.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:16.003017+0000", 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.588 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:19:21.191067+0000", 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 65536, 2024-09-16T10:20:24.589 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:19:31.238634+0000", 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:01.806103+0000", 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.590 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:19:24.951610+0000", 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.591 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:06.766983+0000", 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:10.106604+0000", 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:04.244984+0000", 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [], 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:20:24.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:20:24.594 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.9", 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 1 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:20:24.596 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:20:24.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:24.597 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:20:24.597 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.598 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:20:24.599 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:24.600 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:20:24.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:24.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:24.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:24.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:24.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:24.874 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:24.907 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:24.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:25.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:25.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:25.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:25.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:25.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:25.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:25.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:25.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:25 smithi063 ceph-mon[29993]: 6.1b scrub starts 2024-09-16T10:20:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:25 smithi063 ceph-mon[29993]: 6.1b scrub ok 2024-09-16T10:20:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/507137432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:20:25.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:25 smithi063 ceph-mon[39470]: 6.1b scrub starts 2024-09-16T10:20:25.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:25 smithi063 ceph-mon[39470]: 6.1b scrub ok 2024-09-16T10:20:25.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/507137432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:20:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.453 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:25.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:25 smithi165 ceph-mon[32696]: 6.1b scrub starts 2024-09-16T10:20:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:25 smithi165 ceph-mon[32696]: 6.1b scrub ok 2024-09-16T10:20:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/507137432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:20:25.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:25.564 INFO:tasks.thrashosds.thrasher:j is {'epoch': 216, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:20:20.823283+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:18:51.797277+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '214', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '211', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-16T10:18:32.502087+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '168', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:16.003017+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:19:21.191067+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T10:19:31.238634+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:01.806103+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:19:24.951610+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:06.766983+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:10.106604+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:04.244984+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.9', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:20:25.565 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.9 2024-09-16T10:20:25.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap-items 6.9 2024-09-16T10:20:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.590 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:25.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:25.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:25.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.731 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:25.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:25.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:25.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:25.924 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:25.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:25.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:25.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:26.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:26.128 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:26.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:26.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:26.300 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:26.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:26.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:26.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[29993]: 8.f scrub ok 2024-09-16T10:20:26.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[29993]: pgmap v335: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.94k op/s 2024-09-16T10:20:26.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-16T10:20:26.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:20:26.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:20:26.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:20:26.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:20:26.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[39470]: 8.f scrub ok 2024-09-16T10:20:26.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[39470]: pgmap v335: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.94k op/s 2024-09-16T10:20:26.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-16T10:20:26.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:20:26.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:20:26.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:20:26.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:26 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:20:26.444 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:26.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:26.531 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:26.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:26 smithi165 ceph-mon[32696]: 8.f scrub ok 2024-09-16T10:20:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:26 smithi165 ceph-mon[32696]: pgmap v335: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.94k op/s 2024-09-16T10:20:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:26 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-16T10:20:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:26 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:20:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:26 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:20:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:26 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:20:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:26 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:39764 7.6 e97/81 7:680231a8:::benchmark_data_smithi063_73086_object39763:head v 97'2523, mlcod=97'2521)", 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.247108+0000", 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 317.32136653399999, 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13401402800000001, 2024-09-16T10:20:26.574 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247108+0000", 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247108+0000", 2024-09-16T10:20:26.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247109+0000", 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.102e-06 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247127+0000", 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7719000000000001e-05 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247135+0000", 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8030000000000003e-06 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247141+0000", 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1959999999999999e-06 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247196+0000", 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5050999999999999e-05 2024-09-16T10:20:26.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247209+0000", 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2826999999999999e-05 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381064+0000", 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13385559399999999 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381122+0000", 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7735999999999997e-05 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39762 7.a 7:5fb7c7c3:::benchmark_data_smithi063_73086_object39761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 317.32069685200003, 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.135043899, 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.579 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39762 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247778+0000", 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.16e-07 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247809+0000", 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0239999999999998e-05 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247817+0000", 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1280000000000008e-06 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247823+0000", 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1360000000000002e-06 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249271+0000", 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001448193 2024-09-16T10:20:26.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249391+0000", 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011986500000000001 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,5", 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249517+0000", 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125967 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252815+0000", 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003298282 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382711+0000", 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12989600700000001 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382722+0000", 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0957e-05 2024-09-16T10:20:26.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382773+0000", 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1057999999999998e-05 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382821+0000", 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8250000000000001e-05 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.586 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39769 7.3 7:c947093a:::benchmark_data_smithi063_73086_object39768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:20:26.586 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250737+0000", 2024-09-16T10:20:26.586 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 317.31773723700002, 2024-09-16T10:20:26.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.134149398, 2024-09-16T10:20:26.586 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.586 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.587 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.587 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39769 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250737+0000", 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250737+0000", 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250739+0000", 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.995e-06 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250851+0000", 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112194 2024-09-16T10:20:26.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250870+0000", 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8774999999999999e-05 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250877+0000", 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2099999999999996e-06 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251917+0000", 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001039704 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252030+0000", 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113205 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252158+0000", 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012766000000000001 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254112+0000", 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001953761 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254277+0000", 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000165625 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384796+0000", 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13051881200000001 2024-09-16T10:20:26.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384848+0000", 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2370999999999997e-05 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.384887+0000", 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8086000000000001e-05 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39773 7.2 7:405c1126:::benchmark_data_smithi063_73086_object39772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250895+0000", 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 317.31757962400002, 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13210709400000001, 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39773 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250895+0000", 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250895+0000", 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250896+0000", 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6500000000000008e-07 2024-09-16T10:20:26.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250920+0000", 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4595000000000001e-05 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250927+0000", 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2139999999999999e-06 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250934+0000", 2024-09-16T10:20:26.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8800000000000002e-06 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251195+0000", 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026043099999999999 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251316+0000", 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121551 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.251444+0000", 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012788900000000001 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382829+0000", 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13138502499999999 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382861+0000", 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1996000000000003e-05 2024-09-16T10:20:26.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382925+0000", 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3392999999999994e-05 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382969+0000", 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4174000000000001e-05 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383002+0000", 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2981000000000002e-05 2024-09-16T10:20:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65064 7.2 7:47084aaf:::benchmark_data_smithi063_73086_object65063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 299.02867291699999, 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.140792484, 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.600 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65064 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539802+0000", 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1500000000000004e-07 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539887+0000", 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5189e-05 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539896+0000", 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2810000000000008e-06 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539908+0000", 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2140999999999999e-05 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545096+0000", 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0051880040000000004 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545228+0000", 2024-09-16T10:20:26.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:20:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013243200000000001 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545886+0000", 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00065781499999999996 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.554263+0000", 2024-09-16T10:20:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0083769059999999999 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.554308+0000", 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4771000000000002e-05 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680481+0000", 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12617313499999999 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680544+0000", 2024-09-16T10:20:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3608000000000005e-05 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680594+0000", 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9586999999999998e-05 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65067 7.2 7:48a7aec7:::benchmark_data_smithi063_73086_object65066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:26.608 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 299.02750331300001, 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.139936807, 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65067 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540972+0000", 2024-09-16T10:20:26.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.282e-06 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540992+0000", 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9539999999999999e-05 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541001+0000", 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.704e-06 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541007+0000", 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4269999999999996e-06 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548200+0000", 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071930350000000004 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548316+0000", 2024-09-16T10:20:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116468 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548442+0000", 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012523999999999999 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563514+0000", 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015071782000000001 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680655+0000", 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.117141604 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680817+0000", 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016221599999999999 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680870+0000", 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2448999999999999e-05 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680908+0000", 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8059999999999998e-05 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65068 7.2 7:4e1ab327:::benchmark_data_smithi063_73086_object65067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 299.02745754699998, 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14002576999999999, 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.616 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65068 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6400000000000002e-07 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541038+0000", 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0228999999999999e-05 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541044+0000", 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3770000000000001e-06 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541050+0000", 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.485e-06 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.550940+0000", 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0098900169999999992 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.551033+0000", 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3523000000000006e-05 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.551110+0000", 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7131000000000004e-05 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680659+0000", 2024-09-16T10:20:26.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.129549215 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680696+0000", 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6424000000000002e-05 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680964+0000", 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026829000000000002 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681008+0000", 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3399000000000002e-05 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681043+0000", 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5015999999999998e-05 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.622 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65075 7.2 7:4c929240:::benchmark_data_smithi063_73086_object65074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 299.022517601, 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13568592600000001, 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65075 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545959+0000", 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8649999999999999e-06 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545980+0000", 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1696e-05 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545991+0000", 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0359e-05 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545999+0000", 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1370000000000002e-06 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559321+0000", 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013321932 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559528+0000", 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020748899999999999 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559651+0000", 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123013 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680667+0000", 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12101582399999999 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681281+0000", 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061402500000000003 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681562+0000", 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00028057400000000001 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681607+0000", 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5578000000000003e-05 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681643+0000", 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5434e-05 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.628 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65082 7.a 7:552f4fed:::benchmark_data_smithi063_73086_object65081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 299.01557315500003, 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.144873693, 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.629 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65082 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552904+0000", 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4499999999999998e-06 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552927+0000", 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3240000000000001e-05 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552940+0000", 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3579e-05 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552949+0000", 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1790000000000008e-06 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681082+0000", 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12813308800000001 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681222+0000", 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000140464 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,5", 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681372+0000", 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000149369 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.685517+0000", 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041449700000000004 2024-09-16T10:20:26.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.685528+0000", 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1440999999999999e-05 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697681+0000", 2024-09-16T10:20:26.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012153051 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697731+0000", 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0453e-05 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697775+0000", 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3408999999999997e-05 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.635 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65094 7.2 7:41461aa3:::benchmark_data_smithi063_73086_object65093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.562052+0000", 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 299.006422246, 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13390104, 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.636 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65094 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562052+0000", 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562052+0000", 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562054+0000", 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.68e-06 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562079+0000", 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4989000000000001e-05 2024-09-16T10:20:26.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562087+0000", 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.473e-06 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.562093+0000", 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1820000000000003e-06 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681672+0000", 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119578091 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.684384+0000", 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027128970000000001 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.684688+0000", 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030324399999999999 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692119+0000", 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0074318259999999999 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692659+0000", 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00053903700000000002 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692736+0000", 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7001000000000004e-05 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.692816+0000", 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0469000000000005e-05 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.695953+0000", 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031371509999999999 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.642 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65095 7.2 7:414dfc36:::benchmark_data_smithi063_73086_object65094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 299.004944137, 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13620320899999999, 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:20:26.643 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65095 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563532+0000", 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5659999999999999e-06 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563555+0000", 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2747000000000001e-05 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680763+0000", 2024-09-16T10:20:26.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.117208851 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680777+0000", 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3148e-05 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689111+0000", 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0083343150000000001 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689240+0000", 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012911199999999999 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689823+0000", 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058294800000000002 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.695962+0000", 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0061393569999999998 2024-09-16T10:20:26.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697651+0000", 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016887930000000001 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699637+0000", 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986032 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699691+0000", 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4326000000000001e-05 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699733+0000", 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2014000000000001e-05 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90038 7.6 e112/81 7:605b4ba6:::benchmark_data_smithi063_73086_object90037:head v 112'5732, mlcod=112'5730)", 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:20:26.649 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 277.06667449600002, 2024-09-16T10:20:26.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14626494100000001, 2024-09-16T10:20:26.650 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.650 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:26.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:26.651 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.651 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501801+0000", 2024-09-16T10:20:26.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.47e-07 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501838+0000", 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7397000000000001e-05 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501846+0000", 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2619999999999998e-06 2024-09-16T10:20:26.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501852+0000", 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4210000000000003e-06 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504465+0000", 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026126890000000001 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504484+0000", 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9619999999999998e-05 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646628+0000", 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.142143565 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648065+0000", 2024-09-16T10:20:26.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014370400000000001 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90048 7.6 e112/81 7:6327a166:::benchmark_data_smithi063_73086_object90047:head v 112'5733, mlcod=112'5730)", 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 277.06045630099999, 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.140110544, 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.656 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508020+0000", 2024-09-16T10:20:26.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.815e-06 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508049+0000", 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8785999999999998e-05 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508060+0000", 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1155000000000001e-05 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508067+0000", 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.289e-06 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508275+0000", 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020751799999999999 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508293+0000", 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7975e-05 2024-09-16T10:20:26.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648068+0000", 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13977563800000001 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648129+0000", 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0368000000000002e-05 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.660 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90050 7.6 e112/81 7:60112b96:::benchmark_data_smithi063_73086_object90049:head v 112'5734, mlcod=112'5730)", 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 277.05863004700001, 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13835439699999999, 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.661 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509846+0000", 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2079999999999999e-06 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509888+0000", 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.244e-05 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509897+0000", 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3610000000000007e-06 2024-09-16T10:20:26.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509904+0000", 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5420000000000002e-06 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.511017+0000", 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0011129860000000001 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.511034+0000", 2024-09-16T10:20:26.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6891000000000001e-05 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648132+0000", 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13709860700000001 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648199+0000", 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6361999999999999e-05 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90056 7.0 e112/81 7:081070f4:::benchmark_data_smithi063_73086_object90055:head v 112'5520, mlcod=112'5518)", 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.514081+0000", 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 277.05439287600001, 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.132391548, 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514081+0000", 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514081+0000", 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514083+0000", 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.128e-06 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514166+0000", 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2911999999999995e-05 2024-09-16T10:20:26.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514175+0000", 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0780000000000002e-06 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514179+0000", 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6589999999999998e-06 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514223+0000", 2024-09-16T10:20:26.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3744e-05 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514242+0000", 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8726999999999999e-05 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646382+0000", 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13214073300000001 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646473+0000", 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0567000000000002e-05 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51200 8.f 8:fe221df4:::benchmark_data_smithi063_101290_object51199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.306440+0000", 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.262034245999999, 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99471615099999999, 2024-09-16T10:20:26.670 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51200 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306440+0000", 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306440+0000", 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306442+0000", 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.486e-06 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306461+0000", 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8940000000000002e-05 2024-09-16T10:20:26.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306472+0000", 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1015999999999999e-05 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306477+0000", 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1630000000000004e-06 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306529+0000", 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2510999999999999e-05 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306536+0000", 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6200000000000001e-06 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.295906+0000", 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98937029099999996 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296049+0000", 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014290799999999999 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296156+0000", 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106758 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300931+0000", 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047748139999999996 2024-09-16T10:20:26.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300999+0000", 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8221999999999998e-05 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301050+0000", 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1183000000000001e-05 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301104+0000", 2024-09-16T10:20:26.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3686000000000001e-05 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301156+0000", 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2553000000000003e-05 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51210 8.f 8:f185333d:::benchmark_data_smithi063_101290_object51209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.320115+0000", 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.248359549, 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98597720799999999, 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51210 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320115+0000", 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320115+0000", 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320117+0000", 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8610000000000001e-06 2024-09-16T10:20:26.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320140+0000", 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3788e-05 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320155+0000", 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4246999999999999e-05 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320172+0000", 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7533e-05 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320244+0000", 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1761000000000006e-05 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320255+0000", 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1044e-05 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.298864+0000", 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97860911399999995 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.298997+0000", 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000132562 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299145+0000", 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014797599999999999 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303492+0000", 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043472229999999999 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303669+0000", 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017695800000000001 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305992+0000", 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023235019999999999 2024-09-16T10:20:26.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306047+0000", 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4920999999999998e-05 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306092+0000", 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4718000000000001e-05 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51245 8.f 8:fbb4951d:::benchmark_data_smithi063_101290_object51244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.345508+0000", 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.222966119000006, 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96301515500000001, 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51245 2024-09-16T10:20:26.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345508+0000", 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345508+0000", 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345510+0000", 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.933e-06 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345534+0000", 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3702000000000001e-05 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345547+0000", 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3266e-05 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345555+0000", 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7610000000000007e-06 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345618+0000", 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3330999999999994e-05 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345631+0000", 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2758999999999999e-05 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301191+0000", 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95556000500000005 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301334+0000", 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014330199999999999 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301564+0000", 2024-09-16T10:20:26.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00022991799999999999 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306110+0000", 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045453710000000003 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308381+0000", 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002271669 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308425+0000", 2024-09-16T10:20:26.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3473000000000002e-05 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308481+0000", 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6150000000000003e-05 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308523+0000", 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2515000000000002e-05 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51248 8.f 8:fa504e1d:::benchmark_data_smithi063_101290_object51247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.347607+0000", 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.220866999999998, 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96156481999999999, 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51248 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.692 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347607+0000", 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347607+0000", 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347609+0000", 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8300000000000001e-06 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347634+0000", 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4355e-05 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347650+0000", 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5988e-05 2024-09-16T10:20:26.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347658+0000", 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9999999999999996e-06 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347736+0000", 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8182999999999993e-05 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347748+0000", 2024-09-16T10:20:26.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2157999999999999e-05 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303734+0000", 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95598619699999998 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303866+0000", 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131519 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304017+0000", 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015108600000000001 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308562+0000", 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045456350000000001 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309060+0000", 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00049744100000000005 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309072+0000", 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2463000000000001e-05 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309130+0000", 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7555e-05 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309172+0000", 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2410000000000002e-05 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51266 8.f 8:ff3eb03d:::benchmark_data_smithi063_101290_object51265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.376408+0000", 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.192066585000006, 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93575255499999999, 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:26.700 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51266 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376408+0000", 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376408+0000", 2024-09-16T10:20:26.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376409+0000", 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1599999999999999e-06 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376432+0000", 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3329e-05 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376441+0000", 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1209999999999996e-06 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376446+0000", 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5050000000000001e-06 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376495+0000", 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8776999999999997e-05 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376501+0000", 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4999999999999996e-06 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306171+0000", 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92966961199999998 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306291+0000", 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119981 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306420+0000", 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000128925 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309180+0000", 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027602690000000001 2024-09-16T10:20:26.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309272+0000", 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1778000000000001e-05 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312052+0000", 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027807169999999998 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312112+0000", 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9951000000000002e-05 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312160+0000", 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7929999999999997e-05 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:20:26.707 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:20:26.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:26.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:26.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:26.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:26.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:26.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:27.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:27.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:27.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:27.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:27.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:27.205 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:20:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:27.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:27.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:27.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:27.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:27.445 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:27.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[29993]: 5.5 scrub starts 2024-09-16T10:20:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[29993]: 5.5 scrub ok 2024-09-16T10:20:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2735500330' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[29993]: from='client.45371 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[29993]: 1.a scrub starts 2024-09-16T10:20:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[29993]: 1.a scrub ok 2024-09-16T10:20:27.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:20:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:20:27.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[39470]: 5.5 scrub starts 2024-09-16T10:20:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[39470]: 5.5 scrub ok 2024-09-16T10:20:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2735500330' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[39470]: from='client.45371 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[39470]: 1.a scrub starts 2024-09-16T10:20:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:27 smithi063 ceph-mon[39470]: 1.a scrub ok 2024-09-16T10:20:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:27.482 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:27.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:27.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:27 smithi165 ceph-mon[32696]: 5.5 scrub starts 2024-09-16T10:20:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:27 smithi165 ceph-mon[32696]: 5.5 scrub ok 2024-09-16T10:20:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2735500330' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:27 smithi165 ceph-mon[32696]: from='client.45371 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:27 smithi165 ceph-mon[32696]: 1.a scrub starts 2024-09-16T10:20:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:27 smithi165 ceph-mon[32696]: 1.a scrub ok 2024-09-16T10:20:27.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:27.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:27.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:27.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:27.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:27.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:27.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:27.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:27.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:27.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:28.014 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:28.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:28.150 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:28.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:28.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:28.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:28.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:28.501 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[29993]: pgmap v336: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 865 op/s 2024-09-16T10:20:28.501 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:20:28.501 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:20:28.501 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[29993]: from='client.45371 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:28.501 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[29993]: osdmap e217: 8 total, 8 up, 7 in 2024-09-16T10:20:28.501 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[29993]: 1.8 scrub starts 2024-09-16T10:20:28.501 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[29993]: 1.8 scrub ok 2024-09-16T10:20:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[39470]: pgmap v336: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 865 op/s 2024-09-16T10:20:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:20:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:20:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[39470]: from='client.45371 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[39470]: osdmap e217: 8 total, 8 up, 7 in 2024-09-16T10:20:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[39470]: 1.8 scrub starts 2024-09-16T10:20:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:28 smithi063 ceph-mon[39470]: 1.8 scrub ok 2024-09-16T10:20:28.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:28.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:20:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-mon[32696]: pgmap v336: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 865 op/s 2024-09-16T10:20:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:20:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:20:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-mon[32696]: from='client.45371 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-mon[32696]: osdmap e217: 8 total, 8 up, 7 in 2024-09-16T10:20:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-mon[32696]: 1.8 scrub starts 2024-09-16T10:20:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:28 smithi165 ceph-mon[32696]: 1.8 scrub ok 2024-09-16T10:20:28.564 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:28.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:28.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:28.803 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:28.836 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:28.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:28.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:28.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:29.072 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:29.114 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:29.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:29.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:29.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:20:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:29.261 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:29.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:29.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:29.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:29.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:29.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:20:29.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:29.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:29.529 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:29.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:29.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:29 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:20:29.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:29 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:20:29.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:29 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:20:29.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:29 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:20:29.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:29.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:29.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:29.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:29.674 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:29.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:29.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:29.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:29.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:29.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:29.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:30.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:30.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:30.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:30.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:30.220 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:30.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:30.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:30.360 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:30.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:30.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:30.475 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:30.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:30 smithi165 ceph-mon[32696]: pgmap v338: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.76k op/s 2024-09-16T10:20:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:30 smithi165 ceph-mon[32696]: 3.12 scrub starts 2024-09-16T10:20:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:30 smithi165 ceph-mon[32696]: 3.12 scrub ok 2024-09-16T10:20:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:30 smithi165 ceph-mon[32696]: 5.c scrub starts 2024-09-16T10:20:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:30 smithi165 ceph-mon[32696]: 5.c scrub ok 2024-09-16T10:20:30.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[29993]: pgmap v338: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.76k op/s 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[29993]: 3.12 scrub starts 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[29993]: 3.12 scrub ok 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[29993]: 5.c scrub starts 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[29993]: 5.c scrub ok 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[39470]: pgmap v338: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.76k op/s 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[39470]: 3.12 scrub starts 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[39470]: 3.12 scrub ok 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[39470]: 5.c scrub starts 2024-09-16T10:20:30.614 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:30 smithi063 ceph-mon[39470]: 5.c scrub ok 2024-09-16T10:20:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:30.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:30.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:30.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:30.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:30.850 INFO:teuthology.orchestra.run.smithi063.stderr:clear 6.9 pg_upmap_items mapping 2024-09-16T10:20:30.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:30.925 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:30.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:31.027 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:31.061 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:31.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:31.095 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:31.095 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:31.125 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:31.125 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:31.155 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:31.155 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:31.162 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:31.181 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:31.182 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:31.191 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:31.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:31.205 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:31.205 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:31.228 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:31.228 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:31.245 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:31.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:31.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:31.318 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:31.318 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:31.326 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:31.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:31.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:31.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:20:31.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3026694585' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"} : dispatch 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: from='client.45377 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"} : dispatch 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: from='client.45377 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"}]': finished 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: osdmap e218: 8 total, 8 up, 7 in 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2816607635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: 3.d scrub starts 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[29993]: 3.d scrub ok 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3026694585' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"} : dispatch 2024-09-16T10:20:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: from='client.45377 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"} : dispatch 2024-09-16T10:20:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: from='client.45377 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"}]': finished 2024-09-16T10:20:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: osdmap e218: 8 total, 8 up, 7 in 2024-09-16T10:20:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2816607635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: 3.d scrub starts 2024-09-16T10:20:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:31 smithi063 ceph-mon[39470]: 3.d scrub ok 2024-09-16T10:20:31.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:31.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:20:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:20:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3026694585' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"} : dispatch 2024-09-16T10:20:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: from='client.45377 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"} : dispatch 2024-09-16T10:20:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: from='client.45377 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.9"}]': finished 2024-09-16T10:20:31.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: osdmap e218: 8 total, 8 up, 7 in 2024-09-16T10:20:31.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2816607635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:31.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: 3.d scrub starts 2024-09-16T10:20:31.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:31 smithi165 ceph-mon[32696]: 3.d scrub ok 2024-09-16T10:20:31.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:31.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:31.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:31.704 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:31.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:31.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:31.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:31.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:20:31.841 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:31.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:31.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:32.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:32.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:32.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:20:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.166 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:32.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:32.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:32.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:32.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 4.e scrub starts 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 4.e scrub ok 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: pgmap v340: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.78k op/s 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 5.3 scrub starts 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 4.19 scrub starts 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 4.19 scrub ok 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 5.3 scrub ok 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 3.4 scrub starts 2024-09-16T10:20:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 3.4 scrub ok 2024-09-16T10:20:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 4.5 scrub starts 2024-09-16T10:20:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: 4.5 scrub ok 2024-09-16T10:20:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[29993]: osdmap e219: 8 total, 8 up, 7 in 2024-09-16T10:20:32.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 4.e scrub starts 2024-09-16T10:20:32.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 4.e scrub ok 2024-09-16T10:20:32.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: pgmap v340: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.78k op/s 2024-09-16T10:20:32.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 5.3 scrub starts 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 4.19 scrub starts 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 4.19 scrub ok 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 5.3 scrub ok 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 3.4 scrub starts 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 3.4 scrub ok 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 4.5 scrub starts 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: 4.5 scrub ok 2024-09-16T10:20:32.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:32 smithi063 ceph-mon[39470]: osdmap e219: 8 total, 8 up, 7 in 2024-09-16T10:20:32.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:32.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 4.e scrub starts 2024-09-16T10:20:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 4.e scrub ok 2024-09-16T10:20:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: pgmap v340: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.78k op/s 2024-09-16T10:20:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 5.3 scrub starts 2024-09-16T10:20:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 4.19 scrub starts 2024-09-16T10:20:32.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 4.19 scrub ok 2024-09-16T10:20:32.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 5.3 scrub ok 2024-09-16T10:20:32.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 3.4 scrub starts 2024-09-16T10:20:32.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 3.4 scrub ok 2024-09-16T10:20:32.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 4.5 scrub starts 2024-09-16T10:20:32.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: 4.5 scrub ok 2024-09-16T10:20:32.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:32 smithi165 ceph-mon[32696]: osdmap e219: 8 total, 8 up, 7 in 2024-09-16T10:20:32.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:32.589 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:32.591 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:32.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:32.725 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:32.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:32.861 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:32.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:32.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:32.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:32.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:33.100 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:33.128 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:33.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:33.219 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:33.219 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:33.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:33.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:33.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:33.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:33.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:33.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:33.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:33.539 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:33.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:33 smithi165 ceph-mon[32696]: 4.1b scrub starts 2024-09-16T10:20:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:33 smithi165 ceph-mon[32696]: 4.1b scrub ok 2024-09-16T10:20:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:33 smithi165 ceph-mon[32696]: 5.17 scrub starts 2024-09-16T10:20:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:33 smithi165 ceph-mon[32696]: 5.17 scrub ok 2024-09-16T10:20:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:33 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:20:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:33 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:20:33.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:33.672 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:33.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:33.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[29993]: 4.1b scrub starts 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[29993]: 4.1b scrub ok 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[29993]: 5.17 scrub starts 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[29993]: 5.17 scrub ok 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[39470]: 4.1b scrub starts 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[39470]: 4.1b scrub ok 2024-09-16T10:20:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[39470]: 5.17 scrub starts 2024-09-16T10:20:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[39470]: 5.17 scrub ok 2024-09-16T10:20:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:20:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:33 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:20:33.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:33.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:33.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:33.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:33.948 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:33.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:33.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:20:34.050 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:34.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:34.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:34.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:34.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:34.242 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:20:34.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:34.367 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:34.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:34.414 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:34.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:34.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:34.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:34 smithi165 ceph-mon[32696]: pgmap v342: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-16T10:20:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:34 smithi165 ceph-mon[32696]: 6.18 scrub starts 2024-09-16T10:20:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:34 smithi165 ceph-mon[32696]: 3.1e scrub starts 2024-09-16T10:20:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:34 smithi165 ceph-mon[32696]: 3.1e scrub ok 2024-09-16T10:20:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:34 smithi165 ceph-mon[32696]: 6.18 scrub ok 2024-09-16T10:20:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/380496561' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:20:34.616 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:34.647 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[29993]: pgmap v342: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-16T10:20:34.647 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[29993]: 6.18 scrub starts 2024-09-16T10:20:34.647 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[29993]: 3.1e scrub starts 2024-09-16T10:20:34.647 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[29993]: 3.1e scrub ok 2024-09-16T10:20:34.647 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[29993]: 6.18 scrub ok 2024-09-16T10:20:34.647 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/380496561' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:20:34.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[39470]: pgmap v342: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-16T10:20:34.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[39470]: 6.18 scrub starts 2024-09-16T10:20:34.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[39470]: 3.1e scrub starts 2024-09-16T10:20:34.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[39470]: 3.1e scrub ok 2024-09-16T10:20:34.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[39470]: 6.18 scrub ok 2024-09-16T10:20:34.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/380496561' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:20:34.655 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:34.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:34.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:34.789 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:34.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:34.847 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:20:34.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:34.931 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:34.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:35.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:35.051 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:35.051 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:35.065 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:35.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:35.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:35.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:35.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:35.276 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:35.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:35.347 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:35.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:35.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:35.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:35.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: 3.9 scrub starts 2024-09-16T10:20:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: 3.9 scrub ok 2024-09-16T10:20:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:35.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/380496561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:20:35.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: osdmap e220: 8 total, 8 up, 7 in 2024-09-16T10:20:35.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:35.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:35 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:35.583 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: 3.9 scrub starts 2024-09-16T10:20:35.583 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: 3.9 scrub ok 2024-09-16T10:20:35.583 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:35.583 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:35.583 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:35.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/380496561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:20:35.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: osdmap e220: 8 total, 8 up, 7 in 2024-09-16T10:20:35.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:35.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:35.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: 3.9 scrub starts 2024-09-16T10:20:35.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: 3.9 scrub ok 2024-09-16T10:20:35.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:35.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:35.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:35.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/380496561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:20:35.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: osdmap e220: 8 total, 8 up, 7 in 2024-09-16T10:20:35.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:35.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:35 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:35.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:35.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:35.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:35.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:35.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:35.766 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:35.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:35.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:20:35.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:35.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:35.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:36.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:36.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:36.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:36.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:36.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:36.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:36.224 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:36.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:36.347 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:36.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:36.415 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[29993]: pgmap v344: 168 pgs: 168 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:36.415 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[29993]: osdmap e221: 8 total, 8 up, 7 in 2024-09-16T10:20:36.415 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[29993]: osdmap e222: 8 total, 8 up, 7 in 2024-09-16T10:20:36.415 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4045348369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:36.415 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[39470]: pgmap v344: 168 pgs: 168 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:36.415 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[39470]: osdmap e221: 8 total, 8 up, 7 in 2024-09-16T10:20:36.415 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[39470]: osdmap e222: 8 total, 8 up, 7 in 2024-09-16T10:20:36.416 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4045348369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:36.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:36.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:36.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:36.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:36 smithi165 ceph-mon[32696]: pgmap v344: 168 pgs: 168 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:36.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:36 smithi165 ceph-mon[32696]: osdmap e221: 8 total, 8 up, 7 in 2024-09-16T10:20:36.544 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:36 smithi165 ceph-mon[32696]: osdmap e222: 8 total, 8 up, 7 in 2024-09-16T10:20:36.544 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4045348369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:36.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:36.608 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:36.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:36.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:36.738 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:36.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:36.833 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:20:36.833 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:20:36.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.833 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:163630 8.d 8:bdff9b9f:::benchmark_data_smithi063_101290_object36310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-16T10:20:36.833 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:36.823555+0000", 2024-09-16T10:20:36.833 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0068336630000000002, 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068635420000000003, 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 163630 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:36.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.823555+0000", 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.823555+0000", 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.823556+0000", 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3969999999999999e-06 2024-09-16T10:20:36.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.823560+0000", 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0589999999999996e-06 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.823567+0000", 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1509999999999998e-06 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.823574+0000", 2024-09-16T10:20:36.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.823650+0000", 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6154000000000003e-05 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.828030+0000", 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043801339999999999 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.828147+0000", 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011697999999999999 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:163641 8.d 8.deb0a41d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:36.829434+0000", 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00095406799999999997, 2024-09-16T10:20:36.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00105543, 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 163641 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.829434+0000", 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.829434+0000", 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.829436+0000", 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5209999999999999e-06 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:36.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.829440+0000", 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4279999999999998e-06 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.829449+0000", 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9660000000000002e-06 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.829456+0000", 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.841 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:163642 8.4 8:2eecb353:::benchmark_data_smithi063_101290_object36322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:36.830391+0000", 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9999976, 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015134999999999999, 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:36.842 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 163642 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830391+0000", 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830391+0000", 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830393+0000", 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6109999999999999e-06 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830398+0000", 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6289999999999998e-06 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:36.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830409+0000", 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1090999999999999e-05 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830417+0000", 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.559e-06 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830506+0000", 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9114000000000001e-05 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:36.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:163643 8.4 8.c4be1754 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:36.830434+0000", 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9999547, 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159772, 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:36.846 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 163643 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830434+0000", 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830434+0000", 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830434+0000", 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0399999999999996e-07 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830437+0000", 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8150000000000002e-06 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:36.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830443+0000", 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830448+0000", 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0769999999999998e-06 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.849 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:163644 8.8 8:1b170c11:::benchmark_data_smithi063_101290_object36324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:36.830454+0000", 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9999352, 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000198669, 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 163644 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830454+0000", 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830454+0000", 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830454+0000", 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4000000000000002e-07 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830456+0000", 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1179999999999999e-06 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830461+0000", 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.425e-06 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830465+0000", 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.216e-06 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830541+0000", 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6222000000000003e-05 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:36.830653+0000", 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112367 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:20:36.853 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 5 2024-09-16T10:20:36.854 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:20:36.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:36.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:36.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:37.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:20:37.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:37.036 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:37.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:37.120 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:20:37.120 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:20:37.120 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-09-16T10:20:37.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:20:37.138 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:37.195 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:37.195 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:37.195 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:37.196 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:37.197 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:37.197 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:37.275 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:37.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:37.290 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:20:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:20:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:20:37.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:37.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:37.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:37.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:37 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:20:37.566 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:37.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:37.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:37.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:37.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:37.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:37 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:20:37.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:37 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:20:37.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:37.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:37.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:37.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:37.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:20:37.972 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:37.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:38.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:38.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:38.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:38.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:38.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:38.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:38.354 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:38.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:38.421 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:38.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:38.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:38.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:38.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:38.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:38.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:38.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:38.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:38.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:38.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:38 smithi165 ceph-mon[32696]: pgmap v347: 168 pgs: 168 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 965 B/s rd, 0 B/s wr, 1.36k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:38.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:38.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:20:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:20:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:38.831 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:38.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:38.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:38.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[29993]: pgmap v347: 168 pgs: 168 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 965 B/s rd, 0 B/s wr, 1.36k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:38.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[39470]: pgmap v347: 168 pgs: 168 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 965 B/s rd, 0 B/s wr, 1.36k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:38.968 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:20:38.968 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:20:38.968 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:20:38.968 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:20:38.969 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:20:38.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:38.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:39.072 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:39.105 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:39.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:39.211 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:39.245 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:39.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:39.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:39.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:39.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:39.485 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:39.522 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:39.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:39.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:39.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:39.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:39.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:39.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:39.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:39.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:39.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:39.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:39.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:39.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:39.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:40.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:40.066 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:40.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:40.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:20:40.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:40.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:40.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:40.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:40.344 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:40.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:40.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:40.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:40.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:40.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:40.548 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:40.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:40.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:40.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:40.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:40.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:40.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:40.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:40 smithi165 ceph-mon[32696]: pgmap v348: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.27k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:40.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:40 smithi063 ceph-mon[29993]: pgmap v348: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.27k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:40.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:40 smithi063 ceph-mon[39470]: pgmap v348: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.27k op/s; 2 B/s, 0 objects/s recovering 2024-09-16T10:20:40.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:40.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:40.917 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:40.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:41.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:41.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:41.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:41.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90056 7.0 7:081070f4:::benchmark_data_smithi063_73086_object90055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 291.67770365899997, 2024-09-16T10:20:41.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13312953299999999, 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90056 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:20:41.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513590+0000", 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.531e-06 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513668+0000", 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8187000000000002e-05 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513679+0000", 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0781999999999999e-05 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513685+0000", 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.93e-06 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513734+0000", 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9539000000000001e-05 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513851+0000", 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116749 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513939+0000", 2024-09-16T10:20:41.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8208999999999995e-05 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519242+0000", 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053023430000000002 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523299+0000", 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040579250000000004 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646650+0000", 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.123350551 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646686+0000", 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6316999999999998e-05 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646718+0000", 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1470000000000002e-05 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.202 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90058 7.0 7:0216797c:::benchmark_data_smithi063_73086_object90057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 291.67750497200001, 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13450020500000001, 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:20:41.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90058 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513788+0000", 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1289999999999999e-06 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516073+0000", 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022849010000000002 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516088+0000", 2024-09-16T10:20:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5031000000000001e-05 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516096+0000", 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9680000000000006e-06 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516691+0000", 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00059533899999999996 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516812+0000", 2024-09-16T10:20:41.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120388 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516932+0000", 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120689 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519245+0000", 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023127080000000001 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523333+0000", 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040881149999999998 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648185+0000", 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124851428 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648246+0000", 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1354999999999998e-05 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648287+0000", 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1153999999999999e-05 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90059 7.0 7:0d97e368:::benchmark_data_smithi063_73086_object90058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 291.67517625099998, 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.1397098, 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.209 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.210 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:20:41.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:20:41.210 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90059 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516117+0000", 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5400000000000001e-06 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516153+0000", 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5546000000000002e-05 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516161+0000", 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8539999999999997e-06 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516166+0000", 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2310000000000004e-06 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519295+0000", 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031290430000000002 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519373+0000", 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8521000000000003e-05 2024-09-16T10:20:41.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519458+0000", 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4351000000000006e-05 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521937+0000", 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024794130000000002 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523362+0000", 2024-09-16T10:20:41.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001425228 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655714+0000", 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13235127199999999 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655770+0000", 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6622000000000001e-05 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655826+0000", 2024-09-16T10:20:41.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5179000000000001e-05 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51212 8.4 8:27b2b5e1:::benchmark_data_smithi063_101290_object51211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.870580054000001, 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98431940500000004, 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.216 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51212 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:20:41.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320713+0000", 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.559e-06 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320735+0000", 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2010000000000001e-05 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320745+0000", 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6690000000000001e-06 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320750+0000", 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2939999999999996e-06 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320816+0000", 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5090000000000002e-05 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320824+0000", 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8599999999999999e-06 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295578+0000", 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97475403100000002 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295697+0000", 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118301 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295799+0000", 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102376 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304751+0000", 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089521540000000004 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304762+0000", 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1044e-05 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304922+0000", 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015943199999999999 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304966+0000", 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4545000000000001e-05 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305031+0000", 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5040000000000001e-05 2024-09-16T10:20:41.222 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51220 8.4 8:2e2a097c:::benchmark_data_smithi063_101290_object51219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.865412122999999, 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97951371499999995, 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.223 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51220 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325882+0000", 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.827e-06 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325937+0000", 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5121000000000003e-05 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325948+0000", 2024-09-16T10:20:41.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1066999999999999e-05 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325954+0000", 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999999e-06 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326011+0000", 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6937000000000001e-05 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326023+0000", 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1344e-05 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298310+0000", 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97228777099999997 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298432+0000", 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012194200000000001 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298661+0000", 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000228497 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304876+0000", 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062151819999999997 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305227+0000", 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035094899999999998 2024-09-16T10:20:41.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305288+0000", 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0804000000000003e-05 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305361+0000", 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2706999999999994e-05 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305394+0000", 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2972999999999998e-05 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51223 8.4 8:287d1d16:::benchmark_data_smithi063_101290_object51222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.864773080999996, 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98451053700000002, 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.230 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51223 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326520+0000", 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.079e-06 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326573+0000", 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2924999999999999e-05 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326581+0000", 2024-09-16T10:20:41.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3659999999999992e-06 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326586+0000", 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8369999999999996e-06 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326636+0000", 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9667999999999999e-05 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326646+0000", 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0389e-05 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302523+0000", 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.975876682 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302647+0000", 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123745 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302774+0000", 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012772799999999999 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305245+0000", 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024711160000000002 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309074+0000", 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038286169999999999 2024-09-16T10:20:41.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.310943+0000", 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018688979999999999 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.310987+0000", 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4057999999999997e-05 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311029+0000", 2024-09-16T10:20:41.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2429000000000003e-05 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51230 8.4 8:23e3d41e:::benchmark_data_smithi063_101290_object51229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.853135296999994, 2024-09-16T10:20:41.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97448984100000002, 2024-09-16T10:20:41.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:41.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:41.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:41.239 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.239 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.239 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51230 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338159+0000", 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1610000000000001e-06 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338222+0000", 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3161000000000001e-05 2024-09-16T10:20:41.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338235+0000", 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3129000000000001e-05 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338243+0000", 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8420000000000005e-06 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338299+0000", 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6477999999999997e-05 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338313+0000", 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3852000000000001e-05 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305424+0000", 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96711084599999997 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305539+0000", 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115146 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305678+0000", 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013881700000000001 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311084+0000", 2024-09-16T10:20:41.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054059110000000002 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311113+0000", 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9405999999999999e-05 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312548+0000", 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001434533 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312609+0000", 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1503999999999999e-05 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312646+0000", 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7054999999999997e-05 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51243 8.4 8:253d325f:::benchmark_data_smithi063_101290_object51242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:41.246 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.848856306000002, 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97387738800000001, 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51243 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342437+0000", 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.505e-06 2024-09-16T10:20:41.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342507+0000", 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9720000000000003e-05 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342520+0000", 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3478999999999999e-05 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342529+0000", 2024-09-16T10:20:41.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3380000000000005e-06 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342652+0000", 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012287600000000001 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342658+0000", 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5889999999999999e-06 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307308+0000", 2024-09-16T10:20:41.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96465014199999999 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307405+0000", 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6510999999999998e-05 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307533+0000", 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012844199999999999 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312655+0000", 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051218239999999996 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.315330+0000", 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026750229999999999 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316199+0000", 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086935199999999999 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316263+0000", 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3326999999999999e-05 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316313+0000", 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0260000000000003e-05 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.253 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51249 8.4 8:208cf65c:::benchmark_data_smithi063_101290_object51248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.842452365, 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96761772599999996, 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51249 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348841+0000", 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5460000000000001e-06 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348879+0000", 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7716999999999998e-05 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348889+0000", 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0327e-05 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348896+0000", 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.742e-06 2024-09-16T10:20:41.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348953+0000", 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7028999999999999e-05 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348960+0000", 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4209999999999998e-06 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309120+0000", 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96015961500000002 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309226+0000", 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106158 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309348+0000", 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012231699999999999 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.315758+0000", 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064091460000000001 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316153+0000", 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039516799999999999 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316354+0000", 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000201452 2024-09-16T10:20:41.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316402+0000", 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7577999999999997e-05 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316457+0000", 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5510000000000002e-05 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51270 8.d 8:b5d1ccff:::benchmark_data_smithi063_101290_object51269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.885180684000005, 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.001723546, 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51270 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306113+0000", 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.136e-06 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306261+0000", 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147632 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306274+0000", 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2850000000000001e-05 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306282+0000", 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.388e-06 2024-09-16T10:20:41.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306347+0000", 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4898000000000001e-05 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306359+0000", 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2149e-05 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.296824+0000", 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99046429999999996 2024-09-16T10:20:41.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.296988+0000", 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164017 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.297156+0000", 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016855799999999999 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307591+0000", 2024-09-16T10:20:41.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010434652000000001 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307602+0000", 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.083e-05 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307749+0000", 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014768800000000001 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307800+0000", 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0800999999999997e-05 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307835+0000", 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4647000000000002e-05 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51294 8.d 8:b3f59715:::benchmark_data_smithi063_101290_object51293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.870292395999996, 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.986972195, 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51294 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.268 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321002+0000", 2024-09-16T10:20:41.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2060000000000001e-06 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321046+0000", 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4345000000000003e-05 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321063+0000", 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6688e-05 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321071+0000", 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321156+0000", 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4980000000000003e-05 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321163+0000", 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9949999999999999e-06 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299344+0000", 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97818064299999996 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299480+0000", 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013635700000000001 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299684+0000", 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000203857 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307681+0000", 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079967979999999994 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307717+0000", 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6297999999999997e-05 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307904+0000", 2024-09-16T10:20:41.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018723000000000001 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307943+0000", 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8831999999999997e-05 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307972+0000", 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8555999999999999e-05 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51296 8.d 8:b655e86a:::benchmark_data_smithi063_101290_object51295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.867893893000002, 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99276492999999999, 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51296 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.275 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323400+0000", 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4789999999999999e-06 2024-09-16T10:20:41.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323445+0000", 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5637999999999999e-05 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323455+0000", 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0127e-05 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323461+0000", 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.339e-06 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323504+0000", 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3461999999999998e-05 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323513+0000", 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2269999999999999e-06 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302178+0000", 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97866425099999999 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302303+0000", 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125862 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302439+0000", 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013551200000000001 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307685+0000", 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052464349999999998 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307870+0000", 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184811 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316064+0000", 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081937899999999994 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316116+0000", 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1690999999999997e-05 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316163+0000", 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7305999999999997e-05 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.282 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51303 8.d 8:b2b2f5d7:::benchmark_data_smithi063_101290_object51302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.862492477000004, 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99301229000000002, 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51303 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328801+0000", 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2470000000000001e-06 2024-09-16T10:20:41.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328903+0000", 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102128 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328928+0000", 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4760999999999999e-05 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328936+0000", 2024-09-16T10:20:41.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4400000000000005e-06 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328998+0000", 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1628e-05 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.329009+0000", 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1758e-05 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97591637799999997 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.305040+0000", 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113761 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.305141+0000", 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101047 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312025+0000", 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068839119999999998 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312061+0000", 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6918999999999997e-05 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321718+0000", 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096562799999999997 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321768+0000", 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0266000000000003e-05 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321812+0000", 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3764999999999999e-05 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.289 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51322 8.d 8:b9f45f41:::benchmark_data_smithi063_101290_object51321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.849826733, 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.980450343, 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51322 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341467+0000", 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1409999999999999e-06 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341501+0000", 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3157000000000001e-05 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341513+0000", 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2296999999999999e-05 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341520+0000", 2024-09-16T10:20:41.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5290000000000003e-06 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341603+0000", 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2180000000000003e-05 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341614+0000", 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1748e-05 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307997+0000", 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96638288500000002 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.308109+0000", 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112235 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.308233+0000", 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012397800000000001 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312071+0000", 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003837619 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.315989+0000", 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039176020000000001 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321846+0000", 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058578290000000002 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321886+0000", 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9400999999999998e-05 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321916+0000", 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9742000000000001e-05 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51332 8.d 8:baf4d594:::benchmark_data_smithi063_101290_object51331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.841086464, 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97184618199999995, 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51332 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.297 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350208+0000", 2024-09-16T10:20:41.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.176e-06 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350266+0000", 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7840999999999997e-05 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350279+0000", 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3718e-05 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350287+0000", 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5240000000000003e-06 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350342+0000", 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4755e-05 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350350+0000", 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1259999999999998e-06 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.309962+0000", 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95961197799999998 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.310060+0000", 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8022000000000004e-05 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.310179+0000", 2024-09-16T10:20:41.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119713 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.315951+0000", 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057717239999999998 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316034+0000", 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2670000000000006e-05 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321980+0000", 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059458560000000002 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322021+0000", 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1328999999999997e-05 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322052+0000", 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0750000000000002e-05 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.303 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51340 8.d 8:be7e8c60:::benchmark_data_smithi063_101290_object51339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.834614197999997, 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96553988000000002, 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.304 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51340 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356679+0000", 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356716+0000", 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6591000000000003e-05 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356726+0000", 2024-09-16T10:20:41.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6560000000000004e-06 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356733+0000", 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9789999999999996e-06 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356809+0000", 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6790000000000002e-05 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356819+0000", 2024-09-16T10:20:41.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8269999999999993e-06 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312094+0000", 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.955275333 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312207+0000", 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112341 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312340+0000", 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013320399999999999 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316004+0000", 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036634889999999998 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316206+0000", 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020242200000000001 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322131+0000", 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059253600000000002 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322183+0000", 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1372999999999998e-05 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322218+0000", 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4950000000000002e-05 2024-09-16T10:20:41.310 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51359 8.d 8:be832237:::benchmark_data_smithi063_101290_object51358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.754736808000004, 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88882225199999998, 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.311 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51359 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436556+0000", 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.328e-06 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436612+0000", 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5040999999999997e-05 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436622+0000", 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0491e-05 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436629+0000", 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9929999999999998e-06 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436696+0000", 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6544000000000005e-05 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436710+0000", 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4058e-05 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314089+0000", 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87737920700000005 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314198+0000", 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109348 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314324+0000", 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012599 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321682+0000", 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073583529999999998 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322263+0000", 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058049700000000002 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325295+0000", 2024-09-16T10:20:41.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030324670000000001 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325342+0000", 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6499999999999999e-05 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325377+0000", 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5435000000000002e-05 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51370 8.d 8:b1e0f19d:::benchmark_data_smithi063_101290_object51369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.746528603000002, 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88139332999999997, 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51370 2024-09-16T10:20:41.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444765+0000", 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.412e-06 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444815+0000", 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0201999999999998e-05 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444825+0000", 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0175e-05 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444830+0000", 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1429999999999997e-06 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444910+0000", 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9300000000000003e-05 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444919+0000", 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2499999999999995e-06 2024-09-16T10:20:41.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316237+0000", 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87131822199999998 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316359+0000", 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012239799999999999 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316474+0000", 2024-09-16T10:20:41.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011482800000000001 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321948+0000", 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054734090000000003 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322268+0000", 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032003100000000003 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326071+0000", 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038029539999999999 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326120+0000", 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9326999999999997e-05 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326157+0000", 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6678999999999999e-05 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.324 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51382 8.d 8:b6fd2994:::benchmark_data_smithi063_101290_object51381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.456229+0000", 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.735062968999998, 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87003440600000004, 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51382 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456229+0000", 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456229+0000", 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456231+0000", 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1900000000000002e-06 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456282+0000", 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0651e-05 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456296+0000", 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4289e-05 2024-09-16T10:20:41.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456305+0000", 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4179999999999997e-06 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456374+0000", 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9666999999999996e-05 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456388+0000", 2024-09-16T10:20:41.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4032000000000001e-05 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.318084+0000", 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86169573099999996 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.318200+0000", 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115563 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:20:41.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.318337+0000", 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013709599999999999 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322275+0000", 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039379439999999996 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325445+0000", 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031708299999999999 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326189+0000", 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074375799999999996 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326233+0000", 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3921000000000002e-05 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326263+0000", 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0315999999999999e-05 2024-09-16T10:20:41.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:164640 8.8 8:14c99ab6:::benchmark_data_smithi063_101290_object37320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:37.595518+0000", 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5957735729999998, 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12907907199999999, 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:41.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164640 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595518+0000", 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595518+0000", 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:41.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595520+0000", 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8109999999999999e-06 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595526+0000", 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5589999999999996e-06 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595536+0000", 2024-09-16T10:20:41.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9979999999999998e-06 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595542+0000", 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5509999999999996e-06 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595608+0000", 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5627e-05 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:41.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595749+0000", 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140712 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595832+0000", 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3505999999999996e-05 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.598157+0000", 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023244089999999999 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.598587+0000", 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000430145 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.724500+0000", 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12591281200000001 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.724552+0000", 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2630000000000003e-05 2024-09-16T10:20:41.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.724597+0000", 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5312000000000003e-05 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:41.338 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:20:41.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:41.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:41.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:41.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:41.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:41.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:41.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:41.658 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:41.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:41.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:41.802 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:41.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:41.881 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:20:41.885 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:20:41.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:41.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:41.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:42.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:42.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:41 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:20:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:41 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:20:42.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:41 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:42.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:41 smithi165 ceph-mon[32696]: pgmap v349: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 988 op/s 2024-09-16T10:20:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:42.085 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:42.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:42.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:42.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:20:42.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:20:42.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:42.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[29993]: pgmap v349: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 988 op/s 2024-09-16T10:20:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:20:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:20:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:20:42.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:41 smithi063 ceph-mon[39470]: pgmap v349: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 988 op/s 2024-09-16T10:20:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:42.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:42.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:42.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:20:42.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:42.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:42.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:42.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:42.517 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:42.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:42.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:42.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:42.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:42.727 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:42.760 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-16T10:20:41.161456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5836162269,"num_objects":89242,"num_object_clones":0,"num_object_copies":267726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89242,"num_whiteouts":0,"num_read":1137,"num_read_kb":889,"num_write":166029,"num_write_kb":8148334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10190,"num_bytes_recovered":663357726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69303,"ondisk_log_size":69303,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19035576,"kb_used_data":17216572,"kb_used_omap":10,"kb_used_meta":1818869,"kb_avail":637180488,"statfs":{"total":671965249536,"available":652472819712,"internally_reserved":0,"allocated":17629769728,"data_stored":17626168240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":1862521964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-447807488,"num_objects":-6833,"num_object_clones":0,"num_object_copies":-20499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6833,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6833,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.915153"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":371,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839508+0000","last_change":"2024-09-16T10:18:52.830567+0000","last_active":"2024-09-16T10:20:35.839508+0000","last_peered":"2024-09-16T10:20:35.839508+0000","last_clean":"2024-09-16T10:20:35.839508+0000","last_became_active":"2024-09-16T10:18:52.830260+0000","last_became_peered":"2024-09-16T10:18:52.830260+0000","last_unstale":"2024-09-16T10:20:35.839508+0000","last_undegraded":"2024-09-16T10:20:35.839508+0000","last_fullsized":"2024-09-16T10:20:35.839508+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:07.119072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.119072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.092096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":417,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.834765+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:20:35.834765+0000","last_peered":"2024-09-16T10:20:35.834765+0000","last_clean":"2024-09-16T10:20:35.834765+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:20:35.834765+0000","last_undegraded":"2024-09-16T10:20:35.834765+0000","last_fullsized":"2024-09-16T10:20:35.834765+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:25.723031+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837027+0000","last_change":"2024-09-16T10:18:40.752790+0000","last_active":"2024-09-16T10:20:35.837027+0000","last_peered":"2024-09-16T10:20:35.837027+0000","last_clean":"2024-09-16T10:20:35.837027+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:20:35.837027+0000","last_undegraded":"2024-09-16T10:20:35.837027+0000","last_fullsized":"2024-09-16T10:20:35.837027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:40.752739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:40.752739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.494552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":343,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.834814+0000","last_change":"2024-09-16T10:19:23.784420+0000","last_active":"2024-09-16T10:20:35.834814+0000","last_peered":"2024-09-16T10:20:35.834814+0000","last_clean":"2024-09-16T10:20:35.834814+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:20:35.834814+0000","last_undegraded":"2024-09-16T10:20:35.834814+0000","last_fullsized":"2024-09-16T10:20:35.834814+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.784347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.784347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843338+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:20:35.843338+0000","last_peered":"2024-09-16T10:20:35.843338+0000","last_clean":"2024-09-16T10:20:35.843338+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:20:35.843338+0000","last_undegraded":"2024-09-16T10:20:35.843338+0000","last_fullsized":"2024-09-16T10:20:35.843338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:39.443408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":372,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843957+0000","last_change":"2024-09-16T10:20:33.257003+0000","last_active":"2024-09-16T10:20:35.843957+0000","last_peered":"2024-09-16T10:20:35.843957+0000","last_clean":"2024-09-16T10:20:35.843957+0000","last_became_active":"2024-09-16T10:18:52.836507+0000","last_became_peered":"2024-09-16T10:18:52.836507+0000","last_unstale":"2024-09-16T10:20:35.843957+0000","last_undegraded":"2024-09-16T10:20:35.843957+0000","last_fullsized":"2024-09-16T10:20:35.843957+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.256934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.256934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:55.621693+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":408,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843038+0000","last_change":"2024-09-16T10:20:31.270635+0000","last_active":"2024-09-16T10:20:35.843038+0000","last_peered":"2024-09-16T10:20:35.843038+0000","last_clean":"2024-09-16T10:20:35.843038+0000","last_became_active":"2024-09-16T10:18:52.835715+0000","last_became_peered":"2024-09-16T10:18:52.835715+0000","last_unstale":"2024-09-16T10:20:35.843038+0000","last_undegraded":"2024-09-16T10:20:35.843038+0000","last_fullsized":"2024-09-16T10:20:35.843038+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:20:31.270548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.270548+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:59.374039+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.834682+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:20:35.834682+0000","last_peered":"2024-09-16T10:20:35.834682+0000","last_clean":"2024-09-16T10:20:35.834682+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:20:35.834682+0000","last_undegraded":"2024-09-16T10:20:35.834682+0000","last_fullsized":"2024-09-16T10:20:35.834682+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:07.843544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840155+0000","last_change":"2024-09-16T10:19:57.272422+0000","last_active":"2024-09-16T10:20:35.840155+0000","last_peered":"2024-09-16T10:20:35.840155+0000","last_clean":"2024-09-16T10:20:35.840155+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:20:35.840155+0000","last_undegraded":"2024-09-16T10:20:35.840155+0000","last_fullsized":"2024-09-16T10:20:35.840155+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.272362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.272362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:13.881596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":375,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.835186+0000","last_change":"2024-09-16T10:20:30.008294+0000","last_active":"2024-09-16T10:20:35.835186+0000","last_peered":"2024-09-16T10:20:35.835186+0000","last_clean":"2024-09-16T10:20:35.835186+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:20:35.835186+0000","last_undegraded":"2024-09-16T10:20:35.835186+0000","last_fullsized":"2024-09-16T10:20:35.835186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:30.008215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:30.008215+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.975169+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.846459+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:20:35.846459+0000","last_peered":"2024-09-16T10:20:35.846459+0000","last_clean":"2024-09-16T10:20:35.846459+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:20:35.846459+0000","last_undegraded":"2024-09-16T10:20:35.846459+0000","last_fullsized":"2024-09-16T10:20:35.846459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.206970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":351,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837284+0000","last_change":"2024-09-16T10:20:33.259383+0000","last_active":"2024-09-16T10:20:35.837284+0000","last_peered":"2024-09-16T10:20:35.837284+0000","last_clean":"2024-09-16T10:20:35.837284+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:20:35.837284+0000","last_undegraded":"2024-09-16T10:20:35.837284+0000","last_fullsized":"2024-09-16T10:20:35.837284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.259340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.259340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:59.265290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":340,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.841281+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:20:35.841281+0000","last_peered":"2024-09-16T10:20:35.841281+0000","last_clean":"2024-09-16T10:20:35.841281+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:20:35.841281+0000","last_undegraded":"2024-09-16T10:20:35.841281+0000","last_fullsized":"2024-09-16T10:20:35.841281+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":369,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836415+0000","last_change":"2024-09-16T10:20:32.001472+0000","last_active":"2024-09-16T10:20:35.836415+0000","last_peered":"2024-09-16T10:20:35.836415+0000","last_clean":"2024-09-16T10:20:35.836415+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:20:35.836415+0000","last_undegraded":"2024-09-16T10:20:35.836415+0000","last_fullsized":"2024-09-16T10:20:35.836415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:20:32.001398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:32.001398+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.377925+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":348,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849233+0000","last_change":"2024-09-16T10:19:23.390542+0000","last_active":"2024-09-16T10:20:35.849233+0000","last_peered":"2024-09-16T10:20:35.849233+0000","last_clean":"2024-09-16T10:20:35.849233+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:20:35.849233+0000","last_undegraded":"2024-09-16T10:20:35.849233+0000","last_fullsized":"2024-09-16T10:20:35.849233+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:49.356783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.841077+0000","last_change":"2024-09-16T10:19:46.109382+0000","last_active":"2024-09-16T10:20:35.841077+0000","last_peered":"2024-09-16T10:20:35.841077+0000","last_clean":"2024-09-16T10:20:35.841077+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:20:35.841077+0000","last_undegraded":"2024-09-16T10:20:35.841077+0000","last_fullsized":"2024-09-16T10:20:35.841077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:46.109310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.109310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.462251+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836774+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:20:35.836774+0000","last_peered":"2024-09-16T10:20:35.836774+0000","last_clean":"2024-09-16T10:20:35.836774+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:20:35.836774+0000","last_undegraded":"2024-09-16T10:20:35.836774+0000","last_fullsized":"2024-09-16T10:20:35.836774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.949799+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":347,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838456+0000","last_change":"2024-09-16T10:19:15.182388+0000","last_active":"2024-09-16T10:20:35.838456+0000","last_peered":"2024-09-16T10:20:35.838456+0000","last_clean":"2024-09-16T10:20:35.838456+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:20:35.838456+0000","last_undegraded":"2024-09-16T10:20:35.838456+0000","last_fullsized":"2024-09-16T10:20:35.838456+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.182335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.182335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":176,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837520+0000","last_change":"2024-09-16T10:19:14.579717+0000","last_active":"2024-09-16T10:20:35.837520+0000","last_peered":"2024-09-16T10:20:35.837520+0000","last_clean":"2024-09-16T10:20:35.837520+0000","last_became_active":"2024-09-16T10:18:52.841218+0000","last_became_peered":"2024-09-16T10:18:52.841218+0000","last_unstale":"2024-09-16T10:20:35.837520+0000","last_undegraded":"2024-09-16T10:20:35.837520+0000","last_fullsized":"2024-09-16T10:20:35.837520+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837981+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:20:35.837981+0000","last_peered":"2024-09-16T10:20:35.837981+0000","last_clean":"2024-09-16T10:20:35.837981+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:20:35.837981+0000","last_undegraded":"2024-09-16T10:20:35.837981+0000","last_fullsized":"2024-09-16T10:20:35.837981+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.434331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":302,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844109+0000","last_change":"2024-09-16T10:19:48.110737+0000","last_active":"2024-09-16T10:20:35.844109+0000","last_peered":"2024-09-16T10:20:35.844109+0000","last_clean":"2024-09-16T10:20:35.844109+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:20:35.844109+0000","last_undegraded":"2024-09-16T10:20:35.844109+0000","last_fullsized":"2024-09-16T10:20:35.844109+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:48.110674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.110674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.967276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":354,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.835302+0000","last_change":"2024-09-16T10:19:22.803999+0000","last_active":"2024-09-16T10:20:35.835302+0000","last_peered":"2024-09-16T10:20:35.835302+0000","last_clean":"2024-09-16T10:20:35.835302+0000","last_became_active":"2024-09-16T10:18:52.834723+0000","last_became_peered":"2024-09-16T10:18:52.834723+0000","last_unstale":"2024-09-16T10:20:35.835302+0000","last_undegraded":"2024-09-16T10:20:35.835302+0000","last_fullsized":"2024-09-16T10:20:35.835302+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.204568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":402,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843335+0000","last_change":"2024-09-16T10:19:24.829364+0000","last_active":"2024-09-16T10:20:35.843335+0000","last_peered":"2024-09-16T10:20:35.843335+0000","last_clean":"2024-09-16T10:20:35.843335+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:20:35.843335+0000","last_undegraded":"2024-09-16T10:20:35.843335+0000","last_fullsized":"2024-09-16T10:20:35.843335+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:24.829255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.829255+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":362,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837231+0000","last_change":"2024-09-16T10:18:52.839387+0000","last_active":"2024-09-16T10:20:35.837231+0000","last_peered":"2024-09-16T10:20:35.837231+0000","last_clean":"2024-09-16T10:20:35.837231+0000","last_became_active":"2024-09-16T10:18:52.838765+0000","last_became_peered":"2024-09-16T10:18:52.838765+0000","last_unstale":"2024-09-16T10:20:35.837231+0000","last_undegraded":"2024-09-16T10:20:35.837231+0000","last_fullsized":"2024-09-16T10:20:35.837231+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:05.795954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.795954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:04.825505+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":348,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.852474+0000","last_change":"2024-09-16T10:18:52.847158+0000","last_active":"2024-09-16T10:20:35.852474+0000","last_peered":"2024-09-16T10:20:35.852474+0000","last_clean":"2024-09-16T10:20:35.852474+0000","last_became_active":"2024-09-16T10:18:52.838789+0000","last_became_peered":"2024-09-16T10:18:52.838789+0000","last_unstale":"2024-09-16T10:20:35.852474+0000","last_undegraded":"2024-09-16T10:20:35.852474+0000","last_fullsized":"2024-09-16T10:20:35.852474+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":363,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837199+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:20:35.837199+0000","last_peered":"2024-09-16T10:20:35.837199+0000","last_clean":"2024-09-16T10:20:35.837199+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:20:35.837199+0000","last_undegraded":"2024-09-16T10:20:35.837199+0000","last_fullsized":"2024-09-16T10:20:35.837199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.407870+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":375,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839554+0000","last_change":"2024-09-16T10:19:49.067027+0000","last_active":"2024-09-16T10:20:35.839554+0000","last_peered":"2024-09-16T10:20:35.839554+0000","last_clean":"2024-09-16T10:20:35.839554+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:20:35.839554+0000","last_undegraded":"2024-09-16T10:20:35.839554+0000","last_fullsized":"2024-09-16T10:20:35.839554+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:49.066953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:49.066953+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.060028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":452,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843581+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:20:35.843581+0000","last_peered":"2024-09-16T10:20:35.843581+0000","last_clean":"2024-09-16T10:20:35.843581+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:20:35.843581+0000","last_undegraded":"2024-09-16T10:20:35.843581+0000","last_fullsized":"2024-09-16T10:20:35.843581+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.301174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"222'10492","reported_seq":10618,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.453696+0000","last_change":"2024-09-16T10:18:33.825761+0000","last_active":"2024-09-16T10:20:38.453696+0000","last_peered":"2024-09-16T10:20:38.453696+0000","last_clean":"2024-09-16T10:20:38.453696+0000","last_became_active":"2024-09-16T10:18:33.825267+0000","last_became_peered":"2024-09-16T10:18:33.825267+0000","last_unstale":"2024-09-16T10:20:38.453696+0000","last_undegraded":"2024-09-16T10:20:38.453696+0000","last_fullsized":"2024-09-16T10:20:38.453696+0000","mapping_epoch":165,"log_start":"202'6413","ondisk_log_start":"202'6413","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4079,"log_dups_size":0,"ondisk_log_size":4079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.403700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364642304,"num_objects":5564,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10492,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":169,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839541+0000","last_change":"2024-09-16T10:20:26.408565+0000","last_active":"2024-09-16T10:20:35.839541+0000","last_peered":"2024-09-16T10:20:35.839541+0000","last_clean":"2024-09-16T10:20:35.839541+0000","last_became_active":"2024-09-16T10:18:52.825707+0000","last_became_peered":"2024-09-16T10:18:52.825707+0000","last_unstale":"2024-09-16T10:20:35.839541+0000","last_undegraded":"2024-09-16T10:20:35.839541+0000","last_fullsized":"2024-09-16T10:20:35.839541+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:26.408503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.408503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:31.713460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":378,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837915+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:20:35.837915+0000","last_peered":"2024-09-16T10:20:35.837915+0000","last_clean":"2024-09-16T10:20:35.837915+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:20:35.837915+0000","last_undegraded":"2024-09-16T10:20:35.837915+0000","last_fullsized":"2024-09-16T10:20:35.837915+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.449585+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844676+0000","last_change":"2024-09-16T10:19:43.028508+0000","last_active":"2024-09-16T10:20:35.844676+0000","last_peered":"2024-09-16T10:20:35.844676+0000","last_clean":"2024-09-16T10:20:35.844676+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:20:35.844676+0000","last_undegraded":"2024-09-16T10:20:35.844676+0000","last_fullsized":"2024-09-16T10:20:35.844676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:43.028436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:43.028436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.274639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":433,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836895+0000","last_change":"2024-09-16T10:18:36.623362+0000","last_active":"2024-09-16T10:20:35.836895+0000","last_peered":"2024-09-16T10:20:35.836895+0000","last_clean":"2024-09-16T10:20:35.836895+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:20:35.836895+0000","last_undegraded":"2024-09-16T10:20:35.836895+0000","last_fullsized":"2024-09-16T10:20:35.836895+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:36.623302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:36.623302+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:12.928759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":345,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839693+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:20:35.839693+0000","last_peered":"2024-09-16T10:20:35.839693+0000","last_clean":"2024-09-16T10:20:35.839693+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:20:35.839693+0000","last_undegraded":"2024-09-16T10:20:35.839693+0000","last_fullsized":"2024-09-16T10:20:35.839693+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.040726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"222'10403","reported_seq":10528,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.588021+0000","last_change":"2024-09-16T10:18:33.893627+0000","last_active":"2024-09-16T10:20:38.588021+0000","last_peered":"2024-09-16T10:20:38.588021+0000","last_clean":"2024-09-16T10:20:38.588021+0000","last_became_active":"2024-09-16T10:18:33.893281+0000","last_became_peered":"2024-09-16T10:18:33.893281+0000","last_unstale":"2024-09-16T10:20:38.588021+0000","last_undegraded":"2024-09-16T10:20:38.588021+0000","last_fullsized":"2024-09-16T10:20:38.588021+0000","mapping_epoch":165,"log_start":"201'5984","ondisk_log_start":"201'5984","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355008512,"num_objects":5417,"num_object_clones":0,"num_object_copies":16251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":173,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836075+0000","last_change":"2024-09-16T10:20:18.852161+0000","last_active":"2024-09-16T10:20:35.836075+0000","last_peered":"2024-09-16T10:20:35.836075+0000","last_clean":"2024-09-16T10:20:35.836075+0000","last_became_active":"2024-09-16T10:20:15.864502+0000","last_became_peered":"2024-09-16T10:20:15.864502+0000","last_unstale":"2024-09-16T10:20:35.836075+0000","last_undegraded":"2024-09-16T10:20:35.836075+0000","last_fullsized":"2024-09-16T10:20:35.836075+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.752706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.752706+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:20:47.966037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":385,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844549+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:20:35.844549+0000","last_peered":"2024-09-16T10:20:35.844549+0000","last_clean":"2024-09-16T10:20:35.844549+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:20:35.844549+0000","last_undegraded":"2024-09-16T10:20:35.844549+0000","last_fullsized":"2024-09-16T10:20:35.844549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:03.147111+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":321,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844417+0000","last_change":"2024-09-16T10:19:50.114162+0000","last_active":"2024-09-16T10:20:35.844417+0000","last_peered":"2024-09-16T10:20:35.844417+0000","last_clean":"2024-09-16T10:20:35.844417+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:20:35.844417+0000","last_undegraded":"2024-09-16T10:20:35.844417+0000","last_fullsized":"2024-09-16T10:20:35.844417+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.114098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.114098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:16.082035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":102,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.841566+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:20:35.841566+0000","last_peered":"2024-09-16T10:20:35.841566+0000","last_clean":"2024-09-16T10:20:35.841566+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:20:35.841566+0000","last_undegraded":"2024-09-16T10:20:35.841566+0000","last_fullsized":"2024-09-16T10:20:35.841566+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:36.618613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":367,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.835107+0000","last_change":"2024-09-16T10:19:44.711933+0000","last_active":"2024-09-16T10:20:35.835107+0000","last_peered":"2024-09-16T10:20:35.835107+0000","last_clean":"2024-09-16T10:20:35.835107+0000","last_became_active":"2024-09-16T10:18:52.835843+0000","last_became_peered":"2024-09-16T10:18:52.835843+0000","last_unstale":"2024-09-16T10:20:35.835107+0000","last_undegraded":"2024-09-16T10:20:35.835107+0000","last_fullsized":"2024-09-16T10:20:35.835107+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.727533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"222'10459","reported_seq":14093,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.253064+0000","last_change":"2024-09-16T10:19:15.299139+0000","last_active":"2024-09-16T10:20:38.253064+0000","last_peered":"2024-09-16T10:20:38.253064+0000","last_clean":"2024-09-16T10:20:38.253064+0000","last_became_active":"2024-09-16T10:19:15.293347+0000","last_became_peered":"2024-09-16T10:19:15.293347+0000","last_unstale":"2024-09-16T10:20:38.253064+0000","last_undegraded":"2024-09-16T10:20:38.253064+0000","last_fullsized":"2024-09-16T10:20:38.253064+0000","mapping_epoch":188,"log_start":"201'6334","ondisk_log_start":"201'6334","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4125,"log_dups_size":0,"ondisk_log_size":4125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:09.209374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371064856,"num_objects":5663,"num_object_clones":0,"num_object_copies":16989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10459,"num_write_kb":515841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1748,"num_bytes_recovered":114556928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":171,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844165+0000","last_change":"2024-09-16T10:20:18.860186+0000","last_active":"2024-09-16T10:20:35.844165+0000","last_peered":"2024-09-16T10:20:35.844165+0000","last_clean":"2024-09-16T10:20:35.844165+0000","last_became_active":"2024-09-16T10:20:15.858791+0000","last_became_peered":"2024-09-16T10:20:15.858791+0000","last_unstale":"2024-09-16T10:20:35.844165+0000","last_undegraded":"2024-09-16T10:20:35.844165+0000","last_fullsized":"2024-09-16T10:20:35.844165+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.024848+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:20:48.158381+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":389,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.834954+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:20:35.834954+0000","last_peered":"2024-09-16T10:20:35.834954+0000","last_clean":"2024-09-16T10:20:35.834954+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:20:35.834954+0000","last_undegraded":"2024-09-16T10:20:35.834954+0000","last_fullsized":"2024-09-16T10:20:35.834954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:15.756451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":365,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839713+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:20:35.839713+0000","last_peered":"2024-09-16T10:20:35.839713+0000","last_clean":"2024-09-16T10:20:35.839713+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:20:35.839713+0000","last_undegraded":"2024-09-16T10:20:35.839713+0000","last_fullsized":"2024-09-16T10:20:35.839713+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:14.162775+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":445,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844533+0000","last_change":"2024-09-16T10:18:04.023115+0000","last_active":"2024-09-16T10:20:35.844533+0000","last_peered":"2024-09-16T10:20:35.844533+0000","last_clean":"2024-09-16T10:20:35.844533+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:20:35.844533+0000","last_undegraded":"2024-09-16T10:20:35.844533+0000","last_fullsized":"2024-09-16T10:20:35.844533+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:04.022579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.022579+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:00.209183+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":344,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849411+0000","last_change":"2024-09-16T10:18:52.824583+0000","last_active":"2024-09-16T10:20:35.849411+0000","last_peered":"2024-09-16T10:20:35.849411+0000","last_clean":"2024-09-16T10:20:35.849411+0000","last_became_active":"2024-09-16T10:18:52.824265+0000","last_became_peered":"2024-09-16T10:18:52.824265+0000","last_unstale":"2024-09-16T10:20:35.849411+0000","last_undegraded":"2024-09-16T10:20:35.849411+0000","last_fullsized":"2024-09-16T10:20:35.849411+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"222'10210","reported_seq":13648,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.556645+0000","last_change":"2024-09-16T10:19:14.298863+0000","last_active":"2024-09-16T10:20:38.556645+0000","last_peered":"2024-09-16T10:20:38.556645+0000","last_clean":"2024-09-16T10:20:38.556645+0000","last_became_active":"2024-09-16T10:19:14.293461+0000","last_became_peered":"2024-09-16T10:19:14.293461+0000","last_unstale":"2024-09-16T10:20:38.556645+0000","last_undegraded":"2024-09-16T10:20:38.556645+0000","last_fullsized":"2024-09-16T10:20:38.556645+0000","mapping_epoch":187,"log_start":"201'5776","ondisk_log_start":"201'5776","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361234432,"num_objects":5512,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":242,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843344+0000","last_change":"2024-09-16T10:20:18.867503+0000","last_active":"2024-09-16T10:20:35.843344+0000","last_peered":"2024-09-16T10:20:35.843344+0000","last_clean":"2024-09-16T10:20:35.843344+0000","last_became_active":"2024-09-16T10:20:15.854970+0000","last_became_peered":"2024-09-16T10:20:15.854970+0000","last_unstale":"2024-09-16T10:20:35.843344+0000","last_undegraded":"2024-09-16T10:20:35.843344+0000","last_fullsized":"2024-09-16T10:20:35.843344+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.192777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.192777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:20:48.572116+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":384,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838054+0000","last_change":"2024-09-16T10:19:54.487297+0000","last_active":"2024-09-16T10:20:35.838054+0000","last_peered":"2024-09-16T10:20:35.838054+0000","last_clean":"2024-09-16T10:20:35.838054+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:20:35.838054+0000","last_undegraded":"2024-09-16T10:20:35.838054+0000","last_fullsized":"2024-09-16T10:20:35.838054+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:54.487219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.487219+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.245307+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840234+0000","last_change":"2024-09-16T10:18:52.836576+0000","last_active":"2024-09-16T10:20:35.840234+0000","last_peered":"2024-09-16T10:20:35.840234+0000","last_clean":"2024-09-16T10:20:35.840234+0000","last_became_active":"2024-09-16T10:18:52.836260+0000","last_became_peered":"2024-09-16T10:18:52.836260+0000","last_unstale":"2024-09-16T10:20:35.840234+0000","last_undegraded":"2024-09-16T10:20:35.840234+0000","last_fullsized":"2024-09-16T10:20:35.840234+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.187181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.187181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":441,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838085+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:20:35.838085+0000","last_peered":"2024-09-16T10:20:35.838085+0000","last_clean":"2024-09-16T10:20:35.838085+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:20:35.838085+0000","last_undegraded":"2024-09-16T10:20:35.838085+0000","last_fullsized":"2024-09-16T10:20:35.838085+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.577364+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":359,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837499+0000","last_change":"2024-09-16T10:19:50.044357+0000","last_active":"2024-09-16T10:20:35.837499+0000","last_peered":"2024-09-16T10:20:35.837499+0000","last_clean":"2024-09-16T10:20:35.837499+0000","last_became_active":"2024-09-16T10:18:52.842382+0000","last_became_peered":"2024-09-16T10:18:52.842382+0000","last_unstale":"2024-09-16T10:20:35.837499+0000","last_undegraded":"2024-09-16T10:20:35.837499+0000","last_fullsized":"2024-09-16T10:20:35.837499+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.044297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.044297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.918339+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"222'10347","reported_seq":13883,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.576261+0000","last_change":"2024-09-16T10:20:24.650095+0000","last_active":"2024-09-16T10:20:38.576261+0000","last_peered":"2024-09-16T10:20:38.576261+0000","last_clean":"2024-09-16T10:20:38.576261+0000","last_became_active":"2024-09-16T10:19:14.292726+0000","last_became_peered":"2024-09-16T10:19:14.292726+0000","last_unstale":"2024-09-16T10:20:38.576261+0000","last_undegraded":"2024-09-16T10:20:38.576261+0000","last_fullsized":"2024-09-16T10:20:38.576261+0000","mapping_epoch":187,"log_start":"201'6273","ondisk_log_start":"201'6273","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'9052","last_scrub_stamp":"2024-09-16T10:20:24.649693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.649693+0000","objects_scrubbed":6970,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:33.914769+0000","scrub_duration":2436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365101056,"num_objects":5571,"num_object_clones":0,"num_object_copies":16713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10347,"num_write_kb":509376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":108724224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":244,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844272+0000","last_change":"2024-09-16T10:20:19.537775+0000","last_active":"2024-09-16T10:20:35.844272+0000","last_peered":"2024-09-16T10:20:35.844272+0000","last_clean":"2024-09-16T10:20:35.844272+0000","last_became_active":"2024-09-16T10:20:15.856484+0000","last_became_peered":"2024-09-16T10:20:15.856484+0000","last_unstale":"2024-09-16T10:20:35.844272+0000","last_undegraded":"2024-09-16T10:20:35.844272+0000","last_fullsized":"2024-09-16T10:20:35.844272+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.537727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.537727+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.959881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":396,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838222+0000","last_change":"2024-09-16T10:19:22.293172+0000","last_active":"2024-09-16T10:20:35.838222+0000","last_peered":"2024-09-16T10:20:35.838222+0000","last_clean":"2024-09-16T10:20:35.838222+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:20:35.838222+0000","last_undegraded":"2024-09-16T10:20:35.838222+0000","last_fullsized":"2024-09-16T10:20:35.838222+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:22.293076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.293076+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.645714+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":372,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838836+0000","last_change":"2024-09-16T10:20:31.548535+0000","last_active":"2024-09-16T10:20:35.838836+0000","last_peered":"2024-09-16T10:20:35.838836+0000","last_clean":"2024-09-16T10:20:35.838836+0000","last_became_active":"2024-09-16T10:18:52.832749+0000","last_became_peered":"2024-09-16T10:18:52.832749+0000","last_unstale":"2024-09-16T10:20:35.838836+0000","last_undegraded":"2024-09-16T10:20:35.838836+0000","last_fullsized":"2024-09-16T10:20:35.838836+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:31.548465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.548465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.221683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.842720+0000","last_change":"2024-09-16T10:19:55.433215+0000","last_active":"2024-09-16T10:20:35.842720+0000","last_peered":"2024-09-16T10:20:35.842720+0000","last_clean":"2024-09-16T10:20:35.842720+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:20:35.842720+0000","last_undegraded":"2024-09-16T10:20:35.842720+0000","last_fullsized":"2024-09-16T10:20:35.842720+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:55.433149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.433149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.272672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836932+0000","last_change":"2024-09-16T10:18:52.827865+0000","last_active":"2024-09-16T10:20:35.836932+0000","last_peered":"2024-09-16T10:20:35.836932+0000","last_clean":"2024-09-16T10:20:35.836932+0000","last_became_active":"2024-09-16T10:18:52.827511+0000","last_became_peered":"2024-09-16T10:18:52.827511+0000","last_unstale":"2024-09-16T10:20:35.836932+0000","last_undegraded":"2024-09-16T10:20:35.836932+0000","last_fullsized":"2024-09-16T10:20:35.836932+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.486130+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"222'10504","reported_seq":10629,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.253901+0000","last_change":"2024-09-16T10:18:33.893596+0000","last_active":"2024-09-16T10:20:38.253901+0000","last_peered":"2024-09-16T10:20:38.253901+0000","last_clean":"2024-09-16T10:20:38.253901+0000","last_became_active":"2024-09-16T10:18:33.893254+0000","last_became_peered":"2024-09-16T10:18:33.893254+0000","last_unstale":"2024-09-16T10:20:38.253901+0000","last_undegraded":"2024-09-16T10:20:38.253901+0000","last_fullsized":"2024-09-16T10:20:38.253901+0000","mapping_epoch":165,"log_start":"201'6385","ondisk_log_start":"201'6385","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4119,"log_dups_size":0,"ondisk_log_size":4119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:04.275575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374472704,"num_objects":5714,"num_object_clones":0,"num_object_copies":17142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":173,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840905+0000","last_change":"2024-09-16T10:20:18.857559+0000","last_active":"2024-09-16T10:20:35.840905+0000","last_peered":"2024-09-16T10:20:35.840905+0000","last_clean":"2024-09-16T10:20:35.840905+0000","last_became_active":"2024-09-16T10:20:15.850975+0000","last_became_peered":"2024-09-16T10:20:15.850975+0000","last_unstale":"2024-09-16T10:20:35.840905+0000","last_undegraded":"2024-09-16T10:20:35.840905+0000","last_fullsized":"2024-09-16T10:20:35.840905+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.131122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.131122+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:20:48.280688+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"175'29","reported_seq":531,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838724+0000","last_change":"2024-09-16T10:18:52.860272+0000","last_active":"2024-09-16T10:20:35.838724+0000","last_peered":"2024-09-16T10:20:35.838724+0000","last_clean":"2024-09-16T10:20:35.838724+0000","last_became_active":"2024-09-16T10:18:52.841202+0000","last_became_peered":"2024-09-16T10:18:52.841202+0000","last_unstale":"2024-09-16T10:20:35.838724+0000","last_undegraded":"2024-09-16T10:20:35.838724+0000","last_fullsized":"2024-09-16T10:20:35.838724+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:18:45.654714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.654714+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:01.512318+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837289+0000","last_change":"2024-09-16T10:17:35.928758+0000","last_active":"2024-09-16T10:20:35.837289+0000","last_peered":"2024-09-16T10:20:35.837289+0000","last_clean":"2024-09-16T10:20:35.837289+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:20:35.837289+0000","last_undegraded":"2024-09-16T10:20:35.837289+0000","last_fullsized":"2024-09-16T10:20:35.837289+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.928701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.928701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:41.782908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":367,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840544+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:20:35.840544+0000","last_peered":"2024-09-16T10:20:35.840544+0000","last_clean":"2024-09-16T10:20:35.840544+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:20:35.840544+0000","last_undegraded":"2024-09-16T10:20:35.840544+0000","last_fullsized":"2024-09-16T10:20:35.840544+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":348,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838775+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:20:35.838775+0000","last_peered":"2024-09-16T10:20:35.838775+0000","last_clean":"2024-09-16T10:20:35.838775+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:20:35.838775+0000","last_undegraded":"2024-09-16T10:20:35.838775+0000","last_fullsized":"2024-09-16T10:20:35.838775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:29.172677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":361,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843720+0000","last_change":"2024-09-16T10:20:28.240790+0000","last_active":"2024-09-16T10:20:35.843720+0000","last_peered":"2024-09-16T10:20:35.843720+0000","last_clean":"2024-09-16T10:20:35.843720+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:20:35.843720+0000","last_undegraded":"2024-09-16T10:20:35.843720+0000","last_fullsized":"2024-09-16T10:20:35.843720+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:28.240558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:28.240558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:54.592999+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"222'10459","reported_seq":10663,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.579584+0000","last_change":"2024-09-16T10:20:22.211158+0000","last_active":"2024-09-16T10:20:38.579584+0000","last_peered":"2024-09-16T10:20:38.579584+0000","last_clean":"2024-09-16T10:20:38.579584+0000","last_became_active":"2024-09-16T10:18:33.820857+0000","last_became_peered":"2024-09-16T10:18:33.820857+0000","last_unstale":"2024-09-16T10:20:38.579584+0000","last_undegraded":"2024-09-16T10:20:38.579584+0000","last_fullsized":"2024-09-16T10:20:38.579584+0000","mapping_epoch":165,"log_start":"201'6359","ondisk_log_start":"201'6359","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'8979","last_scrub_stamp":"2024-09-16T10:20:22.210682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:22.210682+0000","objects_scrubbed":7115,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.792498+0000","scrub_duration":2075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363397120,"num_objects":5545,"num_object_clones":0,"num_object_copies":16635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":512128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":441,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849229+0000","last_change":"2024-09-16T10:20:18.856484+0000","last_active":"2024-09-16T10:20:35.849229+0000","last_peered":"2024-09-16T10:20:35.849229+0000","last_clean":"2024-09-16T10:20:35.849229+0000","last_became_active":"2024-09-16T10:20:15.845263+0000","last_became_peered":"2024-09-16T10:20:15.845263+0000","last_unstale":"2024-09-16T10:20:35.849229+0000","last_undegraded":"2024-09-16T10:20:35.849229+0000","last_fullsized":"2024-09-16T10:20:35.849229+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:51.588464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.588464+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:20:48.204414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":372,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838189+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:20:35.838189+0000","last_peered":"2024-09-16T10:20:35.838189+0000","last_clean":"2024-09-16T10:20:35.838189+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:20:35.838189+0000","last_undegraded":"2024-09-16T10:20:35.838189+0000","last_fullsized":"2024-09-16T10:20:35.838189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.443985+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.841637+0000","last_change":"2024-09-16T10:19:16.602946+0000","last_active":"2024-09-16T10:20:35.841637+0000","last_peered":"2024-09-16T10:20:35.841637+0000","last_clean":"2024-09-16T10:20:35.841637+0000","last_became_active":"2024-09-16T10:18:52.833059+0000","last_became_peered":"2024-09-16T10:18:52.833059+0000","last_unstale":"2024-09-16T10:20:35.841637+0000","last_undegraded":"2024-09-16T10:20:35.841637+0000","last_fullsized":"2024-09-16T10:20:35.841637+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.602895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.602895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":131,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843144+0000","last_change":"2024-09-16T10:19:58.199140+0000","last_active":"2024-09-16T10:20:35.843144+0000","last_peered":"2024-09-16T10:20:35.843144+0000","last_clean":"2024-09-16T10:20:35.843144+0000","last_became_active":"2024-09-16T10:18:52.836346+0000","last_became_peered":"2024-09-16T10:18:52.836346+0000","last_unstale":"2024-09-16T10:20:35.843144+0000","last_undegraded":"2024-09-16T10:20:35.843144+0000","last_fullsized":"2024-09-16T10:20:35.843144+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:21.557184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":115,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840090+0000","last_change":"2024-09-16T10:19:16.940617+0000","last_active":"2024-09-16T10:20:35.840090+0000","last_peered":"2024-09-16T10:20:35.840090+0000","last_clean":"2024-09-16T10:20:35.840090+0000","last_became_active":"2024-09-16T10:18:52.827678+0000","last_became_peered":"2024-09-16T10:18:52.827678+0000","last_unstale":"2024-09-16T10:20:35.840090+0000","last_undegraded":"2024-09-16T10:20:35.840090+0000","last_fullsized":"2024-09-16T10:20:35.840090+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.940551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.940551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851670+0000","last_change":"2024-09-16T10:18:37.611634+0000","last_active":"2024-09-16T10:20:35.851670+0000","last_peered":"2024-09-16T10:20:35.851670+0000","last_clean":"2024-09-16T10:20:35.851670+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:20:35.851670+0000","last_undegraded":"2024-09-16T10:20:35.851670+0000","last_fullsized":"2024-09-16T10:20:35.851670+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.611578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.611578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"222'10456","reported_seq":14130,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.415020+0000","last_change":"2024-09-16T10:19:57.299979+0000","last_active":"2024-09-16T10:20:38.415020+0000","last_peered":"2024-09-16T10:20:38.415020+0000","last_clean":"2024-09-16T10:20:38.415020+0000","last_became_active":"2024-09-16T10:19:13.300503+0000","last_became_peered":"2024-09-16T10:19:13.300503+0000","last_unstale":"2024-09-16T10:20:38.415020+0000","last_undegraded":"2024-09-16T10:20:38.415020+0000","last_fullsized":"2024-09-16T10:20:38.415020+0000","mapping_epoch":186,"log_start":"202'6348","ondisk_log_start":"202'6348","created":165,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6795","last_scrub_stamp":"2024-09-16T10:19:57.299850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.299850+0000","objects_scrubbed":6727,"log_size":4108,"log_dups_size":0,"ondisk_log_size":4108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:23.983417+0000","scrub_duration":1826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362151936,"num_objects":5526,"num_object_clones":0,"num_object_copies":16578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10456,"num_write_kb":511424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1730,"num_bytes_recovered":113377280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":166,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838484+0000","last_change":"2024-09-16T10:20:18.865256+0000","last_active":"2024-09-16T10:20:35.838484+0000","last_peered":"2024-09-16T10:20:35.838484+0000","last_clean":"2024-09-16T10:20:35.838484+0000","last_became_active":"2024-09-16T10:20:15.858298+0000","last_became_peered":"2024-09-16T10:20:15.858298+0000","last_unstale":"2024-09-16T10:20:35.838484+0000","last_undegraded":"2024-09-16T10:20:35.838484+0000","last_fullsized":"2024-09-16T10:20:35.838484+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:18:07.544697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.544697+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:20:48.543733+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":407,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839616+0000","last_change":"2024-09-16T10:19:19.231411+0000","last_active":"2024-09-16T10:20:35.839616+0000","last_peered":"2024-09-16T10:20:35.839616+0000","last_clean":"2024-09-16T10:20:35.839616+0000","last_became_active":"2024-09-16T10:18:52.837634+0000","last_became_peered":"2024-09-16T10:18:52.837634+0000","last_unstale":"2024-09-16T10:20:35.839616+0000","last_undegraded":"2024-09-16T10:20:35.839616+0000","last_fullsized":"2024-09-16T10:20:35.839616+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:19:19.231125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:19.231125+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":372,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840579+0000","last_change":"2024-09-16T10:18:45.207140+0000","last_active":"2024-09-16T10:20:35.840579+0000","last_peered":"2024-09-16T10:20:35.840579+0000","last_clean":"2024-09-16T10:20:35.840579+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:20:35.840579+0000","last_undegraded":"2024-09-16T10:20:35.840579+0000","last_fullsized":"2024-09-16T10:20:35.840579+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:45.207071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.207071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":791,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:36.575505+0000","last_change":"2024-09-16T10:20:33.058993+0000","last_active":"2024-09-16T10:20:36.575505+0000","last_peered":"2024-09-16T10:20:36.575505+0000","last_clean":"2024-09-16T10:20:36.575505+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:20:36.575505+0000","last_undegraded":"2024-09-16T10:20:36.575505+0000","last_fullsized":"2024-09-16T10:20:36.575505+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:20:33.058923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.058923+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.147877+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":415,"num_read_kb":410,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840412+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:20:35.840412+0000","last_peered":"2024-09-16T10:20:35.840412+0000","last_clean":"2024-09-16T10:20:35.840412+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:20:35.840412+0000","last_undegraded":"2024-09-16T10:20:35.840412+0000","last_fullsized":"2024-09-16T10:20:35.840412+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.497294+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":361,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838532+0000","last_change":"2024-09-16T10:20:25.566580+0000","last_active":"2024-09-16T10:20:35.838532+0000","last_peered":"2024-09-16T10:20:35.838532+0000","last_clean":"2024-09-16T10:20:35.838532+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:20:35.838532+0000","last_undegraded":"2024-09-16T10:20:35.838532+0000","last_fullsized":"2024-09-16T10:20:35.838532+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:25.566513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:25.566513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:53.062951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"222'10155","reported_seq":10352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.211489+0000","last_change":"2024-09-16T10:19:56.367287+0000","last_active":"2024-09-16T10:20:38.211489+0000","last_peered":"2024-09-16T10:20:38.211489+0000","last_clean":"2024-09-16T10:20:38.211489+0000","last_became_active":"2024-09-16T10:18:33.825120+0000","last_became_peered":"2024-09-16T10:18:33.825120+0000","last_unstale":"2024-09-16T10:20:38.211489+0000","last_undegraded":"2024-09-16T10:20:38.211489+0000","last_fullsized":"2024-09-16T10:20:38.211489+0000","mapping_epoch":165,"log_start":"201'5761","ondisk_log_start":"201'5761","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6534","last_scrub_stamp":"2024-09-16T10:19:56.367108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.367108+0000","objects_scrubbed":6469,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:20.198070+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354222080,"num_objects":5405,"num_object_clones":0,"num_object_copies":16215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10155,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":459,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.842893+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:20:35.842893+0000","last_peered":"2024-09-16T10:20:35.842893+0000","last_clean":"2024-09-16T10:20:35.842893+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:20:35.842893+0000","last_undegraded":"2024-09-16T10:20:35.842893+0000","last_fullsized":"2024-09-16T10:20:35.842893+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.998293+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":330,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843193+0000","last_change":"2024-09-16T10:19:44.015480+0000","last_active":"2024-09-16T10:20:35.843193+0000","last_peered":"2024-09-16T10:20:35.843193+0000","last_clean":"2024-09-16T10:20:35.843193+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:20:35.843193+0000","last_undegraded":"2024-09-16T10:20:35.843193+0000","last_fullsized":"2024-09-16T10:20:35.843193+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.015430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.015430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.463886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":429,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849748+0000","last_change":"2024-09-16T10:20:19.229860+0000","last_active":"2024-09-16T10:20:35.849748+0000","last_peered":"2024-09-16T10:20:35.849748+0000","last_clean":"2024-09-16T10:20:35.849748+0000","last_became_active":"2024-09-16T10:20:15.855504+0000","last_became_peered":"2024-09-16T10:20:15.855504+0000","last_unstale":"2024-09-16T10:20:35.849748+0000","last_undegraded":"2024-09-16T10:20:35.849748+0000","last_fullsized":"2024-09-16T10:20:35.849748+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.229771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.229771+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.171723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.852684+0000","last_change":"2024-09-16T10:18:52.827862+0000","last_active":"2024-09-16T10:20:35.852684+0000","last_peered":"2024-09-16T10:20:35.852684+0000","last_clean":"2024-09-16T10:20:35.852684+0000","last_became_active":"2024-09-16T10:18:52.827467+0000","last_became_peered":"2024-09-16T10:18:52.827467+0000","last_unstale":"2024-09-16T10:20:35.852684+0000","last_undegraded":"2024-09-16T10:20:35.852684+0000","last_fullsized":"2024-09-16T10:20:35.852684+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":345,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840248+0000","last_change":"2024-09-16T10:19:23.923755+0000","last_active":"2024-09-16T10:20:35.840248+0000","last_peered":"2024-09-16T10:20:35.840248+0000","last_clean":"2024-09-16T10:20:35.840248+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:20:35.840248+0000","last_undegraded":"2024-09-16T10:20:35.840248+0000","last_fullsized":"2024-09-16T10:20:35.840248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.923690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.923690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":177,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837523+0000","last_change":"2024-09-16T10:20:19.546115+0000","last_active":"2024-09-16T10:20:35.837523+0000","last_peered":"2024-09-16T10:20:35.837523+0000","last_clean":"2024-09-16T10:20:35.837523+0000","last_became_active":"2024-09-16T10:20:18.869939+0000","last_became_peered":"2024-09-16T10:20:18.869939+0000","last_unstale":"2024-09-16T10:20:35.837523+0000","last_undegraded":"2024-09-16T10:20:35.837523+0000","last_fullsized":"2024-09-16T10:20:35.837523+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.546053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.546053+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.095718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"222'10226","reported_seq":10351,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.145914+0000","last_change":"2024-09-16T10:18:33.825497+0000","last_active":"2024-09-16T10:20:38.145914+0000","last_peered":"2024-09-16T10:20:38.145914+0000","last_clean":"2024-09-16T10:20:38.145914+0000","last_became_active":"2024-09-16T10:18:33.824979+0000","last_became_peered":"2024-09-16T10:18:33.824979+0000","last_unstale":"2024-09-16T10:20:38.145914+0000","last_undegraded":"2024-09-16T10:20:38.145914+0000","last_fullsized":"2024-09-16T10:20:38.145914+0000","mapping_epoch":165,"log_start":"201'6078","ondisk_log_start":"201'6078","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.796343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370147328,"num_objects":5648,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":507968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":398,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836942+0000","last_change":"2024-09-16T10:20:31.013510+0000","last_active":"2024-09-16T10:20:35.836942+0000","last_peered":"2024-09-16T10:20:35.836942+0000","last_clean":"2024-09-16T10:20:35.836942+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:20:35.836942+0000","last_undegraded":"2024-09-16T10:20:35.836942+0000","last_fullsized":"2024-09-16T10:20:35.836942+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:20:31.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.013316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:48.982676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":363,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.835305+0000","last_change":"2024-09-16T10:20:33.973275+0000","last_active":"2024-09-16T10:20:35.835305+0000","last_peered":"2024-09-16T10:20:35.835305+0000","last_clean":"2024-09-16T10:20:35.835305+0000","last_became_active":"2024-09-16T10:18:52.835626+0000","last_became_peered":"2024-09-16T10:18:52.835626+0000","last_unstale":"2024-09-16T10:20:35.835305+0000","last_undegraded":"2024-09-16T10:20:35.835305+0000","last_fullsized":"2024-09-16T10:20:35.835305+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.973201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.973201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:42.032781+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":349,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839907+0000","last_change":"2024-09-16T10:19:18.014961+0000","last_active":"2024-09-16T10:20:35.839907+0000","last_peered":"2024-09-16T10:20:35.839907+0000","last_clean":"2024-09-16T10:20:35.839907+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:20:35.839907+0000","last_undegraded":"2024-09-16T10:20:35.839907+0000","last_fullsized":"2024-09-16T10:20:35.839907+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.014882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.014882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":309,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844060+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:20:35.844060+0000","last_peered":"2024-09-16T10:20:35.844060+0000","last_clean":"2024-09-16T10:20:35.844060+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:20:35.844060+0000","last_undegraded":"2024-09-16T10:20:35.844060+0000","last_fullsized":"2024-09-16T10:20:35.844060+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.607172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"222'10279","reported_seq":13857,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.455266+0000","last_change":"2024-09-16T10:19:48.403185+0000","last_active":"2024-09-16T10:20:38.455266+0000","last_peered":"2024-09-16T10:20:38.455266+0000","last_clean":"2024-09-16T10:20:38.455266+0000","last_became_active":"2024-09-16T10:19:15.290152+0000","last_became_peered":"2024-09-16T10:19:15.290152+0000","last_unstale":"2024-09-16T10:20:38.455266+0000","last_undegraded":"2024-09-16T10:20:38.455266+0000","last_fullsized":"2024-09-16T10:20:38.455266+0000","mapping_epoch":188,"log_start":"201'6152","ondisk_log_start":"201'6152","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'6091","last_scrub_stamp":"2024-09-16T10:19:48.403076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.403076+0000","objects_scrubbed":6009,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:13.478114+0000","scrub_duration":1968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364445696,"num_objects":5561,"num_object_clones":0,"num_object_copies":16683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10279,"num_write_kb":506880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":110428160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":180,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851779+0000","last_change":"2024-09-16T10:20:27.405769+0000","last_active":"2024-09-16T10:20:35.851779+0000","last_peered":"2024-09-16T10:20:35.851779+0000","last_clean":"2024-09-16T10:20:35.851779+0000","last_became_active":"2024-09-16T10:18:52.842646+0000","last_became_peered":"2024-09-16T10:18:52.842646+0000","last_unstale":"2024-09-16T10:20:35.851779+0000","last_undegraded":"2024-09-16T10:20:35.851779+0000","last_fullsized":"2024-09-16T10:20:35.851779+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:27.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:27.405599+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.546066+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":405,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.835948+0000","last_change":"2024-09-16T10:20:22.959397+0000","last_active":"2024-09-16T10:20:35.835948+0000","last_peered":"2024-09-16T10:20:35.835948+0000","last_clean":"2024-09-16T10:20:35.835948+0000","last_became_active":"2024-09-16T10:18:52.826328+0000","last_became_peered":"2024-09-16T10:18:52.826328+0000","last_unstale":"2024-09-16T10:20:35.835948+0000","last_undegraded":"2024-09-16T10:20:35.835948+0000","last_fullsized":"2024-09-16T10:20:35.835948+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:20:22.959311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:22.959311+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.280982+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843081+0000","last_change":"2024-09-16T10:19:21.116145+0000","last_active":"2024-09-16T10:20:35.843081+0000","last_peered":"2024-09-16T10:20:35.843081+0000","last_clean":"2024-09-16T10:20:35.843081+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:20:35.843081+0000","last_undegraded":"2024-09-16T10:20:35.843081+0000","last_fullsized":"2024-09-16T10:20:35.843081+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:21.116098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:21.116098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.939990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":349,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837437+0000","last_change":"2024-09-16T10:20:29.600095+0000","last_active":"2024-09-16T10:20:35.837437+0000","last_peered":"2024-09-16T10:20:35.837437+0000","last_clean":"2024-09-16T10:20:35.837437+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:20:35.837437+0000","last_undegraded":"2024-09-16T10:20:35.837437+0000","last_fullsized":"2024-09-16T10:20:35.837437+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:29.600025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:29.600025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:51.499818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":108,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843909+0000","last_change":"2024-09-16T10:18:52.838759+0000","last_active":"2024-09-16T10:20:35.843909+0000","last_peered":"2024-09-16T10:20:35.843909+0000","last_clean":"2024-09-16T10:20:35.843909+0000","last_became_active":"2024-09-16T10:18:52.838070+0000","last_became_peered":"2024-09-16T10:18:52.838070+0000","last_unstale":"2024-09-16T10:20:35.843909+0000","last_undegraded":"2024-09-16T10:20:35.843909+0000","last_fullsized":"2024-09-16T10:20:35.843909+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:50.491234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.491234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.059410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"222'10690","reported_seq":10816,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.531692+0000","last_change":"2024-09-16T10:18:33.823010+0000","last_active":"2024-09-16T10:20:38.531692+0000","last_peered":"2024-09-16T10:20:38.531692+0000","last_clean":"2024-09-16T10:20:38.531692+0000","last_became_active":"2024-09-16T10:18:33.822701+0000","last_became_peered":"2024-09-16T10:18:33.822701+0000","last_unstale":"2024-09-16T10:20:38.531692+0000","last_undegraded":"2024-09-16T10:20:38.531692+0000","last_fullsized":"2024-09-16T10:20:38.531692+0000","mapping_epoch":165,"log_start":"202'6679","ondisk_log_start":"202'6679","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4011,"log_dups_size":0,"ondisk_log_size":4011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:09.156717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373161984,"num_objects":5694,"num_object_clones":0,"num_object_copies":17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10690,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":436,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849220+0000","last_change":"2024-09-16T10:19:55.550754+0000","last_active":"2024-09-16T10:20:35.849220+0000","last_peered":"2024-09-16T10:20:35.849220+0000","last_clean":"2024-09-16T10:20:35.849220+0000","last_became_active":"2024-09-16T10:18:52.838011+0000","last_became_peered":"2024-09-16T10:18:52.838011+0000","last_unstale":"2024-09-16T10:20:35.849220+0000","last_undegraded":"2024-09-16T10:20:35.849220+0000","last_fullsized":"2024-09-16T10:20:35.849220+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:55.550670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.550670+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:19.176131+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":359,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837361+0000","last_change":"2024-09-16T10:19:16.768415+0000","last_active":"2024-09-16T10:20:35.837361+0000","last_peered":"2024-09-16T10:20:35.837361+0000","last_clean":"2024-09-16T10:20:35.837361+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:20:35.837361+0000","last_undegraded":"2024-09-16T10:20:35.837361+0000","last_fullsized":"2024-09-16T10:20:35.837361+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:16.768062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.768062+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.827070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":349,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844113+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:20:35.844113+0000","last_peered":"2024-09-16T10:20:35.844113+0000","last_clean":"2024-09-16T10:20:35.844113+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:20:35.844113+0000","last_undegraded":"2024-09-16T10:20:35.844113+0000","last_fullsized":"2024-09-16T10:20:35.844113+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.020575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"175'9","reported_seq":167,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851829+0000","last_change":"2024-09-16T10:19:15.342743+0000","last_active":"2024-09-16T10:20:35.851829+0000","last_peered":"2024-09-16T10:20:35.851829+0000","last_clean":"2024-09-16T10:20:35.851829+0000","last_became_active":"2024-09-16T10:18:52.825124+0000","last_became_peered":"2024-09-16T10:18:52.825124+0000","last_unstale":"2024-09-16T10:20:35.851829+0000","last_undegraded":"2024-09-16T10:20:35.851829+0000","last_fullsized":"2024-09-16T10:20:35.851829+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:19:15.342674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.342674+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":351,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836686+0000","last_change":"2024-09-16T10:18:52.834703+0000","last_active":"2024-09-16T10:20:35.836686+0000","last_peered":"2024-09-16T10:20:35.836686+0000","last_clean":"2024-09-16T10:20:35.836686+0000","last_became_active":"2024-09-16T10:18:52.834312+0000","last_became_peered":"2024-09-16T10:18:52.834312+0000","last_unstale":"2024-09-16T10:20:35.836686+0000","last_undegraded":"2024-09-16T10:20:35.836686+0000","last_fullsized":"2024-09-16T10:20:35.836686+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:43.963621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:43.963621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.773176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":34,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836494+0000","last_change":"2024-09-16T10:20:18.941681+0000","last_active":"2024-09-16T10:20:35.836494+0000","last_peered":"2024-09-16T10:20:35.836494+0000","last_clean":"2024-09-16T10:20:35.836494+0000","last_became_active":"2024-09-16T10:20:18.867971+0000","last_became_peered":"2024-09-16T10:20:18.867971+0000","last_unstale":"2024-09-16T10:20:35.836494+0000","last_undegraded":"2024-09-16T10:20:35.836494+0000","last_fullsized":"2024-09-16T10:20:35.836494+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:18.941543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:18.941543+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.174342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"222'10236","reported_seq":10378,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.442347+0000","last_change":"2024-09-16T10:19:59.077503+0000","last_active":"2024-09-16T10:20:38.442347+0000","last_peered":"2024-09-16T10:20:38.442347+0000","last_clean":"2024-09-16T10:20:38.442347+0000","last_became_active":"2024-09-16T10:18:33.823198+0000","last_became_peered":"2024-09-16T10:18:33.823198+0000","last_unstale":"2024-09-16T10:20:38.442347+0000","last_undegraded":"2024-09-16T10:20:38.442347+0000","last_fullsized":"2024-09-16T10:20:38.442347+0000","mapping_epoch":165,"log_start":"201'6174","ondisk_log_start":"201'6174","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":1399,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.486802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368050176,"num_objects":5616,"num_object_clones":0,"num_object_copies":16848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10236,"num_write_kb":507264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":368,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838136+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:20:35.838136+0000","last_peered":"2024-09-16T10:20:35.838136+0000","last_clean":"2024-09-16T10:20:35.838136+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:20:35.838136+0000","last_undegraded":"2024-09-16T10:20:35.838136+0000","last_fullsized":"2024-09-16T10:20:35.838136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.233999+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":365,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.841335+0000","last_change":"2024-09-16T10:20:28.364361+0000","last_active":"2024-09-16T10:20:35.841335+0000","last_peered":"2024-09-16T10:20:35.841335+0000","last_clean":"2024-09-16T10:20:35.841335+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:20:35.841335+0000","last_undegraded":"2024-09-16T10:20:35.841335+0000","last_fullsized":"2024-09-16T10:20:35.841335+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:28.364296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:28.364296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.891566+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":115,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.834929+0000","last_change":"2024-09-16T10:19:18.748438+0000","last_active":"2024-09-16T10:20:35.834929+0000","last_peered":"2024-09-16T10:20:35.834929+0000","last_clean":"2024-09-16T10:20:35.834929+0000","last_became_active":"2024-09-16T10:18:52.835884+0000","last_became_peered":"2024-09-16T10:18:52.835884+0000","last_unstale":"2024-09-16T10:20:35.834929+0000","last_undegraded":"2024-09-16T10:20:35.834929+0000","last_fullsized":"2024-09-16T10:20:35.834929+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":378,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840484+0000","last_change":"2024-09-16T10:20:31.889357+0000","last_active":"2024-09-16T10:20:35.840484+0000","last_peered":"2024-09-16T10:20:35.840484+0000","last_clean":"2024-09-16T10:20:35.840484+0000","last_became_active":"2024-09-16T10:20:31.869705+0000","last_became_peered":"2024-09-16T10:20:31.869705+0000","last_unstale":"2024-09-16T10:20:35.840484+0000","last_undegraded":"2024-09-16T10:20:35.840484+0000","last_fullsized":"2024-09-16T10:20:35.840484+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":252,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843629+0000","last_change":"2024-09-16T10:20:20.538346+0000","last_active":"2024-09-16T10:20:35.843629+0000","last_peered":"2024-09-16T10:20:35.843629+0000","last_clean":"2024-09-16T10:20:35.843629+0000","last_became_active":"2024-09-16T10:20:18.870789+0000","last_became_peered":"2024-09-16T10:20:18.870789+0000","last_unstale":"2024-09-16T10:20:35.843629+0000","last_undegraded":"2024-09-16T10:20:35.843629+0000","last_fullsized":"2024-09-16T10:20:35.843629+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.538076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.538076+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.192429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.6","version":"222'10328","reported_seq":10453,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.533150+0000","last_change":"2024-09-16T10:18:33.894422+0000","last_active":"2024-09-16T10:20:38.533150+0000","last_peered":"2024-09-16T10:20:38.533150+0000","last_clean":"2024-09-16T10:20:38.533150+0000","last_became_active":"2024-09-16T10:18:33.893961+0000","last_became_peered":"2024-09-16T10:18:33.893961+0000","last_unstale":"2024-09-16T10:20:38.533150+0000","last_undegraded":"2024-09-16T10:20:38.533150+0000","last_fullsized":"2024-09-16T10:20:38.533150+0000","mapping_epoch":165,"log_start":"201'5894","ondisk_log_start":"201'5894","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366346240,"num_objects":5590,"num_object_clones":0,"num_object_copies":16770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":509376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":391,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.842828+0000","last_change":"2024-09-16T10:19:24.115166+0000","last_active":"2024-09-16T10:20:35.842828+0000","last_peered":"2024-09-16T10:20:35.842828+0000","last_clean":"2024-09-16T10:20:35.842828+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:20:35.842828+0000","last_undegraded":"2024-09-16T10:20:35.842828+0000","last_fullsized":"2024-09-16T10:20:35.842828+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:24.115075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:24.115075+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.363776+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.852077+0000","last_change":"2024-09-16T10:20:31.091077+0000","last_active":"2024-09-16T10:20:35.852077+0000","last_peered":"2024-09-16T10:20:35.852077+0000","last_clean":"2024-09-16T10:20:35.852077+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:20:35.852077+0000","last_undegraded":"2024-09-16T10:20:35.852077+0000","last_fullsized":"2024-09-16T10:20:35.852077+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:31.090914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.090914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:49.345216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":115,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837949+0000","last_change":"2024-09-16T10:19:17.184744+0000","last_active":"2024-09-16T10:20:35.837949+0000","last_peered":"2024-09-16T10:20:35.837949+0000","last_clean":"2024-09-16T10:20:35.837949+0000","last_became_active":"2024-09-16T10:18:52.831796+0000","last_became_peered":"2024-09-16T10:18:52.831796+0000","last_unstale":"2024-09-16T10:20:35.837949+0000","last_undegraded":"2024-09-16T10:20:35.837949+0000","last_fullsized":"2024-09-16T10:20:35.837949+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.184674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.184674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849651+0000","last_change":"2024-09-16T10:18:52.841267+0000","last_active":"2024-09-16T10:20:35.849651+0000","last_peered":"2024-09-16T10:20:35.849651+0000","last_clean":"2024-09-16T10:20:35.849651+0000","last_became_active":"2024-09-16T10:18:52.840958+0000","last_became_peered":"2024-09-16T10:18:52.840958+0000","last_unstale":"2024-09-16T10:20:35.849651+0000","last_undegraded":"2024-09-16T10:20:35.849651+0000","last_fullsized":"2024-09-16T10:20:35.849651+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:08.596349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:08.596349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.495303+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":185,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843693+0000","last_change":"2024-09-16T10:20:19.166368+0000","last_active":"2024-09-16T10:20:35.843693+0000","last_peered":"2024-09-16T10:20:35.843693+0000","last_clean":"2024-09-16T10:20:35.843693+0000","last_became_active":"2024-09-16T10:20:18.865956+0000","last_became_peered":"2024-09-16T10:20:18.865956+0000","last_unstale":"2024-09-16T10:20:35.843693+0000","last_undegraded":"2024-09-16T10:20:35.843693+0000","last_fullsized":"2024-09-16T10:20:35.843693+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.166321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.166321+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:21.079811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"222'10324","reported_seq":10519,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.212179+0000","last_change":"2024-09-16T10:19:55.377394+0000","last_active":"2024-09-16T10:20:38.212179+0000","last_peered":"2024-09-16T10:20:38.212179+0000","last_clean":"2024-09-16T10:20:38.212179+0000","last_became_active":"2024-09-16T10:18:33.894026+0000","last_became_peered":"2024-09-16T10:18:33.894026+0000","last_unstale":"2024-09-16T10:20:38.212179+0000","last_undegraded":"2024-09-16T10:20:38.212179+0000","last_fullsized":"2024-09-16T10:20:38.212179+0000","mapping_epoch":165,"log_start":"201'5945","ondisk_log_start":"201'5945","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6662","last_scrub_stamp":"2024-09-16T10:19:55.377245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.377245+0000","objects_scrubbed":6589,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.117653+0000","scrub_duration":2000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367788032,"num_objects":5612,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10324,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":375,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.835177+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:20:35.835177+0000","last_peered":"2024-09-16T10:20:35.835177+0000","last_clean":"2024-09-16T10:20:35.835177+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:20:35.835177+0000","last_undegraded":"2024-09-16T10:20:35.835177+0000","last_fullsized":"2024-09-16T10:20:35.835177+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.799332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.855802+0000","last_change":"2024-09-16T10:19:17.357917+0000","last_active":"2024-09-16T10:20:35.855802+0000","last_peered":"2024-09-16T10:20:35.855802+0000","last_clean":"2024-09-16T10:20:35.855802+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:20:35.855802+0000","last_undegraded":"2024-09-16T10:20:35.855802+0000","last_fullsized":"2024-09-16T10:20:35.855802+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.357866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.357866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":107,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837194+0000","last_change":"2024-09-16T10:18:52.830267+0000","last_active":"2024-09-16T10:20:35.837194+0000","last_peered":"2024-09-16T10:20:35.837194+0000","last_clean":"2024-09-16T10:20:35.837194+0000","last_became_active":"2024-09-16T10:18:52.829845+0000","last_became_peered":"2024-09-16T10:18:52.829845+0000","last_unstale":"2024-09-16T10:20:35.837194+0000","last_undegraded":"2024-09-16T10:20:35.837194+0000","last_fullsized":"2024-09-16T10:20:35.837194+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.955280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843428+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:20:35.843428+0000","last_peered":"2024-09-16T10:20:35.843428+0000","last_clean":"2024-09-16T10:20:35.843428+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:20:35.843428+0000","last_undegraded":"2024-09-16T10:20:35.843428+0000","last_fullsized":"2024-09-16T10:20:35.843428+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.174140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":34,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851362+0000","last_change":"2024-09-16T10:20:20.186263+0000","last_active":"2024-09-16T10:20:35.851362+0000","last_peered":"2024-09-16T10:20:35.851362+0000","last_clean":"2024-09-16T10:20:35.851362+0000","last_became_active":"2024-09-16T10:20:18.863901+0000","last_became_peered":"2024-09-16T10:20:18.863901+0000","last_unstale":"2024-09-16T10:20:35.851362+0000","last_undegraded":"2024-09-16T10:20:35.851362+0000","last_fullsized":"2024-09-16T10:20:35.851362+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.186192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.186192+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:23.904161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"222'10025","reported_seq":13453,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:38.251932+0000","last_change":"2024-09-16T10:19:14.298104+0000","last_active":"2024-09-16T10:20:38.251932+0000","last_peered":"2024-09-16T10:20:38.251932+0000","last_clean":"2024-09-16T10:20:38.251932+0000","last_became_active":"2024-09-16T10:19:14.291692+0000","last_became_peered":"2024-09-16T10:19:14.291692+0000","last_unstale":"2024-09-16T10:20:38.251932+0000","last_undegraded":"2024-09-16T10:20:38.251932+0000","last_fullsized":"2024-09-16T10:20:38.251932+0000","mapping_epoch":187,"log_start":"201'5870","ondisk_log_start":"201'5870","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4155,"log_dups_size":0,"ondisk_log_size":4155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.169232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354222080,"num_objects":5405,"num_object_clones":0,"num_object_copies":16215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":493760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":362,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851942+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:20:35.851942+0000","last_peered":"2024-09-16T10:20:35.851942+0000","last_clean":"2024-09-16T10:20:35.851942+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:20:35.851942+0000","last_undegraded":"2024-09-16T10:20:35.851942+0000","last_fullsized":"2024-09-16T10:20:35.851942+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.365771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":369,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851422+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:20:35.851422+0000","last_peered":"2024-09-16T10:20:35.851422+0000","last_clean":"2024-09-16T10:20:35.851422+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:20:35.851422+0000","last_undegraded":"2024-09-16T10:20:35.851422+0000","last_fullsized":"2024-09-16T10:20:35.851422+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:15.178355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":442,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849541+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:20:35.849541+0000","last_peered":"2024-09-16T10:20:35.849541+0000","last_clean":"2024-09-16T10:20:35.849541+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:20:35.849541+0000","last_undegraded":"2024-09-16T10:20:35.849541+0000","last_fullsized":"2024-09-16T10:20:35.849541+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.564166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840123+0000","last_change":"2024-09-16T10:19:20.941840+0000","last_active":"2024-09-16T10:20:35.840123+0000","last_peered":"2024-09-16T10:20:35.840123+0000","last_clean":"2024-09-16T10:20:35.840123+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:20:35.840123+0000","last_undegraded":"2024-09-16T10:20:35.840123+0000","last_fullsized":"2024-09-16T10:20:35.840123+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.941753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.941753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:39.997567+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":34,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840473+0000","last_change":"2024-09-16T10:20:19.314302+0000","last_active":"2024-09-16T10:20:35.840473+0000","last_peered":"2024-09-16T10:20:35.840473+0000","last_clean":"2024-09-16T10:20:35.840473+0000","last_became_active":"2024-09-16T10:20:18.879077+0000","last_became_peered":"2024-09-16T10:20:18.879077+0000","last_unstale":"2024-09-16T10:20:35.840473+0000","last_undegraded":"2024-09-16T10:20:35.840473+0000","last_fullsized":"2024-09-16T10:20:35.840473+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.314234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.314234+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.829443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.842049+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:20:35.842049+0000","last_peered":"2024-09-16T10:20:35.842049+0000","last_clean":"2024-09-16T10:20:35.842049+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:20:35.842049+0000","last_undegraded":"2024-09-16T10:20:35.842049+0000","last_fullsized":"2024-09-16T10:20:35.842049+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":369,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844295+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:20:35.844295+0000","last_peered":"2024-09-16T10:20:35.844295+0000","last_clean":"2024-09-16T10:20:35.844295+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:20:35.844295+0000","last_undegraded":"2024-09-16T10:20:35.844295+0000","last_fullsized":"2024-09-16T10:20:35.844295+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:40.155197+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":340,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840346+0000","last_change":"2024-09-16T10:19:15.972005+0000","last_active":"2024-09-16T10:20:35.840346+0000","last_peered":"2024-09-16T10:20:35.840346+0000","last_clean":"2024-09-16T10:20:35.840346+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:20:35.840346+0000","last_undegraded":"2024-09-16T10:20:35.840346+0000","last_fullsized":"2024-09-16T10:20:35.840346+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:15.971951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:15.971951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":336,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851578+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:20:35.851578+0000","last_peered":"2024-09-16T10:20:35.851578+0000","last_clean":"2024-09-16T10:20:35.851578+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:20:35.851578+0000","last_undegraded":"2024-09-16T10:20:35.851578+0000","last_fullsized":"2024-09-16T10:20:35.851578+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":174,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837737+0000","last_change":"2024-09-16T10:20:20.591626+0000","last_active":"2024-09-16T10:20:35.837737+0000","last_peered":"2024-09-16T10:20:35.837737+0000","last_clean":"2024-09-16T10:20:35.837737+0000","last_became_active":"2024-09-16T10:20:18.873396+0000","last_became_peered":"2024-09-16T10:20:18.873396+0000","last_unstale":"2024-09-16T10:20:35.837737+0000","last_undegraded":"2024-09-16T10:20:35.837737+0000","last_fullsized":"2024-09-16T10:20:35.837737+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.591566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.591566+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.585524+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":375,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839648+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:20:35.839648+0000","last_peered":"2024-09-16T10:20:35.839648+0000","last_clean":"2024-09-16T10:20:35.839648+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:20:35.839648+0000","last_undegraded":"2024-09-16T10:20:35.839648+0000","last_fullsized":"2024-09-16T10:20:35.839648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.020779+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":357,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849463+0000","last_change":"2024-09-16T10:19:22.431194+0000","last_active":"2024-09-16T10:20:35.849463+0000","last_peered":"2024-09-16T10:20:35.849463+0000","last_clean":"2024-09-16T10:20:35.849463+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:20:35.849463+0000","last_undegraded":"2024-09-16T10:20:35.849463+0000","last_fullsized":"2024-09-16T10:20:35.849463+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.431126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.431126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:48.492625+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":348,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844157+0000","last_change":"2024-09-16T10:20:32.229638+0000","last_active":"2024-09-16T10:20:35.844157+0000","last_peered":"2024-09-16T10:20:35.844157+0000","last_clean":"2024-09-16T10:20:35.844157+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:20:35.844157+0000","last_undegraded":"2024-09-16T10:20:35.844157+0000","last_fullsized":"2024-09-16T10:20:35.844157+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:32.229564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:32.229564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.951807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":116,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837439+0000","last_change":"2024-09-16T10:20:19.951949+0000","last_active":"2024-09-16T10:20:35.837439+0000","last_peered":"2024-09-16T10:20:35.837439+0000","last_clean":"2024-09-16T10:20:35.837439+0000","last_became_active":"2024-09-16T10:18:52.822465+0000","last_became_peered":"2024-09-16T10:18:52.822465+0000","last_unstale":"2024-09-16T10:20:35.837439+0000","last_undegraded":"2024-09-16T10:20:35.837439+0000","last_fullsized":"2024-09-16T10:20:35.837439+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.200411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":34,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.847857+0000","last_change":"2024-09-16T10:20:20.322528+0000","last_active":"2024-09-16T10:20:35.847857+0000","last_peered":"2024-09-16T10:20:35.847857+0000","last_clean":"2024-09-16T10:20:35.847857+0000","last_became_active":"2024-09-16T10:20:18.878681+0000","last_became_peered":"2024-09-16T10:20:18.878681+0000","last_unstale":"2024-09-16T10:20:35.847857+0000","last_undegraded":"2024-09-16T10:20:35.847857+0000","last_fullsized":"2024-09-16T10:20:35.847857+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.322463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.322463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:31.968089+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":368,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840071+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:20:35.840071+0000","last_peered":"2024-09-16T10:20:35.840071+0000","last_clean":"2024-09-16T10:20:35.840071+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:20:35.840071+0000","last_undegraded":"2024-09-16T10:20:35.840071+0000","last_fullsized":"2024-09-16T10:20:35.840071+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840129+0000","last_change":"2024-09-16T10:20:29.177634+0000","last_active":"2024-09-16T10:20:35.840129+0000","last_peered":"2024-09-16T10:20:35.840129+0000","last_clean":"2024-09-16T10:20:35.840129+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:20:35.840129+0000","last_undegraded":"2024-09-16T10:20:35.840129+0000","last_fullsized":"2024-09-16T10:20:35.840129+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:29.177563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:29.177563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.824549+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"175'9","reported_seq":468,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843949+0000","last_change":"2024-09-16T10:20:25.284531+0000","last_active":"2024-09-16T10:20:35.843949+0000","last_peered":"2024-09-16T10:20:35.843949+0000","last_clean":"2024-09-16T10:20:35.843949+0000","last_became_active":"2024-09-16T10:18:52.845153+0000","last_became_peered":"2024-09-16T10:18:52.845153+0000","last_unstale":"2024-09-16T10:20:35.843949+0000","last_undegraded":"2024-09-16T10:20:35.843949+0000","last_fullsized":"2024-09-16T10:20:35.843949+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:20:25.284462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:25.284462+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.278781+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":350,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.835783+0000","last_change":"2024-09-16T10:18:52.821349+0000","last_active":"2024-09-16T10:20:35.835783+0000","last_peered":"2024-09-16T10:20:35.835783+0000","last_clean":"2024-09-16T10:20:35.835783+0000","last_became_active":"2024-09-16T10:18:52.821050+0000","last_became_peered":"2024-09-16T10:18:52.821050+0000","last_unstale":"2024-09-16T10:20:35.835783+0000","last_undegraded":"2024-09-16T10:20:35.835783+0000","last_fullsized":"2024-09-16T10:20:35.835783+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.915613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.915613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:42.778010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":110,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840665+0000","last_change":"2024-09-16T10:20:19.113511+0000","last_active":"2024-09-16T10:20:35.840665+0000","last_peered":"2024-09-16T10:20:35.840665+0000","last_clean":"2024-09-16T10:20:35.840665+0000","last_became_active":"2024-09-16T10:20:18.868687+0000","last_became_peered":"2024-09-16T10:20:18.868687+0000","last_unstale":"2024-09-16T10:20:35.840665+0000","last_undegraded":"2024-09-16T10:20:35.840665+0000","last_fullsized":"2024-09-16T10:20:35.840665+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.113367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.113367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:48.886516+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":375,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843419+0000","last_change":"2024-09-16T10:19:45.045388+0000","last_active":"2024-09-16T10:20:35.843419+0000","last_peered":"2024-09-16T10:20:35.843419+0000","last_clean":"2024-09-16T10:20:35.843419+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:20:35.843419+0000","last_undegraded":"2024-09-16T10:20:35.843419+0000","last_fullsized":"2024-09-16T10:20:35.843419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:45.045164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.045164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.833432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849695+0000","last_change":"2024-09-16T10:18:52.820977+0000","last_active":"2024-09-16T10:20:35.849695+0000","last_peered":"2024-09-16T10:20:35.849695+0000","last_clean":"2024-09-16T10:20:35.849695+0000","last_became_active":"2024-09-16T10:18:52.820693+0000","last_became_peered":"2024-09-16T10:18:52.820693+0000","last_unstale":"2024-09-16T10:20:35.849695+0000","last_undegraded":"2024-09-16T10:20:35.849695+0000","last_fullsized":"2024-09-16T10:20:35.849695+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.355305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":449,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839872+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:20:35.839872+0000","last_peered":"2024-09-16T10:20:35.839872+0000","last_clean":"2024-09-16T10:20:35.839872+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:20:35.839872+0000","last_undegraded":"2024-09-16T10:20:35.839872+0000","last_fullsized":"2024-09-16T10:20:35.839872+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.555917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":343,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839428+0000","last_change":"2024-09-16T10:18:09.148508+0000","last_active":"2024-09-16T10:20:35.839428+0000","last_peered":"2024-09-16T10:20:35.839428+0000","last_clean":"2024-09-16T10:20:35.839428+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:20:35.839428+0000","last_undegraded":"2024-09-16T10:20:35.839428+0000","last_fullsized":"2024-09-16T10:20:35.839428+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:09.148447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.148447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:16.056190+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":362,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.836830+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:20:35.836830+0000","last_peered":"2024-09-16T10:20:35.836830+0000","last_clean":"2024-09-16T10:20:35.836830+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:20:35.836830+0000","last_undegraded":"2024-09-16T10:20:35.836830+0000","last_fullsized":"2024-09-16T10:20:35.836830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.805984+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837563+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:20:35.837563+0000","last_peered":"2024-09-16T10:20:35.837563+0000","last_clean":"2024-09-16T10:20:35.837563+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:20:35.837563+0000","last_undegraded":"2024-09-16T10:20:35.837563+0000","last_fullsized":"2024-09-16T10:20:35.837563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837024+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:20:35.837024+0000","last_peered":"2024-09-16T10:20:35.837024+0000","last_clean":"2024-09-16T10:20:35.837024+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:20:35.837024+0000","last_undegraded":"2024-09-16T10:20:35.837024+0000","last_fullsized":"2024-09-16T10:20:35.837024+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:04.473585+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":336,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.842608+0000","last_change":"2024-09-16T10:19:17.803486+0000","last_active":"2024-09-16T10:20:35.842608+0000","last_peered":"2024-09-16T10:20:35.842608+0000","last_clean":"2024-09-16T10:20:35.842608+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:20:35.842608+0000","last_undegraded":"2024-09-16T10:20:35.842608+0000","last_fullsized":"2024-09-16T10:20:35.842608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.803412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.803412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":385,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.838148+0000","last_change":"2024-09-16T10:19:18.621273+0000","last_active":"2024-09-16T10:20:35.838148+0000","last_peered":"2024-09-16T10:20:35.838148+0000","last_clean":"2024-09-16T10:20:35.838148+0000","last_became_active":"2024-09-16T10:18:52.832688+0000","last_became_peered":"2024-09-16T10:18:52.832688+0000","last_unstale":"2024-09-16T10:20:35.838148+0000","last_undegraded":"2024-09-16T10:20:35.838148+0000","last_fullsized":"2024-09-16T10:20:35.838148+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:18.620605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.620605+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.851941+0000","last_change":"2024-09-16T10:19:47.608315+0000","last_active":"2024-09-16T10:20:35.851941+0000","last_peered":"2024-09-16T10:20:35.851941+0000","last_clean":"2024-09-16T10:20:35.851941+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:20:35.851941+0000","last_undegraded":"2024-09-16T10:20:35.851941+0000","last_fullsized":"2024-09-16T10:20:35.851941+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.608243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.608243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.564159+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":313,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843519+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:20:35.843519+0000","last_peered":"2024-09-16T10:20:35.843519+0000","last_clean":"2024-09-16T10:20:35.843519+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:20:35.843519+0000","last_undegraded":"2024-09-16T10:20:35.843519+0000","last_fullsized":"2024-09-16T10:20:35.843519+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:36.046746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":335,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837074+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:20:35.837074+0000","last_peered":"2024-09-16T10:20:35.837074+0000","last_clean":"2024-09-16T10:20:35.837074+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:20:35.837074+0000","last_undegraded":"2024-09-16T10:20:35.837074+0000","last_fullsized":"2024-09-16T10:20:35.837074+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:00.958267+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":301,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.848959+0000","last_change":"2024-09-16T10:20:21.141380+0000","last_active":"2024-09-16T10:20:35.848959+0000","last_peered":"2024-09-16T10:20:35.848959+0000","last_clean":"2024-09-16T10:20:35.848959+0000","last_became_active":"2024-09-16T10:20:18.867938+0000","last_became_peered":"2024-09-16T10:20:18.867938+0000","last_unstale":"2024-09-16T10:20:35.848959+0000","last_undegraded":"2024-09-16T10:20:35.848959+0000","last_fullsized":"2024-09-16T10:20:35.848959+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:21.141324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:21.141324+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:31.692320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":363,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.842738+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:20:35.842738+0000","last_peered":"2024-09-16T10:20:35.842738+0000","last_clean":"2024-09-16T10:20:35.842738+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:20:35.842738+0000","last_undegraded":"2024-09-16T10:20:35.842738+0000","last_fullsized":"2024-09-16T10:20:35.842738+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.840567+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:20:35.840567+0000","last_peered":"2024-09-16T10:20:35.840567+0000","last_clean":"2024-09-16T10:20:35.840567+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:20:35.840567+0000","last_undegraded":"2024-09-16T10:20:35.840567+0000","last_fullsized":"2024-09-16T10:20:35.840567+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.195400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849673+0000","last_change":"2024-09-16T10:19:45.534275+0000","last_active":"2024-09-16T10:20:35.849673+0000","last_peered":"2024-09-16T10:20:35.849673+0000","last_clean":"2024-09-16T10:20:35.849673+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:20:35.849673+0000","last_undegraded":"2024-09-16T10:20:35.849673+0000","last_fullsized":"2024-09-16T10:20:35.849673+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:45.533998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.533998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:47.342382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843720+0000","last_change":"2024-09-16T10:19:47.129026+0000","last_active":"2024-09-16T10:20:35.843720+0000","last_peered":"2024-09-16T10:20:35.843720+0000","last_clean":"2024-09-16T10:20:35.843720+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:20:35.843720+0000","last_undegraded":"2024-09-16T10:20:35.843720+0000","last_fullsized":"2024-09-16T10:20:35.843720+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.128964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.128964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:52.566335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":385,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844499+0000","last_change":"2024-09-16T10:19:56.200787+0000","last_active":"2024-09-16T10:20:35.844499+0000","last_peered":"2024-09-16T10:20:35.844499+0000","last_clean":"2024-09-16T10:20:35.844499+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:20:35.844499+0000","last_undegraded":"2024-09-16T10:20:35.844499+0000","last_fullsized":"2024-09-16T10:20:35.844499+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:19:56.200704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.200704+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.350145+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":361,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839598+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:20:35.839598+0000","last_peered":"2024-09-16T10:20:35.839598+0000","last_clean":"2024-09-16T10:20:35.839598+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:20:35.839598+0000","last_undegraded":"2024-09-16T10:20:35.839598+0000","last_fullsized":"2024-09-16T10:20:35.839598+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.848146+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:20:35.848146+0000","last_peered":"2024-09-16T10:20:35.848146+0000","last_clean":"2024-09-16T10:20:35.848146+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:20:35.848146+0000","last_undegraded":"2024-09-16T10:20:35.848146+0000","last_fullsized":"2024-09-16T10:20:35.848146+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":114,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.849129+0000","last_change":"2024-09-16T10:19:44.487481+0000","last_active":"2024-09-16T10:20:35.849129+0000","last_peered":"2024-09-16T10:20:35.849129+0000","last_clean":"2024-09-16T10:20:35.849129+0000","last_became_active":"2024-09-16T10:18:52.838002+0000","last_became_peered":"2024-09-16T10:18:52.838002+0000","last_unstale":"2024-09-16T10:20:35.849129+0000","last_undegraded":"2024-09-16T10:20:35.849129+0000","last_fullsized":"2024-09-16T10:20:35.849129+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:51.584073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.837696+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:20:35.837696+0000","last_peered":"2024-09-16T10:20:35.837696+0000","last_clean":"2024-09-16T10:20:35.837696+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:20:35.837696+0000","last_undegraded":"2024-09-16T10:20:35.837696+0000","last_fullsized":"2024-09-16T10:20:35.837696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:43.126929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.844652+0000","last_change":"2024-09-16T10:19:20.738852+0000","last_active":"2024-09-16T10:20:35.844652+0000","last_peered":"2024-09-16T10:20:35.844652+0000","last_clean":"2024-09-16T10:20:35.844652+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:20:35.844652+0000","last_undegraded":"2024-09-16T10:20:35.844652+0000","last_fullsized":"2024-09-16T10:20:35.844652+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.738779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.738779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":387,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.843716+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:20:35.843716+0000","last_peered":"2024-09-16T10:20:35.843716+0000","last_clean":"2024-09-16T10:20:35.843716+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:20:35.843716+0000","last_undegraded":"2024-09-16T10:20:35.843716+0000","last_fullsized":"2024-09-16T10:20:35.843716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.553581+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":363,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T10:20:35.839528+0000","last_change":"2024-09-16T10:19:16.217856+0000","last_active":"2024-09-16T10:20:35.839528+0000","last_peered":"2024-09-16T10:20:35.839528+0000","last_clean":"2024-09-16T10:20:35.839528+0000","last_became_active":"2024-09-16T10:18:52.834424+0000","last_became_peered":"2024-09-16T10:18:52.834424+0000","last_unstale":"2024-09-16T10:20:35.839528+0000","last_undegraded":"2024-09-16T10:20:35.839528+0000","last_fullsized":"2024-09-16T10:20:35.839528+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:37.131801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5835456536,"num_objects":89043,"num_object_clones":0,"num_object_copies":267129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165593,"num_write_kb":8148289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10118,"num_bytes_recovered":663093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18089062400,"data_stored":18089050184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67168,"ondisk_log_size":67168,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":415,"num_read_kb":410,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":699594,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234072,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2384544,"kb_used_data":2158992,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":91360608,"statfs":{"total":95995035648,"available":93553262592,"internally_reserved":0,"allocated":2210807808,"data_stored":2210303728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":230947274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528412,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2786152,"kb_used_data":2512016,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":90959000,"statfs":{"total":95995035648,"available":93142016000,"internally_reserved":0,"allocated":2572304384,"data_stored":2571800218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790050,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3488040,"kb_used_data":3254164,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90257112,"statfs":{"total":95995035648,"available":92423282688,"internally_reserved":0,"allocated":3332263936,"data_stored":3331756998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986279,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2402660,"kb_used_data":2146580,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91342492,"statfs":{"total":95995035648,"available":93534711808,"internally_reserved":0,"allocated":2198097920,"data_stored":2197589663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215212,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2790908,"kb_used_data":2511212,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":90954244,"statfs":{"total":95995035648,"available":93137145856,"internally_reserved":0,"allocated":2571481088,"data_stored":2570950494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":286390728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2788604,"kb_used_data":2504748,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":90956548,"statfs":{"total":95995035648,"available":93139505152,"internally_reserved":0,"allocated":2564861952,"data_stored":2564329969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2394668,"kb_used_data":2128860,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91350484,"statfs":{"total":95995035648,"available":93542895616,"internally_reserved":0,"allocated":2179952640,"data_stored":2179437170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2165899264,"data_stored":2165899264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2562850816,"data_stored":2562850816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":562823168,"data_stored":562823168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2558656512,"data_stored":2558656512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2190540800,"data_stored":2190540800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3292270592,"data_stored":3292266520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2550992896,"data_stored":2550988824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2205028352,"data_stored":2205024280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:20:42.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-09-16T10:20:42.769 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:42.802 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:42.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:42.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:42.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:20:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:42.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:42.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:43.037 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:43.102 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:43.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:43.141 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:43 smithi063 ceph-mon[29993]: from='client.15561 v1:172.21.15.63:0/3697809188' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:43.141 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1852930904' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:43.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:43 smithi063 ceph-mon[39470]: from='client.15561 v1:172.21.15.63:0/3697809188' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:43.141 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1852930904' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:43.156 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:20:43.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:43.226 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:43.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:43.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:43.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:43.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:43.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:43.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:43.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:43 smithi165 ceph-mon[32696]: from='client.15561 v1:172.21.15.63:0/3697809188' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:20:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1852930904' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:43.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:43.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:43.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:43.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:43.771 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:43.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:43.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:43.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:43.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:43.941 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:44.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:44.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:44.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:44.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:44.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:44.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:44.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:44.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1852930904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:44.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[29993]: osdmap e223: 8 total, 8 up, 7 in 2024-09-16T10:20:44.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[29993]: pgmap v351: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 920 op/s 2024-09-16T10:20:44.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:44.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1852930904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:44.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[39470]: osdmap e223: 8 total, 8 up, 7 in 2024-09-16T10:20:44.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:44 smithi063 ceph-mon[39470]: pgmap v351: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 920 op/s 2024-09-16T10:20:44.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:44.315 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:44.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:44.416 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:44.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:44 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:44.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:44 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1852930904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:44.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:44 smithi165 ceph-mon[32696]: osdmap e223: 8 total, 8 up, 7 in 2024-09-16T10:20:44.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:44 smithi165 ceph-mon[32696]: pgmap v351: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 920 op/s 2024-09-16T10:20:44.459 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:44.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:44.521 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:44.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:44.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:173965 8.9 8:979dc554:::benchmark_data_smithi063_101290_object46645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:44.605621+0000", 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0065356090000000004, 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065639039999999997, 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:44.615 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173965 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605621+0000", 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605621+0000", 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605623+0000", 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605630+0000", 2024-09-16T10:20:44.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3879999999999997e-06 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605641+0000", 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1681e-05 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605648+0000", 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0659999999999998e-06 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:44.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605704+0000", 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5872999999999999e-05 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605814+0000", 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109656 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.605918+0000", 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010357999999999999 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.608402+0000", 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002484928 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.608473+0000", 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0009e-05 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.620 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.15333.0:173961 8.d e223/188 8:b18eeea0:::benchmark_data_smithi063_101290_object46641:head v 223'10974, mlcod=223'10972)", 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:44.607037+0000", 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0051201679999999996, 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005222969, 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.607037+0000", 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.607037+0000", 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.607038+0000", 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6799999999999999e-07 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.607042+0000", 2024-09-16T10:20:44.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9640000000000002e-06 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.607050+0000", 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2880000000000004e-06 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.607055+0000", 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.113e-06 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.607986+0000", 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093093799999999997 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.608006+0000", 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9966000000000002e-05 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.15333.0:173971 8.7 e223/165 8:e7319c0b:::benchmark_data_smithi063_101290_object46651:head v 223'10791, mlcod=223'10789)", 2024-09-16T10:20:44.624 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:44.610130+0000", 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0020272020000000001, 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021748549999999998, 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610130+0000", 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610130+0000", 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610131+0000", 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3039999999999999e-06 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610137+0000", 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3790000000000003e-06 2024-09-16T10:20:44.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610151+0000", 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3609e-05 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610158+0000", 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.029e-06 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610220+0000", 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2477999999999994e-05 2024-09-16T10:20:44.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610241+0000", 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0355000000000001e-05 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:173970 8.5 8:ac396c2b:::benchmark_data_smithi063_101290_object46650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:44.610190+0000", 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0019672079999999998, 2024-09-16T10:20:44.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021600370000000001, 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173970 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610190+0000", 2024-09-16T10:20:44.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610190+0000", 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610192+0000", 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0339999999999999e-06 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610197+0000", 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7610000000000004e-06 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610209+0000", 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1369000000000001e-05 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610216+0000", 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8040000000000004e-06 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610390+0000", 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174735 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610507+0000", 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116273 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:44.610626+0000", 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119349 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.632 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:20:44.633 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.633 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:20:44.633 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:20:44.633 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 4 2024-09-16T10:20:44.633 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:20:44.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:44.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:44.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:44.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:44.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:44.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:44.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:45.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:45.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:45.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:45.159 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 8 pgp_num to 16 2024-09-16T10:20:45.173 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:45.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:45.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:45.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:45 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1066781' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:20:45.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:45.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:45.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:20:45.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:45.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:45 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1066781' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:20:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:45 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1066781' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:20:45.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:45.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:45.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:45.521 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:45.548 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:45.548 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:45.578 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:45.578 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:45.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:45.603 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:45.603 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:45.636 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:45.636 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:45.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:45.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:45.669 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:45.705 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:45.705 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:45.737 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:45.737 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:45.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:45.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:45.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:45.838 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:45.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:45.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:45.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:45.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:20:46.030 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:46.078 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:46.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:46.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:46.211 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:46.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1066781' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:20:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:46 smithi165 ceph-mon[32696]: osdmap e224: 8 total, 8 up, 7 in 2024-09-16T10:20:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:46 smithi165 ceph-mon[32696]: pgmap v353: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-16T10:20:46.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1025447460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:46.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:46.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:46.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:46.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1066781' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[29993]: osdmap e224: 8 total, 8 up, 7 in 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[29993]: pgmap v353: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1025447460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1066781' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[39470]: osdmap e224: 8 total, 8 up, 7 in 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[39470]: pgmap v353: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-16T10:20:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1025447460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:46.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:46.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:46.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:46.628 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:46.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:46.721 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:46.722 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:46.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:46.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:46.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:46.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:46.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:46.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:47.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:47.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:47.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:47.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:47.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:47.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:47.323 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:47.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:47.336 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:20:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:20:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:20:47.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:47.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:47.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:47.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:20:47.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:47.594 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:47.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:47.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:47.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:47.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:47.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:47.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:47.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:47.936 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:47.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:48.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:48.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:48.170 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:48.195 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:20:48.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:48.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:48.392 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:48 smithi165 ceph-mon[32696]: pgmap v354: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s 2024-09-16T10:20:48.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/708935306' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:48.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:48 smithi165 ceph-mon[32696]: from='client.45422 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:48.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:48.440 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:48 smithi063 ceph-mon[29993]: pgmap v354: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s 2024-09-16T10:20:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/708935306' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:48 smithi063 ceph-mon[29993]: from='client.45422 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:48 smithi063 ceph-mon[39470]: pgmap v354: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s 2024-09-16T10:20:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/708935306' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:48 smithi063 ceph-mon[39470]: from='client.45422 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:20:48.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:48.558 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:48.558 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:48.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:48.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:48.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:48.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:20:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:20:48.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:48.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:48.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:48.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:48.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:48.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:49.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:20:49.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:49.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:49.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:49.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:49.249 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:49.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:49.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:49.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:20:49.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:49.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:49.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:49.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:49.403 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: from='client.45422 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: osdmap e225: 8 total, 8 up, 7 in 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.4 scrub ok 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.1 scrub ok 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.7 scrub ok 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:20:49.404 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: from='client.45422 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: osdmap e225: 8 total, 8 up, 7 in 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.4 scrub ok 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.1 scrub ok 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.7 scrub ok 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:20:49.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:49 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:20:49.492 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:49.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:20:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: from='client.45422 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:20:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: osdmap e225: 8 total, 8 up, 7 in 2024-09-16T10:20:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:20:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.4 scrub ok 2024-09-16T10:20:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:20:49.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.1 scrub ok 2024-09-16T10:20:49.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:49.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:20:49.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.7 scrub ok 2024-09-16T10:20:49.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:20:49.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:49 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:20:49.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:49.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:49.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:49.722 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:49.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:49.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:49.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:49.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:49.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:49.956 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:49.967 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:49.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:50.069 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:50.102 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:50.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:50.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:50.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:50.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:50.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:50.346 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:50.346 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:50.368 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:50.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:50.435 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:20:50.435 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[29993]: 1.5 scrub ok 2024-09-16T10:20:50.435 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[29993]: pgmap v356: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.32k op/s 2024-09-16T10:20:50.435 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:20:50.435 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:20:50.435 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:20:50.435 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[39470]: 1.5 scrub ok 2024-09-16T10:20:50.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[39470]: pgmap v356: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.32k op/s 2024-09-16T10:20:50.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:20:50.436 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:50 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:20:50.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:50.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:50.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:50 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:20:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:50 smithi165 ceph-mon[32696]: 1.5 scrub ok 2024-09-16T10:20:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:50 smithi165 ceph-mon[32696]: pgmap v356: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.32k op/s 2024-09-16T10:20:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:50 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:20:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:50 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:20:50.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:50.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:50.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:50.701 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:50.701 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:50.701 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:50.701 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:50.702 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:50.702 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:50.705 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:50.705 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:50.736 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:50.737 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:50.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:50.767 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:50.767 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:50.781 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:50.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:50.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:50.800 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:50.831 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:50.832 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:50.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:50.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:50.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:51.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:51.054 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:51.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:51.156 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:51.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:51.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:20:51.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:51.320 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:51.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:51.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:51.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:51.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:51.544 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:20:51.544 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:20:51.544 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T10:20:51.544 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T10:20:51.544 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T10:20:51.544 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-16T10:20:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:51 smithi165 ceph-mon[32696]: 4.12 scrub starts 2024-09-16T10:20:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:51 smithi165 ceph-mon[32696]: 4.12 scrub ok 2024-09-16T10:20:51.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2090614973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:51.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:51 smithi165 ceph-mon[32696]: osdmap e226: 8 total, 8 up, 7 in 2024-09-16T10:20:51.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:51 smithi165 ceph-mon[32696]: 6.8 scrub starts 2024-09-16T10:20:51.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:51 smithi165 ceph-mon[32696]: 6.8 scrub ok 2024-09-16T10:20:51.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:51.577 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[29993]: 4.12 scrub starts 2024-09-16T10:20:51.577 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[29993]: 4.12 scrub ok 2024-09-16T10:20:51.577 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2090614973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:51.577 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[29993]: osdmap e226: 8 total, 8 up, 7 in 2024-09-16T10:20:51.577 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[29993]: 6.8 scrub starts 2024-09-16T10:20:51.578 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[29993]: 6.8 scrub ok 2024-09-16T10:20:51.578 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[39470]: 4.12 scrub starts 2024-09-16T10:20:51.578 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[39470]: 4.12 scrub ok 2024-09-16T10:20:51.578 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2090614973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:51.578 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[39470]: osdmap e226: 8 total, 8 up, 7 in 2024-09-16T10:20:51.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[39470]: 6.8 scrub starts 2024-09-16T10:20:51.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:51 smithi063 ceph-mon[39470]: 6.8 scrub ok 2024-09-16T10:20:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:51.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:51.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:51.636 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:51.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:51.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:51.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:51.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:20:51.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:51.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:51.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:51.942 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:52.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:52.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:52.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:52.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:52.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:52.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:52.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:52.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:52.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:52.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:52.456 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:52.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:52.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[29993]: pgmap v358: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-16T10:20:52.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[29993]: 8.c scrub starts 2024-09-16T10:20:52.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[29993]: 8.d scrub starts 2024-09-16T10:20:52.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[29993]: 4.a scrub starts 2024-09-16T10:20:52.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[29993]: 4.a scrub ok 2024-09-16T10:20:52.497 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:20:52.497 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[39470]: pgmap v358: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-16T10:20:52.497 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[39470]: 8.c scrub starts 2024-09-16T10:20:52.497 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[39470]: 8.d scrub starts 2024-09-16T10:20:52.497 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[39470]: 4.a scrub starts 2024-09-16T10:20:52.497 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[39470]: 4.a scrub ok 2024-09-16T10:20:52.497 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:52 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:20:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:52 smithi165 ceph-mon[32696]: pgmap v358: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-16T10:20:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:52 smithi165 ceph-mon[32696]: 8.c scrub starts 2024-09-16T10:20:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:52 smithi165 ceph-mon[32696]: 8.d scrub starts 2024-09-16T10:20:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:52 smithi165 ceph-mon[32696]: 4.a scrub starts 2024-09-16T10:20:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:52 smithi165 ceph-mon[32696]: 4.a scrub ok 2024-09-16T10:20:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:52 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:20:52.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:52.559 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:52.560 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:20:52.560 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T10:20:52.560 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:20:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:52.603 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:52.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:52.704 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:52.736 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:52.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:52.838 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:52.871 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:52.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:52.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:53.000 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:53.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:53.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:53.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:53.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:53.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:53.421 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:53.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:20:53.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:20:53.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:20:53.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: 8.c scrub ok 2024-09-16T10:20:53.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:20:53.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:20:53.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: 8.4 scrub starts 2024-09-16T10:20:53.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:53.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:53.555 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:20:53.555 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:20:53.555 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:20:53.555 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:20:53.556 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:20:53.561 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:53.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:53.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:53.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: 8.c scrub ok 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: 8.4 scrub starts 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:20:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: 8.c scrub ok 2024-09-16T10:20:53.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:20:53.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:20:53.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: 8.4 scrub starts 2024-09-16T10:20:53.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:20:53.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:20:53.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:53.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:53.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:53.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:53.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:54.076 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:54.117 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:54.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:54.203 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:54.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:54.241 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:20:54.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:54.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:54.361 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:54.398 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:54.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:54.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:20:54.498 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: 8.d scrub ok 2024-09-16T10:20:54.498 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: pgmap v359: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s 2024-09-16T10:20:54.498 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: 8.e scrub starts 2024-09-16T10:20:54.499 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:20:54.499 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:20:54.499 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: 8.8 scrub starts 2024-09-16T10:20:54.499 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1254762779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:54.499 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:54 smithi165 ceph-mon[32696]: from='client.45434 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:54.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: 8.d scrub ok 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: pgmap v359: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: 8.e scrub starts 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: 8.8 scrub starts 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1254762779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:54.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[29993]: from='client.45434 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: 8.d scrub ok 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: pgmap v359: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: 8.e scrub starts 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: 8.8 scrub starts 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1254762779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:54.508 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:54 smithi063 ceph-mon[39470]: from='client.45434 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:20:54.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:54.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:54.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:54.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:54.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:54.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:54.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:54.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:54.903 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:20:54.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:54.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:54.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:55.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:55.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:55.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:55.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:55.232 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:55.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: 8.4 scrub ok 2024-09-16T10:20:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:20:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:20:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: from='client.45434 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:55.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: osdmap e227: 8 total, 8 up, 7 in 2024-09-16T10:20:55.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:20:55.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:20:55.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: 6.11 scrub starts 2024-09-16T10:20:55.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: 6.11 scrub ok 2024-09-16T10:20:55.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:55 smithi165 ceph-mon[32696]: 8.e scrub ok 2024-09-16T10:20:55.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:55.369 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:55.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: 8.4 scrub ok 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: from='client.45434 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: osdmap e227: 8 total, 8 up, 7 in 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:20:55.470 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: 6.11 scrub starts 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: 6.11 scrub ok 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[29993]: 8.e scrub ok 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: 8.4 scrub ok 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: from='client.45434 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: osdmap e227: 8 total, 8 up, 7 in 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: 6.11 scrub starts 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: 6.11 scrub ok 2024-09-16T10:20:55.471 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:55 smithi063 ceph-mon[39470]: 8.e scrub ok 2024-09-16T10:20:55.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:55.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:55.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:55.537 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:20:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:20:55.573 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:20:55.573 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:20:55.601 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:20:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:20:55.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:55.632 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:20:55.632 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:20:55.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:55.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:55.657 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:20:55.658 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:20:55.704 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:20:55.704 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:20:55.741 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:20:55.741 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:20:55.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:55.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:20:55.775 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:20:55.786 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:55.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:55.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:55.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:55.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:56.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:56.062 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:56.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:56.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:56.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:56.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:56.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:56.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:56.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:56.448 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:56.477 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:56.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:56.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 5.16 scrub starts 2024-09-16T10:20:56.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 5.16 scrub ok 2024-09-16T10:20:56.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 8.8 scrub ok 2024-09-16T10:20:56.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: pgmap v361: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 733 B/s rd, 0 B/s wr, 1.90k op/s 2024-09-16T10:20:56.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 5.d scrub starts 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 5.d scrub ok 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 3.0 scrub starts 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 3.0 scrub ok 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4252846546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:56.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:56 smithi165 ceph-mon[32696]: osdmap e228: 8 total, 8 up, 7 in 2024-09-16T10:20:56.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:56.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:56.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:56.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 5.16 scrub starts 2024-09-16T10:20:56.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 5.16 scrub ok 2024-09-16T10:20:56.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 8.8 scrub ok 2024-09-16T10:20:56.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: pgmap v361: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 733 B/s rd, 0 B/s wr, 1.90k op/s 2024-09-16T10:20:56.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 5.d scrub starts 2024-09-16T10:20:56.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 5.d scrub ok 2024-09-16T10:20:56.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 3.0 scrub starts 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 3.0 scrub ok 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4252846546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[29993]: osdmap e228: 8 total, 8 up, 7 in 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 5.16 scrub starts 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 5.16 scrub ok 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 8.8 scrub ok 2024-09-16T10:20:56.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: pgmap v361: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 733 B/s rd, 0 B/s wr, 1.90k op/s 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 5.d scrub starts 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 5.d scrub ok 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 3.0 scrub starts 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 3.0 scrub ok 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4252846546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:20:56.680 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:56 smithi063 ceph-mon[39470]: osdmap e228: 8 total, 8 up, 7 in 2024-09-16T10:20:56.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:56.744 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:56.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:56.799 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:56.800 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:56.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:56.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:56.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:56.979 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:57.011 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:57.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:57.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:57.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:57.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:57.238 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:20:57.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:57.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:57.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:57.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:57.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:57.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:57.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:57 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:20:57.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:20:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:20:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:20:57.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:57 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:20:57.496 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:20:57.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:20:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:57 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:20:57.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:57.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:20:57.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:57.667 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:57.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:57.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:57.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:57.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:57.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:57.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:58.002 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:58.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:58.148 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:58.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:58.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:58.324 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:20:58.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:20:58.423 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:20:58.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 3.3 scrub starts 2024-09-16T10:20:58.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 3.3 scrub ok 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: pgmap v363: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 808 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 3.e scrub starts 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 3.e scrub ok 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 3.1b scrub starts 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 3.1b scrub ok 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[29993]: 8.5 scrub starts 2024-09-16T10:20:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 3.3 scrub starts 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 3.3 scrub ok 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: pgmap v363: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 808 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 3.e scrub starts 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 3.e scrub ok 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 3.1b scrub starts 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 3.1b scrub ok 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:20:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:58 smithi063 ceph-mon[39470]: 8.5 scrub starts 2024-09-16T10:20:58.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 3.3 scrub starts 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 3.3 scrub ok 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: pgmap v363: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 808 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 3.e scrub starts 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 3.e scrub ok 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 3.1b scrub starts 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 3.1b scrub ok 2024-09-16T10:20:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:20:58.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:20:58.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-mon[32696]: 8.5 scrub starts 2024-09-16T10:20:58.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:20:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:20:58.568 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:58.669 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.703 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:58.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:58.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:20:58.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:20:58.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:20:58.923 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:20:58.923 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:20:58.923 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-16T10:20:58.923 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-16T10:20:58.923 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:20:58.935 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:20:58.935 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:20:58.942 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:58.980 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:58.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:59.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:59.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:59.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:59.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:20:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:59.254 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:59.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:20:59.276 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:20:59.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:20:59.389 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:59.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:20:59.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:20:59.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:59.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:59.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 5.10 scrub starts 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 5.10 scrub ok 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 5.6 scrub starts 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 5.6 scrub ok 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 3.18 scrub starts 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 3.18 scrub ok 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2652830176' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:20:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:20:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:20:59 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:20:59.633 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:59.666 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:59.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:20:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:20:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:20:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 5.10 scrub starts 2024-09-16T10:20:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 5.10 scrub ok 2024-09-16T10:20:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 5.6 scrub starts 2024-09-16T10:20:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 5.6 scrub ok 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 3.18 scrub starts 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 3.18 scrub ok 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2652830176' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 5.10 scrub starts 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 5.10 scrub ok 2024-09-16T10:20:59.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 5.6 scrub starts 2024-09-16T10:20:59.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 5.6 scrub ok 2024-09-16T10:20:59.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 3.18 scrub starts 2024-09-16T10:20:59.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 3.18 scrub ok 2024-09-16T10:20:59.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:20:59.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2652830176' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:20:59.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:20:59.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:20:59 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:20:59.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:20:59.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:20:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:59.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:20:59.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:20:59.939 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:20:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:20:59.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:00.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:21:00.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:00.133 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:00.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:00.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:00.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:00.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:00.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:00.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:00.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:00.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:00 smithi165 ceph-mon[32696]: 8.2 scrub starts 2024-09-16T10:21:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:00 smithi165 ceph-mon[32696]: 8.5 scrub ok 2024-09-16T10:21:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:00 smithi165 ceph-mon[32696]: pgmap v364: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:00 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2652830176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:21:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:00 smithi165 ceph-mon[32696]: osdmap e229: 8 total, 8 up, 7 in 2024-09-16T10:21:00.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:00.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[29993]: 8.2 scrub starts 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[29993]: 8.5 scrub ok 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[29993]: pgmap v364: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2652830176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[29993]: osdmap e229: 8 total, 8 up, 7 in 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[39470]: 8.2 scrub starts 2024-09-16T10:21:00.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[39470]: 8.5 scrub ok 2024-09-16T10:21:00.640 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[39470]: pgmap v364: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:00.640 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:00.640 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2652830176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:21:00.640 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:00 smithi063 ceph-mon[39470]: osdmap e229: 8 total, 8 up, 7 in 2024-09-16T10:21:00.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:00.726 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:00.735 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:00.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:00.747 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:00.800 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:00.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:00.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:00.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:00.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:01.012 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:01.114 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:01.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:01.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:01.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:01.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:01.427 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:01.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:01.563 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:01.666 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:01.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:01.834 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:01.936 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:01.972 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:01.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:02.074 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:02.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:02.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:02.183 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:01 smithi063 ceph-mon[29993]: osdmap e230: 8 total, 8 up, 7 in 2024-09-16T10:21:02.184 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:01 smithi063 ceph-mon[29993]: pgmap v367: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-16T10:21:02.184 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3639099621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:02.184 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:01 smithi063 ceph-mon[39470]: osdmap e230: 8 total, 8 up, 7 in 2024-09-16T10:21:02.184 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:01 smithi063 ceph-mon[39470]: pgmap v367: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-16T10:21:02.184 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3639099621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:02.184 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:02.185 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:02.186 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:02.186 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:02.186 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:02.186 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:02.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:02.238 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:02.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:01 smithi165 ceph-mon[32696]: osdmap e230: 8 total, 8 up, 7 in 2024-09-16T10:21:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:01 smithi165 ceph-mon[32696]: pgmap v367: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-16T10:21:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3639099621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:02.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:02.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:02.376 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:02.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:02.477 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:02.509 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:02.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:02.610 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:02.636 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:02.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:02.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:21:02.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:02.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:02.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:02.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:02.909 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:02.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:03.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:03.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:03.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:03.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:03.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:03.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:03.278 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:03.314 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:03.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:03.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:03.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:03.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:03.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:03.603 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:03.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:03.704 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:03.749 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:03.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:03.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:03.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:03.885 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:03.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:03.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:04.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:04.065 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:04.065 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:21:04.065 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:21:04.065 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:21:04.065 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:04.124 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:04.163 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:04.265 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:04.296 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:04.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:04.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:04.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:04 smithi063 ceph-mon[39470]: pgmap v368: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 698 B/s rd, 0 B/s wr, 956 op/s 2024-09-16T10:21:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:04 smithi063 ceph-mon[29993]: pgmap v368: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 698 B/s rd, 0 B/s wr, 956 op/s 2024-09-16T10:21:04.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:04.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:04 smithi165 ceph-mon[32696]: pgmap v368: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 698 B/s rd, 0 B/s wr, 956 op/s 2024-09-16T10:21:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.579 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:04.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:04.680 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.716 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:04.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:04.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:04.854 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:04.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:04.981 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:21:04.983 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:21:04.992 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:04.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:05.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:05.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:05.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:05.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:05.277 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:05.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:05.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:05.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:05.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:05.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:05.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:05.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:05.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:05.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:05.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:05.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:05.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:21:05.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:05.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:05.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:05.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:05.979 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:05.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:06.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:06.122 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:06.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:06.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:06.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:06.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:06.353 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:06 smithi063 ceph-mon[39470]: from='client.15621 v1:172.21.15.63:0/2704762778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:21:06.353 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:06 smithi063 ceph-mon[39470]: pgmap v369: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s 2024-09-16T10:21:06.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:06 smithi063 ceph-mon[29993]: from='client.15621 v1:172.21.15.63:0/2704762778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:21:06.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:06 smithi063 ceph-mon[29993]: pgmap v369: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s 2024-09-16T10:21:06.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:06.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:06.400 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-16T10:21:03.175165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4035648051,"num_objects":61770,"num_object_clones":0,"num_object_copies":185310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61770,"num_whiteouts":0,"num_read":1157,"num_read_kb":909,"num_write":193501,"num_write_kb":8148334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10190,"num_bytes_recovered":663357726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69475,"ondisk_log_size":69475,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13982184,"kb_used_data":11992180,"kb_used_omap":10,"kb_used_meta":1989621,"kb_avail":642233880,"statfs":{"total":671965249536,"available":657647493120,"internally_reserved":0,"allocated":12279992320,"data_stored":12276165418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":2037372012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-459407360,"num_objects":-7010,"num_object_clones":0,"num_object_copies":-21030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7010,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.331956"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":383,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847897+0000","last_change":"2024-09-16T10:18:52.830567+0000","last_active":"2024-09-16T10:20:55.847897+0000","last_peered":"2024-09-16T10:20:55.847897+0000","last_clean":"2024-09-16T10:20:55.847897+0000","last_became_active":"2024-09-16T10:18:52.830260+0000","last_became_peered":"2024-09-16T10:18:52.830260+0000","last_unstale":"2024-09-16T10:20:55.847897+0000","last_undegraded":"2024-09-16T10:20:55.847897+0000","last_fullsized":"2024-09-16T10:20:55.847897+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:07.119072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.119072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.092096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":430,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841501+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:20:55.841501+0000","last_peered":"2024-09-16T10:20:55.841501+0000","last_clean":"2024-09-16T10:20:55.841501+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:20:55.841501+0000","last_undegraded":"2024-09-16T10:20:55.841501+0000","last_fullsized":"2024-09-16T10:20:55.841501+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:25.723031+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846409+0000","last_change":"2024-09-16T10:18:40.752790+0000","last_active":"2024-09-16T10:20:55.846409+0000","last_peered":"2024-09-16T10:20:55.846409+0000","last_clean":"2024-09-16T10:20:55.846409+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:20:55.846409+0000","last_undegraded":"2024-09-16T10:20:55.846409+0000","last_fullsized":"2024-09-16T10:20:55.846409+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:40.752739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:40.752739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.494552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":355,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842713+0000","last_change":"2024-09-16T10:19:23.784420+0000","last_active":"2024-09-16T10:20:55.842713+0000","last_peered":"2024-09-16T10:20:55.842713+0000","last_clean":"2024-09-16T10:20:55.842713+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:20:55.842713+0000","last_undegraded":"2024-09-16T10:20:55.842713+0000","last_fullsized":"2024-09-16T10:20:55.842713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.784347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.784347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":364,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846010+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:20:55.846010+0000","last_peered":"2024-09-16T10:20:55.846010+0000","last_clean":"2024-09-16T10:20:55.846010+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:20:55.846010+0000","last_undegraded":"2024-09-16T10:20:55.846010+0000","last_fullsized":"2024-09-16T10:20:55.846010+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:39.443408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":384,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844754+0000","last_change":"2024-09-16T10:20:33.257003+0000","last_active":"2024-09-16T10:20:55.844754+0000","last_peered":"2024-09-16T10:20:55.844754+0000","last_clean":"2024-09-16T10:20:55.844754+0000","last_became_active":"2024-09-16T10:18:52.836507+0000","last_became_peered":"2024-09-16T10:18:52.836507+0000","last_unstale":"2024-09-16T10:20:55.844754+0000","last_undegraded":"2024-09-16T10:20:55.844754+0000","last_fullsized":"2024-09-16T10:20:55.844754+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.256934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.256934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:55.621693+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":421,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844513+0000","last_change":"2024-09-16T10:20:31.270635+0000","last_active":"2024-09-16T10:20:55.844513+0000","last_peered":"2024-09-16T10:20:55.844513+0000","last_clean":"2024-09-16T10:20:55.844513+0000","last_became_active":"2024-09-16T10:18:52.835715+0000","last_became_peered":"2024-09-16T10:18:52.835715+0000","last_unstale":"2024-09-16T10:20:55.844513+0000","last_undegraded":"2024-09-16T10:20:55.844513+0000","last_fullsized":"2024-09-16T10:20:55.844513+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:20:31.270548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.270548+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:59.374039+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":367,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841377+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:20:55.841377+0000","last_peered":"2024-09-16T10:20:55.841377+0000","last_clean":"2024-09-16T10:20:55.841377+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:20:55.841377+0000","last_undegraded":"2024-09-16T10:20:55.841377+0000","last_fullsized":"2024-09-16T10:20:55.841377+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:07.843544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845890+0000","last_change":"2024-09-16T10:19:57.272422+0000","last_active":"2024-09-16T10:20:55.845890+0000","last_peered":"2024-09-16T10:20:55.845890+0000","last_clean":"2024-09-16T10:20:55.845890+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:20:55.845890+0000","last_undegraded":"2024-09-16T10:20:55.845890+0000","last_fullsized":"2024-09-16T10:20:55.845890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.272362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.272362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:13.881596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":387,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.843098+0000","last_change":"2024-09-16T10:20:30.008294+0000","last_active":"2024-09-16T10:20:55.843098+0000","last_peered":"2024-09-16T10:20:55.843098+0000","last_clean":"2024-09-16T10:20:55.843098+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:20:55.843098+0000","last_undegraded":"2024-09-16T10:20:55.843098+0000","last_fullsized":"2024-09-16T10:20:55.843098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:30.008215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:30.008215+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.975169+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846772+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:20:55.846772+0000","last_peered":"2024-09-16T10:20:55.846772+0000","last_clean":"2024-09-16T10:20:55.846772+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:20:55.846772+0000","last_undegraded":"2024-09-16T10:20:55.846772+0000","last_fullsized":"2024-09-16T10:20:55.846772+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.206970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":363,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845647+0000","last_change":"2024-09-16T10:20:33.259383+0000","last_active":"2024-09-16T10:20:55.845647+0000","last_peered":"2024-09-16T10:20:55.845647+0000","last_clean":"2024-09-16T10:20:55.845647+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:20:55.845647+0000","last_undegraded":"2024-09-16T10:20:55.845647+0000","last_fullsized":"2024-09-16T10:20:55.845647+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.259340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.259340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:59.265290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":352,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846158+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:20:55.846158+0000","last_peered":"2024-09-16T10:20:55.846158+0000","last_clean":"2024-09-16T10:20:55.846158+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:20:55.846158+0000","last_undegraded":"2024-09-16T10:20:55.846158+0000","last_fullsized":"2024-09-16T10:20:55.846158+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.350155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":381,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841384+0000","last_change":"2024-09-16T10:20:32.001472+0000","last_active":"2024-09-16T10:20:55.841384+0000","last_peered":"2024-09-16T10:20:55.841384+0000","last_clean":"2024-09-16T10:20:55.841384+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:20:55.841384+0000","last_undegraded":"2024-09-16T10:20:55.841384+0000","last_fullsized":"2024-09-16T10:20:55.841384+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:20:32.001398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:32.001398+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.377925+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":360,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852183+0000","last_change":"2024-09-16T10:19:23.390542+0000","last_active":"2024-09-16T10:20:55.852183+0000","last_peered":"2024-09-16T10:20:55.852183+0000","last_clean":"2024-09-16T10:20:55.852183+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:20:55.852183+0000","last_undegraded":"2024-09-16T10:20:55.852183+0000","last_fullsized":"2024-09-16T10:20:55.852183+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":364,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.849194+0000","last_change":"2024-09-16T10:19:46.109382+0000","last_active":"2024-09-16T10:20:55.849194+0000","last_peered":"2024-09-16T10:20:55.849194+0000","last_clean":"2024-09-16T10:20:55.849194+0000","last_became_active":"2024-09-16T10:13:43.121379+0000","last_became_peered":"2024-09-16T10:13:43.121379+0000","last_unstale":"2024-09-16T10:20:55.849194+0000","last_undegraded":"2024-09-16T10:20:55.849194+0000","last_fullsized":"2024-09-16T10:20:55.849194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:46.109310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.109310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.462251+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":364,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842453+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:20:55.842453+0000","last_peered":"2024-09-16T10:20:55.842453+0000","last_clean":"2024-09-16T10:20:55.842453+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:20:55.842453+0000","last_undegraded":"2024-09-16T10:20:55.842453+0000","last_fullsized":"2024-09-16T10:20:55.842453+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.949799+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":367,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:57.263993+0000","last_change":"2024-09-16T10:20:57.263993+0000","last_active":"2024-09-16T10:20:57.263993+0000","last_peered":"2024-09-16T10:20:57.263993+0000","last_clean":"2024-09-16T10:20:57.263993+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:20:57.263993+0000","last_undegraded":"2024-09-16T10:20:57.263993+0000","last_fullsized":"2024-09-16T10:20:57.263993+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.263916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.263916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:25.361547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":188,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.840575+0000","last_change":"2024-09-16T10:19:14.579717+0000","last_active":"2024-09-16T10:20:55.840575+0000","last_peered":"2024-09-16T10:20:55.840575+0000","last_clean":"2024-09-16T10:20:55.840575+0000","last_became_active":"2024-09-16T10:18:52.841218+0000","last_became_peered":"2024-09-16T10:18:52.841218+0000","last_unstale":"2024-09-16T10:20:55.840575+0000","last_undegraded":"2024-09-16T10:20:55.840575+0000","last_fullsized":"2024-09-16T10:20:55.840575+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.441526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841101+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:20:55.841101+0000","last_peered":"2024-09-16T10:20:55.841101+0000","last_clean":"2024-09-16T10:20:55.841101+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:20:55.841101+0000","last_undegraded":"2024-09-16T10:20:55.841101+0000","last_fullsized":"2024-09-16T10:20:55.841101+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.434331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":314,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845004+0000","last_change":"2024-09-16T10:19:48.110737+0000","last_active":"2024-09-16T10:20:55.845004+0000","last_peered":"2024-09-16T10:20:55.845004+0000","last_clean":"2024-09-16T10:20:55.845004+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:20:55.845004+0000","last_undegraded":"2024-09-16T10:20:55.845004+0000","last_fullsized":"2024-09-16T10:20:55.845004+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:48.110674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.110674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:10.967276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":366,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.843234+0000","last_change":"2024-09-16T10:19:22.803999+0000","last_active":"2024-09-16T10:20:55.843234+0000","last_peered":"2024-09-16T10:20:55.843234+0000","last_clean":"2024-09-16T10:20:55.843234+0000","last_became_active":"2024-09-16T10:18:52.834723+0000","last_became_peered":"2024-09-16T10:18:52.834723+0000","last_unstale":"2024-09-16T10:20:55.843234+0000","last_undegraded":"2024-09-16T10:20:55.843234+0000","last_fullsized":"2024-09-16T10:20:55.843234+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":422,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:57.423064+0000","last_change":"2024-09-16T10:20:57.423064+0000","last_active":"2024-09-16T10:20:57.423064+0000","last_peered":"2024-09-16T10:20:57.423064+0000","last_clean":"2024-09-16T10:20:57.423064+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:20:57.423064+0000","last_undegraded":"2024-09-16T10:20:57.423064+0000","last_fullsized":"2024-09-16T10:20:57.423064+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:20:57.422968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.422968+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:21.471811+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":374,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842590+0000","last_change":"2024-09-16T10:18:52.839387+0000","last_active":"2024-09-16T10:20:55.842590+0000","last_peered":"2024-09-16T10:20:55.842590+0000","last_clean":"2024-09-16T10:20:55.842590+0000","last_became_active":"2024-09-16T10:18:52.838765+0000","last_became_peered":"2024-09-16T10:18:52.838765+0000","last_unstale":"2024-09-16T10:20:55.842590+0000","last_undegraded":"2024-09-16T10:20:55.842590+0000","last_fullsized":"2024-09-16T10:20:55.842590+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:05.795954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.795954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:04.825505+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":360,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852598+0000","last_change":"2024-09-16T10:18:52.847158+0000","last_active":"2024-09-16T10:20:55.852598+0000","last_peered":"2024-09-16T10:20:55.852598+0000","last_clean":"2024-09-16T10:20:55.852598+0000","last_became_active":"2024-09-16T10:18:52.838789+0000","last_became_peered":"2024-09-16T10:18:52.838789+0000","last_unstale":"2024-09-16T10:20:55.852598+0000","last_undegraded":"2024-09-16T10:20:55.852598+0000","last_fullsized":"2024-09-16T10:20:55.852598+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.084681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846517+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:20:55.846517+0000","last_peered":"2024-09-16T10:20:55.846517+0000","last_clean":"2024-09-16T10:20:55.846517+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:20:55.846517+0000","last_undegraded":"2024-09-16T10:20:55.846517+0000","last_fullsized":"2024-09-16T10:20:55.846517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:37.450673+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":387,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847542+0000","last_change":"2024-09-16T10:19:49.067027+0000","last_active":"2024-09-16T10:20:55.847542+0000","last_peered":"2024-09-16T10:20:55.847542+0000","last_clean":"2024-09-16T10:20:55.847542+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:20:55.847542+0000","last_undegraded":"2024-09-16T10:20:55.847542+0000","last_fullsized":"2024-09-16T10:20:55.847542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:49.066953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:49.066953+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.060028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":468,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.849828+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:20:55.849828+0000","last_peered":"2024-09-16T10:20:55.849828+0000","last_clean":"2024-09-16T10:20:55.849828+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:20:55.849828+0000","last_undegraded":"2024-09-16T10:20:55.849828+0000","last_fullsized":"2024-09-16T10:20:55.849828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.628369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"228'12181","reported_seq":12320,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.454687+0000","last_change":"2024-09-16T10:18:33.825761+0000","last_active":"2024-09-16T10:20:58.454687+0000","last_peered":"2024-09-16T10:20:58.454687+0000","last_clean":"2024-09-16T10:20:58.454687+0000","last_became_active":"2024-09-16T10:18:33.825267+0000","last_became_peered":"2024-09-16T10:18:33.825267+0000","last_unstale":"2024-09-16T10:20:58.454687+0000","last_undegraded":"2024-09-16T10:20:58.454687+0000","last_fullsized":"2024-09-16T10:20:58.454687+0000","mapping_epoch":165,"log_start":"209'8113","ondisk_log_start":"209'8113","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:45.304329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253952000,"num_objects":3875,"num_object_clones":0,"num_object_copies":11625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12181,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":181,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845664+0000","last_change":"2024-09-16T10:20:26.408565+0000","last_active":"2024-09-16T10:20:55.845664+0000","last_peered":"2024-09-16T10:20:55.845664+0000","last_clean":"2024-09-16T10:20:55.845664+0000","last_became_active":"2024-09-16T10:18:52.825707+0000","last_became_peered":"2024-09-16T10:18:52.825707+0000","last_unstale":"2024-09-16T10:20:55.845664+0000","last_undegraded":"2024-09-16T10:20:55.845664+0000","last_fullsized":"2024-09-16T10:20:55.845664+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:26.408503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.408503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:31.713460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":390,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841012+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:20:55.841012+0000","last_peered":"2024-09-16T10:20:55.841012+0000","last_clean":"2024-09-16T10:20:55.841012+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:20:55.841012+0000","last_undegraded":"2024-09-16T10:20:55.841012+0000","last_fullsized":"2024-09-16T10:20:55.841012+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.426229+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848394+0000","last_change":"2024-09-16T10:19:43.028508+0000","last_active":"2024-09-16T10:20:55.848394+0000","last_peered":"2024-09-16T10:20:55.848394+0000","last_clean":"2024-09-16T10:20:55.848394+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:20:55.848394+0000","last_undegraded":"2024-09-16T10:20:55.848394+0000","last_fullsized":"2024-09-16T10:20:55.848394+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:43.028436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:43.028436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":449,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842991+0000","last_change":"2024-09-16T10:18:36.623362+0000","last_active":"2024-09-16T10:20:55.842991+0000","last_peered":"2024-09-16T10:20:55.842991+0000","last_clean":"2024-09-16T10:20:55.842991+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:20:55.842991+0000","last_undegraded":"2024-09-16T10:20:55.842991+0000","last_fullsized":"2024-09-16T10:20:55.842991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:36.623302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:36.623302+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:12.928759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":357,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845538+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:20:55.845538+0000","last_peered":"2024-09-16T10:20:55.845538+0000","last_clean":"2024-09-16T10:20:55.845538+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:20:55.845538+0000","last_undegraded":"2024-09-16T10:20:55.845538+0000","last_fullsized":"2024-09-16T10:20:55.845538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"228'12086","reported_seq":12275,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.634831+0000","last_change":"2024-09-16T10:20:52.238795+0000","last_active":"2024-09-16T10:20:58.634831+0000","last_peered":"2024-09-16T10:20:58.634831+0000","last_clean":"2024-09-16T10:20:58.634831+0000","last_became_active":"2024-09-16T10:18:33.893281+0000","last_became_peered":"2024-09-16T10:18:33.893281+0000","last_unstale":"2024-09-16T10:20:58.634831+0000","last_undegraded":"2024-09-16T10:20:58.634831+0000","last_fullsized":"2024-09-16T10:20:58.634831+0000","mapping_epoch":165,"log_start":"206'7584","ondisk_log_start":"206'7584","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'11541","last_scrub_stamp":"2024-09-16T10:20:52.238522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.238522+0000","objects_scrubbed":4319,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:08.553927+0000","scrub_duration":1012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844046+0000","last_change":"2024-09-16T10:20:48.845379+0000","last_active":"2024-09-16T10:20:55.844046+0000","last_peered":"2024-09-16T10:20:55.844046+0000","last_clean":"2024-09-16T10:20:55.844046+0000","last_became_active":"2024-09-16T10:20:15.864502+0000","last_became_peered":"2024-09-16T10:20:15.864502+0000","last_unstale":"2024-09-16T10:20:55.844046+0000","last_undegraded":"2024-09-16T10:20:55.844046+0000","last_fullsized":"2024-09-16T10:20:55.844046+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.845331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.845331+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.651147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":397,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845030+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:20:55.845030+0000","last_peered":"2024-09-16T10:20:55.845030+0000","last_clean":"2024-09-16T10:20:55.845030+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:20:55.845030+0000","last_undegraded":"2024-09-16T10:20:55.845030+0000","last_fullsized":"2024-09-16T10:20:55.845030+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:03.147111+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":334,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845815+0000","last_change":"2024-09-16T10:19:50.114162+0000","last_active":"2024-09-16T10:20:55.845815+0000","last_peered":"2024-09-16T10:20:55.845815+0000","last_clean":"2024-09-16T10:20:55.845815+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:20:55.845815+0000","last_undegraded":"2024-09-16T10:20:55.845815+0000","last_fullsized":"2024-09-16T10:20:55.845815+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.114098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.114098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:16.082035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":114,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847135+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:20:55.847135+0000","last_peered":"2024-09-16T10:20:55.847135+0000","last_clean":"2024-09-16T10:20:55.847135+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:20:55.847135+0000","last_undegraded":"2024-09-16T10:20:55.847135+0000","last_fullsized":"2024-09-16T10:20:55.847135+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:36.618613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":379,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842988+0000","last_change":"2024-09-16T10:19:44.711933+0000","last_active":"2024-09-16T10:20:55.842988+0000","last_peered":"2024-09-16T10:20:55.842988+0000","last_clean":"2024-09-16T10:20:55.842988+0000","last_became_active":"2024-09-16T10:18:52.835843+0000","last_became_peered":"2024-09-16T10:18:52.835843+0000","last_unstale":"2024-09-16T10:20:55.842988+0000","last_undegraded":"2024-09-16T10:20:55.842988+0000","last_fullsized":"2024-09-16T10:20:55.842988+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"228'12174","reported_seq":15875,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.285245+0000","last_change":"2024-09-16T10:20:52.637052+0000","last_active":"2024-09-16T10:20:58.285245+0000","last_peered":"2024-09-16T10:20:58.285245+0000","last_clean":"2024-09-16T10:20:58.285245+0000","last_became_active":"2024-09-16T10:19:15.293347+0000","last_became_peered":"2024-09-16T10:19:15.293347+0000","last_unstale":"2024-09-16T10:20:58.285245+0000","last_undegraded":"2024-09-16T10:20:58.285245+0000","last_fullsized":"2024-09-16T10:20:58.285245+0000","mapping_epoch":188,"log_start":"209'8034","ondisk_log_start":"209'8034","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'11682","last_scrub_stamp":"2024-09-16T10:20:52.636902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.636902+0000","objects_scrubbed":4492,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:56.892704+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258670616,"num_objects":3948,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12174,"num_write_kb":515841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1748,"num_bytes_recovered":114556928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848210+0000","last_change":"2024-09-16T10:20:48.378293+0000","last_active":"2024-09-16T10:20:55.848210+0000","last_peered":"2024-09-16T10:20:55.848210+0000","last_clean":"2024-09-16T10:20:55.848210+0000","last_became_active":"2024-09-16T10:20:15.858791+0000","last_became_peered":"2024-09-16T10:20:15.858791+0000","last_unstale":"2024-09-16T10:20:55.848210+0000","last_undegraded":"2024-09-16T10:20:55.848210+0000","last_fullsized":"2024-09-16T10:20:55.848210+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.378229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.378229+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.046361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":401,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841645+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:20:55.841645+0000","last_peered":"2024-09-16T10:20:55.841645+0000","last_clean":"2024-09-16T10:20:55.841645+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:20:55.841645+0000","last_undegraded":"2024-09-16T10:20:55.841645+0000","last_fullsized":"2024-09-16T10:20:55.841645+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:15.756451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":377,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847771+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:20:55.847771+0000","last_peered":"2024-09-16T10:20:55.847771+0000","last_clean":"2024-09-16T10:20:55.847771+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:20:55.847771+0000","last_undegraded":"2024-09-16T10:20:55.847771+0000","last_fullsized":"2024-09-16T10:20:55.847771+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:14.162775+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":461,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844887+0000","last_change":"2024-09-16T10:18:04.023115+0000","last_active":"2024-09-16T10:20:55.844887+0000","last_peered":"2024-09-16T10:20:55.844887+0000","last_clean":"2024-09-16T10:20:55.844887+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:20:55.844887+0000","last_undegraded":"2024-09-16T10:20:55.844887+0000","last_fullsized":"2024-09-16T10:20:55.844887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:04.022579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.022579+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:00.209183+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":356,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851159+0000","last_change":"2024-09-16T10:18:52.824583+0000","last_active":"2024-09-16T10:20:55.851159+0000","last_peered":"2024-09-16T10:20:55.851159+0000","last_clean":"2024-09-16T10:20:55.851159+0000","last_became_active":"2024-09-16T10:18:52.824265+0000","last_became_peered":"2024-09-16T10:18:52.824265+0000","last_unstale":"2024-09-16T10:20:55.851159+0000","last_undegraded":"2024-09-16T10:20:55.851159+0000","last_fullsized":"2024-09-16T10:20:55.851159+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:37.026545+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"228'11944","reported_seq":15446,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.570747+0000","last_change":"2024-09-16T10:20:54.577288+0000","last_active":"2024-09-16T10:20:58.570747+0000","last_peered":"2024-09-16T10:20:58.570747+0000","last_clean":"2024-09-16T10:20:58.570747+0000","last_became_active":"2024-09-16T10:19:14.293461+0000","last_became_peered":"2024-09-16T10:19:14.293461+0000","last_unstale":"2024-09-16T10:20:58.570747+0000","last_undegraded":"2024-09-16T10:20:58.570747+0000","last_fullsized":"2024-09-16T10:20:58.570747+0000","mapping_epoch":187,"log_start":"206'7476","ondisk_log_start":"206'7476","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11610","last_scrub_stamp":"2024-09-16T10:20:54.577223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.577223+0000","objects_scrubbed":4152,"log_size":4468,"log_dups_size":0,"ondisk_log_size":4468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:02.760584+0000","scrub_duration":1252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247595008,"num_objects":3778,"num_object_clones":0,"num_object_copies":11334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":262,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848567+0000","last_change":"2024-09-16T10:20:48.578602+0000","last_active":"2024-09-16T10:20:55.848567+0000","last_peered":"2024-09-16T10:20:55.848567+0000","last_clean":"2024-09-16T10:20:55.848567+0000","last_became_active":"2024-09-16T10:20:15.854970+0000","last_became_peered":"2024-09-16T10:20:15.854970+0000","last_unstale":"2024-09-16T10:20:55.848567+0000","last_undegraded":"2024-09-16T10:20:55.848567+0000","last_fullsized":"2024-09-16T10:20:55.848567+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:01.210031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":396,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842095+0000","last_change":"2024-09-16T10:19:54.487297+0000","last_active":"2024-09-16T10:20:55.842095+0000","last_peered":"2024-09-16T10:20:55.842095+0000","last_clean":"2024-09-16T10:20:55.842095+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:20:55.842095+0000","last_undegraded":"2024-09-16T10:20:55.842095+0000","last_fullsized":"2024-09-16T10:20:55.842095+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:54.487219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.487219+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.245307+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845971+0000","last_change":"2024-09-16T10:18:52.836576+0000","last_active":"2024-09-16T10:20:55.845971+0000","last_peered":"2024-09-16T10:20:55.845971+0000","last_clean":"2024-09-16T10:20:55.845971+0000","last_became_active":"2024-09-16T10:18:52.836260+0000","last_became_peered":"2024-09-16T10:18:52.836260+0000","last_unstale":"2024-09-16T10:20:55.845971+0000","last_undegraded":"2024-09-16T10:20:55.845971+0000","last_fullsized":"2024-09-16T10:20:55.845971+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.187181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.187181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:42.402210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":457,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846542+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:20:55.846542+0000","last_peered":"2024-09-16T10:20:55.846542+0000","last_clean":"2024-09-16T10:20:55.846542+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:20:55.846542+0000","last_undegraded":"2024-09-16T10:20:55.846542+0000","last_fullsized":"2024-09-16T10:20:55.846542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.577364+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":371,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845850+0000","last_change":"2024-09-16T10:19:50.044357+0000","last_active":"2024-09-16T10:20:55.845850+0000","last_peered":"2024-09-16T10:20:55.845850+0000","last_clean":"2024-09-16T10:20:55.845850+0000","last_became_active":"2024-09-16T10:18:52.842382+0000","last_became_peered":"2024-09-16T10:18:52.842382+0000","last_unstale":"2024-09-16T10:20:55.845850+0000","last_undegraded":"2024-09-16T10:20:55.845850+0000","last_fullsized":"2024-09-16T10:20:55.845850+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.044297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.044297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"228'12056","reported_seq":15605,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.568057+0000","last_change":"2024-09-16T10:20:24.650095+0000","last_active":"2024-09-16T10:20:58.568057+0000","last_peered":"2024-09-16T10:20:58.568057+0000","last_clean":"2024-09-16T10:20:58.568057+0000","last_became_active":"2024-09-16T10:19:14.292726+0000","last_became_peered":"2024-09-16T10:19:14.292726+0000","last_unstale":"2024-09-16T10:20:58.568057+0000","last_undegraded":"2024-09-16T10:20:58.568057+0000","last_fullsized":"2024-09-16T10:20:58.568057+0000","mapping_epoch":187,"log_start":"209'7973","ondisk_log_start":"209'7973","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'9052","last_scrub_stamp":"2024-09-16T10:20:24.649693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.649693+0000","objects_scrubbed":6970,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.170127+0000","scrub_duration":2436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253100032,"num_objects":3862,"num_object_clones":0,"num_object_copies":11586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12056,"num_write_kb":509376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":108724224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":256,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848385+0000","last_change":"2024-09-16T10:20:19.537775+0000","last_active":"2024-09-16T10:20:55.848385+0000","last_peered":"2024-09-16T10:20:55.848385+0000","last_clean":"2024-09-16T10:20:55.848385+0000","last_became_active":"2024-09-16T10:20:15.856484+0000","last_became_peered":"2024-09-16T10:20:15.856484+0000","last_unstale":"2024-09-16T10:20:55.848385+0000","last_undegraded":"2024-09-16T10:20:55.848385+0000","last_fullsized":"2024-09-16T10:20:55.848385+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.537727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.537727+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.959881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":408,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846650+0000","last_change":"2024-09-16T10:19:22.293172+0000","last_active":"2024-09-16T10:20:55.846650+0000","last_peered":"2024-09-16T10:20:55.846650+0000","last_clean":"2024-09-16T10:20:55.846650+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:20:55.846650+0000","last_undegraded":"2024-09-16T10:20:55.846650+0000","last_fullsized":"2024-09-16T10:20:55.846650+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:19:22.293076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.293076+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":384,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.840891+0000","last_change":"2024-09-16T10:20:31.548535+0000","last_active":"2024-09-16T10:20:55.840891+0000","last_peered":"2024-09-16T10:20:55.840891+0000","last_clean":"2024-09-16T10:20:55.840891+0000","last_became_active":"2024-09-16T10:18:52.832749+0000","last_became_peered":"2024-09-16T10:18:52.832749+0000","last_unstale":"2024-09-16T10:20:55.840891+0000","last_undegraded":"2024-09-16T10:20:55.840891+0000","last_fullsized":"2024-09-16T10:20:55.840891+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:31.548465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.548465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.221683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848226+0000","last_change":"2024-09-16T10:19:55.433215+0000","last_active":"2024-09-16T10:20:55.848226+0000","last_peered":"2024-09-16T10:20:55.848226+0000","last_clean":"2024-09-16T10:20:55.848226+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:20:55.848226+0000","last_undegraded":"2024-09-16T10:20:55.848226+0000","last_fullsized":"2024-09-16T10:20:55.848226+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:55.433149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.433149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.272672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":364,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846281+0000","last_change":"2024-09-16T10:18:52.827865+0000","last_active":"2024-09-16T10:20:55.846281+0000","last_peered":"2024-09-16T10:20:55.846281+0000","last_clean":"2024-09-16T10:20:55.846281+0000","last_became_active":"2024-09-16T10:18:52.827511+0000","last_became_peered":"2024-09-16T10:18:52.827511+0000","last_unstale":"2024-09-16T10:20:55.846281+0000","last_undegraded":"2024-09-16T10:20:55.846281+0000","last_fullsized":"2024-09-16T10:20:55.846281+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.486130+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"228'12313","reported_seq":12504,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.307899+0000","last_change":"2024-09-16T10:20:54.830991+0000","last_active":"2024-09-16T10:20:58.307899+0000","last_peered":"2024-09-16T10:20:58.307899+0000","last_clean":"2024-09-16T10:20:58.307899+0000","last_became_active":"2024-09-16T10:18:33.893254+0000","last_became_peered":"2024-09-16T10:18:33.893254+0000","last_unstale":"2024-09-16T10:20:58.307899+0000","last_undegraded":"2024-09-16T10:20:58.307899+0000","last_fullsized":"2024-09-16T10:20:58.307899+0000","mapping_epoch":165,"log_start":"209'8185","ondisk_log_start":"209'8185","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11985","last_scrub_stamp":"2024-09-16T10:20:54.830858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.830858+0000","objects_scrubbed":4288,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.069310+0000","scrub_duration":1423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255918080,"num_objects":3905,"num_object_clones":0,"num_object_copies":11715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12313,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.849024+0000","last_change":"2024-09-16T10:20:48.395772+0000","last_active":"2024-09-16T10:20:55.849024+0000","last_peered":"2024-09-16T10:20:55.849024+0000","last_clean":"2024-09-16T10:20:55.849024+0000","last_became_active":"2024-09-16T10:20:15.850975+0000","last_became_peered":"2024-09-16T10:20:15.850975+0000","last_unstale":"2024-09-16T10:20:55.849024+0000","last_undegraded":"2024-09-16T10:20:55.849024+0000","last_fullsized":"2024-09-16T10:20:55.849024+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:49.930446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"175'29","reported_seq":547,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.840774+0000","last_change":"2024-09-16T10:18:52.860272+0000","last_active":"2024-09-16T10:20:55.840774+0000","last_peered":"2024-09-16T10:20:55.840774+0000","last_clean":"2024-09-16T10:20:55.840774+0000","last_became_active":"2024-09-16T10:18:52.841202+0000","last_became_peered":"2024-09-16T10:18:52.841202+0000","last_unstale":"2024-09-16T10:20:55.840774+0000","last_undegraded":"2024-09-16T10:20:55.840774+0000","last_fullsized":"2024-09-16T10:20:55.840774+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:18:45.654714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.654714+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:01.512318+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:56.965996+0000","last_change":"2024-09-16T10:20:56.965996+0000","last_active":"2024-09-16T10:20:56.965996+0000","last_peered":"2024-09-16T10:20:56.965996+0000","last_clean":"2024-09-16T10:20:56.965996+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:20:56.965996+0000","last_undegraded":"2024-09-16T10:20:56.965996+0000","last_fullsized":"2024-09-16T10:20:56.965996+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:56.965933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:56.965933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.271911+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":379,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846576+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:20:55.846576+0000","last_peered":"2024-09-16T10:20:55.846576+0000","last_clean":"2024-09-16T10:20:55.846576+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:20:55.846576+0000","last_undegraded":"2024-09-16T10:20:55.846576+0000","last_fullsized":"2024-09-16T10:20:55.846576+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.398498+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":360,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847391+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:20:55.847391+0000","last_peered":"2024-09-16T10:20:55.847391+0000","last_clean":"2024-09-16T10:20:55.847391+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:20:55.847391+0000","last_undegraded":"2024-09-16T10:20:55.847391+0000","last_fullsized":"2024-09-16T10:20:55.847391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:29.172677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":373,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844457+0000","last_change":"2024-09-16T10:20:28.240790+0000","last_active":"2024-09-16T10:20:55.844457+0000","last_peered":"2024-09-16T10:20:55.844457+0000","last_clean":"2024-09-16T10:20:55.844457+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:20:55.844457+0000","last_undegraded":"2024-09-16T10:20:55.844457+0000","last_fullsized":"2024-09-16T10:20:55.844457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:28.240558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:28.240558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:54.592999+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"228'12147","reported_seq":12366,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.557642+0000","last_change":"2024-09-16T10:20:22.211158+0000","last_active":"2024-09-16T10:20:58.557642+0000","last_peered":"2024-09-16T10:20:58.557642+0000","last_clean":"2024-09-16T10:20:58.557642+0000","last_became_active":"2024-09-16T10:18:33.820857+0000","last_became_peered":"2024-09-16T10:18:33.820857+0000","last_unstale":"2024-09-16T10:20:58.557642+0000","last_undegraded":"2024-09-16T10:20:58.557642+0000","last_fullsized":"2024-09-16T10:20:58.557642+0000","mapping_epoch":165,"log_start":"209'8059","ondisk_log_start":"209'8059","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'8979","last_scrub_stamp":"2024-09-16T10:20:22.210682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:22.210682+0000","objects_scrubbed":7115,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:32.400224+0000","scrub_duration":2075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252772352,"num_objects":3857,"num_object_clones":0,"num_object_copies":11571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12147,"num_write_kb":512128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":461,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852072+0000","last_change":"2024-09-16T10:20:49.064455+0000","last_active":"2024-09-16T10:20:55.852072+0000","last_peered":"2024-09-16T10:20:55.852072+0000","last_clean":"2024-09-16T10:20:55.852072+0000","last_became_active":"2024-09-16T10:20:15.845263+0000","last_became_peered":"2024-09-16T10:20:15.845263+0000","last_unstale":"2024-09-16T10:20:55.852072+0000","last_undegraded":"2024-09-16T10:20:55.852072+0000","last_fullsized":"2024-09-16T10:20:55.852072+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.064404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.064404+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:53.238156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":384,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841105+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:20:55.841105+0000","last_peered":"2024-09-16T10:20:55.841105+0000","last_clean":"2024-09-16T10:20:55.841105+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:20:55.841105+0000","last_undegraded":"2024-09-16T10:20:55.841105+0000","last_fullsized":"2024-09-16T10:20:55.841105+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.411556+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842584+0000","last_change":"2024-09-16T10:20:55.257286+0000","last_active":"2024-09-16T10:20:55.842584+0000","last_peered":"2024-09-16T10:20:55.842584+0000","last_clean":"2024-09-16T10:20:55.842584+0000","last_became_active":"2024-09-16T10:18:52.833059+0000","last_became_peered":"2024-09-16T10:18:52.833059+0000","last_unstale":"2024-09-16T10:20:55.842584+0000","last_undegraded":"2024-09-16T10:20:55.842584+0000","last_fullsized":"2024-09-16T10:20:55.842584+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:56.920867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":144,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844617+0000","last_change":"2024-09-16T10:19:58.199140+0000","last_active":"2024-09-16T10:20:55.844617+0000","last_peered":"2024-09-16T10:20:55.844617+0000","last_clean":"2024-09-16T10:20:55.844617+0000","last_became_active":"2024-09-16T10:18:52.836346+0000","last_became_peered":"2024-09-16T10:18:52.836346+0000","last_unstale":"2024-09-16T10:20:55.844617+0000","last_undegraded":"2024-09-16T10:20:55.844617+0000","last_fullsized":"2024-09-16T10:20:55.844617+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:21.557184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":127,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846136+0000","last_change":"2024-09-16T10:19:16.940617+0000","last_active":"2024-09-16T10:20:55.846136+0000","last_peered":"2024-09-16T10:20:55.846136+0000","last_clean":"2024-09-16T10:20:55.846136+0000","last_became_active":"2024-09-16T10:18:52.827678+0000","last_became_peered":"2024-09-16T10:18:52.827678+0000","last_unstale":"2024-09-16T10:20:55.846136+0000","last_undegraded":"2024-09-16T10:20:55.846136+0000","last_fullsized":"2024-09-16T10:20:55.846136+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.940551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.940551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":373,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851905+0000","last_change":"2024-09-16T10:20:52.145844+0000","last_active":"2024-09-16T10:20:55.851905+0000","last_peered":"2024-09-16T10:20:55.851905+0000","last_clean":"2024-09-16T10:20:55.851905+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:20:55.851905+0000","last_undegraded":"2024-09-16T10:20:55.851905+0000","last_fullsized":"2024-09-16T10:20:55.851905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:52.145765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.145765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:52.409028+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"228'12150","reported_seq":15838,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.467079+0000","last_change":"2024-09-16T10:19:57.299979+0000","last_active":"2024-09-16T10:20:58.467079+0000","last_peered":"2024-09-16T10:20:58.467079+0000","last_clean":"2024-09-16T10:20:58.467079+0000","last_became_active":"2024-09-16T10:19:13.300503+0000","last_became_peered":"2024-09-16T10:19:13.300503+0000","last_unstale":"2024-09-16T10:20:58.467079+0000","last_undegraded":"2024-09-16T10:20:58.467079+0000","last_fullsized":"2024-09-16T10:20:58.467079+0000","mapping_epoch":186,"log_start":"209'8048","ondisk_log_start":"209'8048","created":165,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6795","last_scrub_stamp":"2024-09-16T10:19:57.299850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.299850+0000","objects_scrubbed":6727,"log_size":4102,"log_dups_size":0,"ondisk_log_size":4102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:25.776420+0000","scrub_duration":1826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251133952,"num_objects":3832,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":511424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1730,"num_bytes_recovered":113377280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":186,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842509+0000","last_change":"2024-09-16T10:20:49.328913+0000","last_active":"2024-09-16T10:20:55.842509+0000","last_peered":"2024-09-16T10:20:55.842509+0000","last_clean":"2024-09-16T10:20:55.842509+0000","last_became_active":"2024-09-16T10:20:15.858298+0000","last_became_peered":"2024-09-16T10:20:15.858298+0000","last_unstale":"2024-09-16T10:20:55.842509+0000","last_undegraded":"2024-09-16T10:20:55.842509+0000","last_fullsized":"2024-09-16T10:20:55.842509+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:56.845624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":427,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847644+0000","last_change":"2024-09-16T10:20:55.304934+0000","last_active":"2024-09-16T10:20:55.847644+0000","last_peered":"2024-09-16T10:20:55.847644+0000","last_clean":"2024-09-16T10:20:55.847644+0000","last_became_active":"2024-09-16T10:18:52.837634+0000","last_became_peered":"2024-09-16T10:18:52.837634+0000","last_unstale":"2024-09-16T10:20:55.847644+0000","last_undegraded":"2024-09-16T10:20:55.847644+0000","last_fullsized":"2024-09-16T10:20:55.847644+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:20:55.304837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.304837+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.147025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":392,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.849028+0000","last_change":"2024-09-16T10:20:52.288156+0000","last_active":"2024-09-16T10:20:55.849028+0000","last_peered":"2024-09-16T10:20:55.849028+0000","last_clean":"2024-09-16T10:20:55.849028+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:20:55.849028+0000","last_undegraded":"2024-09-16T10:20:55.849028+0000","last_fullsized":"2024-09-16T10:20:55.849028+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:52.288090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.288090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:10.544341+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":823,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:56.622686+0000","last_change":"2024-09-16T10:20:33.058993+0000","last_active":"2024-09-16T10:20:56.622686+0000","last_peered":"2024-09-16T10:20:56.622686+0000","last_clean":"2024-09-16T10:20:56.622686+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:20:56.622686+0000","last_undegraded":"2024-09-16T10:20:56.622686+0000","last_fullsized":"2024-09-16T10:20:56.622686+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:20:33.058923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.058923+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.147877+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":435,"num_read_kb":430,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845821+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:20:55.845821+0000","last_peered":"2024-09-16T10:20:55.845821+0000","last_clean":"2024-09-16T10:20:55.845821+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:20:55.845821+0000","last_undegraded":"2024-09-16T10:20:55.845821+0000","last_fullsized":"2024-09-16T10:20:55.845821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:56.497294+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":373,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.840390+0000","last_change":"2024-09-16T10:20:25.566580+0000","last_active":"2024-09-16T10:20:55.840390+0000","last_peered":"2024-09-16T10:20:55.840390+0000","last_clean":"2024-09-16T10:20:55.840390+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:20:55.840390+0000","last_undegraded":"2024-09-16T10:20:55.840390+0000","last_fullsized":"2024-09-16T10:20:55.840390+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:25.566513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:25.566513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:53.062951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"228'11751","reported_seq":11962,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.199923+0000","last_change":"2024-09-16T10:19:56.367287+0000","last_active":"2024-09-16T10:20:58.199923+0000","last_peered":"2024-09-16T10:20:58.199923+0000","last_clean":"2024-09-16T10:20:58.199923+0000","last_became_active":"2024-09-16T10:18:33.825120+0000","last_became_peered":"2024-09-16T10:18:33.825120+0000","last_unstale":"2024-09-16T10:20:58.199923+0000","last_undegraded":"2024-09-16T10:20:58.199923+0000","last_fullsized":"2024-09-16T10:20:58.199923+0000","mapping_epoch":165,"log_start":"205'7361","ondisk_log_start":"205'7361","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6534","last_scrub_stamp":"2024-09-16T10:19:56.367108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.367108+0000","objects_scrubbed":6469,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:05.193357+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249626624,"num_objects":3809,"num_object_clones":0,"num_object_copies":11427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11751,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":475,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848567+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:20:55.848567+0000","last_peered":"2024-09-16T10:20:55.848567+0000","last_clean":"2024-09-16T10:20:55.848567+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:20:55.848567+0000","last_undegraded":"2024-09-16T10:20:55.848567+0000","last_fullsized":"2024-09-16T10:20:55.848567+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.998293+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":342,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848108+0000","last_change":"2024-09-16T10:19:44.015480+0000","last_active":"2024-09-16T10:20:55.848108+0000","last_peered":"2024-09-16T10:20:55.848108+0000","last_clean":"2024-09-16T10:20:55.848108+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:20:55.848108+0000","last_undegraded":"2024-09-16T10:20:55.848108+0000","last_fullsized":"2024-09-16T10:20:55.848108+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.015430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.015430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.463886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":441,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851514+0000","last_change":"2024-09-16T10:20:19.229860+0000","last_active":"2024-09-16T10:20:55.851514+0000","last_peered":"2024-09-16T10:20:55.851514+0000","last_clean":"2024-09-16T10:20:55.851514+0000","last_became_active":"2024-09-16T10:20:15.855504+0000","last_became_peered":"2024-09-16T10:20:15.855504+0000","last_unstale":"2024-09-16T10:20:55.851514+0000","last_undegraded":"2024-09-16T10:20:55.851514+0000","last_fullsized":"2024-09-16T10:20:55.851514+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.229771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.229771+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.171723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852804+0000","last_change":"2024-09-16T10:18:52.827862+0000","last_active":"2024-09-16T10:20:55.852804+0000","last_peered":"2024-09-16T10:20:55.852804+0000","last_clean":"2024-09-16T10:20:55.852804+0000","last_became_active":"2024-09-16T10:18:52.827467+0000","last_became_peered":"2024-09-16T10:18:52.827467+0000","last_unstale":"2024-09-16T10:20:55.852804+0000","last_undegraded":"2024-09-16T10:20:55.852804+0000","last_fullsized":"2024-09-16T10:20:55.852804+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.015337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":357,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845619+0000","last_change":"2024-09-16T10:19:23.923755+0000","last_active":"2024-09-16T10:20:55.845619+0000","last_peered":"2024-09-16T10:20:55.845619+0000","last_clean":"2024-09-16T10:20:55.845619+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:20:55.845619+0000","last_undegraded":"2024-09-16T10:20:55.845619+0000","last_fullsized":"2024-09-16T10:20:55.845619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.923690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.923690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.471730+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":189,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841940+0000","last_change":"2024-09-16T10:20:19.546115+0000","last_active":"2024-09-16T10:20:55.841940+0000","last_peered":"2024-09-16T10:20:55.841940+0000","last_clean":"2024-09-16T10:20:55.841940+0000","last_became_active":"2024-09-16T10:20:18.869939+0000","last_became_peered":"2024-09-16T10:20:18.869939+0000","last_unstale":"2024-09-16T10:20:55.841940+0000","last_undegraded":"2024-09-16T10:20:55.841940+0000","last_fullsized":"2024-09-16T10:20:55.841940+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.546053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.546053+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.095718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"228'11965","reported_seq":12104,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.112748+0000","last_change":"2024-09-16T10:18:33.825497+0000","last_active":"2024-09-16T10:20:58.112748+0000","last_peered":"2024-09-16T10:20:58.112748+0000","last_clean":"2024-09-16T10:20:58.112748+0000","last_became_active":"2024-09-16T10:18:33.824979+0000","last_became_peered":"2024-09-16T10:18:33.824979+0000","last_unstale":"2024-09-16T10:20:58.112748+0000","last_undegraded":"2024-09-16T10:20:58.112748+0000","last_fullsized":"2024-09-16T10:20:58.112748+0000","mapping_epoch":165,"log_start":"208'7878","ondisk_log_start":"208'7878","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256180224,"num_objects":3909,"num_object_clones":0,"num_object_copies":11727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11965,"num_write_kb":507968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":410,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.843036+0000","last_change":"2024-09-16T10:20:31.013510+0000","last_active":"2024-09-16T10:20:55.843036+0000","last_peered":"2024-09-16T10:20:55.843036+0000","last_clean":"2024-09-16T10:20:55.843036+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:20:55.843036+0000","last_undegraded":"2024-09-16T10:20:55.843036+0000","last_fullsized":"2024-09-16T10:20:55.843036+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:20:31.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.013316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:48.982676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":375,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842048+0000","last_change":"2024-09-16T10:20:33.973275+0000","last_active":"2024-09-16T10:20:55.842048+0000","last_peered":"2024-09-16T10:20:55.842048+0000","last_clean":"2024-09-16T10:20:55.842048+0000","last_became_active":"2024-09-16T10:18:52.835626+0000","last_became_peered":"2024-09-16T10:18:52.835626+0000","last_unstale":"2024-09-16T10:20:55.842048+0000","last_undegraded":"2024-09-16T10:20:55.842048+0000","last_fullsized":"2024-09-16T10:20:55.842048+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.973201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.973201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:42.032781+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845488+0000","last_change":"2024-09-16T10:20:55.449343+0000","last_active":"2024-09-16T10:20:55.845488+0000","last_peered":"2024-09-16T10:20:55.845488+0000","last_clean":"2024-09-16T10:20:55.845488+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:20:55.845488+0000","last_undegraded":"2024-09-16T10:20:55.845488+0000","last_fullsized":"2024-09-16T10:20:55.845488+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.449274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.449274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:10.343283+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":322,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848834+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:20:55.848834+0000","last_peered":"2024-09-16T10:20:55.848834+0000","last_clean":"2024-09-16T10:20:55.848834+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:20:55.848834+0000","last_undegraded":"2024-09-16T10:20:55.848834+0000","last_fullsized":"2024-09-16T10:20:55.848834+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.607172+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"228'12040","reported_seq":15631,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.432637+0000","last_change":"2024-09-16T10:19:48.403185+0000","last_active":"2024-09-16T10:20:58.432637+0000","last_peered":"2024-09-16T10:20:58.432637+0000","last_clean":"2024-09-16T10:20:58.432637+0000","last_became_active":"2024-09-16T10:19:15.290152+0000","last_became_peered":"2024-09-16T10:19:15.290152+0000","last_unstale":"2024-09-16T10:20:58.432637+0000","last_undegraded":"2024-09-16T10:20:58.432637+0000","last_fullsized":"2024-09-16T10:20:58.432637+0000","mapping_epoch":188,"log_start":"209'7952","ondisk_log_start":"209'7952","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'6091","last_scrub_stamp":"2024-09-16T10:19:48.403076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.403076+0000","objects_scrubbed":6009,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:05.609265+0000","scrub_duration":1968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249036800,"num_objects":3800,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12040,"num_write_kb":506880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":110428160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847181+0000","last_change":"2024-09-16T10:20:27.405769+0000","last_active":"2024-09-16T10:20:55.847181+0000","last_peered":"2024-09-16T10:20:55.847181+0000","last_clean":"2024-09-16T10:20:55.847181+0000","last_became_active":"2024-09-16T10:18:52.842646+0000","last_became_peered":"2024-09-16T10:18:52.842646+0000","last_unstale":"2024-09-16T10:20:55.847181+0000","last_undegraded":"2024-09-16T10:20:55.847181+0000","last_fullsized":"2024-09-16T10:20:55.847181+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:27.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:27.405599+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.546066+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":417,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.843918+0000","last_change":"2024-09-16T10:20:22.959397+0000","last_active":"2024-09-16T10:20:55.843918+0000","last_peered":"2024-09-16T10:20:55.843918+0000","last_clean":"2024-09-16T10:20:55.843918+0000","last_became_active":"2024-09-16T10:18:52.826328+0000","last_became_peered":"2024-09-16T10:18:52.826328+0000","last_unstale":"2024-09-16T10:20:55.843918+0000","last_undegraded":"2024-09-16T10:20:55.843918+0000","last_fullsized":"2024-09-16T10:20:55.843918+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:20:22.959311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:22.959311+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.280982+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848663+0000","last_change":"2024-09-16T10:19:21.116145+0000","last_active":"2024-09-16T10:20:55.848663+0000","last_peered":"2024-09-16T10:20:55.848663+0000","last_clean":"2024-09-16T10:20:55.848663+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:20:55.848663+0000","last_undegraded":"2024-09-16T10:20:55.848663+0000","last_fullsized":"2024-09-16T10:20:55.848663+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:21.116098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:21.116098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.615403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":361,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.840493+0000","last_change":"2024-09-16T10:20:29.600095+0000","last_active":"2024-09-16T10:20:55.840493+0000","last_peered":"2024-09-16T10:20:55.840493+0000","last_clean":"2024-09-16T10:20:55.840493+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:20:55.840493+0000","last_undegraded":"2024-09-16T10:20:55.840493+0000","last_fullsized":"2024-09-16T10:20:55.840493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:29.600025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:29.600025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:51.499818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":120,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844854+0000","last_change":"2024-09-16T10:18:52.838759+0000","last_active":"2024-09-16T10:20:55.844854+0000","last_peered":"2024-09-16T10:20:55.844854+0000","last_clean":"2024-09-16T10:20:55.844854+0000","last_became_active":"2024-09-16T10:18:52.838070+0000","last_became_peered":"2024-09-16T10:18:52.838070+0000","last_unstale":"2024-09-16T10:20:55.844854+0000","last_undegraded":"2024-09-16T10:20:55.844854+0000","last_fullsized":"2024-09-16T10:20:55.844854+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:50.491234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.491234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.383400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"228'12477","reported_seq":12616,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.584626+0000","last_change":"2024-09-16T10:18:33.823010+0000","last_active":"2024-09-16T10:20:58.584626+0000","last_peered":"2024-09-16T10:20:58.584626+0000","last_clean":"2024-09-16T10:20:58.584626+0000","last_became_active":"2024-09-16T10:18:33.822701+0000","last_became_peered":"2024-09-16T10:18:33.822701+0000","last_unstale":"2024-09-16T10:20:58.584626+0000","last_undegraded":"2024-09-16T10:20:58.584626+0000","last_fullsized":"2024-09-16T10:20:58.584626+0000","mapping_epoch":165,"log_start":"210'8379","ondisk_log_start":"210'8379","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.329041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256049152,"num_objects":3907,"num_object_clones":0,"num_object_copies":11721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":448,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.850754+0000","last_change":"2024-09-16T10:19:55.550754+0000","last_active":"2024-09-16T10:20:55.850754+0000","last_peered":"2024-09-16T10:20:55.850754+0000","last_clean":"2024-09-16T10:20:55.850754+0000","last_became_active":"2024-09-16T10:18:52.838011+0000","last_became_peered":"2024-09-16T10:18:52.838011+0000","last_unstale":"2024-09-16T10:20:55.850754+0000","last_undegraded":"2024-09-16T10:20:55.850754+0000","last_fullsized":"2024-09-16T10:20:55.850754+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:19:55.550670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.550670+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:19.176131+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":371,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842739+0000","last_change":"2024-09-16T10:19:16.768415+0000","last_active":"2024-09-16T10:20:55.842739+0000","last_peered":"2024-09-16T10:20:55.842739+0000","last_clean":"2024-09-16T10:20:55.842739+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:20:55.842739+0000","last_undegraded":"2024-09-16T10:20:55.842739+0000","last_fullsized":"2024-09-16T10:20:55.842739+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:16.768062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.768062+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":361,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846445+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:20:55.846445+0000","last_peered":"2024-09-16T10:20:55.846445+0000","last_clean":"2024-09-16T10:20:55.846445+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:20:55.846445+0000","last_undegraded":"2024-09-16T10:20:55.846445+0000","last_fullsized":"2024-09-16T10:20:55.846445+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"175'9","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852043+0000","last_change":"2024-09-16T10:20:55.177381+0000","last_active":"2024-09-16T10:20:55.852043+0000","last_peered":"2024-09-16T10:20:55.852043+0000","last_clean":"2024-09-16T10:20:55.852043+0000","last_became_active":"2024-09-16T10:18:52.825124+0000","last_became_peered":"2024-09-16T10:18:52.825124+0000","last_unstale":"2024-09-16T10:20:55.852043+0000","last_undegraded":"2024-09-16T10:20:55.852043+0000","last_fullsized":"2024-09-16T10:20:55.852043+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:20:55.177089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.177089+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:06.263297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":371,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842362+0000","last_change":"2024-09-16T10:20:51.873893+0000","last_active":"2024-09-16T10:20:55.842362+0000","last_peered":"2024-09-16T10:20:55.842362+0000","last_clean":"2024-09-16T10:20:55.842362+0000","last_became_active":"2024-09-16T10:18:52.834312+0000","last_became_peered":"2024-09-16T10:18:52.834312+0000","last_unstale":"2024-09-16T10:20:55.842362+0000","last_undegraded":"2024-09-16T10:20:55.842362+0000","last_fullsized":"2024-09-16T10:20:55.842362+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.873833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.873833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:14.931229+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":46,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842143+0000","last_change":"2024-09-16T10:20:18.941681+0000","last_active":"2024-09-16T10:20:55.842143+0000","last_peered":"2024-09-16T10:20:55.842143+0000","last_clean":"2024-09-16T10:20:55.842143+0000","last_became_active":"2024-09-16T10:20:18.867971+0000","last_became_peered":"2024-09-16T10:20:18.867971+0000","last_unstale":"2024-09-16T10:20:55.842143+0000","last_undegraded":"2024-09-16T10:20:55.842143+0000","last_fullsized":"2024-09-16T10:20:55.842143+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:18.941543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:18.941543+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.174342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"228'11986","reported_seq":12142,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.461033+0000","last_change":"2024-09-16T10:19:59.077503+0000","last_active":"2024-09-16T10:20:58.461033+0000","last_peered":"2024-09-16T10:20:58.461033+0000","last_clean":"2024-09-16T10:20:58.461033+0000","last_became_active":"2024-09-16T10:18:33.823198+0000","last_became_peered":"2024-09-16T10:18:33.823198+0000","last_unstale":"2024-09-16T10:20:58.461033+0000","last_undegraded":"2024-09-16T10:20:58.461033+0000","last_fullsized":"2024-09-16T10:20:58.461033+0000","mapping_epoch":165,"log_start":"209'7874","ondisk_log_start":"209'7874","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":1399,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.319001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253362176,"num_objects":3866,"num_object_clones":0,"num_object_copies":11598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11986,"num_write_kb":507264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":380,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846592+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:20:55.846592+0000","last_peered":"2024-09-16T10:20:55.846592+0000","last_clean":"2024-09-16T10:20:55.846592+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:20:55.846592+0000","last_undegraded":"2024-09-16T10:20:55.846592+0000","last_fullsized":"2024-09-16T10:20:55.846592+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.233999+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":377,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846251+0000","last_change":"2024-09-16T10:20:28.364361+0000","last_active":"2024-09-16T10:20:55.846251+0000","last_peered":"2024-09-16T10:20:55.846251+0000","last_clean":"2024-09-16T10:20:55.846251+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:20:55.846251+0000","last_undegraded":"2024-09-16T10:20:55.846251+0000","last_fullsized":"2024-09-16T10:20:55.846251+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:28.364296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:28.364296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.891566+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":127,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842804+0000","last_change":"2024-09-16T10:19:18.748438+0000","last_active":"2024-09-16T10:20:55.842804+0000","last_peered":"2024-09-16T10:20:55.842804+0000","last_clean":"2024-09-16T10:20:55.842804+0000","last_became_active":"2024-09-16T10:18:52.835884+0000","last_became_peered":"2024-09-16T10:18:52.835884+0000","last_unstale":"2024-09-16T10:20:55.842804+0000","last_undegraded":"2024-09-16T10:20:55.842804+0000","last_fullsized":"2024-09-16T10:20:55.842804+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":390,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848932+0000","last_change":"2024-09-16T10:20:31.889357+0000","last_active":"2024-09-16T10:20:55.848932+0000","last_peered":"2024-09-16T10:20:55.848932+0000","last_clean":"2024-09-16T10:20:55.848932+0000","last_became_active":"2024-09-16T10:20:31.869705+0000","last_became_peered":"2024-09-16T10:20:31.869705+0000","last_unstale":"2024-09-16T10:20:55.848932+0000","last_undegraded":"2024-09-16T10:20:55.848932+0000","last_fullsized":"2024-09-16T10:20:55.848932+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:37.267422+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":264,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.849873+0000","last_change":"2024-09-16T10:20:20.538346+0000","last_active":"2024-09-16T10:20:55.849873+0000","last_peered":"2024-09-16T10:20:55.849873+0000","last_clean":"2024-09-16T10:20:55.849873+0000","last_became_active":"2024-09-16T10:20:18.870789+0000","last_became_peered":"2024-09-16T10:20:18.870789+0000","last_unstale":"2024-09-16T10:20:55.849873+0000","last_undegraded":"2024-09-16T10:20:55.849873+0000","last_fullsized":"2024-09-16T10:20:55.849873+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.538076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.538076+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.192429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.6","version":"228'12012","reported_seq":12151,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.620195+0000","last_change":"2024-09-16T10:18:33.894422+0000","last_active":"2024-09-16T10:20:58.620195+0000","last_peered":"2024-09-16T10:20:58.620195+0000","last_clean":"2024-09-16T10:20:58.620195+0000","last_became_active":"2024-09-16T10:18:33.893961+0000","last_became_peered":"2024-09-16T10:18:33.893961+0000","last_unstale":"2024-09-16T10:20:58.620195+0000","last_undegraded":"2024-09-16T10:20:58.620195+0000","last_fullsized":"2024-09-16T10:20:58.620195+0000","mapping_epoch":165,"log_start":"206'7594","ondisk_log_start":"206'7594","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.292676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255983616,"num_objects":3906,"num_object_clones":0,"num_object_copies":11718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":509376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":411,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848319+0000","last_change":"2024-09-16T10:20:51.673782+0000","last_active":"2024-09-16T10:20:55.848319+0000","last_peered":"2024-09-16T10:20:55.848319+0000","last_clean":"2024-09-16T10:20:55.848319+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:20:55.848319+0000","last_undegraded":"2024-09-16T10:20:55.848319+0000","last_fullsized":"2024-09-16T10:20:55.848319+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:20:51.673686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.673686+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:16.084083+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852530+0000","last_change":"2024-09-16T10:20:31.091077+0000","last_active":"2024-09-16T10:20:55.852530+0000","last_peered":"2024-09-16T10:20:55.852530+0000","last_clean":"2024-09-16T10:20:55.852530+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:20:55.852530+0000","last_undegraded":"2024-09-16T10:20:55.852530+0000","last_fullsized":"2024-09-16T10:20:55.852530+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:31.090914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.090914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:49.345216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":135,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846414+0000","last_change":"2024-09-16T10:20:54.330571+0000","last_active":"2024-09-16T10:20:55.846414+0000","last_peered":"2024-09-16T10:20:55.846414+0000","last_clean":"2024-09-16T10:20:55.846414+0000","last_became_active":"2024-09-16T10:18:52.831796+0000","last_became_peered":"2024-09-16T10:18:52.831796+0000","last_unstale":"2024-09-16T10:20:55.846414+0000","last_undegraded":"2024-09-16T10:20:55.846414+0000","last_fullsized":"2024-09-16T10:20:55.846414+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.330503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.330503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:18.648114+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":372,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851418+0000","last_change":"2024-09-16T10:20:51.105297+0000","last_active":"2024-09-16T10:20:55.851418+0000","last_peered":"2024-09-16T10:20:55.851418+0000","last_clean":"2024-09-16T10:20:55.851418+0000","last_became_active":"2024-09-16T10:18:52.840958+0000","last_became_peered":"2024-09-16T10:18:52.840958+0000","last_unstale":"2024-09-16T10:20:55.851418+0000","last_undegraded":"2024-09-16T10:20:55.851418+0000","last_fullsized":"2024-09-16T10:20:55.851418+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.105240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.105240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:12.132076+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":197,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846824+0000","last_change":"2024-09-16T10:20:19.166368+0000","last_active":"2024-09-16T10:20:55.846824+0000","last_peered":"2024-09-16T10:20:55.846824+0000","last_clean":"2024-09-16T10:20:55.846824+0000","last_became_active":"2024-09-16T10:20:18.865956+0000","last_became_peered":"2024-09-16T10:20:18.865956+0000","last_unstale":"2024-09-16T10:20:55.846824+0000","last_undegraded":"2024-09-16T10:20:55.846824+0000","last_fullsized":"2024-09-16T10:20:55.846824+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.166321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.166321+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:21.079811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"228'12078","reported_seq":12310,"reported_epoch":228,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:20:58.250467+0000","last_change":"2024-09-16T10:20:57.699943+0000","last_active":"2024-09-16T10:20:58.250467+0000","last_peered":"2024-09-16T10:20:58.250467+0000","last_clean":"2024-09-16T10:20:58.250467+0000","last_became_active":"2024-09-16T10:18:33.894026+0000","last_became_peered":"2024-09-16T10:18:33.894026+0000","last_unstale":"2024-09-16T10:20:58.250467+0000","last_undegraded":"2024-09-16T10:20:58.250467+0000","last_fullsized":"2024-09-16T10:20:58.250467+0000","mapping_epoch":165,"log_start":"206'7645","ondisk_log_start":"206'7645","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6662","last_scrub_stamp":"2024-09-16T10:19:55.377245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.377245+0000","objects_scrubbed":1699,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":2000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252837888,"num_objects":3858,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12078,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":387,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841870+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:20:55.841870+0000","last_peered":"2024-09-16T10:20:55.841870+0000","last_clean":"2024-09-16T10:20:55.841870+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:20:55.841870+0000","last_undegraded":"2024-09-16T10:20:55.841870+0000","last_fullsized":"2024-09-16T10:20:55.841870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.841996+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":377,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:57.246050+0000","last_change":"2024-09-16T10:20:57.246050+0000","last_active":"2024-09-16T10:20:57.246050+0000","last_peered":"2024-09-16T10:20:57.246050+0000","last_clean":"2024-09-16T10:20:57.246050+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:20:57.246050+0000","last_undegraded":"2024-09-16T10:20:57.246050+0000","last_fullsized":"2024-09-16T10:20:57.246050+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.245939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.245939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:00.669214+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":119,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845542+0000","last_change":"2024-09-16T10:18:52.830267+0000","last_active":"2024-09-16T10:20:55.845542+0000","last_peered":"2024-09-16T10:20:55.845542+0000","last_clean":"2024-09-16T10:20:55.845542+0000","last_became_active":"2024-09-16T10:18:52.829845+0000","last_became_peered":"2024-09-16T10:18:52.829845+0000","last_unstale":"2024-09-16T10:20:55.845542+0000","last_undegraded":"2024-09-16T10:20:55.845542+0000","last_fullsized":"2024-09-16T10:20:55.845542+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:59.955280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":365,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846095+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:20:55.846095+0000","last_peered":"2024-09-16T10:20:55.846095+0000","last_clean":"2024-09-16T10:20:55.846095+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:20:55.846095+0000","last_undegraded":"2024-09-16T10:20:55.846095+0000","last_fullsized":"2024-09-16T10:20:55.846095+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.174140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":46,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851620+0000","last_change":"2024-09-16T10:20:20.186263+0000","last_active":"2024-09-16T10:20:55.851620+0000","last_peered":"2024-09-16T10:20:55.851620+0000","last_clean":"2024-09-16T10:20:55.851620+0000","last_became_active":"2024-09-16T10:20:18.863901+0000","last_became_peered":"2024-09-16T10:20:18.863901+0000","last_unstale":"2024-09-16T10:20:55.851620+0000","last_undegraded":"2024-09-16T10:20:55.851620+0000","last_fullsized":"2024-09-16T10:20:55.851620+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.186192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.186192+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:23.904161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"228'11705","reported_seq":15197,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.310629+0000","last_change":"2024-09-16T10:20:53.772044+0000","last_active":"2024-09-16T10:20:58.310629+0000","last_peered":"2024-09-16T10:20:58.310629+0000","last_clean":"2024-09-16T10:20:58.310629+0000","last_became_active":"2024-09-16T10:19:14.291692+0000","last_became_peered":"2024-09-16T10:19:14.291692+0000","last_unstale":"2024-09-16T10:20:58.310629+0000","last_undegraded":"2024-09-16T10:20:58.310629+0000","last_fullsized":"2024-09-16T10:20:58.310629+0000","mapping_epoch":187,"log_start":"207'7570","ondisk_log_start":"207'7570","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'11311","last_scrub_stamp":"2024-09-16T10:20:53.771722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:53.771722+0000","objects_scrubbed":4176,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:56.039707+0000","scrub_duration":1405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244121600,"num_objects":3725,"num_object_clones":0,"num_object_copies":11175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11705,"num_write_kb":493760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":374,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847312+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:20:55.847312+0000","last_peered":"2024-09-16T10:20:55.847312+0000","last_clean":"2024-09-16T10:20:55.847312+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:20:55.847312+0000","last_undegraded":"2024-09-16T10:20:55.847312+0000","last_fullsized":"2024-09-16T10:20:55.847312+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.365771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":381,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851843+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:20:55.851843+0000","last_peered":"2024-09-16T10:20:55.851843+0000","last_clean":"2024-09-16T10:20:55.851843+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:20:55.851843+0000","last_undegraded":"2024-09-16T10:20:55.851843+0000","last_fullsized":"2024-09-16T10:20:55.851843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:15.178355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":458,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.850702+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:20:55.850702+0000","last_peered":"2024-09-16T10:20:55.850702+0000","last_clean":"2024-09-16T10:20:55.850702+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:20:55.850702+0000","last_undegraded":"2024-09-16T10:20:55.850702+0000","last_fullsized":"2024-09-16T10:20:55.850702+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.056619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":365,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846115+0000","last_change":"2024-09-16T10:19:20.941840+0000","last_active":"2024-09-16T10:20:55.846115+0000","last_peered":"2024-09-16T10:20:55.846115+0000","last_clean":"2024-09-16T10:20:55.846115+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:20:55.846115+0000","last_undegraded":"2024-09-16T10:20:55.846115+0000","last_fullsized":"2024-09-16T10:20:55.846115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.941753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.941753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":46,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846430+0000","last_change":"2024-09-16T10:20:19.314302+0000","last_active":"2024-09-16T10:20:55.846430+0000","last_peered":"2024-09-16T10:20:55.846430+0000","last_clean":"2024-09-16T10:20:55.846430+0000","last_became_active":"2024-09-16T10:20:18.879077+0000","last_became_peered":"2024-09-16T10:20:18.879077+0000","last_unstale":"2024-09-16T10:20:55.846430+0000","last_undegraded":"2024-09-16T10:20:55.846430+0000","last_fullsized":"2024-09-16T10:20:55.846430+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.314234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.314234+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.829443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847560+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:20:55.847560+0000","last_peered":"2024-09-16T10:20:55.847560+0000","last_clean":"2024-09-16T10:20:55.847560+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:20:55.847560+0000","last_undegraded":"2024-09-16T10:20:55.847560+0000","last_fullsized":"2024-09-16T10:20:55.847560+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.631488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":382,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845625+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:20:55.845625+0000","last_peered":"2024-09-16T10:20:55.845625+0000","last_clean":"2024-09-16T10:20:55.845625+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:20:55.845625+0000","last_undegraded":"2024-09-16T10:20:55.845625+0000","last_fullsized":"2024-09-16T10:20:55.845625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:42.398743+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":361,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846217+0000","last_change":"2024-09-16T10:20:54.453057+0000","last_active":"2024-09-16T10:20:55.846217+0000","last_peered":"2024-09-16T10:20:55.846217+0000","last_clean":"2024-09-16T10:20:55.846217+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:20:55.846217+0000","last_undegraded":"2024-09-16T10:20:55.846217+0000","last_fullsized":"2024-09-16T10:20:55.846217+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.452998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.452998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:01.123347+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":349,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852366+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:20:55.852366+0000","last_peered":"2024-09-16T10:20:55.852366+0000","last_clean":"2024-09-16T10:20:55.852366+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:20:55.852366+0000","last_undegraded":"2024-09-16T10:20:55.852366+0000","last_fullsized":"2024-09-16T10:20:55.852366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:45.047690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":186,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842155+0000","last_change":"2024-09-16T10:20:20.591626+0000","last_active":"2024-09-16T10:20:55.842155+0000","last_peered":"2024-09-16T10:20:55.842155+0000","last_clean":"2024-09-16T10:20:55.842155+0000","last_became_active":"2024-09-16T10:20:18.873396+0000","last_became_peered":"2024-09-16T10:20:18.873396+0000","last_unstale":"2024-09-16T10:20:55.842155+0000","last_undegraded":"2024-09-16T10:20:55.842155+0000","last_fullsized":"2024-09-16T10:20:55.842155+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.591566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.591566+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.585524+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":388,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847863+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:20:55.847863+0000","last_peered":"2024-09-16T10:20:55.847863+0000","last_clean":"2024-09-16T10:20:55.847863+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:20:55.847863+0000","last_undegraded":"2024-09-16T10:20:55.847863+0000","last_fullsized":"2024-09-16T10:20:55.847863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.020779+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.850950+0000","last_change":"2024-09-16T10:19:22.431194+0000","last_active":"2024-09-16T10:20:55.850950+0000","last_peered":"2024-09-16T10:20:55.850950+0000","last_clean":"2024-09-16T10:20:55.850950+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:20:55.850950+0000","last_undegraded":"2024-09-16T10:20:55.850950+0000","last_fullsized":"2024-09-16T10:20:55.850950+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.431126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.431126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":361,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845443+0000","last_change":"2024-09-16T10:20:32.229638+0000","last_active":"2024-09-16T10:20:55.845443+0000","last_peered":"2024-09-16T10:20:55.845443+0000","last_clean":"2024-09-16T10:20:55.845443+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:20:55.845443+0000","last_undegraded":"2024-09-16T10:20:55.845443+0000","last_fullsized":"2024-09-16T10:20:55.845443+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:32.229564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:32.229564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.951807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":128,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842838+0000","last_change":"2024-09-16T10:20:19.951949+0000","last_active":"2024-09-16T10:20:55.842838+0000","last_peered":"2024-09-16T10:20:55.842838+0000","last_clean":"2024-09-16T10:20:55.842838+0000","last_became_active":"2024-09-16T10:18:52.822465+0000","last_became_peered":"2024-09-16T10:18:52.822465+0000","last_unstale":"2024-09-16T10:20:55.842838+0000","last_undegraded":"2024-09-16T10:20:55.842838+0000","last_fullsized":"2024-09-16T10:20:55.842838+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.200411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":46,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846810+0000","last_change":"2024-09-16T10:20:20.322528+0000","last_active":"2024-09-16T10:20:55.846810+0000","last_peered":"2024-09-16T10:20:55.846810+0000","last_clean":"2024-09-16T10:20:55.846810+0000","last_became_active":"2024-09-16T10:20:18.878681+0000","last_became_peered":"2024-09-16T10:20:18.878681+0000","last_unstale":"2024-09-16T10:20:55.846810+0000","last_undegraded":"2024-09-16T10:20:55.846810+0000","last_fullsized":"2024-09-16T10:20:55.846810+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.322463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.322463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:31.968089+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":381,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845850+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:20:55.845850+0000","last_peered":"2024-09-16T10:20:55.845850+0000","last_clean":"2024-09-16T10:20:55.845850+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:20:55.845850+0000","last_undegraded":"2024-09-16T10:20:55.845850+0000","last_fullsized":"2024-09-16T10:20:55.845850+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:39.395073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847397+0000","last_change":"2024-09-16T10:20:29.177634+0000","last_active":"2024-09-16T10:20:55.847397+0000","last_peered":"2024-09-16T10:20:55.847397+0000","last_clean":"2024-09-16T10:20:55.847397+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:20:55.847397+0000","last_undegraded":"2024-09-16T10:20:55.847397+0000","last_fullsized":"2024-09-16T10:20:55.847397+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:29.177563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:29.177563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.824549+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"175'9","reported_seq":485,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847156+0000","last_change":"2024-09-16T10:20:25.284531+0000","last_active":"2024-09-16T10:20:55.847156+0000","last_peered":"2024-09-16T10:20:55.847156+0000","last_clean":"2024-09-16T10:20:55.847156+0000","last_became_active":"2024-09-16T10:18:52.845153+0000","last_became_peered":"2024-09-16T10:18:52.845153+0000","last_unstale":"2024-09-16T10:20:55.847156+0000","last_undegraded":"2024-09-16T10:20:55.847156+0000","last_fullsized":"2024-09-16T10:20:55.847156+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:20:25.284462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:25.284462+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.278781+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":371,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:57.931848+0000","last_change":"2024-09-16T10:20:57.931764+0000","last_active":"2024-09-16T10:20:57.931848+0000","last_peered":"2024-09-16T10:20:57.931848+0000","last_clean":"2024-09-16T10:20:57.931848+0000","last_became_active":"2024-09-16T10:18:52.821050+0000","last_became_peered":"2024-09-16T10:18:52.821050+0000","last_unstale":"2024-09-16T10:20:57.931848+0000","last_undegraded":"2024-09-16T10:20:57.931848+0000","last_fullsized":"2024-09-16T10:20:57.931848+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.931714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.931714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:26.199004+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.849111+0000","last_change":"2024-09-16T10:20:19.113511+0000","last_active":"2024-09-16T10:20:55.849111+0000","last_peered":"2024-09-16T10:20:55.849111+0000","last_clean":"2024-09-16T10:20:55.849111+0000","last_became_active":"2024-09-16T10:20:18.868687+0000","last_became_peered":"2024-09-16T10:20:18.868687+0000","last_unstale":"2024-09-16T10:20:55.849111+0000","last_undegraded":"2024-09-16T10:20:55.849111+0000","last_fullsized":"2024-09-16T10:20:55.849111+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.113367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.113367+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:48.886516+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":388,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844916+0000","last_change":"2024-09-16T10:19:45.045388+0000","last_active":"2024-09-16T10:20:55.844916+0000","last_peered":"2024-09-16T10:20:55.844916+0000","last_clean":"2024-09-16T10:20:55.844916+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:20:55.844916+0000","last_undegraded":"2024-09-16T10:20:55.844916+0000","last_fullsized":"2024-09-16T10:20:55.844916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:45.045164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.045164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:08.833432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851449+0000","last_change":"2024-09-16T10:18:52.820977+0000","last_active":"2024-09-16T10:20:55.851449+0000","last_peered":"2024-09-16T10:20:55.851449+0000","last_clean":"2024-09-16T10:20:55.851449+0000","last_became_active":"2024-09-16T10:18:52.820693+0000","last_became_peered":"2024-09-16T10:18:52.820693+0000","last_unstale":"2024-09-16T10:20:55.851449+0000","last_undegraded":"2024-09-16T10:20:55.851449+0000","last_fullsized":"2024-09-16T10:20:55.851449+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:06.355305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":466,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845714+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:20:55.845714+0000","last_peered":"2024-09-16T10:20:55.845714+0000","last_clean":"2024-09-16T10:20:55.845714+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:20:55.845714+0000","last_undegraded":"2024-09-16T10:20:55.845714+0000","last_fullsized":"2024-09-16T10:20:55.845714+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.555917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":356,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847414+0000","last_change":"2024-09-16T10:18:09.148508+0000","last_active":"2024-09-16T10:20:55.847414+0000","last_peered":"2024-09-16T10:20:55.847414+0000","last_clean":"2024-09-16T10:20:55.847414+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:20:55.847414+0000","last_undegraded":"2024-09-16T10:20:55.847414+0000","last_fullsized":"2024-09-16T10:20:55.847414+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:09.148447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.148447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:16.056190+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":375,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842008+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:20:55.842008+0000","last_peered":"2024-09-16T10:20:55.842008+0000","last_clean":"2024-09-16T10:20:55.842008+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:20:55.842008+0000","last_undegraded":"2024-09-16T10:20:55.842008+0000","last_fullsized":"2024-09-16T10:20:55.842008+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.859976+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842973+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:20:55.842973+0000","last_peered":"2024-09-16T10:20:55.842973+0000","last_clean":"2024-09-16T10:20:55.842973+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:20:55.842973+0000","last_undegraded":"2024-09-16T10:20:55.842973+0000","last_fullsized":"2024-09-16T10:20:55.842973+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:36.922512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.841866+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:20:55.841866+0000","last_peered":"2024-09-16T10:20:55.841866+0000","last_clean":"2024-09-16T10:20:55.841866+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:20:55.841866+0000","last_undegraded":"2024-09-16T10:20:55.841866+0000","last_fullsized":"2024-09-16T10:20:55.841866+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:04.473585+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":357,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.843843+0000","last_change":"2024-09-16T10:20:54.392034+0000","last_active":"2024-09-16T10:20:55.843843+0000","last_peered":"2024-09-16T10:20:55.843843+0000","last_clean":"2024-09-16T10:20:55.843843+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:20:55.843843+0000","last_undegraded":"2024-09-16T10:20:55.843843+0000","last_fullsized":"2024-09-16T10:20:55.843843+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.391970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.391970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.175737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":406,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.842189+0000","last_change":"2024-09-16T10:20:50.296501+0000","last_active":"2024-09-16T10:20:55.842189+0000","last_peered":"2024-09-16T10:20:55.842189+0000","last_clean":"2024-09-16T10:20:55.842189+0000","last_became_active":"2024-09-16T10:18:52.832688+0000","last_became_peered":"2024-09-16T10:18:52.832688+0000","last_unstale":"2024-09-16T10:20:55.842189+0000","last_undegraded":"2024-09-16T10:20:55.842189+0000","last_fullsized":"2024-09-16T10:20:55.842189+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:16.056306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":377,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852407+0000","last_change":"2024-09-16T10:19:47.608315+0000","last_active":"2024-09-16T10:20:55.852407+0000","last_peered":"2024-09-16T10:20:55.852407+0000","last_clean":"2024-09-16T10:20:55.852407+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:20:55.852407+0000","last_undegraded":"2024-09-16T10:20:55.852407+0000","last_fullsized":"2024-09-16T10:20:55.852407+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.608243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.608243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:20:58.564159+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":326,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846170+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:20:55.846170+0000","last_peered":"2024-09-16T10:20:55.846170+0000","last_clean":"2024-09-16T10:20:55.846170+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:20:55.846170+0000","last_undegraded":"2024-09-16T10:20:55.846170+0000","last_fullsized":"2024-09-16T10:20:55.846170+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.331834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":348,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.845532+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:20:55.845532+0000","last_peered":"2024-09-16T10:20:55.845532+0000","last_clean":"2024-09-16T10:20:55.845532+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:20:55.845532+0000","last_undegraded":"2024-09-16T10:20:55.845532+0000","last_fullsized":"2024-09-16T10:20:55.845532+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:00.958267+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":313,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.852341+0000","last_change":"2024-09-16T10:20:21.141380+0000","last_active":"2024-09-16T10:20:55.852341+0000","last_peered":"2024-09-16T10:20:55.852341+0000","last_clean":"2024-09-16T10:20:55.852341+0000","last_became_active":"2024-09-16T10:20:18.867938+0000","last_became_peered":"2024-09-16T10:20:18.867938+0000","last_unstale":"2024-09-16T10:20:55.852341+0000","last_undegraded":"2024-09-16T10:20:55.852341+0000","last_fullsized":"2024-09-16T10:20:55.852341+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:21.141324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:21.141324+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:31.692320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":376,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844081+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:20:55.844081+0000","last_peered":"2024-09-16T10:20:55.844081+0000","last_clean":"2024-09-16T10:20:55.844081+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:20:55.844081+0000","last_undegraded":"2024-09-16T10:20:55.844081+0000","last_fullsized":"2024-09-16T10:20:55.844081+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.343855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":377,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846359+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:20:55.846359+0000","last_peered":"2024-09-16T10:20:55.846359+0000","last_clean":"2024-09-16T10:20:55.846359+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:20:55.846359+0000","last_undegraded":"2024-09-16T10:20:55.846359+0000","last_fullsized":"2024-09-16T10:20:55.846359+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.195400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":369,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.850973+0000","last_change":"2024-09-16T10:19:45.534275+0000","last_active":"2024-09-16T10:20:55.850973+0000","last_peered":"2024-09-16T10:20:55.850973+0000","last_clean":"2024-09-16T10:20:55.850973+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:20:55.850973+0000","last_undegraded":"2024-09-16T10:20:55.850973+0000","last_fullsized":"2024-09-16T10:20:55.850973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:45.533998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.533998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":365,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.846365+0000","last_change":"2024-09-16T10:19:47.129026+0000","last_active":"2024-09-16T10:20:55.846365+0000","last_peered":"2024-09-16T10:20:55.846365+0000","last_clean":"2024-09-16T10:20:55.846365+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:20:55.846365+0000","last_undegraded":"2024-09-16T10:20:55.846365+0000","last_fullsized":"2024-09-16T10:20:55.846365+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.128964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.128964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":397,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.844937+0000","last_change":"2024-09-16T10:19:56.200787+0000","last_active":"2024-09-16T10:20:55.844937+0000","last_peered":"2024-09-16T10:20:55.844937+0000","last_clean":"2024-09-16T10:20:55.844937+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:20:55.844937+0000","last_undegraded":"2024-09-16T10:20:55.844937+0000","last_fullsized":"2024-09-16T10:20:55.844937+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:19:56.200704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.200704+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.350145+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":374,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847979+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:20:55.847979+0000","last_peered":"2024-09-16T10:20:55.847979+0000","last_clean":"2024-09-16T10:20:55.847979+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:20:55.847979+0000","last_undegraded":"2024-09-16T10:20:55.847979+0000","last_fullsized":"2024-09-16T10:20:55.847979+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.328316+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":368,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.848605+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:20:55.848605+0000","last_peered":"2024-09-16T10:20:55.848605+0000","last_clean":"2024-09-16T10:20:55.848605+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:20:55.848605+0000","last_undegraded":"2024-09-16T10:20:55.848605+0000","last_fullsized":"2024-09-16T10:20:55.848605+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.662531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":126,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.851974+0000","last_change":"2024-09-16T10:19:44.487481+0000","last_active":"2024-09-16T10:20:55.851974+0000","last_peered":"2024-09-16T10:20:55.851974+0000","last_clean":"2024-09-16T10:20:55.851974+0000","last_became_active":"2024-09-16T10:18:52.838002+0000","last_became_peered":"2024-09-16T10:18:52.838002+0000","last_unstale":"2024-09-16T10:20:55.851974+0000","last_undegraded":"2024-09-16T10:20:55.851974+0000","last_fullsized":"2024-09-16T10:20:55.851974+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":364,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.840853+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:20:55.840853+0000","last_peered":"2024-09-16T10:20:55.840853+0000","last_clean":"2024-09-16T10:20:55.840853+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:20:55.840853+0000","last_undegraded":"2024-09-16T10:20:55.840853+0000","last_fullsized":"2024-09-16T10:20:55.840853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":377,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:58.426297+0000","last_change":"2024-09-16T10:20:58.426297+0000","last_active":"2024-09-16T10:20:58.426297+0000","last_peered":"2024-09-16T10:20:58.426297+0000","last_clean":"2024-09-16T10:20:58.426297+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:20:58.426297+0000","last_undegraded":"2024-09-16T10:20:58.426297+0000","last_fullsized":"2024-09-16T10:20:58.426297+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.426235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.426235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.172313+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":399,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.849953+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:20:55.849953+0000","last_peered":"2024-09-16T10:20:55.849953+0000","last_clean":"2024-09-16T10:20:55.849953+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:20:55.849953+0000","last_undegraded":"2024-09-16T10:20:55.849953+0000","last_fullsized":"2024-09-16T10:20:55.849953+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:11.553581+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":375,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T10:20:55.847664+0000","last_change":"2024-09-16T10:19:16.217856+0000","last_active":"2024-09-16T10:20:55.847664+0000","last_peered":"2024-09-16T10:20:55.847664+0000","last_clean":"2024-09-16T10:20:55.847664+0000","last_became_active":"2024-09-16T10:18:52.834424+0000","last_became_peered":"2024-09-16T10:18:52.834424+0000","last_unstale":"2024-09-16T10:20:55.847664+0000","last_undegraded":"2024-09-16T10:20:55.847664+0000","last_fullsized":"2024-09-16T10:20:55.847664+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4035051544,"num_objects":61571,"num_object_clones":0,"num_object_copies":184713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193065,"num_write_kb":8148289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10118,"num_bytes_recovered":663093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12685815808,"data_stored":12685803592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67340,"ondisk_log_size":67340,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":435,"num_read_kb":430,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234076,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1791080,"kb_used_data":1525208,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91954072,"statfs":{"total":95995035648,"available":94160969728,"internally_reserved":0,"allocated":1561812992,"data_stored":1561276710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":272169418},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528416,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2058152,"kb_used_data":1756184,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91687000,"statfs":{"total":95995035648,"available":93887488000,"internally_reserved":0,"allocated":1798332416,"data_stored":1797796048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309131718},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790054,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2526636,"kb_used_data":2270428,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91218516,"statfs":{"total":95995035648,"available":93407760384,"internally_reserved":0,"allocated":2324918272,"data_stored":2324379132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986283,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736812,"kb_used_data":1489052,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92008340,"statfs":{"total":95995035648,"available":94216540160,"internally_reserved":0,"allocated":1524789248,"data_stored":1524248789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215216,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2060420,"kb_used_data":1732148,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91684732,"statfs":{"total":95995035648,"available":93885165568,"internally_reserved":0,"allocated":1773719552,"data_stored":1773156756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":336132552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2022532,"kb_used_data":1742836,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91722620,"statfs":{"total":95995035648,"available":93923962880,"internally_reserved":0,"allocated":1784664064,"data_stored":1784099879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738495,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1786552,"kb_used_data":1476324,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91958600,"statfs":{"total":95995035648,"available":94165606400,"internally_reserved":0,"allocated":1511755776,"data_stored":1511208104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1497038848,"data_stored":1497038848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1770848256,"data_stored":1770848256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":562823168,"data_stored":562823168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1763573760,"data_stored":1763573760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1515126784,"data_stored":1515126784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2277511168,"data_stored":2277507096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1773932544,"data_stored":1773928472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1524961280,"data_stored":1524957208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:21:06.407 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-09-16T10:21:06.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-16T10:21:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:06.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:06.461 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:06.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:06.525 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:06.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:06.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:06.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:06 smithi165 ceph-mon[32696]: from='client.15621 v1:172.21.15.63:0/2704762778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:21:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:06 smithi165 ceph-mon[32696]: pgmap v369: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s 2024-09-16T10:21:06.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:06.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:06.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:06.761 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:21:06.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:06.849 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:06.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:06.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:07.015 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:07.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:07.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:07.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:07.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:07.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:21:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:21:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:21:07.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:07.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:07.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:07.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:07.449 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:07.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:07.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:07.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:07.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:07.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:07.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:07.753 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:07.753 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:21:07.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:21:07.753 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39764 7.6 7:680231a8:::benchmark_data_smithi063_73086_object39763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 358.50053353999999, 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13555234199999999, 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39764 2024-09-16T10:21:07.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246056+0000", 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246057+0000", 2024-09-16T10:21:07.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3999999999999999e-06 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246080+0000", 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2629999999999998e-05 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246091+0000", 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0978999999999999e-05 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246097+0000", 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6579999999999996e-06 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246213+0000", 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115435 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246353+0000", 2024-09-16T10:21:07.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014010199999999999 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.246471+0000", 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117877 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249103+0000", 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026325020000000001 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249476+0000", 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037260100000000002 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381493+0000", 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13201689999999999 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381556+0000", 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3516000000000006e-05 2024-09-16T10:21:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381608+0000", 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1742e-05 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39772 7.6 7:6310c551:::benchmark_data_smithi063_73086_object39771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 358.49608467199999, 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13122466799999999, 2024-09-16T10:21:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39772 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250504+0000", 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250506+0000", 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.897e-06 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250528+0000", 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1883999999999999e-05 2024-09-16T10:21:07.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250537+0000", 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.285e-06 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250545+0000", 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.08e-06 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250610+0000", 2024-09-16T10:21:07.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5287e-05 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.250734+0000", 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012402299999999999 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252617+0000", 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018828919999999999 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.254471+0000", 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001854138 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.255185+0000", 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00071378999999999995 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381651+0000", 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12646639400000001 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381696+0000", 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5364999999999997e-05 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.381729+0000", 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2632999999999997e-05 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.766 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39783 7.f 7:fb2e33c9:::benchmark_data_smithi063_73086_object39782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 358.48898633599998, 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13116349199999999, 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39783 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257603+0000", 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257604+0000", 2024-09-16T10:21:07.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7710000000000001e-06 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257627+0000", 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2589999999999999e-05 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257636+0000", 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.302e-06 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257641+0000", 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257700+0000", 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8791e-05 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.257856+0000", 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015637100000000001 2024-09-16T10:21:07.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.258009+0000", 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015277300000000001 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383671+0000", 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12566208600000001 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.383705+0000", 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3278999999999998e-05 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388613+0000", 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049080310000000002 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388687+0000", 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4381000000000005e-05 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.388766+0000", 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9198999999999998e-05 2024-09-16T10:21:07.772 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:40097 7.f 7:f223a0ce:::benchmark_data_smithi063_73086_object40096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 358.20127746999998, 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.133293199, 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.773 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 40097 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545312+0000", 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545313+0000", 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0470000000000001e-06 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545333+0000", 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0778e-05 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545341+0000", 2024-09-16T10:21:07.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0739999999999993e-06 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.545348+0000", 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3049999999999998e-06 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549229+0000", 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038809500000000002 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549377+0000", 2024-09-16T10:21:07.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014856200000000001 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549537+0000", 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015973699999999999 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659510+0000", 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.109972575 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.663519+0000", 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040092649999999997 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678483+0000", 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0149646 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678559+0000", 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5891999999999998e-05 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.678605+0000", 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5414000000000003e-05 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90038 7.6 7:605b4ba6:::benchmark_data_smithi063_73086_object90037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 318.24855244399998, 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.151200375, 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.779 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90038 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498037+0000", 2024-09-16T10:21:07.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498038+0000", 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3200000000000001e-06 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498061+0000", 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2646e-05 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498070+0000", 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.217e-06 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498076+0000", 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1759999999999998e-06 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498121+0000", 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4870000000000002e-05 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498227+0000", 2024-09-16T10:21:07.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106239 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.498358+0000", 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013121899999999999 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501482+0000", 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031239000000000002 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.503423+0000", 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019405819999999999 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648604+0000", 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14518165099999999 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648977+0000", 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000372641 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649237+0000", 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025991399999999998 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90048 7.6 7:6327a166:::benchmark_data_smithi063_73086_object90047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 318.24233729100001, 2024-09-16T10:21:07.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14568709799999999, 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90048 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:21:07.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504252+0000", 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504254+0000", 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1409999999999999e-06 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504278+0000", 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3711999999999999e-05 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504289+0000", 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1331e-05 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504296+0000", 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4599999999999997e-06 2024-09-16T10:21:07.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504413+0000", 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011668299999999999 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504522+0000", 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010877 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504652+0000", 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013002399999999999 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509094+0000", 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044417069999999996 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514152+0000", 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0050583470000000004 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649421+0000", 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13526861600000001 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649711+0000", 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00029075100000000001 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.649939+0000", 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00022755599999999999 2024-09-16T10:21:07.792 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90050 7.6 7:60112b96:::benchmark_data_smithi063_73086_object90049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 318.23738279000003, 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14140651900000001, 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.793 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90050 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509206+0000", 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509208+0000", 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.407e-06 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509230+0000", 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1861e-05 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509237+0000", 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3159999999999999e-06 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509242+0000", 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5820000000000001e-06 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509424+0000", 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018168599999999999 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509616+0000", 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019154600000000001 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509757+0000", 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000141485 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514172+0000", 2024-09-16T10:21:07.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044148629999999998 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.514209+0000", 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6706999999999999e-05 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650127+0000", 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.135917818 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650433+0000", 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030668399999999999 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650613+0000", 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000179564 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.799 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90064 7.6 7:67c07ba3:::benchmark_data_smithi063_73086_object90063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 318.22603343899999, 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.130695903, 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90064 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520556+0000", 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520558+0000", 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.984e-06 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520581+0000", 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3700999999999999e-05 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520593+0000", 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.128e-05 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520600+0000", 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520651+0000", 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1133e-05 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520775+0000", 2024-09-16T10:21:07.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012390099999999999 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.520940+0000", 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016520200000000001 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523864+0000", 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029241139999999998 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524288+0000", 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00042372299999999998 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.650781+0000", 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12649292200000001 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.651082+0000", 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030106300000000001 2024-09-16T10:21:07.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.651252+0000", 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016956999999999999 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90066 7.f 7:fafdc57c:::benchmark_data_smithi063_73086_object90065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:21:07.806 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 318.22473713599999, 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.132960727, 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90066 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521852+0000", 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521853+0000", 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4419999999999999e-06 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521876+0000", 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2430999999999999e-05 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521885+0000", 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6989999999999998e-06 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521892+0000", 2024-09-16T10:21:07.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7909999999999999e-06 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.521943+0000", 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0321999999999997e-05 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.522046+0000", 2024-09-16T10:21:07.811 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:07.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010361600000000001 2024-09-16T10:21:07.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.522161+0000", 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000114833 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.641264+0000", 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11910251299999999 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.641301+0000", 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7638000000000001e-05 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.652401+0000", 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011099367000000001 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.654749+0000", 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023483110000000001 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.654813+0000", 2024-09-16T10:21:07.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3763999999999995e-05 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90070 7.6 7:6e0aafb6:::benchmark_data_smithi063_73086_object90069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 318.22230433599998, 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13177412199999999, 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.814 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90070 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524285+0000", 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524286+0000", 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3489999999999999e-06 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524308+0000", 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.228e-05 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524317+0000", 2024-09-16T10:21:07.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5310000000000001e-06 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524323+0000", 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3010000000000003e-06 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524360+0000", 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6721000000000003e-05 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524466+0000", 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106162 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640787+0000", 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116321087 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.645632+0000", 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0048448600000000003 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.647100+0000", 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001468254 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.655959+0000", 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088585079999999993 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.656015+0000", 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6378000000000001e-05 2024-09-16T10:21:07.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.656059+0000", 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3690999999999999e-05 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.820 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201820 7.9 7:907729a7:::benchmark_data_smithi063_73086_object82541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.350555+0000", 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 206.39603377399999, 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.148878706, 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201820 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350555+0000", 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350555+0000", 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350556+0000", 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0200000000000001e-07 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350559+0000", 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9739999999999998e-06 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350565+0000", 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.19e-06 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.350570+0000", 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0849999999999996e-06 2024-09-16T10:21:07.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.367568+0000", 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016997412999999999 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.367703+0000", 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013530899999999999 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.367862+0000", 2024-09-16T10:21:07.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015915300000000001 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377527+0000", 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0096649560000000006 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387412+0000", 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.009884716 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.499312+0000", 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11189982900000001 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.499380+0000", 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8617999999999999e-05 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.499434+0000", 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3661e-05 2024-09-16T10:21:07.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201823 7.9 7:91c75bba:::benchmark_data_smithi063_73086_object82544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.357374+0000", 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 206.38921474599999, 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14531649399999999, 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.827 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201823 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357374+0000", 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357374+0000", 2024-09-16T10:21:07.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357377+0000", 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2520000000000002e-06 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357382+0000", 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.383e-06 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357393+0000", 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0865e-05 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.357400+0000", 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5530000000000004e-06 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.370715+0000", 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.01331473 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.370834+0000", 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011859299999999999 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.370930+0000", 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5977000000000006e-05 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.381075+0000", 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010145711999999999 2024-09-16T10:21:07.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387417+0000", 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063419729999999999 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.502472+0000", 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11505420800000001 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.502611+0000", 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013925600000000001 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.502691+0000", 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9992000000000003e-05 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201826 7.4 7:2bb12e82:::benchmark_data_smithi063_73086_object82547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.363029+0000", 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 206.38355956500001, 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13118088, 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201826 2024-09-16T10:21:07.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363029+0000", 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363029+0000", 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363032+0000", 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6910000000000002e-06 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363038+0000", 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0399999999999998e-06 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363050+0000", 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1824000000000001e-05 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363058+0000", 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.588e-06 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363115+0000", 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6997999999999999e-05 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363230+0000", 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011523600000000001 2024-09-16T10:21:07.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.363338+0000", 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108513 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.368289+0000", 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049509360000000004 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.371441+0000", 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031520110000000001 2024-09-16T10:21:07.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.494133+0000", 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.122692074 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.494167+0000", 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3141e-05 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.494210+0000", 2024-09-16T10:21:07.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3828000000000001e-05 2024-09-16T10:21:07.840 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.840 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.840 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.840 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:07.841 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:07.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.841 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201831 7.9 7:99be6e1c:::benchmark_data_smithi063_73086_object82552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:07.841 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.365317+0000", 2024-09-16T10:21:07.841 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 206.38127158, 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.141925095, 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201831 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.842 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365317+0000", 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365317+0000", 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365319+0000", 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.215e-06 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365322+0000", 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7979999999999999e-06 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365329+0000", 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5869999999999998e-06 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365334+0000", 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3770000000000004e-06 2024-09-16T10:21:07.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.373135+0000", 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0078000819999999998 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.373243+0000", 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108107 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.373345+0000", 2024-09-16T10:21:07.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102804 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.381109+0000", 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0077639320000000003 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387422+0000", 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063129229999999998 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507118+0000", 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11969548200000001 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507184+0000", 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6002999999999998e-05 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507243+0000", 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8785e-05 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201833 7.9 7:92434f80:::benchmark_data_smithi063_73086_object82554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.365699+0000", 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 206.380889849, 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14236796500000001, 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201833 2024-09-16T10:21:07.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365699+0000", 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365699+0000", 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365700+0000", 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.24e-06 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365705+0000", 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5689999999999998e-06 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365713+0000", 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2549999999999997e-06 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.365720+0000", 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6449999999999999e-06 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377556+0000", 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011835714000000001 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377656+0000", 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000100579 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.377769+0000", 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112946 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.384425+0000", 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0066563439999999998 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387428+0000", 2024-09-16T10:21:07.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003002041 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507290+0000", 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119862684 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.507367+0000", 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7121999999999998e-05 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.508067+0000", 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069982599999999996 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51238 8.e 8:7236be56:::benchmark_data_smithi063_101290_object51237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 114.404953621, 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96343729, 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.854 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51238 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.855 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341637+0000", 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.621e-06 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341663+0000", 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6026000000000001e-05 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341674+0000", 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0871e-05 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341682+0000", 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8390000000000007e-06 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341780+0000", 2024-09-16T10:21:07.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8233999999999995e-05 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341791+0000", 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1426999999999999e-05 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296465+0000", 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95467394000000005 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296604+0000", 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013824400000000001 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296727+0000", 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012366699999999999 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304672+0000", 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079445339999999996 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304896+0000", 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000224117 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304941+0000", 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5374999999999998e-05 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305028+0000", 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6447999999999996e-05 2024-09-16T10:21:07.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305073+0000", 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4947000000000002e-05 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51239 8.e 8:7166e593:::benchmark_data_smithi063_101290_object51238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 114.404535444, 2024-09-16T10:21:07.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96520916099999998, 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51239 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.862 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342055+0000", 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.051e-06 2024-09-16T10:21:07.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342073+0000", 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8325000000000002e-05 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342083+0000", 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9850000000000001e-06 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342090+0000", 2024-09-16T10:21:07.864 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2119999999999997e-06 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342196+0000", 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010593400000000001 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342244+0000", 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8272000000000001e-05 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299084+0000", 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95683958300000005 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299216+0000", 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131595 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299311+0000", 2024-09-16T10:21:07.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5445000000000005e-05 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305117+0000", 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058058169999999996 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305157+0000", 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9836999999999998e-05 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307167+0000", 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020101390000000002 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307221+0000", 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4317000000000002e-05 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307263+0000", 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1649000000000001e-05 2024-09-16T10:21:07.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51244 8.e 8:7a51d24a:::benchmark_data_smithi063_101290_object51243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 114.401314313, 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96214958799999994, 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.869 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51244 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.870 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345277+0000", 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9810000000000002e-06 2024-09-16T10:21:07.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345305+0000", 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7858999999999999e-05 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345320+0000", 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5268e-05 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345328+0000", 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2900000000000002e-06 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345404+0000", 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6241000000000004e-05 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345415+0000", 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0672e-05 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300950+0000", 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95553457900000005 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301082+0000", 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013229299999999999 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301242+0000", 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015961099999999999 2024-09-16T10:21:07.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307271+0000", 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060297889999999998 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307310+0000", 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8288999999999999e-05 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307349+0000", 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9499999999999998e-05 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307391+0000", 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1779000000000001e-05 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307424+0000", 2024-09-16T10:21:07.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3436999999999999e-05 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51256 8.e 8:745a54ba:::benchmark_data_smithi063_101290_object51255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 114.389475834, 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95048443400000004, 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51256 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357116+0000", 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8820000000000001e-06 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357147+0000", 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1263999999999997e-05 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357167+0000", 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9164000000000002e-05 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357176+0000", 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8309999999999996e-06 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357255+0000", 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8213999999999993e-05 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357267+0000", 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2729e-05 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303081+0000", 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94581367599999999 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303176+0000", 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5471999999999997e-05 2024-09-16T10:21:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303272+0000", 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5874999999999999e-05 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307320+0000", 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040474079999999997 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307459+0000", 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013938500000000001 2024-09-16T10:21:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307514+0000", 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4948999999999999e-05 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307560+0000", 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5803999999999997e-05 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307598+0000", 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7781000000000003e-05 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51260 8.e 8:7df47e01:::benchmark_data_smithi063_101290_object51259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 114.38435339599999, 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95708149499999995, 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51260 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362238+0000", 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4140000000000001e-06 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362265+0000", 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7183999999999999e-05 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362279+0000", 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3832000000000001e-05 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362287+0000", 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6990000000000003e-06 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362347+0000", 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0560000000000003e-05 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362356+0000", 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1990000000000006e-06 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305199+0000", 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94284296700000003 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305311+0000", 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011281900000000001 2024-09-16T10:21:07.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305449+0000", 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000137708 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309902+0000", 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044530100000000003 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312979+0000", 2024-09-16T10:21:07.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030772859999999998 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317533+0000", 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045533479999999996 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319259+0000", 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017263129999999999 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:07.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319317+0000", 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8155999999999997e-05 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:07.891 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:07.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:08.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:08.037 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:08.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:08.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:08.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:08.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:08.249 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:21:08.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:08.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:08.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:08.416 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:08.454 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:08.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:08 smithi165 ceph-mon[32696]: pgmap v370: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 870 op/s 2024-09-16T10:21:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3353091687' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:08.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:21:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:21:08.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:08.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:08.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:08.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:08.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:08 smithi063 ceph-mon[29993]: pgmap v370: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 870 op/s 2024-09-16T10:21:08.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3353091687' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:08.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:08 smithi063 ceph-mon[39470]: pgmap v370: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 870 op/s 2024-09-16T10:21:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3353091687' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:08.728 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:08.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:08.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:08.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:08.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:08.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:09.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:09.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:09.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:09.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:09.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:09.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:09.419 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:09.521 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:09 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3353091687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:09 smithi165 ceph-mon[32696]: osdmap e231: 8 total, 8 up, 7 in 2024-09-16T10:21:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]} : dispatch 2024-09-16T10:21:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]} : dispatch 2024-09-16T10:21:09.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3353091687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[29993]: osdmap e231: 8 total, 8 up, 7 in 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]} : dispatch 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]} : dispatch 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3353091687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[39470]: osdmap e231: 8 total, 8 up, 7 in 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:09.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]} : dispatch 2024-09-16T10:21:09.559 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]} : dispatch 2024-09-16T10:21:09.560 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:09.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:09.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:09.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:09.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:09.936 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:09.971 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:09.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:10.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:10.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:10.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:10.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:10.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:10.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:10.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:10.391 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:10.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:10.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:10.529 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:10.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:10 smithi165 ceph-mon[32696]: pgmap v372: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-16T10:21:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:10 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-09-16T10:21:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:10 smithi165 ceph-mon[32696]: osdmap e232: 8 total, 8 up, 7 in 2024-09-16T10:21:10.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:10.653 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:10.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:10.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:10 smithi063 ceph-mon[29993]: pgmap v372: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-16T10:21:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:10 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-09-16T10:21:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:10 smithi063 ceph-mon[29993]: osdmap e232: 8 total, 8 up, 7 in 2024-09-16T10:21:10.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:10 smithi063 ceph-mon[39470]: pgmap v372: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-16T10:21:10.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:10 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-09-16T10:21:10.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:10 smithi063 ceph-mon[39470]: osdmap e232: 8 total, 8 up, 7 in 2024-09-16T10:21:10.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:10.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:10.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:10.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:10.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:10.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:11.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:11.053 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:11.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:11.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:11.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:11.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:11.290 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:11.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:21:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:11.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:11.416 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:11.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:11.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:11 smithi165 ceph-mon[32696]: osdmap e233: 8 total, 8 up, 7 in 2024-09-16T10:21:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3126573358' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:21:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3126573358' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:21:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:11 smithi165 ceph-mon[32696]: osdmap e234: 8 total, 8 up, 7 in 2024-09-16T10:21:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:11.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:11.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:11.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[29993]: osdmap e233: 8 total, 8 up, 7 in 2024-09-16T10:21:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3126573358' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:21:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3126573358' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:21:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[29993]: osdmap e234: 8 total, 8 up, 7 in 2024-09-16T10:21:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[39470]: osdmap e233: 8 total, 8 up, 7 in 2024-09-16T10:21:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3126573358' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:21:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3126573358' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:21:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[39470]: osdmap e234: 8 total, 8 up, 7 in 2024-09-16T10:21:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:11.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:11.704 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:11.717 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:21:11.735 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:11.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:11.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:11.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:11.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:11.982 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:12.014 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:12.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:12.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:12.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:12.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:12.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:12.409 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 1 pg_num to 11 2024-09-16T10:21:12.430 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:12.532 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:12.554 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:12.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:12.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:12.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:12 smithi063 ceph-mon[29993]: pgmap v376: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.87k op/s 2024-09-16T10:21:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:21:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:12 smithi063 ceph-mon[29993]: osdmap e235: 8 total, 8 up, 7 in 2024-09-16T10:21:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:12 smithi063 ceph-mon[39470]: pgmap v376: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.87k op/s 2024-09-16T10:21:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:21:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:12 smithi063 ceph-mon[39470]: osdmap e235: 8 total, 8 up, 7 in 2024-09-16T10:21:12.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:12.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:12 smithi165 ceph-mon[32696]: pgmap v376: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.87k op/s 2024-09-16T10:21:12.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:21:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:12 smithi165 ceph-mon[32696]: osdmap e235: 8 total, 8 up, 7 in 2024-09-16T10:21:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:12.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:12.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:12.971 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:12.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:13.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:13.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:13.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:13.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:13.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:13.234 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:13.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:13.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:13.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:13.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:13.474 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:13.510 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:13.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:13.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:13.654 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:13.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:13.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:13.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:13.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:13.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:13.927 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:13.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:14.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:14.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:14.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:14.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:14.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[29993]: osdmap e236: 8 total, 8 up, 7 in 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[29993]: pgmap v379: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[39470]: osdmap e236: 8 total, 8 up, 7 in 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[39470]: pgmap v379: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:21:14.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:21:14.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:13 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:21:14.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:14.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:13 smithi165 ceph-mon[32696]: osdmap e236: 8 total, 8 up, 7 in 2024-09-16T10:21:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:13 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:21:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:13 smithi165 ceph-mon[32696]: pgmap v379: 168 pgs: 168 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail 2024-09-16T10:21:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:13 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:21:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:13 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:21:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:13 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:21:14.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:14.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:14.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:14.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:14.475 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:14.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:14.577 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:14.621 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:14.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:14.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:14.726 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:14.726 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:14.770 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:14.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:14.771 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:14.771 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:14.800 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:14.800 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:14.833 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:14.833 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:14.861 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:14.861 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:14.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:14.896 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:14.896 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:14.910 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:14.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:14.930 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:14.930 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:14.963 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:14.963 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:15.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:15.045 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:15.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:15.147 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:15.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:15.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:15.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:14 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:21:15.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1939126101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:14 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:21:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1939126101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:15.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:15.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:14 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:21:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1939126101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:15.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:15.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:15.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:15.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:15.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:15.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:15.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:15.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:15.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:15.735 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:15.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:15.837 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:15.874 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:15.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:15.976 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:16.032 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:16.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:16.077 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:16.077 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:16.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.168 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:16.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[39470]: pgmap v380: 168 pgs: 168 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 6.5 KiB/s rd, 40 KiB/s wr, 1.39k op/s; 80 KiB/s, 0 objects/s recovering 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[29993]: pgmap v380: 168 pgs: 168 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 6.5 KiB/s rd, 40 KiB/s wr, 1.39k op/s; 80 KiB/s, 0 objects/s recovering 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:21:16.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:15 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:21:16.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:15 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:21:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:15 smithi165 ceph-mon[32696]: pgmap v380: 168 pgs: 168 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 6.5 KiB/s rd, 40 KiB/s wr, 1.39k op/s; 80 KiB/s, 0 objects/s recovering 2024-09-16T10:21:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:15 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:21:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:15 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:21:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:16.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:16.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:16.435 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:16.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:16.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:16.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:16.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:16.808 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:16.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:16.863 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:21:16.863 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:21:16.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:16.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:16.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:17.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-16T10:21:17.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[29993]: osdmap e237: 8 total, 8 up, 7 in 2024-09-16T10:21:17.059 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:21:17.060 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:21:17.060 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:21:17.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-16T10:21:17.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[39470]: osdmap e237: 8 total, 8 up, 7 in 2024-09-16T10:21:17.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:21:17.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:21:17.060 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:16 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:21:17.080 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:17.118 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:17.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:17.220 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:17.261 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:17.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:16 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-16T10:21:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:16 smithi165 ceph-mon[32696]: osdmap e237: 8 total, 8 up, 7 in 2024-09-16T10:21:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:16 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:21:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:16 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:21:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:16 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:21:17.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:17.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:17.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:21:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:21:17.502 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:17.540 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:17.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:17.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:17.680 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:17.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:17.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:17.820 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:17.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:17.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:21:17.913 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:21:17.921 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:17.969 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:17.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:18.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:21:18.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:18.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:18.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:18.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:21:18.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[29993]: osdmap e238: 8 total, 8 up, 7 in 2024-09-16T10:21:18.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[29993]: pgmap v383: 168 pgs: 168 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 6.0 KiB/s rd, 37 KiB/s wr, 1.28k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-16T10:21:18.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4201466494' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:18.173 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:21:18.173 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[39470]: osdmap e238: 8 total, 8 up, 7 in 2024-09-16T10:21:18.173 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[39470]: pgmap v383: 168 pgs: 168 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 6.0 KiB/s rd, 37 KiB/s wr, 1.28k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-16T10:21:18.173 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:17 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4201466494' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:18.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:18.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:18.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:17 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:21:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:17 smithi165 ceph-mon[32696]: osdmap e238: 8 total, 8 up, 7 in 2024-09-16T10:21:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:17 smithi165 ceph-mon[32696]: pgmap v383: 168 pgs: 168 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 6.0 KiB/s rd, 37 KiB/s wr, 1.28k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-16T10:21:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:17 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4201466494' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:18.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:18.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:18.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:18.497 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:18.533 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:18.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:18.635 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:18.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:18.681 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:18.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:18.772 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:18.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:18.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:21:18.808 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:18.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:18.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:18.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:18.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:18.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:19.053 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:19.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:19.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:19.186 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4201466494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[29993]: osdmap e239: 8 total, 8 up, 7 in 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[29993]: 1.11 scrub ok 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:19.187 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4201466494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:21:19.188 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[39470]: osdmap e239: 8 total, 8 up, 7 in 2024-09-16T10:21:19.188 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:21:19.188 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[39470]: 1.11 scrub ok 2024-09-16T10:21:19.188 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:18 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:19.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:19.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:19.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:21:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4201466494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:21:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-mon[32696]: osdmap e239: 8 total, 8 up, 7 in 2024-09-16T10:21:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:21:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-mon[32696]: 1.11 scrub ok 2024-09-16T10:21:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:18 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:19.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:19.366 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:19.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:19.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:19.485 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:21:19.485 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:21:19.501 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:19.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:19.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:19.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:19.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:19.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:19.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:19.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:19.880 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:19.913 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:19.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:20.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:20.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:20.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:20.158 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:20.183 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:20.183 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:20.192 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:20.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:20.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.12 scrub ok 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.e scrub ok 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.d scrub ok 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: osdmap e240: 8 total, 8 up, 7 in 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: pgmap v386: 167 pgs: 1 peering, 166 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:21:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[29993]: 1.4 scrub ok 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.12 scrub ok 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.e scrub ok 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.d scrub ok 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: osdmap e240: 8 total, 8 up, 7 in 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: pgmap v386: 167 pgs: 1 peering, 166 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:21:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:21:20.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:21:20.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:21:20.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:21:20.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:19 smithi063 ceph-mon[39470]: 1.4 scrub ok 2024-09-16T10:21:20.217 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:20.218 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:20.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:20.249 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:20.280 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:20.281 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:20.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.12 scrub ok 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.e scrub ok 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.d scrub ok 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: osdmap e240: 8 total, 8 up, 7 in 2024-09-16T10:21:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: pgmap v386: 167 pgs: 1 peering, 166 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-09-16T10:21:20.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:21:20.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:21:20.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:21:20.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:21:20.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:21:20.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:19 smithi165 ceph-mon[32696]: 1.4 scrub ok 2024-09-16T10:21:20.308 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:20.309 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:20.321 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:21:20.321 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:21:20.321 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:21:20.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:21:20.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:20.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:20.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:20.343 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:20.343 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:20.378 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:20.379 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:20.416 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:20.416 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:20.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:20.494 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:20.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:20.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:20.639 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:20.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:20.640 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:21:20.687 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:21:20.740 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:20.813 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:20.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:20.914 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:20.937 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:21:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:20.960 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:20.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:21.061 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:21.104 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:21.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: 1.10 scrub ok 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: osdmap e241: 8 total, 8 up, 7 in 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: 6.16 scrub starts 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: 6.16 scrub ok 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3587231861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[29993]: osdmap e242: 8 total, 8 up, 7 in 2024-09-16T10:21:21.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: 1.10 scrub ok 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: osdmap e241: 8 total, 8 up, 7 in 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: 6.16 scrub starts 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: 6.16 scrub ok 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3587231861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:21:21.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:20 smithi063 ceph-mon[39470]: osdmap e242: 8 total, 8 up, 7 in 2024-09-16T10:21:21.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:21.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:21.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:21.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: 1.10 scrub ok 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: osdmap e241: 8 total, 8 up, 7 in 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: 6.16 scrub starts 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: 6.16 scrub ok 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3587231861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:21:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:20 smithi165 ceph-mon[32696]: osdmap e242: 8 total, 8 up, 7 in 2024-09-16T10:21:21.345 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:21.375 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:21.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:21.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:21.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:21.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:21.536 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:21.594 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:21.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:21.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:21.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:21.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:21.782 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:21.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:21.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:21.921 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:21.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:22.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:22.060 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:22.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:22.162 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:22.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:21:22.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:21:22.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[29993]: pgmap v389: 167 pgs: 1 peering, 166 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-09-16T10:21:22.204 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[29993]: osdmap e243: 8 total, 8 up, 7 in 2024-09-16T10:21:22.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:21:22.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:21:22.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[39470]: pgmap v389: 167 pgs: 1 peering, 166 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-09-16T10:21:22.205 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:21 smithi063 ceph-mon[39470]: osdmap e243: 8 total, 8 up, 7 in 2024-09-16T10:21:22.208 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:22.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:21 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:21:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:21 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:21:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:21 smithi165 ceph-mon[32696]: pgmap v389: 167 pgs: 1 peering, 166 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-09-16T10:21:22.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:21 smithi165 ceph-mon[32696]: osdmap e243: 8 total, 8 up, 7 in 2024-09-16T10:21:22.311 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:22.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:22.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:22.507 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:22.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:22.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 243, 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:21:21.858041+0000", 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:18:51.797277+0000", 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:21:22.571 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 8, 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:21:22.572 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:21:22.573 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 19, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 11, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 11, 2024-09-16T10:21:22.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 11, 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 19, 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.13", 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 243, 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 242, 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 242, 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "243", 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:21:22.575 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "243", 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "241", 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:21:22.576 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.577 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:21:22.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.0899999141693115, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.0899999141693115, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.8400000333786011, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.8400000333786011, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:21:22.579 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:21:22.580 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:21:22.581 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:21:22.582 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:21:22.583 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:21:22.584 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6700000762939453, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6700000762939453, 2024-09-16T10:21:22.585 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.75, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:21:22.586 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:21:22.587 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:21:22.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:21:22.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:21:22.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:21:22.588 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:21:22.588 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:21:22.588 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:21:22.588 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:21:22.589 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:21:22.590 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:21:22.591 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.592 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.4900000095367432, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.4900000095367432, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.309999942779541, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.309999942779541, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.593 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:21:22.594 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:21:22.595 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:21:22.596 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:21:22.597 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:21:22.598 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.2400000095367432, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.2400000095367432, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:21:22.601 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:21:22.602 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:21:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:21:22.604 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:21:22.605 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.4900000095367432, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.4900000095367432, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.309999942779541, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.309999942779541, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:21:22.606 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:21:22.607 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:21:22.608 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:21:22.609 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:21:22.610 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:21:22.611 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.612 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.9900000095367432, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.9900000095367432, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.613 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 8, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_1", 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:18:32.502087+0000", 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:21:22.614 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:21:22.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "224", 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:21:22.616 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:21:22.617 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:21:22.618 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.9900000095367432, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.9900000095367432, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:21:22.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:21:22.620 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 218, 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.621 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:21:22.622 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:21:22.623 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:21:22.624 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:21:22.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:22.625 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:21:22.625 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.626 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 241, 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.627 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:21:22.628 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.630 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 0, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 81, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.631 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.632 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.633 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 239, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.635 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:21:22.636 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:21:22.639 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 235, 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:21:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:21:22.642 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.643 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 235, 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:21:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 235, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.651 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:21:22.652 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 235, 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:21:22.653 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:21:22.654 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.655 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.656 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:16.003017+0000", 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.657 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:34.067289+0000", 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.658 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 65536, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:53.354159+0000", 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.659 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:01.806103+0000", 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:21:22.660 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:34.634689+0000", 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:06.766983+0000", 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:20:10.106604+0000", 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:21:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:21:21.067808+0000", 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [], 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:21:22.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.664 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:21:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:21:22.666 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:21:22.667 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:21:22.668 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:22.669 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:22.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:22.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:22.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:22.811 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:22.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:22.868 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:22.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:22.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:22.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:23.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:23.079 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:23.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:23.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:23.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:21:23.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:21:23.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2967350046' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:21:23.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[29993]: osdmap e244: 8 total, 8 up, 7 in 2024-09-16T10:21:23.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:21:23.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:21:23.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2967350046' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:21:23.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:22 smithi063 ceph-mon[39470]: osdmap e244: 8 total, 8 up, 7 in 2024-09-16T10:21:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:23.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:23.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:22 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:21:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:22 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:21:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:22 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2967350046' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:21:23.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:22 smithi165 ceph-mon[32696]: osdmap e244: 8 total, 8 up, 7 in 2024-09-16T10:21:23.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:23.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:23.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:23.451 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:21:23.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:23.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:23.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:23.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:23.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:23.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:23.691 INFO:tasks.thrashosds.thrasher:j is {'epoch': 243, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:21:21.858041+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:18:51.797277+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 19, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 19, 'last_pg_merge_meta': {'source_pgid': '1.13', 'ready_epoch': 243, 'last_epoch_started': 242, 'last_epoch_clean': 242, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '243', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.0899999141693115, 'score_stable': 2.0899999141693115, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.840000033378601, 'raw_score_stable': 1.840000033378601, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-16T10:18:32.502087+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '224', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 241, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 235, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 235, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 235, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 235, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:16.003017+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:34.067289+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T10:20:53.354159+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:01.806103+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:34.634689+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:06.766983+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:20:10.106604+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:21:21.067808+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:21:23.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:21:23.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:23.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:23.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:23.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:23.930 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:23.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:23.963 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:21:24.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:24.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:24.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:24.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:24.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:24.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:23 smithi165 ceph-mon[32696]: 5.e scrub starts 2024-09-16T10:21:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:23 smithi165 ceph-mon[32696]: 5.e scrub ok 2024-09-16T10:21:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:23 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1068710241' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:23 smithi165 ceph-mon[32696]: pgmap v392: 166 pgs: 1 peering, 165 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail 2024-09-16T10:21:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:21:24.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:23 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:21:24.313 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:24.348 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:24.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:24.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:24.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[29993]: 5.e scrub starts 2024-09-16T10:21:24.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[29993]: 5.e scrub ok 2024-09-16T10:21:24.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1068710241' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:24.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[29993]: pgmap v392: 166 pgs: 1 peering, 165 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[39470]: 5.e scrub starts 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[39470]: 5.e scrub ok 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1068710241' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[39470]: pgmap v392: 166 pgs: 1 peering, 165 active+clean; 2.0 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:21:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:23 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:21:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:24.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:24.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:24.587 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:24.610 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:24.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:24.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:24.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:24.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:24.854 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:24.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:24.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:24.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:25.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:25.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:25.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:25.244 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:25.277 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1068710241' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: osdmap e245: 8 total, 8 up, 7 in 2024-09-16T10:21:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:21:25.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:24 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:21:25.379 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:25.414 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1068710241' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: osdmap e245: 8 total, 8 up, 7 in 2024-09-16T10:21:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1068710241' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: osdmap e245: 8 total, 8 up, 7 in 2024-09-16T10:21:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:21:25.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:24 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:21:25.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:25.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:25.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:25.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:25.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:25.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:25.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:25.966 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:25.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:26.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:26.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:26.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:26.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:26.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:26.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:26.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:25 smithi165 ceph-mon[32696]: osdmap e246: 8 total, 8 up, 7 in 2024-09-16T10:21:26.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:26 smithi165 ceph-mon[32696]: pgmap v395: 166 pgs: 166 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.83k op/s 2024-09-16T10:21:26.337 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:26.374 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:26.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:26.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:25 smithi063 ceph-mon[29993]: osdmap e246: 8 total, 8 up, 7 in 2024-09-16T10:21:26.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:25 smithi063 ceph-mon[29993]: pgmap v395: 166 pgs: 166 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.83k op/s 2024-09-16T10:21:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:25 smithi063 ceph-mon[39470]: osdmap e246: 8 total, 8 up, 7 in 2024-09-16T10:21:26.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:25 smithi063 ceph-mon[39470]: pgmap v395: 166 pgs: 166 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.83k op/s 2024-09-16T10:21:26.475 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:26.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:26.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:26.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:26.635 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:26.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:26.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:26.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:26.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:26.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:26.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:26.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:27.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:27.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:27.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:27.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:27.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:27.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:27.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:27.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:26 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:21:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:26 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:21:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:26 smithi165 ceph-mon[32696]: osdmap e247: 8 total, 8 up, 7 in 2024-09-16T10:21:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:26 smithi165 ceph-mon[32696]: 8.1 scrub starts 2024-09-16T10:21:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:26 smithi165 ceph-mon[32696]: 8.1 scrub ok 2024-09-16T10:21:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:27.325 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:27.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:27.360 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:21:27.361 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:21:27.361 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[29993]: osdmap e247: 8 total, 8 up, 7 in 2024-09-16T10:21:27.361 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[29993]: 8.1 scrub starts 2024-09-16T10:21:27.361 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[29993]: 8.1 scrub ok 2024-09-16T10:21:27.361 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:21:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:21:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:21:27.362 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:21:27.362 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:21:27.362 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[39470]: osdmap e247: 8 total, 8 up, 7 in 2024-09-16T10:21:27.362 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[39470]: 8.1 scrub starts 2024-09-16T10:21:27.362 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:26 smithi063 ceph-mon[39470]: 8.1 scrub ok 2024-09-16T10:21:27.427 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:27.476 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:27.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50949 8.a 8:59c98f0f:::benchmark_data_smithi063_101290_object50948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:21:27.537 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.26817211100001, 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0382288049999999, 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50949 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.539 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264644+0000", 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.978e-06 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264667+0000", 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3649999999999999e-05 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264681+0000", 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3423e-05 2024-09-16T10:21:27.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264688+0000", 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1010000000000003e-06 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265626+0000", 2024-09-16T10:21:27.541 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00093838999999999999 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265747+0000", 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120319 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.267052+0000", 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013057990000000001 2024-09-16T10:21:27.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.281048+0000", 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013995452 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283470+0000", 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024224540000000001 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.543 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283482+0000", 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1581e-05 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302818+0000", 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0193361350000001 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302871+0000", 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2522999999999998e-05 2024-09-16T10:21:27.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50954 8.a 8:57f268c8:::benchmark_data_smithi063_101290_object50953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.26481168399999, 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0425692, 2024-09-16T10:21:27.545 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50954 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:21:27.547 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.548 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.548 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.548 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268004+0000", 2024-09-16T10:21:27.548 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1050000000000002e-06 2024-09-16T10:21:27.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.548 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268031+0000", 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6468000000000001e-05 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268041+0000", 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0532e-05 2024-09-16T10:21:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268048+0000", 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7259999999999996e-06 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.550 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.270471+0000", 2024-09-16T10:21:27.551 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024233789999999998 2024-09-16T10:21:27.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.551 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.551 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275200+0000", 2024-09-16T10:21:27.551 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047289130000000004 2024-09-16T10:21:27.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275360+0000", 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000159531 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.552 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284217+0000", 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088568970000000007 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284226+0000", 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3559999999999999e-06 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.553 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.554 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.303128+0000", 2024-09-16T10:21:27.554 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0189021700000001 2024-09-16T10:21:27.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.554 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:21:27.554 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304057+0000", 2024-09-16T10:21:27.554 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00092923599999999995 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310454+0000", 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063961260000000002 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.555 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310571+0000", 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117761 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50965 8.a 8:5e5d9df4:::benchmark_data_smithi063_101290_object50964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.557 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:21:27.557 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.25435194900001, 2024-09-16T10:21:27.558 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0323928659999999, 2024-09-16T10:21:27.558 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.558 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.558 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.558 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50965 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:21:27.559 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278464+0000", 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.968e-06 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.560 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278491+0000", 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7061999999999999e-05 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278500+0000", 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4970000000000007e-06 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278507+0000", 2024-09-16T10:21:27.561 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5139999999999998e-06 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284249+0000", 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0057418360000000002 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284255+0000", 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4529999999999999e-06 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.562 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304143+0000", 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.019888039 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304299+0000", 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155651 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304463+0000", 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016370800000000001 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308275+0000", 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038120049999999998 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310595+0000", 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023207620000000001 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310754+0000", 2024-09-16T10:21:27.564 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015848500000000001 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310795+0000", 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1655e-05 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310855+0000", 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9231000000000003e-05 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50996 8.a 8:5adb42d1:::benchmark_data_smithi063_101290_object50995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.22772210799999, 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005949921, 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50996 2024-09-16T10:21:27.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305093+0000", 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.779e-06 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305117+0000", 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3370000000000002e-05 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305127+0000", 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0284e-05 2024-09-16T10:21:27.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305134+0000", 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9659999999999999e-06 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305192+0000", 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7587000000000003e-05 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305201+0000", 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2590000000000006e-06 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306679+0000", 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0014777290000001 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306790+0000", 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111675 2024-09-16T10:21:27.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306942+0000", 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015151300000000001 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310399+0000", 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034570529999999999 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310904+0000", 2024-09-16T10:21:27.571 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00050482900000000004 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310943+0000", 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9048000000000003e-05 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310974+0000", 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1164999999999997e-05 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311042+0000", 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7663999999999995e-05 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51001 8.a 8:59406931:::benchmark_data_smithi063_101290_object51000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.22360498899999, 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0021313270000001, 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.573 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51001 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309210+0000", 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2589999999999999e-06 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309227+0000", 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6765999999999998e-05 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309235+0000", 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1969999999999996e-06 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309240+0000", 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6290000000000001e-06 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309279+0000", 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9199999999999997e-05 2024-09-16T10:21:27.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309285+0000", 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8810000000000001e-06 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308323+0000", 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99903792999999996 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308464+0000", 2024-09-16T10:21:27.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014168000000000001 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308635+0000", 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000170383 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310405+0000", 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001770134 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311104+0000", 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069940900000000001 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311178+0000", 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3911000000000004e-05 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311237+0000", 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9113000000000001e-05 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311340+0000", 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102835 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51015 8.a 8:5f39a03a:::benchmark_data_smithi063_101290_object51014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.20729901999999, 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99112391399999999, 2024-09-16T10:21:27.580 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51015 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325516+0000", 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-07 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325537+0000", 2024-09-16T10:21:27.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1900999999999999e-05 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325554+0000", 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6937999999999999e-05 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325562+0000", 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3830000000000003e-06 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325628+0000", 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6418e-05 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325640+0000", 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2157000000000001e-05 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311387+0000", 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98574625999999999 2024-09-16T10:21:27.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311498+0000", 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111768 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311624+0000", 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012560199999999999 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316394+0000", 2024-09-16T10:21:27.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047700279999999999 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316457+0000", 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2819999999999998e-05 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316471+0000", 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4236000000000001e-05 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316522+0000", 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1001000000000002e-05 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316639+0000", 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116641 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51028 8.a 8:504c9504:::benchmark_data_smithi063_101290_object51027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:21:27.587 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.19891702300001, 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98302855600000005, 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51028 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333898+0000", 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.375e-06 2024-09-16T10:21:27.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333916+0000", 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8236e-05 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333925+0000", 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.585e-06 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333931+0000", 2024-09-16T10:21:27.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1070000000000001e-06 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334057+0000", 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012610900000000001 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334083+0000", 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5327e-05 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.312988+0000", 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97890501699999999 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313110+0000", 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012285099999999999 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313254+0000", 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014338900000000001 2024-09-16T10:21:27.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316650+0000", 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033961529999999998 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316706+0000", 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5878000000000003e-05 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316760+0000", 2024-09-16T10:21:27.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4502000000000001e-05 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316833+0000", 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2632000000000005e-05 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316925+0000", 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2394999999999999e-05 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.594 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51060 8.a 8:500fc7dc:::benchmark_data_smithi063_101290_object51059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.17591688900001, 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96027958700000005, 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51060 2024-09-16T10:21:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356899+0000", 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.821e-06 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356917+0000", 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7972999999999999e-05 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356926+0000", 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9679999999999995e-06 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356931+0000", 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356997+0000", 2024-09-16T10:21:27.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6079000000000002e-05 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.357005+0000", 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7569999999999993e-06 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314700+0000", 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95769431000000005 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314827+0000", 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012752799999999999 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314995+0000", 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016789100000000001 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316982+0000", 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986603 2024-09-16T10:21:27.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316999+0000", 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7319000000000001e-05 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317035+0000", 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5655000000000003e-05 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317088+0000", 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3278e-05 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317177+0000", 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8486999999999995e-05 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.603 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51073 8.a 8:53b2a9fa:::benchmark_data_smithi063_101290_object51072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.16478732600001, 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.957136019, 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.604 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51073 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368028+0000", 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5939999999999999e-06 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368053+0000", 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5130000000000002e-05 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368066+0000", 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2397e-05 2024-09-16T10:21:27.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368074+0000", 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2789999999999998e-06 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368143+0000", 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9558e-05 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368154+0000", 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0869999999999999e-05 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317228+0000", 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.949073273 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317346+0000", 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011878400000000001 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317477+0000", 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013062600000000001 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322458+0000", 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049812980000000003 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322513+0000", 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4561999999999997e-05 2024-09-16T10:21:27.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325080+0000", 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025674819999999998 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325125+0000", 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5065000000000003e-05 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325163+0000", 2024-09-16T10:21:27.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7101000000000003e-05 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51087 8.a 8:51205303:::benchmark_data_smithi063_101290_object51086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.153829793, 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94635271099999996, 2024-09-16T10:21:27.611 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51087 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378986+0000", 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9850000000000001e-06 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379013+0000", 2024-09-16T10:21:27.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6911e-05 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379029+0000", 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5928999999999999e-05 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379037+0000", 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7100000000000007e-06 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379115+0000", 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8634999999999995e-05 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379128+0000", 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2903e-05 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319060+0000", 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93993237200000002 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319187+0000", 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012621 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319358+0000", 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017167699999999999 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322553+0000", 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003194625 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325067+0000", 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025139170000000001 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325170+0000", 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102948 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325265+0000", 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5570999999999997e-05 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325337+0000", 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1317999999999997e-05 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51105 8.a 8:558fcaa8:::benchmark_data_smithi063_101290_object51104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.139937538, 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93274633699999998, 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51105 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.619 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392878+0000", 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.516e-06 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392900+0000", 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2575e-05 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392913+0000", 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2196e-05 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392920+0000", 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7170000000000008e-06 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393546+0000", 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00062592699999999997 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393555+0000", 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7809999999999994e-06 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320780+0000", 2024-09-16T10:21:27.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92722495100000002 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320895+0000", 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115375 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.321051+0000", 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155482 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325345+0000", 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042937840000000001 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325384+0000", 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9261999999999998e-05 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325522+0000", 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0001383 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325584+0000", 2024-09-16T10:21:27.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-05 2024-09-16T10:21:27.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325623+0000", 2024-09-16T10:21:27.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8940999999999999e-05 2024-09-16T10:21:27.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.627 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.628 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51170 8.a 8:55be5a3b:::benchmark_data_smithi063_101290_object51169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.628 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:21:27.628 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.08398965699999, 2024-09-16T10:21:27.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87700012299999996, 2024-09-16T10:21:27.628 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.629 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.629 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.629 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.629 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.629 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.629 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51170 2024-09-16T10:21:27.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:21:27.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448826+0000", 2024-09-16T10:21:27.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7749999999999999e-06 2024-09-16T10:21:27.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448849+0000", 2024-09-16T10:21:27.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3298999999999998e-05 2024-09-16T10:21:27.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448867+0000", 2024-09-16T10:21:27.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7467999999999999e-05 2024-09-16T10:21:27.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448874+0000", 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-06 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448936+0000", 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1383999999999994e-05 2024-09-16T10:21:27.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448947+0000", 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1756000000000001e-05 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322695+0000", 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87374773699999997 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322825+0000", 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000129579 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322994+0000", 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016874899999999999 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325349+0000", 2024-09-16T10:21:27.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002355514 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325668+0000", 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000319034 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325723+0000", 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5376e-05 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325775+0000", 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1924e-05 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325824+0000", 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8918000000000002e-05 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.638 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51179 8.a 8:5e35a76f:::benchmark_data_smithi063_101290_object51178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.01969432600001, 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82446452000000003, 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.639 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51179 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513122+0000", 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5579999999999999e-06 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513149+0000", 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7101e-05 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513166+0000", 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7017e-05 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513175+0000", 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6209999999999993e-06 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513268+0000", 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3287000000000003e-05 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513279+0000", 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0889e-05 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325865+0000", 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.812586527 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325989+0000", 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123347 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.326103+0000", 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011419199999999999 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331852+0000", 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005749466 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331901+0000", 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8854999999999999e-05 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337506+0000", 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005604876 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337547+0000", 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0961999999999998e-05 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337584+0000", 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6822000000000001e-05 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.646 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51188 8.a 8:5734c940:::benchmark_data_smithi063_101290_object51187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 135.00967818399999, 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.814521685, 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51188 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523137+0000", 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.804e-06 2024-09-16T10:21:27.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523159+0000", 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1447e-05 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523171+0000", 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2347000000000001e-05 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523179+0000", 2024-09-16T10:21:27.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6310000000000003e-06 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523240+0000", 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1150999999999998e-05 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523248+0000", 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5190000000000003e-06 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.327963+0000", 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.80471533100000003 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328066+0000", 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103287 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328230+0000", 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163521 2024-09-16T10:21:27.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331938+0000", 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037081639999999999 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331973+0000", 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5355999999999998e-05 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337597+0000", 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056237809999999996 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337637+0000", 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.019e-05 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337657+0000", 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0156000000000001e-05 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51286 8.1 8:897e7f6e:::benchmark_data_smithi063_101290_object51285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 133.215787224, 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.983161858, 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51286 2024-09-16T10:21:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317028+0000", 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.141e-06 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317054+0000", 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6186e-05 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317065+0000", 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.08e-05 2024-09-16T10:21:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317072+0000", 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3520000000000001e-06 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317688+0000", 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061638600000000002 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317698+0000", 2024-09-16T10:21:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.346e-06 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292346+0000", 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97464779999999995 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292464+0000", 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118582 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292563+0000", 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9047000000000002e-05 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296570+0000", 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040065379999999996 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296637+0000", 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7547999999999998e-05 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300071+0000", 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003433298 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300139+0000", 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8702999999999996e-05 2024-09-16T10:21:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300188+0000", 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9131000000000001e-05 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51312 8.1 8:8f097c26:::benchmark_data_smithi063_101290_object51311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 133.196753573, 2024-09-16T10:21:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96891611600000005, 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51312 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336063+0000", 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3949999999999999e-06 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336103+0000", 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0754000000000003e-05 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336138+0000", 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4662000000000001e-05 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336160+0000", 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1752000000000001e-05 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336277+0000", 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117586 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336288+0000", 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0852999999999999e-05 2024-09-16T10:21:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294143+0000", 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95785426600000001 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294236+0000", 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3010000000000006e-05 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294323+0000", 2024-09-16T10:21:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7619000000000003e-05 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298376+0000", 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004053175 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300258+0000", 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018817300000000001 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304863+0000", 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046049539999999996 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3490000000000004e-05 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304976+0000", 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9870000000000002e-05 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51316 8.1 8:858e6b14:::benchmark_data_smithi063_101290_object51315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 133.19543359599999, 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96772945600000004, 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51316 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337381+0000", 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.192e-06 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337441+0000", 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9988000000000002e-05 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337452+0000", 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0857e-05 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337460+0000", 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2649999999999999e-06 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337520+0000", 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0377999999999997e-05 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337530+0000", 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0416e-05 2024-09-16T10:21:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295200+0000", 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95766991499999998 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295307+0000", 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010660199999999999 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295446+0000", 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013906699999999999 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300014+0000", 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045680249999999999 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300329+0000", 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031521199999999999 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305018+0000", 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046893569999999999 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305073+0000", 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4243000000000002e-05 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305110+0000", 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6939e-05 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51363 8.1 8:83ceb830:::benchmark_data_smithi063_101290_object51362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:21:27.677 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 133.09390757099999, 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.86632090799999995, 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51363 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438908+0000", 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.454e-06 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438931+0000", 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3197000000000002e-05 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438945+0000", 2024-09-16T10:21:27.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3703000000000001e-05 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438952+0000", 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5259999999999996e-06 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439023+0000", 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0457000000000007e-05 2024-09-16T10:21:27.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439034+0000", 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1141000000000001e-05 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296689+0000", 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.85765570300000005 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296811+0000", 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121229 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296957+0000", 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014621999999999999 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300288+0000", 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033314 2024-09-16T10:21:27.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300372+0000", 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3277999999999998e-05 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305147+0000", 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047752990000000002 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305194+0000", 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6789999999999998e-05 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305227+0000", 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3510999999999998e-05 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51388 8.1 8:8d4128ae:::benchmark_data_smithi063_101290_object51387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 133.06919273700001, 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.84172315399999997, 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.686 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.687 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.687 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51388 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463623+0000", 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463644+0000", 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0937999999999998e-05 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463654+0000", 2024-09-16T10:21:27.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0390999999999999e-05 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463661+0000", 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2400000000000001e-06 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463714+0000", 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3059999999999997e-05 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463722+0000", 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4769999999999997e-06 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298409+0000", 2024-09-16T10:21:27.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.83468682000000005 2024-09-16T10:21:27.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298629+0000", 2024-09-16T10:21:27.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220258 2024-09-16T10:21:27.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:27.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298774+0000", 2024-09-16T10:21:27.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145012 2024-09-16T10:21:27.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300340+0000", 2024-09-16T10:21:27.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001566352 2024-09-16T10:21:27.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302114+0000", 2024-09-16T10:21:27.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001773518 2024-09-16T10:21:27.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305261+0000", 2024-09-16T10:21:27.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003147721 2024-09-16T10:21:27.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305310+0000", 2024-09-16T10:21:27.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9051000000000002e-05 2024-09-16T10:21:27.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305344+0000", 2024-09-16T10:21:27.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3751000000000003e-05 2024-09-16T10:21:27.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.697 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.698 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51396 8.1 8:8a8d0cb2:::benchmark_data_smithi063_101290_object51395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:21:27.698 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:21:27.699 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 133.05040063600001, 2024-09-16T10:21:27.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82304446200000003, 2024-09-16T10:21:27.699 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:27.699 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:27.699 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:27.699 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:27.699 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:27.700 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:27.700 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51396 2024-09-16T10:21:27.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.700 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:27.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:27.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:21:27.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:27.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:21:27.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:27.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:27.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482414+0000", 2024-09-16T10:21:27.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2300000000000001e-06 2024-09-16T10:21:27.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:27.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482433+0000", 2024-09-16T10:21:27.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8876e-05 2024-09-16T10:21:27.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:27.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482445+0000", 2024-09-16T10:21:27.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2170999999999999e-05 2024-09-16T10:21:27.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:27.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482453+0000", 2024-09-16T10:21:27.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8939999999999993e-06 2024-09-16T10:21:27.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482504+0000", 2024-09-16T10:21:27.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0698000000000001e-05 2024-09-16T10:21:27.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:27.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482512+0000", 2024-09-16T10:21:27.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1319999999999994e-06 2024-09-16T10:21:27.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:27.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300411+0000", 2024-09-16T10:21:27.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.81789900699999996 2024-09-16T10:21:27.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:27.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300560+0000", 2024-09-16T10:21:27.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014893799999999999 2024-09-16T10:21:27.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:27.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300718+0000", 2024-09-16T10:21:27.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015829700000000001 2024-09-16T10:21:27.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302254+0000", 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015359429999999999 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302509+0000", 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025458499999999999 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305379+0000", 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028699329999999999 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305426+0000", 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7330000000000003e-05 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305458+0000", 2024-09-16T10:21:27.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1427999999999998e-05 2024-09-16T10:21:27.712 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.712 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.712 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.712 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:27.712 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:27.712 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:27.714 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:27.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:27.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:27.847 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:27.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:27.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:27.983 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:27.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:28.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:28.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:28.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:28.223 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:28.270 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:28.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:28.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:28.273 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: osdmap e248: 8 total, 8 up, 7 in 2024-09-16T10:21:28.273 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: 4.3 scrub starts 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: 4.3 scrub ok 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: pgmap v398: 165 pgs: 165 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.83k op/s 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: 8.a scrub starts 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[29993]: 8.a scrub ok 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: osdmap e248: 8 total, 8 up, 7 in 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: 4.3 scrub starts 2024-09-16T10:21:28.274 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: 4.3 scrub ok 2024-09-16T10:21:28.275 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: pgmap v398: 165 pgs: 165 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.83k op/s 2024-09-16T10:21:28.275 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:21:28.275 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:21:28.275 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: 8.a scrub starts 2024-09-16T10:21:28.275 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:28 smithi063 ceph-mon[39470]: 8.a scrub ok 2024-09-16T10:21:28.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: osdmap e248: 8 total, 8 up, 7 in 2024-09-16T10:21:28.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: 4.3 scrub starts 2024-09-16T10:21:28.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: 4.3 scrub ok 2024-09-16T10:21:28.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: pgmap v398: 165 pgs: 165 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.83k op/s 2024-09-16T10:21:28.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:21:28.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:21:28.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: 8.a scrub starts 2024-09-16T10:21:28.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-mon[32696]: 8.a scrub ok 2024-09-16T10:21:28.372 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:28.408 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:28.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:28.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:28.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:28.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:28.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:28.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:28.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:28.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:28.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:21:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:21:28.832 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:28.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:28.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:28.967 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:28.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:29.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:29.103 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:29.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:29.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:29.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:29.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:29.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:29 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-16T10:21:29.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:29 smithi165 ceph-mon[32696]: osdmap e249: 8 total, 8 up, 7 in 2024-09-16T10:21:29.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:29.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:29 smithi165 ceph-mon[32696]: osdmap e250: 8 total, 8 up, 7 in 2024-09-16T10:21:29.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-16T10:21:29.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[29993]: osdmap e249: 8 total, 8 up, 7 in 2024-09-16T10:21:29.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:29.283 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[29993]: osdmap e250: 8 total, 8 up, 7 in 2024-09-16T10:21:29.285 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-16T10:21:29.285 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[39470]: osdmap e249: 8 total, 8 up, 7 in 2024-09-16T10:21:29.285 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:29.285 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:29 smithi063 ceph-mon[39470]: osdmap e250: 8 total, 8 up, 7 in 2024-09-16T10:21:29.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:21:29.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:29.405 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:29.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:29.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:29.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:29.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:29.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:29.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:29.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:29.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:29.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:29.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:29.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:29.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:29.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:30.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:30.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:30.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:30.130 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[29993]: pgmap v401: 165 pgs: 2 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-16T10:21:30.130 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[29993]: 3.8 scrub starts 2024-09-16T10:21:30.130 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[29993]: 3.8 scrub ok 2024-09-16T10:21:30.130 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3650986540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:30.130 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[39470]: pgmap v401: 165 pgs: 2 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-16T10:21:30.130 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[39470]: 3.8 scrub starts 2024-09-16T10:21:30.130 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[39470]: 3.8 scrub ok 2024-09-16T10:21:30.131 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3650986540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:30.163 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:30.163 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:30.163 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:30.163 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:30.163 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:30.163 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:30.163 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:30.164 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:30.164 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:30.164 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:30.166 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:30.166 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:30.182 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:30.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:30.203 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:30.204 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:30.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:30.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:30.237 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:30.237 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:30.251 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:30.259 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:21:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:30 smithi165 ceph-mon[32696]: pgmap v401: 165 pgs: 2 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-16T10:21:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:30 smithi165 ceph-mon[32696]: 3.8 scrub starts 2024-09-16T10:21:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:30 smithi165 ceph-mon[32696]: 3.8 scrub ok 2024-09-16T10:21:30.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3650986540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:30.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:30.383 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:30.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:30.484 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:30.521 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:30.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:30.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:30.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:30.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:30.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:30.791 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:30.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:30.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:30.923 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:30.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:31.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:31.057 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:31.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:31.159 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:31.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:31.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:31.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:31.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:31 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:21:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:31 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:21:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:31 smithi165 ceph-mon[32696]: osdmap e251: 8 total, 8 up, 7 in 2024-09-16T10:21:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:31 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:21:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:31 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:21:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:31 smithi165 ceph-mon[32696]: osdmap e252: 8 total, 8 up, 7 in 2024-09-16T10:21:31.323 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:31.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:31.425 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:31.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[29993]: osdmap e251: 8 total, 8 up, 7 in 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[29993]: osdmap e252: 8 total, 8 up, 7 in 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[39470]: osdmap e251: 8 total, 8 up, 7 in 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:21:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:21:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:31 smithi063 ceph-mon[39470]: osdmap e252: 8 total, 8 up, 7 in 2024-09-16T10:21:31.465 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:31.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:31.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:31.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:31.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:31.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:31.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:31.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:31.844 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:31.892 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:31.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:31.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:32.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:32.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:32.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:32.168 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:32.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 1.a scrub starts 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 1.a scrub ok 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 8.2 scrub starts 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 6.0 scrub starts 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 6.0 scrub ok 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: pgmap v404: 164 pgs: 1 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 8.2 scrub ok 2024-09-16T10:21:32.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:21:32.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:21:32.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:21:32.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 1.a scrub starts 2024-09-16T10:21:32.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 1.a scrub ok 2024-09-16T10:21:32.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 8.2 scrub starts 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 6.0 scrub starts 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 6.0 scrub ok 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: pgmap v404: 164 pgs: 1 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 8.2 scrub ok 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:21:32.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:32 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:21:32.272 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:32.301 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 1.a scrub starts 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 1.a scrub ok 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 8.2 scrub starts 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 6.0 scrub starts 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 6.0 scrub ok 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: pgmap v404: 164 pgs: 1 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 8.2 scrub ok 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:21:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:21:32.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:32 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:21:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:32.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:21:32.371 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:21:32.404 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:32.457 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:32.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:32.535 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:21:32.535 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:21:32.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:32.596 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:32.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:32.698 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:32.739 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:32.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:32.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:32.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:32.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:32.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.013 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:33.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:33.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:33.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:33.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:33.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:33.353 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[39470]: 4.c scrub starts 2024-09-16T10:21:33.353 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[39470]: 4.c scrub ok 2024-09-16T10:21:33.353 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-16T10:21:33.353 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[39470]: osdmap e253: 8 total, 8 up, 7 in 2024-09-16T10:21:33.354 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[39470]: from='client.45512 v1:172.21.15.63:0/2503843296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:21:33.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[29993]: 4.c scrub starts 2024-09-16T10:21:33.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[29993]: 4.c scrub ok 2024-09-16T10:21:33.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-16T10:21:33.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[29993]: osdmap e253: 8 total, 8 up, 7 in 2024-09-16T10:21:33.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:33 smithi063 ceph-mon[29993]: from='client.45512 v1:172.21.15.63:0/2503843296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:21:33.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:33.400 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-16T10:21:32.039132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1215502899,"num_objects":18738,"num_object_clones":0,"num_object_copies":56214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18738,"num_whiteouts":0,"num_read":1187,"num_read_kb":939,"num_write":236533,"num_write_kb":8148334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10191,"num_bytes_recovered":663685406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70529,"ondisk_log_size":70529,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":496,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5764032,"kb_used_data":3714164,"kb_used_omap":10,"kb_used_meta":2049333,"kb_avail":650452032,"statfs":{"total":671965249536,"available":666062880768,"internally_reserved":0,"allocated":3803303936,"data_stored":3798723092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":2098517100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,9,8,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-447283200,"num_objects":-6825,"num_object_clones":0,"num_object_copies":-20475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6825,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6825,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040456"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":425,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010191+0000","last_change":"2024-09-16T10:18:52.830567+0000","last_active":"2024-09-16T10:21:28.010191+0000","last_peered":"2024-09-16T10:21:28.010191+0000","last_clean":"2024-09-16T10:21:28.010191+0000","last_became_active":"2024-09-16T10:18:52.830260+0000","last_became_peered":"2024-09-16T10:18:52.830260+0000","last_unstale":"2024-09-16T10:21:28.010191+0000","last_undegraded":"2024-09-16T10:21:28.010191+0000","last_fullsized":"2024-09-16T10:21:28.010191+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:07.119072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.119072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:08.252838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":472,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:27.999028+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:21:27.999028+0000","last_peered":"2024-09-16T10:21:27.999028+0000","last_clean":"2024-09-16T10:21:27.999028+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:21:27.999028+0000","last_undegraded":"2024-09-16T10:21:27.999028+0000","last_fullsized":"2024-09-16T10:21:27.999028+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.009632+0000","last_change":"2024-09-16T10:18:40.752790+0000","last_active":"2024-09-16T10:21:28.009632+0000","last_peered":"2024-09-16T10:21:28.009632+0000","last_clean":"2024-09-16T10:21:28.009632+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:21:28.009632+0000","last_undegraded":"2024-09-16T10:21:28.009632+0000","last_fullsized":"2024-09-16T10:21:28.009632+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:40.752739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:40.752739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:02.230537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.000254+0000","last_change":"2024-09-16T10:21:25.880760+0000","last_active":"2024-09-16T10:21:28.000254+0000","last_peered":"2024-09-16T10:21:28.000254+0000","last_clean":"2024-09-16T10:21:28.000254+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:21:28.000254+0000","last_undegraded":"2024-09-16T10:21:28.000254+0000","last_fullsized":"2024-09-16T10:21:28.000254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:25.880690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:25.880690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:35.656146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":407,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.008795+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:21:28.008795+0000","last_peered":"2024-09-16T10:21:28.008795+0000","last_clean":"2024-09-16T10:21:28.008795+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:21:28.008795+0000","last_undegraded":"2024-09-16T10:21:28.008795+0000","last_fullsized":"2024-09-16T10:21:28.008795+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:39.443408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":426,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002843+0000","last_change":"2024-09-16T10:20:33.257003+0000","last_active":"2024-09-16T10:21:28.002843+0000","last_peered":"2024-09-16T10:21:28.002843+0000","last_clean":"2024-09-16T10:21:28.002843+0000","last_became_active":"2024-09-16T10:18:52.836507+0000","last_became_peered":"2024-09-16T10:18:52.836507+0000","last_unstale":"2024-09-16T10:21:28.002843+0000","last_undegraded":"2024-09-16T10:21:28.002843+0000","last_fullsized":"2024-09-16T10:21:28.002843+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.256934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.256934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:55.621693+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":463,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002542+0000","last_change":"2024-09-16T10:20:31.270635+0000","last_active":"2024-09-16T10:21:28.002542+0000","last_peered":"2024-09-16T10:21:28.002542+0000","last_clean":"2024-09-16T10:21:28.002542+0000","last_became_active":"2024-09-16T10:18:52.835715+0000","last_became_peered":"2024-09-16T10:18:52.835715+0000","last_unstale":"2024-09-16T10:21:28.002542+0000","last_undegraded":"2024-09-16T10:21:28.002542+0000","last_fullsized":"2024-09-16T10:21:28.002542+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:20:31.270548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.270548+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:59.374039+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:27.998864+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:21:27.998864+0000","last_peered":"2024-09-16T10:21:27.998864+0000","last_clean":"2024-09-16T10:21:27.998864+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:21:27.998864+0000","last_undegraded":"2024-09-16T10:21:27.998864+0000","last_fullsized":"2024-09-16T10:21:27.998864+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:16.874460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":427,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007998+0000","last_change":"2024-09-16T10:21:19.427397+0000","last_active":"2024-09-16T10:21:28.007998+0000","last_peered":"2024-09-16T10:21:28.007998+0000","last_clean":"2024-09-16T10:21:28.007998+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:21:28.007998+0000","last_undegraded":"2024-09-16T10:21:28.007998+0000","last_fullsized":"2024-09-16T10:21:28.007998+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:19.427214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.427214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:35.414719+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":430,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.000615+0000","last_change":"2024-09-16T10:20:30.008294+0000","last_active":"2024-09-16T10:21:28.000615+0000","last_peered":"2024-09-16T10:21:28.000615+0000","last_clean":"2024-09-16T10:21:28.000615+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:21:28.000615+0000","last_undegraded":"2024-09-16T10:21:28.000615+0000","last_fullsized":"2024-09-16T10:21:28.000615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:30.008215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:30.008215+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.975169+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.005884+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:21:28.005884+0000","last_peered":"2024-09-16T10:21:28.005884+0000","last_clean":"2024-09-16T10:21:28.005884+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:21:28.005884+0000","last_undegraded":"2024-09-16T10:21:28.005884+0000","last_fullsized":"2024-09-16T10:21:28.005884+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.399478+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.005922+0000","last_change":"2024-09-16T10:20:33.259383+0000","last_active":"2024-09-16T10:21:28.005922+0000","last_peered":"2024-09-16T10:21:28.005922+0000","last_clean":"2024-09-16T10:21:28.005922+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:21:28.005922+0000","last_undegraded":"2024-09-16T10:21:28.005922+0000","last_fullsized":"2024-09-16T10:21:28.005922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.259340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.259340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:59.265290+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":395,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.008202+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:21:28.008202+0000","last_peered":"2024-09-16T10:21:28.008202+0000","last_clean":"2024-09-16T10:21:28.008202+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:21:28.008202+0000","last_undegraded":"2024-09-16T10:21:28.008202+0000","last_fullsized":"2024-09-16T10:21:28.008202+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.350155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":424,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.001051+0000","last_change":"2024-09-16T10:20:32.001472+0000","last_active":"2024-09-16T10:21:28.001051+0000","last_peered":"2024-09-16T10:21:28.001051+0000","last_clean":"2024-09-16T10:21:28.001051+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:21:28.001051+0000","last_undegraded":"2024-09-16T10:21:28.001051+0000","last_fullsized":"2024-09-16T10:21:28.001051+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:20:32.001398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:32.001398+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.377925+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012635+0000","last_change":"2024-09-16T10:19:23.390542+0000","last_active":"2024-09-16T10:21:28.012635+0000","last_peered":"2024-09-16T10:21:28.012635+0000","last_clean":"2024-09-16T10:21:28.012635+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:21:28.012635+0000","last_undegraded":"2024-09-16T10:21:28.012635+0000","last_fullsized":"2024-09-16T10:21:28.012635+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:03.365502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":422,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.009095+0000","last_change":"2024-09-16T10:21:20.402146+0000","last_active":"2024-09-16T10:21:28.009095+0000","last_peered":"2024-09-16T10:21:28.009095+0000","last_clean":"2024-09-16T10:21:28.009095+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:21:28.009095+0000","last_undegraded":"2024-09-16T10:21:28.009095+0000","last_fullsized":"2024-09-16T10:21:28.009095+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.402080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.402080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:34.120569+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002358+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:21:28.002358+0000","last_peered":"2024-09-16T10:21:28.002358+0000","last_clean":"2024-09-16T10:21:28.002358+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:21:28.002358+0000","last_undegraded":"2024-09-16T10:21:28.002358+0000","last_fullsized":"2024-09-16T10:21:28.002358+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007219+0000","last_change":"2024-09-16T10:20:57.263993+0000","last_active":"2024-09-16T10:21:28.007219+0000","last_peered":"2024-09-16T10:21:28.007219+0000","last_clean":"2024-09-16T10:21:28.007219+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:21:28.007219+0000","last_undegraded":"2024-09-16T10:21:28.007219+0000","last_fullsized":"2024-09-16T10:21:28.007219+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.263916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.263916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:25.361547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":231,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004828+0000","last_change":"2024-09-16T10:19:14.579717+0000","last_active":"2024-09-16T10:21:28.004828+0000","last_peered":"2024-09-16T10:21:28.004828+0000","last_clean":"2024-09-16T10:21:28.004828+0000","last_became_active":"2024-09-16T10:18:52.841218+0000","last_became_peered":"2024-09-16T10:18:52.841218+0000","last_unstale":"2024-09-16T10:21:28.004828+0000","last_undegraded":"2024-09-16T10:21:28.004828+0000","last_fullsized":"2024-09-16T10:21:28.004828+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.441526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004996+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:21:28.004996+0000","last_peered":"2024-09-16T10:21:28.004996+0000","last_clean":"2024-09-16T10:21:28.004996+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:21:28.004996+0000","last_undegraded":"2024-09-16T10:21:28.004996+0000","last_fullsized":"2024-09-16T10:21:28.004996+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.286414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":356,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003416+0000","last_change":"2024-09-16T10:19:48.110737+0000","last_active":"2024-09-16T10:21:28.003416+0000","last_peered":"2024-09-16T10:21:28.003416+0000","last_clean":"2024-09-16T10:21:28.003416+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:21:28.003416+0000","last_undegraded":"2024-09-16T10:21:28.003416+0000","last_fullsized":"2024-09-16T10:21:28.003416+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:48.110674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:48.110674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":409,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.000747+0000","last_change":"2024-09-16T10:19:22.803999+0000","last_active":"2024-09-16T10:21:28.000747+0000","last_peered":"2024-09-16T10:21:28.000747+0000","last_clean":"2024-09-16T10:21:28.000747+0000","last_became_active":"2024-09-16T10:18:52.834723+0000","last_became_peered":"2024-09-16T10:18:52.834723+0000","last_unstale":"2024-09-16T10:21:28.000747+0000","last_undegraded":"2024-09-16T10:21:28.000747+0000","last_fullsized":"2024-09-16T10:21:28.000747+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":465,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002507+0000","last_change":"2024-09-16T10:20:57.423064+0000","last_active":"2024-09-16T10:21:28.002507+0000","last_peered":"2024-09-16T10:21:28.002507+0000","last_clean":"2024-09-16T10:21:28.002507+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:21:28.002507+0000","last_undegraded":"2024-09-16T10:21:28.002507+0000","last_fullsized":"2024-09-16T10:21:28.002507+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:20:57.422968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.422968+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:21.471811+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":417,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002261+0000","last_change":"2024-09-16T10:18:52.839387+0000","last_active":"2024-09-16T10:21:28.002261+0000","last_peered":"2024-09-16T10:21:28.002261+0000","last_clean":"2024-09-16T10:21:28.002261+0000","last_became_active":"2024-09-16T10:18:52.838765+0000","last_became_peered":"2024-09-16T10:18:52.838765+0000","last_unstale":"2024-09-16T10:21:28.002261+0000","last_undegraded":"2024-09-16T10:21:28.002261+0000","last_fullsized":"2024-09-16T10:21:28.002261+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:05.795954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.795954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.920993+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013700+0000","last_change":"2024-09-16T10:18:52.847158+0000","last_active":"2024-09-16T10:21:28.013700+0000","last_peered":"2024-09-16T10:21:28.013700+0000","last_clean":"2024-09-16T10:21:28.013700+0000","last_became_active":"2024-09-16T10:18:52.838789+0000","last_became_peered":"2024-09-16T10:18:52.838789+0000","last_unstale":"2024-09-16T10:21:28.013700+0000","last_undegraded":"2024-09-16T10:21:28.013700+0000","last_fullsized":"2024-09-16T10:21:28.013700+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.084681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.009782+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:21:28.009782+0000","last_peered":"2024-09-16T10:21:28.009782+0000","last_clean":"2024-09-16T10:21:28.009782+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:21:28.009782+0000","last_undegraded":"2024-09-16T10:21:28.009782+0000","last_fullsized":"2024-09-16T10:21:28.009782+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:37.450673+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":429,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010629+0000","last_change":"2024-09-16T10:19:49.067027+0000","last_active":"2024-09-16T10:21:28.010629+0000","last_peered":"2024-09-16T10:21:28.010629+0000","last_clean":"2024-09-16T10:21:28.010629+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:21:28.010629+0000","last_undegraded":"2024-09-16T10:21:28.010629+0000","last_fullsized":"2024-09-16T10:21:28.010629+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:49.066953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:49.066953+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":516,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012303+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:21:28.012303+0000","last_peered":"2024-09-16T10:21:28.012303+0000","last_clean":"2024-09-16T10:21:28.012303+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:21:28.012303+0000","last_undegraded":"2024-09-16T10:21:28.012303+0000","last_fullsized":"2024-09-16T10:21:28.012303+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.628369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"249'14878","reported_seq":15059,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.484716+0000","last_change":"2024-09-16T10:18:33.825761+0000","last_active":"2024-09-16T10:21:28.484716+0000","last_peered":"2024-09-16T10:21:28.484716+0000","last_clean":"2024-09-16T10:21:28.484716+0000","last_became_active":"2024-09-16T10:18:33.825267+0000","last_became_peered":"2024-09-16T10:18:33.825267+0000","last_unstale":"2024-09-16T10:21:28.484716+0000","last_undegraded":"2024-09-16T10:21:28.484716+0000","last_fullsized":"2024-09-16T10:21:28.484716+0000","mapping_epoch":165,"log_start":"222'10513","ondisk_log_start":"222'10513","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:45.304329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77201408,"num_objects":1178,"num_object_clones":0,"num_object_copies":3534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14878,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":232,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007115+0000","last_change":"2024-09-16T10:20:26.408565+0000","last_active":"2024-09-16T10:21:28.007115+0000","last_peered":"2024-09-16T10:21:28.007115+0000","last_clean":"2024-09-16T10:21:28.007115+0000","last_became_active":"2024-09-16T10:18:52.825707+0000","last_became_peered":"2024-09-16T10:18:52.825707+0000","last_unstale":"2024-09-16T10:21:28.007115+0000","last_undegraded":"2024-09-16T10:21:28.007115+0000","last_fullsized":"2024-09-16T10:21:28.007115+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:26.408503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.408503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:50.763370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":432,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.009895+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:21:28.009895+0000","last_peered":"2024-09-16T10:21:28.009895+0000","last_clean":"2024-09-16T10:21:28.009895+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:21:28.009895+0000","last_undegraded":"2024-09-16T10:21:28.009895+0000","last_fullsized":"2024-09-16T10:21:28.009895+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.426229+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004216+0000","last_change":"2024-09-16T10:19:43.028508+0000","last_active":"2024-09-16T10:21:28.004216+0000","last_peered":"2024-09-16T10:21:28.004216+0000","last_clean":"2024-09-16T10:21:28.004216+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:21:28.004216+0000","last_undegraded":"2024-09-16T10:21:28.004216+0000","last_fullsized":"2024-09-16T10:21:28.004216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:43.028436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:43.028436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":505,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002449+0000","last_change":"2024-09-16T10:21:22.928090+0000","last_active":"2024-09-16T10:21:28.002449+0000","last_peered":"2024-09-16T10:21:28.002449+0000","last_clean":"2024-09-16T10:21:28.002449+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:21:28.002449+0000","last_undegraded":"2024-09-16T10:21:28.002449+0000","last_fullsized":"2024-09-16T10:21:28.002449+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:21:22.928016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:22.928016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:40.025296+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007475+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:21:28.007475+0000","last_peered":"2024-09-16T10:21:28.007475+0000","last_clean":"2024-09-16T10:21:28.007475+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:21:28.007475+0000","last_undegraded":"2024-09-16T10:21:28.007475+0000","last_fullsized":"2024-09-16T10:21:28.007475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:04.486610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"249'14729","reported_seq":14961,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.640743+0000","last_change":"2024-09-16T10:20:52.238795+0000","last_active":"2024-09-16T10:21:28.640743+0000","last_peered":"2024-09-16T10:21:28.640743+0000","last_clean":"2024-09-16T10:21:28.640743+0000","last_became_active":"2024-09-16T10:18:33.893281+0000","last_became_peered":"2024-09-16T10:18:33.893281+0000","last_unstale":"2024-09-16T10:21:28.640743+0000","last_undegraded":"2024-09-16T10:21:28.640743+0000","last_fullsized":"2024-09-16T10:21:28.640743+0000","mapping_epoch":165,"log_start":"222'10285","ondisk_log_start":"222'10285","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'11541","last_scrub_stamp":"2024-09-16T10:20:52.238522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.238522+0000","objects_scrubbed":4319,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:08.553927+0000","scrub_duration":1012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71499776,"num_objects":1091,"num_object_clones":0,"num_object_copies":3273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14729,"num_write_kb":506240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":244,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004909+0000","last_change":"2024-09-16T10:20:48.845379+0000","last_active":"2024-09-16T10:21:28.004909+0000","last_peered":"2024-09-16T10:21:28.004909+0000","last_clean":"2024-09-16T10:21:28.004909+0000","last_became_active":"2024-09-16T10:20:15.864502+0000","last_became_peered":"2024-09-16T10:20:15.864502+0000","last_unstale":"2024-09-16T10:21:28.004909+0000","last_undegraded":"2024-09-16T10:21:28.004909+0000","last_fullsized":"2024-09-16T10:21:28.004909+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.845331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.845331+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:11.387905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":439,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003958+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:21:28.003958+0000","last_peered":"2024-09-16T10:21:28.003958+0000","last_clean":"2024-09-16T10:21:28.003958+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:21:28.003958+0000","last_undegraded":"2024-09-16T10:21:28.003958+0000","last_fullsized":"2024-09-16T10:21:28.003958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:03.424262+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":376,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003994+0000","last_change":"2024-09-16T10:19:50.114162+0000","last_active":"2024-09-16T10:21:28.003994+0000","last_peered":"2024-09-16T10:21:28.003994+0000","last_clean":"2024-09-16T10:21:28.003994+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:21:28.003994+0000","last_undegraded":"2024-09-16T10:21:28.003994+0000","last_fullsized":"2024-09-16T10:21:28.003994+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.114098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.114098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":157,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010981+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:21:28.010981+0000","last_peered":"2024-09-16T10:21:28.010981+0000","last_clean":"2024-09-16T10:21:28.010981+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:21:28.010981+0000","last_undegraded":"2024-09-16T10:21:28.010981+0000","last_fullsized":"2024-09-16T10:21:28.010981+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:36.618613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":421,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.000532+0000","last_change":"2024-09-16T10:19:44.711933+0000","last_active":"2024-09-16T10:21:28.000532+0000","last_peered":"2024-09-16T10:21:28.000532+0000","last_clean":"2024-09-16T10:21:28.000532+0000","last_became_active":"2024-09-16T10:18:52.835843+0000","last_became_peered":"2024-09-16T10:18:52.835843+0000","last_unstale":"2024-09-16T10:21:28.000532+0000","last_undegraded":"2024-09-16T10:21:28.000532+0000","last_fullsized":"2024-09-16T10:21:28.000532+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"249'14953","reported_seq":18696,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.351687+0000","last_change":"2024-09-16T10:20:52.637052+0000","last_active":"2024-09-16T10:21:28.351687+0000","last_peered":"2024-09-16T10:21:28.351687+0000","last_clean":"2024-09-16T10:21:28.351687+0000","last_became_active":"2024-09-16T10:19:15.293347+0000","last_became_peered":"2024-09-16T10:19:15.293347+0000","last_unstale":"2024-09-16T10:21:28.351687+0000","last_undegraded":"2024-09-16T10:21:28.351687+0000","last_fullsized":"2024-09-16T10:21:28.351687+0000","mapping_epoch":188,"log_start":"222'10666","ondisk_log_start":"222'10666","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'11682","last_scrub_stamp":"2024-09-16T10:20:52.636902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.636902+0000","objects_scrubbed":4492,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:56.892704+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76546072,"num_objects":1169,"num_object_clones":0,"num_object_copies":3507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14953,"num_write_kb":515841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1748,"num_bytes_recovered":114556928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":262,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003657+0000","last_change":"2024-09-16T10:21:19.429546+0000","last_active":"2024-09-16T10:21:28.003657+0000","last_peered":"2024-09-16T10:21:28.003657+0000","last_clean":"2024-09-16T10:21:28.003657+0000","last_became_active":"2024-09-16T10:21:18.944665+0000","last_became_peered":"2024-09-16T10:21:18.944665+0000","last_unstale":"2024-09-16T10:21:28.003657+0000","last_undegraded":"2024-09-16T10:21:28.003657+0000","last_fullsized":"2024-09-16T10:21:28.003657+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:35.583189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":443,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:27.999721+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:21:27.999721+0000","last_peered":"2024-09-16T10:21:27.999721+0000","last_clean":"2024-09-16T10:21:27.999721+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:21:27.999721+0000","last_undegraded":"2024-09-16T10:21:27.999721+0000","last_fullsized":"2024-09-16T10:21:27.999721+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010898+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:21:28.010898+0000","last_peered":"2024-09-16T10:21:28.010898+0000","last_clean":"2024-09-16T10:21:28.010898+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:21:28.010898+0000","last_undegraded":"2024-09-16T10:21:28.010898+0000","last_fullsized":"2024-09-16T10:21:28.010898+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":509,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003762+0000","last_change":"2024-09-16T10:18:04.023115+0000","last_active":"2024-09-16T10:21:28.003762+0000","last_peered":"2024-09-16T10:21:28.003762+0000","last_clean":"2024-09-16T10:21:28.003762+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:21:28.003762+0000","last_undegraded":"2024-09-16T10:21:28.003762+0000","last_fullsized":"2024-09-16T10:21:28.003762+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:04.022579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.022579+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:01.460206+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":398,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013020+0000","last_change":"2024-09-16T10:18:52.824583+0000","last_active":"2024-09-16T10:21:28.013020+0000","last_peered":"2024-09-16T10:21:28.013020+0000","last_clean":"2024-09-16T10:21:28.013020+0000","last_became_active":"2024-09-16T10:18:52.824265+0000","last_became_peered":"2024-09-16T10:18:52.824265+0000","last_unstale":"2024-09-16T10:21:28.013020+0000","last_undegraded":"2024-09-16T10:21:28.013020+0000","last_fullsized":"2024-09-16T10:21:28.013020+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:03.498619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:03.498619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:37.026545+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"249'14571","reported_seq":18115,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.620586+0000","last_change":"2024-09-16T10:20:54.577288+0000","last_active":"2024-09-16T10:21:28.620586+0000","last_peered":"2024-09-16T10:21:28.620586+0000","last_clean":"2024-09-16T10:21:28.620586+0000","last_became_active":"2024-09-16T10:19:14.293461+0000","last_became_peered":"2024-09-16T10:19:14.293461+0000","last_unstale":"2024-09-16T10:21:28.620586+0000","last_undegraded":"2024-09-16T10:21:28.620586+0000","last_fullsized":"2024-09-16T10:21:28.620586+0000","mapping_epoch":187,"log_start":"222'10162","ondisk_log_start":"222'10162","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11610","last_scrub_stamp":"2024-09-16T10:20:54.577223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.577223+0000","objects_scrubbed":4152,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:02.760584+0000","scrub_duration":1252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75431936,"num_objects":1151,"num_object_clones":0,"num_object_copies":3453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14571,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1651,"num_bytes_recovered":108199936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":313,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012045+0000","last_change":"2024-09-16T10:20:48.578602+0000","last_active":"2024-09-16T10:21:28.012045+0000","last_peered":"2024-09-16T10:21:28.012045+0000","last_clean":"2024-09-16T10:21:28.012045+0000","last_became_active":"2024-09-16T10:20:15.854970+0000","last_became_peered":"2024-09-16T10:20:15.854970+0000","last_unstale":"2024-09-16T10:21:28.012045+0000","last_undegraded":"2024-09-16T10:21:28.012045+0000","last_fullsized":"2024-09-16T10:21:28.012045+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:55.157197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":438,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003814+0000","last_change":"2024-09-16T10:19:54.487297+0000","last_active":"2024-09-16T10:21:28.003814+0000","last_peered":"2024-09-16T10:21:28.003814+0000","last_clean":"2024-09-16T10:21:28.003814+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:21:28.003814+0000","last_undegraded":"2024-09-16T10:21:28.003814+0000","last_fullsized":"2024-09-16T10:21:28.003814+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:54.487219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.487219+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:15.256397+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.008072+0000","last_change":"2024-09-16T10:18:52.836576+0000","last_active":"2024-09-16T10:21:28.008072+0000","last_peered":"2024-09-16T10:21:28.008072+0000","last_clean":"2024-09-16T10:21:28.008072+0000","last_became_active":"2024-09-16T10:18:52.836260+0000","last_became_peered":"2024-09-16T10:18:52.836260+0000","last_unstale":"2024-09-16T10:21:28.008072+0000","last_undegraded":"2024-09-16T10:21:28.008072+0000","last_fullsized":"2024-09-16T10:21:28.008072+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:04.187181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:04.187181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:42.402210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":505,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006428+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:21:28.006428+0000","last_peered":"2024-09-16T10:21:28.006428+0000","last_clean":"2024-09-16T10:21:28.006428+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:21:28.006428+0000","last_undegraded":"2024-09-16T10:21:28.006428+0000","last_fullsized":"2024-09-16T10:21:28.006428+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.577364+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007304+0000","last_change":"2024-09-16T10:19:50.044357+0000","last_active":"2024-09-16T10:21:28.007304+0000","last_peered":"2024-09-16T10:21:28.007304+0000","last_clean":"2024-09-16T10:21:28.007304+0000","last_became_active":"2024-09-16T10:18:52.842382+0000","last_became_peered":"2024-09-16T10:18:52.842382+0000","last_unstale":"2024-09-16T10:21:28.007304+0000","last_undegraded":"2024-09-16T10:21:28.007304+0000","last_fullsized":"2024-09-16T10:21:28.007304+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:50.044297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:50.044297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"249'14778","reported_seq":18368,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.602313+0000","last_change":"2024-09-16T10:20:24.650095+0000","last_active":"2024-09-16T10:21:28.602313+0000","last_peered":"2024-09-16T10:21:28.602313+0000","last_clean":"2024-09-16T10:21:28.602313+0000","last_became_active":"2024-09-16T10:19:14.292726+0000","last_became_peered":"2024-09-16T10:19:14.292726+0000","last_unstale":"2024-09-16T10:21:28.602313+0000","last_undegraded":"2024-09-16T10:21:28.602313+0000","last_fullsized":"2024-09-16T10:21:28.602313+0000","mapping_epoch":187,"log_start":"222'10573","ondisk_log_start":"222'10573","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'9052","last_scrub_stamp":"2024-09-16T10:20:24.649693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.649693+0000","objects_scrubbed":6970,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.170127+0000","scrub_duration":2436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74711040,"num_objects":1140,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14778,"num_write_kb":509376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1659,"num_bytes_recovered":108724224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":307,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.015680+0000","last_change":"2024-09-16T10:20:19.537775+0000","last_active":"2024-09-16T10:21:28.015680+0000","last_peered":"2024-09-16T10:21:28.015680+0000","last_clean":"2024-09-16T10:21:28.015680+0000","last_became_active":"2024-09-16T10:20:15.856484+0000","last_became_peered":"2024-09-16T10:20:15.856484+0000","last_unstale":"2024-09-16T10:21:28.015680+0000","last_undegraded":"2024-09-16T10:21:28.015680+0000","last_fullsized":"2024-09-16T10:21:28.015680+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.537727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.537727+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":458,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006586+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:21:28.006586+0000","last_peered":"2024-09-16T10:21:28.006586+0000","last_clean":"2024-09-16T10:21:28.006586+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:21:28.006586+0000","last_undegraded":"2024-09-16T10:21:28.006586+0000","last_fullsized":"2024-09-16T10:21:28.006586+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:38.989185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":426,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.005229+0000","last_change":"2024-09-16T10:20:31.548535+0000","last_active":"2024-09-16T10:21:28.005229+0000","last_peered":"2024-09-16T10:21:28.005229+0000","last_clean":"2024-09-16T10:21:28.005229+0000","last_became_active":"2024-09-16T10:18:52.832749+0000","last_became_peered":"2024-09-16T10:18:52.832749+0000","last_unstale":"2024-09-16T10:21:28.005229+0000","last_undegraded":"2024-09-16T10:21:28.005229+0000","last_fullsized":"2024-09-16T10:21:28.005229+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:31.548465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.548465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.221683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012093+0000","last_change":"2024-09-16T10:20:58.707720+0000","last_active":"2024-09-16T10:21:28.012093+0000","last_peered":"2024-09-16T10:21:28.012093+0000","last_clean":"2024-09-16T10:21:28.012093+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:21:28.012093+0000","last_undegraded":"2024-09-16T10:21:28.012093+0000","last_fullsized":"2024-09-16T10:21:28.012093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.707654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.707654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:18.162948+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003235+0000","last_change":"2024-09-16T10:18:52.827865+0000","last_active":"2024-09-16T10:21:28.003235+0000","last_peered":"2024-09-16T10:21:28.003235+0000","last_clean":"2024-09-16T10:21:28.003235+0000","last_became_active":"2024-09-16T10:18:52.827511+0000","last_became_peered":"2024-09-16T10:18:52.827511+0000","last_unstale":"2024-09-16T10:21:28.003235+0000","last_undegraded":"2024-09-16T10:21:28.003235+0000","last_fullsized":"2024-09-16T10:21:28.003235+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:03.057932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:03.057932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.486130+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"249'15051","reported_seq":15284,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.377935+0000","last_change":"2024-09-16T10:20:54.830991+0000","last_active":"2024-09-16T10:21:28.377935+0000","last_peered":"2024-09-16T10:21:28.377935+0000","last_clean":"2024-09-16T10:21:28.377935+0000","last_became_active":"2024-09-16T10:18:33.893254+0000","last_became_peered":"2024-09-16T10:18:33.893254+0000","last_unstale":"2024-09-16T10:21:28.377935+0000","last_undegraded":"2024-09-16T10:21:28.377935+0000","last_fullsized":"2024-09-16T10:21:28.377935+0000","mapping_epoch":165,"log_start":"222'10813","ondisk_log_start":"222'10813","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'11985","last_scrub_stamp":"2024-09-16T10:20:54.830858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.830858+0000","objects_scrubbed":4288,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.069310+0000","scrub_duration":1423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76480512,"num_objects":1167,"num_object_clones":0,"num_object_copies":3501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15051,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":242,"reported_epoch":248,"state":"peering","last_fresh":"2024-09-16T10:21:27.004724+0000","last_change":"2024-09-16T10:21:28.006445+0000","last_active":"2024-09-16T10:21:28.006445+0000","last_peered":"2024-09-16T10:21:27.004724+0000","last_clean":"2024-09-16T10:21:27.004724+0000","last_became_active":"2024-09-16T10:20:15.850975+0000","last_became_peered":"2024-09-16T10:20:15.850975+0000","last_unstale":"2024-09-16T10:21:27.004724+0000","last_undegraded":"2024-09-16T10:21:27.004724+0000","last_fullsized":"2024-09-16T10:21:27.004724+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"175'29","reported_seq":595,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.005098+0000","last_change":"2024-09-16T10:18:52.860272+0000","last_active":"2024-09-16T10:21:28.005098+0000","last_peered":"2024-09-16T10:21:28.005098+0000","last_clean":"2024-09-16T10:21:28.005098+0000","last_became_active":"2024-09-16T10:18:52.841202+0000","last_became_peered":"2024-09-16T10:18:52.841202+0000","last_unstale":"2024-09-16T10:21:28.005098+0000","last_undegraded":"2024-09-16T10:21:28.005098+0000","last_fullsized":"2024-09-16T10:21:28.005098+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-16T10:18:45.654714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:45.654714+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:04.260955+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.001841+0000","last_change":"2024-09-16T10:20:56.965996+0000","last_active":"2024-09-16T10:21:28.001841+0000","last_peered":"2024-09-16T10:21:28.001841+0000","last_clean":"2024-09-16T10:21:28.001841+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:21:28.001841+0000","last_undegraded":"2024-09-16T10:21:28.001841+0000","last_fullsized":"2024-09-16T10:21:28.001841+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:56.965933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:56.965933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.271911+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":421,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.008783+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:21:28.008783+0000","last_peered":"2024-09-16T10:21:28.008783+0000","last_clean":"2024-09-16T10:21:28.008783+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:21:28.008783+0000","last_undegraded":"2024-09-16T10:21:28.008783+0000","last_fullsized":"2024-09-16T10:21:28.008783+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.398498+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":402,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007000+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:21:28.007000+0000","last_peered":"2024-09-16T10:21:28.007000+0000","last_clean":"2024-09-16T10:21:28.007000+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:21:28.007000+0000","last_undegraded":"2024-09-16T10:21:28.007000+0000","last_fullsized":"2024-09-16T10:21:28.007000+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:29.172677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":415,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002584+0000","last_change":"2024-09-16T10:20:28.240790+0000","last_active":"2024-09-16T10:21:28.002584+0000","last_peered":"2024-09-16T10:21:28.002584+0000","last_clean":"2024-09-16T10:21:28.002584+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:21:28.002584+0000","last_undegraded":"2024-09-16T10:21:28.002584+0000","last_fullsized":"2024-09-16T10:21:28.002584+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:28.240558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:28.240558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:54.592999+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"249'14888","reported_seq":15149,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.633003+0000","last_change":"2024-09-16T10:20:22.211158+0000","last_active":"2024-09-16T10:21:28.633003+0000","last_peered":"2024-09-16T10:21:28.633003+0000","last_clean":"2024-09-16T10:21:28.633003+0000","last_became_active":"2024-09-16T10:18:33.820857+0000","last_became_peered":"2024-09-16T10:18:33.820857+0000","last_unstale":"2024-09-16T10:21:28.633003+0000","last_undegraded":"2024-09-16T10:21:28.633003+0000","last_fullsized":"2024-09-16T10:21:28.633003+0000","mapping_epoch":165,"log_start":"222'10726","ondisk_log_start":"222'10726","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'8979","last_scrub_stamp":"2024-09-16T10:20:22.210682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:22.210682+0000","objects_scrubbed":7115,"log_size":4162,"log_dups_size":0,"ondisk_log_size":4162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:32.400224+0000","scrub_duration":2075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73138176,"num_objects":1116,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14888,"num_write_kb":512128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":524,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012391+0000","last_change":"2024-09-16T10:21:27.006377+0000","last_active":"2024-09-16T10:21:28.012391+0000","last_peered":"2024-09-16T10:21:28.012391+0000","last_clean":"2024-09-16T10:21:28.012391+0000","last_became_active":"2024-09-16T10:21:27.006016+0000","last_became_peered":"2024-09-16T10:21:27.006016+0000","last_unstale":"2024-09-16T10:21:28.012391+0000","last_undegraded":"2024-09-16T10:21:28.012391+0000","last_fullsized":"2024-09-16T10:21:28.012391+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.064404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.064404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.237662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":426,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010219+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:21:28.010219+0000","last_peered":"2024-09-16T10:21:28.010219+0000","last_clean":"2024-09-16T10:21:28.010219+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:21:28.010219+0000","last_undegraded":"2024-09-16T10:21:28.010219+0000","last_fullsized":"2024-09-16T10:21:28.010219+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.411556+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002974+0000","last_change":"2024-09-16T10:20:55.257286+0000","last_active":"2024-09-16T10:21:28.002974+0000","last_peered":"2024-09-16T10:21:28.002974+0000","last_clean":"2024-09-16T10:21:28.002974+0000","last_became_active":"2024-09-16T10:18:52.833059+0000","last_became_peered":"2024-09-16T10:18:52.833059+0000","last_unstale":"2024-09-16T10:21:28.002974+0000","last_undegraded":"2024-09-16T10:21:28.002974+0000","last_fullsized":"2024-09-16T10:21:28.002974+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:56.920867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":186,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002637+0000","last_change":"2024-09-16T10:19:58.199140+0000","last_active":"2024-09-16T10:21:28.002637+0000","last_peered":"2024-09-16T10:21:28.002637+0000","last_clean":"2024-09-16T10:21:28.002637+0000","last_became_active":"2024-09-16T10:18:52.836346+0000","last_became_peered":"2024-09-16T10:18:52.836346+0000","last_unstale":"2024-09-16T10:21:28.002637+0000","last_undegraded":"2024-09-16T10:21:28.002637+0000","last_fullsized":"2024-09-16T10:21:28.002637+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":178,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006084+0000","last_change":"2024-09-16T10:20:58.393219+0000","last_active":"2024-09-16T10:21:28.006084+0000","last_peered":"2024-09-16T10:21:28.006084+0000","last_clean":"2024-09-16T10:21:28.006084+0000","last_became_active":"2024-09-16T10:18:52.827678+0000","last_became_peered":"2024-09-16T10:18:52.827678+0000","last_unstale":"2024-09-16T10:21:28.006084+0000","last_undegraded":"2024-09-16T10:21:28.006084+0000","last_fullsized":"2024-09-16T10:21:28.006084+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.393147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.393147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:24.791999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":415,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012990+0000","last_change":"2024-09-16T10:20:52.145844+0000","last_active":"2024-09-16T10:21:28.012990+0000","last_peered":"2024-09-16T10:21:28.012990+0000","last_clean":"2024-09-16T10:21:28.012990+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:21:28.012990+0000","last_undegraded":"2024-09-16T10:21:28.012990+0000","last_fullsized":"2024-09-16T10:21:28.012990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:52.145765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.145765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:52.409028+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"249'14786","reported_seq":18538,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.490456+0000","last_change":"2024-09-16T10:21:27.849644+0000","last_active":"2024-09-16T10:21:28.490456+0000","last_peered":"2024-09-16T10:21:28.490456+0000","last_clean":"2024-09-16T10:21:28.490456+0000","last_became_active":"2024-09-16T10:19:13.300503+0000","last_became_peered":"2024-09-16T10:19:13.300503+0000","last_unstale":"2024-09-16T10:21:28.490456+0000","last_undegraded":"2024-09-16T10:21:28.490456+0000","last_fullsized":"2024-09-16T10:21:28.490456+0000","mapping_epoch":186,"log_start":"222'10448","ondisk_log_start":"222'10448","created":165,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'14726","last_scrub_stamp":"2024-09-16T10:21:27.849520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.849520+0000","objects_scrubbed":1270,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:28.615977+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78381056,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14786,"num_write_kb":511424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1730,"num_bytes_recovered":113377280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":249,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010256+0000","last_change":"2024-09-16T10:21:22.894819+0000","last_active":"2024-09-16T10:21:28.010256+0000","last_peered":"2024-09-16T10:21:28.010256+0000","last_clean":"2024-09-16T10:21:28.010256+0000","last_became_active":"2024-09-16T10:21:22.894381+0000","last_became_peered":"2024-09-16T10:21:22.894381+0000","last_unstale":"2024-09-16T10:21:28.010256+0000","last_undegraded":"2024-09-16T10:21:28.010256+0000","last_fullsized":"2024-09-16T10:21:28.010256+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:01.610913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":469,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010783+0000","last_change":"2024-09-16T10:20:55.304934+0000","last_active":"2024-09-16T10:21:28.010783+0000","last_peered":"2024-09-16T10:21:28.010783+0000","last_clean":"2024-09-16T10:21:28.010783+0000","last_became_active":"2024-09-16T10:18:52.837634+0000","last_became_peered":"2024-09-16T10:18:52.837634+0000","last_unstale":"2024-09-16T10:21:28.010783+0000","last_undegraded":"2024-09-16T10:21:28.010783+0000","last_fullsized":"2024-09-16T10:21:28.010783+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:20:55.304837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.304837+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.147025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":434,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010792+0000","last_change":"2024-09-16T10:20:52.288156+0000","last_active":"2024-09-16T10:21:28.010792+0000","last_peered":"2024-09-16T10:21:28.010792+0000","last_clean":"2024-09-16T10:21:28.010792+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:21:28.010792+0000","last_undegraded":"2024-09-16T10:21:28.010792+0000","last_fullsized":"2024-09-16T10:21:28.010792+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:52.288090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.288090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:10.544341+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":895,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.014565+0000","last_change":"2024-09-16T10:20:33.058993+0000","last_active":"2024-09-16T10:21:28.014565+0000","last_peered":"2024-09-16T10:21:28.014565+0000","last_clean":"2024-09-16T10:21:28.014565+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:21:28.014565+0000","last_undegraded":"2024-09-16T10:21:28.014565+0000","last_fullsized":"2024-09-16T10:21:28.014565+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:20:33.058923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.058923+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.147877+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":465,"num_read_kb":460,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006759+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:21:28.006759+0000","last_peered":"2024-09-16T10:21:28.006759+0000","last_clean":"2024-09-16T10:21:28.006759+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:21:28.006759+0000","last_undegraded":"2024-09-16T10:21:28.006759+0000","last_fullsized":"2024-09-16T10:21:28.006759+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:08.437011+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":415,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003371+0000","last_change":"2024-09-16T10:20:25.566580+0000","last_active":"2024-09-16T10:21:28.003371+0000","last_peered":"2024-09-16T10:21:28.003371+0000","last_clean":"2024-09-16T10:21:28.003371+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:21:28.003371+0000","last_undegraded":"2024-09-16T10:21:28.003371+0000","last_fullsized":"2024-09-16T10:21:28.003371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:25.566513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:25.566513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:53.062951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"249'14359","reported_seq":14612,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.253839+0000","last_change":"2024-09-16T10:19:56.367287+0000","last_active":"2024-09-16T10:21:28.253839+0000","last_peered":"2024-09-16T10:21:28.253839+0000","last_clean":"2024-09-16T10:21:28.253839+0000","last_became_active":"2024-09-16T10:18:33.825120+0000","last_became_peered":"2024-09-16T10:18:33.825120+0000","last_unstale":"2024-09-16T10:21:28.253839+0000","last_undegraded":"2024-09-16T10:21:28.253839+0000","last_fullsized":"2024-09-16T10:21:28.253839+0000","mapping_epoch":165,"log_start":"222'10022","ondisk_log_start":"222'10022","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6534","last_scrub_stamp":"2024-09-16T10:19:56.367108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.367108+0000","objects_scrubbed":6469,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:06.803153+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78708736,"num_objects":1201,"num_object_clones":0,"num_object_copies":3603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14359,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":523,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012031+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:21:28.012031+0000","last_peered":"2024-09-16T10:21:28.012031+0000","last_clean":"2024-09-16T10:21:28.012031+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:21:28.012031+0000","last_undegraded":"2024-09-16T10:21:28.012031+0000","last_fullsized":"2024-09-16T10:21:28.012031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.998293+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":384,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002442+0000","last_change":"2024-09-16T10:19:44.015480+0000","last_active":"2024-09-16T10:21:28.002442+0000","last_peered":"2024-09-16T10:21:28.002442+0000","last_clean":"2024-09-16T10:21:28.002442+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:21:28.002442+0000","last_undegraded":"2024-09-16T10:21:28.002442+0000","last_fullsized":"2024-09-16T10:21:28.002442+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.015430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.015430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":492,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.014771+0000","last_change":"2024-09-16T10:20:19.229860+0000","last_active":"2024-09-16T10:21:28.014771+0000","last_peered":"2024-09-16T10:21:28.014771+0000","last_clean":"2024-09-16T10:21:28.014771+0000","last_became_active":"2024-09-16T10:20:15.855504+0000","last_became_peered":"2024-09-16T10:20:15.855504+0000","last_unstale":"2024-09-16T10:21:28.014771+0000","last_undegraded":"2024-09-16T10:21:28.014771+0000","last_fullsized":"2024-09-16T10:21:28.014771+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.229771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.229771+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013898+0000","last_change":"2024-09-16T10:18:52.827862+0000","last_active":"2024-09-16T10:21:28.013898+0000","last_peered":"2024-09-16T10:21:28.013898+0000","last_clean":"2024-09-16T10:21:28.013898+0000","last_became_active":"2024-09-16T10:18:52.827467+0000","last_became_peered":"2024-09-16T10:18:52.827467+0000","last_unstale":"2024-09-16T10:21:28.013898+0000","last_undegraded":"2024-09-16T10:21:28.013898+0000","last_fullsized":"2024-09-16T10:21:28.013898+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.015337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006574+0000","last_change":"2024-09-16T10:19:23.923755+0000","last_active":"2024-09-16T10:21:28.006574+0000","last_peered":"2024-09-16T10:21:28.006574+0000","last_clean":"2024-09-16T10:21:28.006574+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:21:28.006574+0000","last_undegraded":"2024-09-16T10:21:28.006574+0000","last_fullsized":"2024-09-16T10:21:28.006574+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:23.923690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:23.923690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.471730+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":246,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.005424+0000","last_change":"2024-09-16T10:21:12.867540+0000","last_active":"2024-09-16T10:21:28.005424+0000","last_peered":"2024-09-16T10:21:28.005424+0000","last_clean":"2024-09-16T10:21:28.005424+0000","last_became_active":"2024-09-16T10:21:12.866128+0000","last_became_peered":"2024-09-16T10:21:12.866128+0000","last_unstale":"2024-09-16T10:21:28.005424+0000","last_undegraded":"2024-09-16T10:21:28.005424+0000","last_fullsized":"2024-09-16T10:21:28.005424+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.546053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.546053+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:48.129442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"249'14667","reported_seq":14863,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.184692+0000","last_change":"2024-09-16T10:20:59.290413+0000","last_active":"2024-09-16T10:21:28.184692+0000","last_peered":"2024-09-16T10:21:28.184692+0000","last_clean":"2024-09-16T10:21:28.184692+0000","last_became_active":"2024-09-16T10:18:33.824979+0000","last_became_peered":"2024-09-16T10:18:33.824979+0000","last_unstale":"2024-09-16T10:21:28.184692+0000","last_undegraded":"2024-09-16T10:21:28.184692+0000","last_fullsized":"2024-09-16T10:21:28.184692+0000","mapping_epoch":165,"log_start":"222'10401","ondisk_log_start":"222'10401","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":1149,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:29.288814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79101952,"num_objects":1207,"num_object_clones":0,"num_object_copies":3621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14667,"num_write_kb":507968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":452,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002546+0000","last_change":"2024-09-16T10:20:31.013510+0000","last_active":"2024-09-16T10:21:28.002546+0000","last_peered":"2024-09-16T10:21:28.002546+0000","last_clean":"2024-09-16T10:21:28.002546+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:21:28.002546+0000","last_undegraded":"2024-09-16T10:21:28.002546+0000","last_fullsized":"2024-09-16T10:21:28.002546+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:20:31.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.013316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:48.982676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":417,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.000334+0000","last_change":"2024-09-16T10:20:33.973275+0000","last_active":"2024-09-16T10:21:28.000334+0000","last_peered":"2024-09-16T10:21:28.000334+0000","last_clean":"2024-09-16T10:21:28.000334+0000","last_became_active":"2024-09-16T10:18:52.835626+0000","last_became_peered":"2024-09-16T10:18:52.835626+0000","last_unstale":"2024-09-16T10:21:28.000334+0000","last_undegraded":"2024-09-16T10:21:28.000334+0000","last_fullsized":"2024-09-16T10:21:28.000334+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.973201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.973201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:42.032781+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006497+0000","last_change":"2024-09-16T10:20:55.449343+0000","last_active":"2024-09-16T10:21:28.006497+0000","last_peered":"2024-09-16T10:21:28.006497+0000","last_clean":"2024-09-16T10:21:28.006497+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:21:28.006497+0000","last_undegraded":"2024-09-16T10:21:28.006497+0000","last_fullsized":"2024-09-16T10:21:28.006497+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.449274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.449274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:10.343283+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":364,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003517+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:21:28.003517+0000","last_peered":"2024-09-16T10:21:28.003517+0000","last_clean":"2024-09-16T10:21:28.003517+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:21:28.003517+0000","last_undegraded":"2024-09-16T10:21:28.003517+0000","last_fullsized":"2024-09-16T10:21:28.003517+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"249'14740","reported_seq":18394,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.480658+0000","last_change":"2024-09-16T10:21:26.861792+0000","last_active":"2024-09-16T10:21:28.480658+0000","last_peered":"2024-09-16T10:21:28.480658+0000","last_clean":"2024-09-16T10:21:28.480658+0000","last_became_active":"2024-09-16T10:19:15.290152+0000","last_became_peered":"2024-09-16T10:19:15.290152+0000","last_unstale":"2024-09-16T10:21:28.480658+0000","last_undegraded":"2024-09-16T10:21:28.480658+0000","last_fullsized":"2024-09-16T10:21:28.480658+0000","mapping_epoch":188,"log_start":"222'10452","ondisk_log_start":"222'10452","created":165,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'14581","last_scrub_stamp":"2024-09-16T10:21:26.861601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:21:26.861601+0000","objects_scrubbed":1276,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:46.021198+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14740,"num_write_kb":506880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1685,"num_bytes_recovered":110428160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":243,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006895+0000","last_change":"2024-09-16T10:20:27.405769+0000","last_active":"2024-09-16T10:21:28.006895+0000","last_peered":"2024-09-16T10:21:28.006895+0000","last_clean":"2024-09-16T10:21:28.006895+0000","last_became_active":"2024-09-16T10:18:52.842646+0000","last_became_peered":"2024-09-16T10:18:52.842646+0000","last_unstale":"2024-09-16T10:21:28.006895+0000","last_undegraded":"2024-09-16T10:21:28.006895+0000","last_fullsized":"2024-09-16T10:21:28.006895+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:27.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:27.405599+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:29.866476+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":459,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004402+0000","last_change":"2024-09-16T10:20:22.959397+0000","last_active":"2024-09-16T10:21:28.004402+0000","last_peered":"2024-09-16T10:21:28.004402+0000","last_clean":"2024-09-16T10:21:28.004402+0000","last_became_active":"2024-09-16T10:18:52.826328+0000","last_became_peered":"2024-09-16T10:18:52.826328+0000","last_unstale":"2024-09-16T10:21:28.004402+0000","last_undegraded":"2024-09-16T10:21:28.004402+0000","last_fullsized":"2024-09-16T10:21:28.004402+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:20:22.959311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:22.959311+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.280982+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012406+0000","last_change":"2024-09-16T10:19:21.116145+0000","last_active":"2024-09-16T10:21:28.012406+0000","last_peered":"2024-09-16T10:21:28.012406+0000","last_clean":"2024-09-16T10:21:28.012406+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:21:28.012406+0000","last_undegraded":"2024-09-16T10:21:28.012406+0000","last_fullsized":"2024-09-16T10:21:28.012406+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:21.116098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:21.116098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.615403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004298+0000","last_change":"2024-09-16T10:20:29.600095+0000","last_active":"2024-09-16T10:21:28.004298+0000","last_peered":"2024-09-16T10:21:28.004298+0000","last_clean":"2024-09-16T10:21:28.004298+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:21:28.004298+0000","last_undegraded":"2024-09-16T10:21:28.004298+0000","last_fullsized":"2024-09-16T10:21:28.004298+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:29.600025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:29.600025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:51.499818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":163,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003086+0000","last_change":"2024-09-16T10:18:52.838759+0000","last_active":"2024-09-16T10:21:28.003086+0000","last_peered":"2024-09-16T10:21:28.003086+0000","last_clean":"2024-09-16T10:21:28.003086+0000","last_became_active":"2024-09-16T10:18:52.838070+0000","last_became_peered":"2024-09-16T10:18:52.838070+0000","last_unstale":"2024-09-16T10:21:28.003086+0000","last_undegraded":"2024-09-16T10:21:28.003086+0000","last_fullsized":"2024-09-16T10:21:28.003086+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:50.491234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.491234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.383400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"249'15219","reported_seq":15400,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.632296+0000","last_change":"2024-09-16T10:18:33.823010+0000","last_active":"2024-09-16T10:21:28.632296+0000","last_peered":"2024-09-16T10:21:28.632296+0000","last_clean":"2024-09-16T10:21:28.632296+0000","last_became_active":"2024-09-16T10:18:33.822701+0000","last_became_peered":"2024-09-16T10:18:33.822701+0000","last_unstale":"2024-09-16T10:21:28.632296+0000","last_undegraded":"2024-09-16T10:21:28.632296+0000","last_fullsized":"2024-09-16T10:21:28.632296+0000","mapping_epoch":165,"log_start":"222'11022","ondisk_log_start":"222'11022","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.329041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76349440,"num_objects":1165,"num_object_clones":0,"num_object_copies":3495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15219,"num_write_kb":524288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":507,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012922+0000","last_change":"2024-09-16T10:21:21.240457+0000","last_active":"2024-09-16T10:21:28.012922+0000","last_peered":"2024-09-16T10:21:28.012922+0000","last_clean":"2024-09-16T10:21:28.012922+0000","last_became_active":"2024-09-16T10:18:52.838011+0000","last_became_peered":"2024-09-16T10:18:52.838011+0000","last_unstale":"2024-09-16T10:21:28.012922+0000","last_undegraded":"2024-09-16T10:21:28.012922+0000","last_fullsized":"2024-09-16T10:21:28.012922+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:21.240387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:21.240387+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:22.630130+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002443+0000","last_change":"2024-09-16T10:19:16.768415+0000","last_active":"2024-09-16T10:21:28.002443+0000","last_peered":"2024-09-16T10:21:28.002443+0000","last_clean":"2024-09-16T10:21:28.002443+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:21:28.002443+0000","last_undegraded":"2024-09-16T10:21:28.002443+0000","last_fullsized":"2024-09-16T10:21:28.002443+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:16.768062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.768062+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.009177+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:21:28.009177+0000","last_peered":"2024-09-16T10:21:28.009177+0000","last_clean":"2024-09-16T10:21:28.009177+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:21:28.009177+0000","last_undegraded":"2024-09-16T10:21:28.009177+0000","last_fullsized":"2024-09-16T10:21:28.009177+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:15.355777+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"175'9","reported_seq":240,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013072+0000","last_change":"2024-09-16T10:20:55.177381+0000","last_active":"2024-09-16T10:21:28.013072+0000","last_peered":"2024-09-16T10:21:28.013072+0000","last_clean":"2024-09-16T10:21:28.013072+0000","last_became_active":"2024-09-16T10:18:52.825124+0000","last_became_peered":"2024-09-16T10:18:52.825124+0000","last_unstale":"2024-09-16T10:21:28.013072+0000","last_undegraded":"2024-09-16T10:21:28.013072+0000","last_fullsized":"2024-09-16T10:21:28.013072+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:20:55.177089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.177089+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:06.263297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002222+0000","last_change":"2024-09-16T10:20:51.873893+0000","last_active":"2024-09-16T10:21:28.002222+0000","last_peered":"2024-09-16T10:21:28.002222+0000","last_clean":"2024-09-16T10:21:28.002222+0000","last_became_active":"2024-09-16T10:18:52.834312+0000","last_became_peered":"2024-09-16T10:18:52.834312+0000","last_unstale":"2024-09-16T10:21:28.002222+0000","last_undegraded":"2024-09-16T10:21:28.002222+0000","last_fullsized":"2024-09-16T10:21:28.002222+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.873833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.873833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:14.931229+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":59,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.015525+0000","last_change":"2024-09-16T10:21:18.757238+0000","last_active":"2024-09-16T10:21:28.015525+0000","last_peered":"2024-09-16T10:21:28.015525+0000","last_clean":"2024-09-16T10:21:28.015525+0000","last_became_active":"2024-09-16T10:21:12.869907+0000","last_became_peered":"2024-09-16T10:21:12.869907+0000","last_unstale":"2024-09-16T10:21:28.015525+0000","last_undegraded":"2024-09-16T10:21:28.015525+0000","last_fullsized":"2024-09-16T10:21:28.015525+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:18.757065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:18.757065+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:42.275436+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"249'14691","reported_seq":14888,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.493524+0000","last_change":"2024-09-16T10:19:59.077503+0000","last_active":"2024-09-16T10:21:28.493524+0000","last_peered":"2024-09-16T10:21:28.493524+0000","last_clean":"2024-09-16T10:21:28.493524+0000","last_became_active":"2024-09-16T10:18:33.823198+0000","last_became_peered":"2024-09-16T10:18:33.823198+0000","last_unstale":"2024-09-16T10:21:28.493524+0000","last_undegraded":"2024-09-16T10:21:28.493524+0000","last_fullsized":"2024-09-16T10:21:28.493524+0000","mapping_epoch":165,"log_start":"222'10374","ondisk_log_start":"222'10374","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":1399,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.319001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76087296,"num_objects":1161,"num_object_clones":0,"num_object_copies":3483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14691,"num_write_kb":507264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":422,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006508+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:21:28.006508+0000","last_peered":"2024-09-16T10:21:28.006508+0000","last_clean":"2024-09-16T10:21:28.006508+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:21:28.006508+0000","last_undegraded":"2024-09-16T10:21:28.006508+0000","last_fullsized":"2024-09-16T10:21:28.006508+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.233999+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.008332+0000","last_change":"2024-09-16T10:20:28.364361+0000","last_active":"2024-09-16T10:21:28.008332+0000","last_peered":"2024-09-16T10:21:28.008332+0000","last_clean":"2024-09-16T10:21:28.008332+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:21:28.008332+0000","last_undegraded":"2024-09-16T10:21:28.008332+0000","last_fullsized":"2024-09-16T10:21:28.008332+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:28.364296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:28.364296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:34.891566+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":170,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.000355+0000","last_change":"2024-09-16T10:19:18.748438+0000","last_active":"2024-09-16T10:21:28.000355+0000","last_peered":"2024-09-16T10:21:28.000355+0000","last_clean":"2024-09-16T10:21:28.000355+0000","last_became_active":"2024-09-16T10:18:52.835884+0000","last_became_peered":"2024-09-16T10:18:52.835884+0000","last_unstale":"2024-09-16T10:21:28.000355+0000","last_undegraded":"2024-09-16T10:21:28.000355+0000","last_fullsized":"2024-09-16T10:21:28.000355+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:02.927836+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":432,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010602+0000","last_change":"2024-09-16T10:20:31.889357+0000","last_active":"2024-09-16T10:21:28.010602+0000","last_peered":"2024-09-16T10:21:28.010602+0000","last_clean":"2024-09-16T10:21:28.010602+0000","last_became_active":"2024-09-16T10:20:31.869705+0000","last_became_peered":"2024-09-16T10:20:31.869705+0000","last_unstale":"2024-09-16T10:21:28.010602+0000","last_undegraded":"2024-09-16T10:21:28.010602+0000","last_fullsized":"2024-09-16T10:21:28.010602+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:37.267422+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":321,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012451+0000","last_change":"2024-09-16T10:21:12.878826+0000","last_active":"2024-09-16T10:21:28.012451+0000","last_peered":"2024-09-16T10:21:28.012451+0000","last_clean":"2024-09-16T10:21:28.012451+0000","last_became_active":"2024-09-16T10:21:12.874449+0000","last_became_peered":"2024-09-16T10:21:12.874449+0000","last_unstale":"2024-09-16T10:21:28.012451+0000","last_undegraded":"2024-09-16T10:21:28.012451+0000","last_fullsized":"2024-09-16T10:21:28.012451+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:20.538076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:20.538076+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.6","version":"249'14756","reported_seq":14937,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.641110+0000","last_change":"2024-09-16T10:18:33.894422+0000","last_active":"2024-09-16T10:21:28.641110+0000","last_peered":"2024-09-16T10:21:28.641110+0000","last_clean":"2024-09-16T10:21:28.641110+0000","last_became_active":"2024-09-16T10:18:33.893961+0000","last_became_peered":"2024-09-16T10:18:33.893961+0000","last_unstale":"2024-09-16T10:21:28.641110+0000","last_undegraded":"2024-09-16T10:21:28.641110+0000","last_fullsized":"2024-09-16T10:21:28.641110+0000","mapping_epoch":165,"log_start":"222'10320","ondisk_log_start":"222'10320","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.798588+0000","objects_scrubbed":0,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.292676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76152832,"num_objects":1162,"num_object_clones":0,"num_object_copies":3486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14756,"num_write_kb":509376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":453,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012171+0000","last_change":"2024-09-16T10:20:51.673782+0000","last_active":"2024-09-16T10:21:28.012171+0000","last_peered":"2024-09-16T10:21:28.012171+0000","last_clean":"2024-09-16T10:21:28.012171+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:21:28.012171+0000","last_undegraded":"2024-09-16T10:21:28.012171+0000","last_fullsized":"2024-09-16T10:21:28.012171+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:20:51.673686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.673686+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:16.084083+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013349+0000","last_change":"2024-09-16T10:20:31.091077+0000","last_active":"2024-09-16T10:21:28.013349+0000","last_peered":"2024-09-16T10:21:28.013349+0000","last_clean":"2024-09-16T10:21:28.013349+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:21:28.013349+0000","last_undegraded":"2024-09-16T10:21:28.013349+0000","last_fullsized":"2024-09-16T10:21:28.013349+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:31.090914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.090914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:49.345216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":178,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006347+0000","last_change":"2024-09-16T10:20:54.330571+0000","last_active":"2024-09-16T10:21:28.006347+0000","last_peered":"2024-09-16T10:21:28.006347+0000","last_clean":"2024-09-16T10:21:28.006347+0000","last_became_active":"2024-09-16T10:18:52.831796+0000","last_became_peered":"2024-09-16T10:18:52.831796+0000","last_unstale":"2024-09-16T10:21:28.006347+0000","last_undegraded":"2024-09-16T10:21:28.006347+0000","last_fullsized":"2024-09-16T10:21:28.006347+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.330503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.330503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:18.648114+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":414,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.014644+0000","last_change":"2024-09-16T10:20:51.105297+0000","last_active":"2024-09-16T10:21:28.014644+0000","last_peered":"2024-09-16T10:21:28.014644+0000","last_clean":"2024-09-16T10:21:28.014644+0000","last_became_active":"2024-09-16T10:18:52.840958+0000","last_became_peered":"2024-09-16T10:18:52.840958+0000","last_unstale":"2024-09-16T10:21:28.014644+0000","last_undegraded":"2024-09-16T10:21:28.014644+0000","last_fullsized":"2024-09-16T10:21:28.014644+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.105240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.105240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:12.132076+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":259,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003247+0000","last_change":"2024-09-16T10:21:12.892367+0000","last_active":"2024-09-16T10:21:28.003247+0000","last_peered":"2024-09-16T10:21:28.003247+0000","last_clean":"2024-09-16T10:21:28.003247+0000","last_became_active":"2024-09-16T10:21:12.876854+0000","last_became_peered":"2024-09-16T10:21:12.876854+0000","last_unstale":"2024-09-16T10:21:28.003247+0000","last_undegraded":"2024-09-16T10:21:28.003247+0000","last_fullsized":"2024-09-16T10:21:28.003247+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:19.166321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.166321+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"249'14759","reported_seq":15062,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.235156+0000","last_change":"2024-09-16T10:20:58.987382+0000","last_active":"2024-09-16T10:21:28.235156+0000","last_peered":"2024-09-16T10:21:28.235156+0000","last_clean":"2024-09-16T10:21:28.235156+0000","last_became_active":"2024-09-16T10:18:33.894026+0000","last_became_peered":"2024-09-16T10:18:33.894026+0000","last_unstale":"2024-09-16T10:21:28.235156+0000","last_undegraded":"2024-09-16T10:21:28.235156+0000","last_fullsized":"2024-09-16T10:21:28.235156+0000","mapping_epoch":165,"log_start":"222'10403","ondisk_log_start":"222'10403","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12139","last_scrub_stamp":"2024-09-16T10:20:58.987165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.987165+0000","objects_scrubbed":3864,"log_size":4356,"log_dups_size":0,"ondisk_log_size":4356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:25.474069+0000","scrub_duration":1279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135872,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14759,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":429,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.000184+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:21:28.000184+0000","last_peered":"2024-09-16T10:21:28.000184+0000","last_clean":"2024-09-16T10:21:28.000184+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:21:28.000184+0000","last_undegraded":"2024-09-16T10:21:28.000184+0000","last_fullsized":"2024-09-16T10:21:28.000184+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:38.841996+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013980+0000","last_change":"2024-09-16T10:20:57.246050+0000","last_active":"2024-09-16T10:21:28.013980+0000","last_peered":"2024-09-16T10:21:28.013980+0000","last_clean":"2024-09-16T10:21:28.013980+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:21:28.013980+0000","last_undegraded":"2024-09-16T10:21:28.013980+0000","last_fullsized":"2024-09-16T10:21:28.013980+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.245939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.245939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:00.669214+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":162,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.005751+0000","last_change":"2024-09-16T10:18:52.830267+0000","last_active":"2024-09-16T10:21:28.005751+0000","last_peered":"2024-09-16T10:21:28.005751+0000","last_clean":"2024-09-16T10:21:28.005751+0000","last_became_active":"2024-09-16T10:18:52.829845+0000","last_became_peered":"2024-09-16T10:18:52.829845+0000","last_unstale":"2024-09-16T10:21:28.005751+0000","last_undegraded":"2024-09-16T10:21:28.005751+0000","last_fullsized":"2024-09-16T10:21:28.005751+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:13:40.081041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:00.234519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":407,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.008898+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:21:28.008898+0000","last_peered":"2024-09-16T10:21:28.008898+0000","last_clean":"2024-09-16T10:21:28.008898+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:21:28.008898+0000","last_undegraded":"2024-09-16T10:21:28.008898+0000","last_fullsized":"2024-09-16T10:21:28.008898+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:12.363006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":209,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004206+0000","last_change":"2024-09-16T10:21:18.904473+0000","last_active":"2024-09-16T10:21:28.004206+0000","last_peered":"2024-09-16T10:21:28.004206+0000","last_clean":"2024-09-16T10:21:28.004206+0000","last_became_active":"2024-09-16T10:21:12.879114+0000","last_became_peered":"2024-09-16T10:21:12.879114+0000","last_unstale":"2024-09-16T10:21:28.004206+0000","last_undegraded":"2024-09-16T10:21:28.004206+0000","last_fullsized":"2024-09-16T10:21:28.004206+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:18.904405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:18.904405+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:29.481979+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"249'14272","reported_seq":17806,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.368151+0000","last_change":"2024-09-16T10:20:53.772044+0000","last_active":"2024-09-16T10:21:28.368151+0000","last_peered":"2024-09-16T10:21:28.368151+0000","last_clean":"2024-09-16T10:21:28.368151+0000","last_became_active":"2024-09-16T10:19:14.291692+0000","last_became_peered":"2024-09-16T10:19:14.291692+0000","last_unstale":"2024-09-16T10:21:28.368151+0000","last_undegraded":"2024-09-16T10:21:28.368151+0000","last_fullsized":"2024-09-16T10:21:28.368151+0000","mapping_epoch":187,"log_start":"222'9970","ondisk_log_start":"222'9970","created":165,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'11311","last_scrub_stamp":"2024-09-16T10:20:53.771722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:18:32.798588+0000","last_clean_scrub_stamp":"2024-09-16T10:20:53.771722+0000","objects_scrubbed":4176,"log_size":4302,"log_dups_size":0,"ondisk_log_size":4302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:56.039707+0000","scrub_duration":1405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14272,"num_write_kb":493760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":416,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007036+0000","last_change":"2024-09-16T10:13:39.104706+0000","last_active":"2024-09-16T10:21:28.007036+0000","last_peered":"2024-09-16T10:21:28.007036+0000","last_clean":"2024-09-16T10:21:28.007036+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:21:28.007036+0000","last_undegraded":"2024-09-16T10:21:28.007036+0000","last_fullsized":"2024-09-16T10:21:28.007036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:13:38.076596+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:35.365771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013348+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:21:28.013348+0000","last_peered":"2024-09-16T10:21:28.013348+0000","last_clean":"2024-09-16T10:21:28.013348+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:21:28.013348+0000","last_undegraded":"2024-09-16T10:21:28.013348+0000","last_fullsized":"2024-09-16T10:21:28.013348+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:15.185950+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":506,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012771+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:21:28.012771+0000","last_peered":"2024-09-16T10:21:28.012771+0000","last_clean":"2024-09-16T10:21:28.012771+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:21:28.012771+0000","last_undegraded":"2024-09-16T10:21:28.012771+0000","last_fullsized":"2024-09-16T10:21:28.012771+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.056619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":407,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007930+0000","last_change":"2024-09-16T10:19:20.941840+0000","last_active":"2024-09-16T10:21:28.007930+0000","last_peered":"2024-09-16T10:21:28.007930+0000","last_clean":"2024-09-16T10:21:28.007930+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:21:28.007930+0000","last_undegraded":"2024-09-16T10:21:28.007930+0000","last_fullsized":"2024-09-16T10:21:28.007930+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.941753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.941753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:59.360989+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.011747+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:21:28.011747+0000","last_peered":"2024-09-16T10:21:28.011747+0000","last_clean":"2024-09-16T10:21:28.011747+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:21:28.011747+0000","last_undegraded":"2024-09-16T10:21:28.011747+0000","last_fullsized":"2024-09-16T10:21:28.011747+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.631488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":424,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003736+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:21:28.003736+0000","last_peered":"2024-09-16T10:21:28.003736+0000","last_clean":"2024-09-16T10:21:28.003736+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:21:28.003736+0000","last_undegraded":"2024-09-16T10:21:28.003736+0000","last_fullsized":"2024-09-16T10:21:28.003736+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:42.398743+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006165+0000","last_change":"2024-09-16T10:20:54.453057+0000","last_active":"2024-09-16T10:21:28.006165+0000","last_peered":"2024-09-16T10:21:28.006165+0000","last_clean":"2024-09-16T10:21:28.006165+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:21:28.006165+0000","last_undegraded":"2024-09-16T10:21:28.006165+0000","last_fullsized":"2024-09-16T10:21:28.006165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.452998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.452998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:01.123347+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":391,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012903+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:21:28.012903+0000","last_peered":"2024-09-16T10:21:28.012903+0000","last_clean":"2024-09-16T10:21:28.012903+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:21:28.012903+0000","last_undegraded":"2024-09-16T10:21:28.012903+0000","last_fullsized":"2024-09-16T10:21:28.012903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:45.047690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":430,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010760+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:21:28.010760+0000","last_peered":"2024-09-16T10:21:28.010760+0000","last_clean":"2024-09-16T10:21:28.010760+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:21:28.010760+0000","last_undegraded":"2024-09-16T10:21:28.010760+0000","last_fullsized":"2024-09-16T10:21:28.010760+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:12.148098+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013173+0000","last_change":"2024-09-16T10:19:22.431194+0000","last_active":"2024-09-16T10:21:28.013173+0000","last_peered":"2024-09-16T10:21:28.013173+0000","last_clean":"2024-09-16T10:21:28.013173+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:21:28.013173+0000","last_undegraded":"2024-09-16T10:21:28.013173+0000","last_fullsized":"2024-09-16T10:21:28.013173+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.431126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.431126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:01.301158+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003512+0000","last_change":"2024-09-16T10:20:32.229638+0000","last_active":"2024-09-16T10:21:28.003512+0000","last_peered":"2024-09-16T10:21:28.003512+0000","last_clean":"2024-09-16T10:21:28.003512+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:21:28.003512+0000","last_undegraded":"2024-09-16T10:21:28.003512+0000","last_fullsized":"2024-09-16T10:21:28.003512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:32.229564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:32.229564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:47.951807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":171,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002558+0000","last_change":"2024-09-16T10:20:19.951949+0000","last_active":"2024-09-16T10:21:28.002558+0000","last_peered":"2024-09-16T10:21:28.002558+0000","last_clean":"2024-09-16T10:21:28.002558+0000","last_became_active":"2024-09-16T10:18:52.822465+0000","last_became_peered":"2024-09-16T10:18:52.822465+0000","last_unstale":"2024-09-16T10:21:28.002558+0000","last_undegraded":"2024-09-16T10:21:28.002558+0000","last_fullsized":"2024-09-16T10:21:28.002558+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:43.200411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":460,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.014246+0000","last_change":"2024-09-16T10:21:19.170073+0000","last_active":"2024-09-16T10:21:28.014246+0000","last_peered":"2024-09-16T10:21:28.014246+0000","last_clean":"2024-09-16T10:21:28.014246+0000","last_became_active":"2024-09-16T10:21:12.870669+0000","last_became_peered":"2024-09-16T10:21:12.870669+0000","last_unstale":"2024-09-16T10:21:28.014246+0000","last_undegraded":"2024-09-16T10:21:28.014246+0000","last_fullsized":"2024-09-16T10:21:28.014246+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.170006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.170006+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:21.075615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007914+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:21:28.007914+0000","last_peered":"2024-09-16T10:21:28.007914+0000","last_clean":"2024-09-16T10:21:28.007914+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:21:28.007914+0000","last_undegraded":"2024-09-16T10:21:28.007914+0000","last_fullsized":"2024-09-16T10:21:28.007914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:39.395073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007812+0000","last_change":"2024-09-16T10:20:29.177634+0000","last_active":"2024-09-16T10:21:28.007812+0000","last_peered":"2024-09-16T10:21:28.007812+0000","last_clean":"2024-09-16T10:21:28.007812+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:21:28.007812+0000","last_undegraded":"2024-09-16T10:21:28.007812+0000","last_fullsized":"2024-09-16T10:21:28.007812+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:29.177563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:29.177563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:41.824549+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"175'9","reported_seq":533,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003424+0000","last_change":"2024-09-16T10:20:25.284531+0000","last_active":"2024-09-16T10:21:28.003424+0000","last_peered":"2024-09-16T10:21:28.003424+0000","last_clean":"2024-09-16T10:21:28.003424+0000","last_became_active":"2024-09-16T10:18:52.845153+0000","last_became_peered":"2024-09-16T10:18:52.845153+0000","last_unstale":"2024-09-16T10:21:28.003424+0000","last_undegraded":"2024-09-16T10:21:28.003424+0000","last_fullsized":"2024-09-16T10:21:28.003424+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:20:25.284462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:25.284462+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:28.278781+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002477+0000","last_change":"2024-09-16T10:20:57.931764+0000","last_active":"2024-09-16T10:21:28.002477+0000","last_peered":"2024-09-16T10:21:28.002477+0000","last_clean":"2024-09-16T10:21:28.002477+0000","last_became_active":"2024-09-16T10:18:52.821050+0000","last_became_peered":"2024-09-16T10:18:52.821050+0000","last_unstale":"2024-09-16T10:21:28.002477+0000","last_undegraded":"2024-09-16T10:21:28.002477+0000","last_fullsized":"2024-09-16T10:21:28.002477+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.931714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.931714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:26.199004+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":430,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002897+0000","last_change":"2024-09-16T10:19:45.045388+0000","last_active":"2024-09-16T10:21:28.002897+0000","last_peered":"2024-09-16T10:21:28.002897+0000","last_clean":"2024-09-16T10:21:28.002897+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:21:28.002897+0000","last_undegraded":"2024-09-16T10:21:28.002897+0000","last_fullsized":"2024-09-16T10:21:28.002897+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:19:45.045164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.045164+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013334+0000","last_change":"2024-09-16T10:18:52.820977+0000","last_active":"2024-09-16T10:21:28.013334+0000","last_peered":"2024-09-16T10:21:28.013334+0000","last_clean":"2024-09-16T10:21:28.013334+0000","last_became_active":"2024-09-16T10:18:52.820693+0000","last_became_peered":"2024-09-16T10:18:52.820693+0000","last_unstale":"2024-09-16T10:21:28.013334+0000","last_undegraded":"2024-09-16T10:21:28.013334+0000","last_fullsized":"2024-09-16T10:21:28.013334+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:07.290466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":514,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.007643+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:21:28.007643+0000","last_peered":"2024-09-16T10:21:28.007643+0000","last_clean":"2024-09-16T10:21:28.007643+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:21:28.007643+0000","last_undegraded":"2024-09-16T10:21:28.007643+0000","last_fullsized":"2024-09-16T10:21:28.007643+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:12.483221+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010462+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:21:28.010462+0000","last_peered":"2024-09-16T10:21:28.010462+0000","last_clean":"2024-09-16T10:21:28.010462+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:21:28.010462+0000","last_undegraded":"2024-09-16T10:21:28.010462+0000","last_fullsized":"2024-09-16T10:21:28.010462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:41.183832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":417,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.001266+0000","last_change":"2024-09-16T10:17:34.967648+0000","last_active":"2024-09-16T10:21:28.001266+0000","last_peered":"2024-09-16T10:21:28.001266+0000","last_clean":"2024-09-16T10:21:28.001266+0000","last_became_active":"2024-09-16T10:13:39.093597+0000","last_became_peered":"2024-09-16T10:13:39.093597+0000","last_unstale":"2024-09-16T10:21:28.001266+0000","last_undegraded":"2024-09-16T10:21:28.001266+0000","last_fullsized":"2024-09-16T10:21:28.001266+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:46.859976+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002648+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:21:28.002648+0000","last_peered":"2024-09-16T10:21:28.002648+0000","last_clean":"2024-09-16T10:21:28.002648+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:21:28.002648+0000","last_undegraded":"2024-09-16T10:21:28.002648+0000","last_fullsized":"2024-09-16T10:21:28.002648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:36.922512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003646+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:21:28.003646+0000","last_peered":"2024-09-16T10:21:28.003646+0000","last_clean":"2024-09-16T10:21:28.003646+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:21:28.003646+0000","last_undegraded":"2024-09-16T10:21:28.003646+0000","last_fullsized":"2024-09-16T10:21:28.003646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:12.287430+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.001995+0000","last_change":"2024-09-16T10:20:54.392034+0000","last_active":"2024-09-16T10:21:28.001995+0000","last_peered":"2024-09-16T10:21:28.001995+0000","last_clean":"2024-09-16T10:21:28.001995+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:21:28.001995+0000","last_undegraded":"2024-09-16T10:21:28.001995+0000","last_fullsized":"2024-09-16T10:21:28.001995+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.391970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.391970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.175737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":448,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004093+0000","last_change":"2024-09-16T10:20:50.296501+0000","last_active":"2024-09-16T10:21:28.004093+0000","last_peered":"2024-09-16T10:21:28.004093+0000","last_clean":"2024-09-16T10:21:28.004093+0000","last_became_active":"2024-09-16T10:18:52.832688+0000","last_became_peered":"2024-09-16T10:18:52.832688+0000","last_unstale":"2024-09-16T10:21:28.004093+0000","last_undegraded":"2024-09-16T10:21:28.004093+0000","last_fullsized":"2024-09-16T10:21:28.004093+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:16.056306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.013219+0000","last_change":"2024-09-16T10:19:47.608315+0000","last_active":"2024-09-16T10:21:28.013219+0000","last_peered":"2024-09-16T10:21:28.013219+0000","last_clean":"2024-09-16T10:21:28.013219+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:21:28.013219+0000","last_undegraded":"2024-09-16T10:21:28.013219+0000","last_fullsized":"2024-09-16T10:21:28.013219+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.608243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.608243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:10.274483+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":368,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.008975+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:21:28.008975+0000","last_peered":"2024-09-16T10:21:28.008975+0000","last_clean":"2024-09-16T10:21:28.008975+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:21:28.008975+0000","last_undegraded":"2024-09-16T10:21:28.008975+0000","last_fullsized":"2024-09-16T10:21:28.008975+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.331834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":390,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.005660+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:21:28.005660+0000","last_peered":"2024-09-16T10:21:28.005660+0000","last_clean":"2024-09-16T10:21:28.005660+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:21:28.005660+0000","last_undegraded":"2024-09-16T10:21:28.005660+0000","last_fullsized":"2024-09-16T10:21:28.005660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:06.286296+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.002256+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:21:28.002256+0000","last_peered":"2024-09-16T10:21:28.002256+0000","last_clean":"2024-09-16T10:21:28.002256+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:21:28.002256+0000","last_undegraded":"2024-09-16T10:21:28.002256+0000","last_fullsized":"2024-09-16T10:21:28.002256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.343855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.006296+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:21:28.006296+0000","last_peered":"2024-09-16T10:21:28.006296+0000","last_clean":"2024-09-16T10:21:28.006296+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:21:28.006296+0000","last_undegraded":"2024-09-16T10:21:28.006296+0000","last_fullsized":"2024-09-16T10:21:28.006296+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:11.444035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.014126+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:21:28.014126+0000","last_peered":"2024-09-16T10:21:28.014126+0000","last_clean":"2024-09-16T10:21:28.014126+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:21:28.014126+0000","last_undegraded":"2024-09-16T10:21:28.014126+0000","last_fullsized":"2024-09-16T10:21:28.014126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:06.862710+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":407,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.009102+0000","last_change":"2024-09-16T10:19:47.129026+0000","last_active":"2024-09-16T10:21:28.009102+0000","last_peered":"2024-09-16T10:21:28.009102+0000","last_clean":"2024-09-16T10:21:28.009102+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:21:28.009102+0000","last_undegraded":"2024-09-16T10:21:28.009102+0000","last_fullsized":"2024-09-16T10:21:28.009102+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.128964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.128964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":439,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003876+0000","last_change":"2024-09-16T10:19:56.200787+0000","last_active":"2024-09-16T10:21:28.003876+0000","last_peered":"2024-09-16T10:21:28.003876+0000","last_clean":"2024-09-16T10:21:28.003876+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:21:28.003876+0000","last_undegraded":"2024-09-16T10:21:28.003876+0000","last_fullsized":"2024-09-16T10:21:28.003876+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:19:56.200704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:56.200704+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":416,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010271+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:21:28.010271+0000","last_peered":"2024-09-16T10:21:28.010271+0000","last_clean":"2024-09-16T10:21:28.010271+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:21:28.010271+0000","last_undegraded":"2024-09-16T10:21:28.010271+0000","last_fullsized":"2024-09-16T10:21:28.010271+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:44.328316+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.011268+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:21:28.011268+0000","last_peered":"2024-09-16T10:21:28.011268+0000","last_clean":"2024-09-16T10:21:28.011268+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:21:28.011268+0000","last_undegraded":"2024-09-16T10:21:28.011268+0000","last_fullsized":"2024-09-16T10:21:28.011268+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:21:40.662531+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":169,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.012238+0000","last_change":"2024-09-16T10:19:44.487481+0000","last_active":"2024-09-16T10:21:28.012238+0000","last_peered":"2024-09-16T10:21:28.012238+0000","last_clean":"2024-09-16T10:21:28.012238+0000","last_became_active":"2024-09-16T10:18:52.838002+0000","last_became_peered":"2024-09-16T10:18:52.838002+0000","last_unstale":"2024-09-16T10:21:28.012238+0000","last_undegraded":"2024-09-16T10:21:28.012238+0000","last_fullsized":"2024-09-16T10:21:28.012238+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:09.296075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":407,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.003780+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:21:28.003780+0000","last_peered":"2024-09-16T10:21:28.003780+0000","last_clean":"2024-09-16T10:21:28.003780+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:21:28.003780+0000","last_undegraded":"2024-09-16T10:21:28.003780+0000","last_fullsized":"2024-09-16T10:21:28.003780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:14.257315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.004128+0000","last_change":"2024-09-16T10:20:58.426297+0000","last_active":"2024-09-16T10:21:28.004128+0000","last_peered":"2024-09-16T10:21:28.004128+0000","last_clean":"2024-09-16T10:21:28.004128+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:21:28.004128+0000","last_undegraded":"2024-09-16T10:21:28.004128+0000","last_fullsized":"2024-09-16T10:21:28.004128+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.426235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.426235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:13.172313+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":441,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.014729+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:21:28.014729+0000","last_peered":"2024-09-16T10:21:28.014729+0000","last_clean":"2024-09-16T10:21:28.014729+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:21:28.014729+0000","last_undegraded":"2024-09-16T10:21:28.014729+0000","last_fullsized":"2024-09-16T10:21:28.014729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:17.736482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":417,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-16T10:21:28.010575+0000","last_change":"2024-09-16T10:19:16.217856+0000","last_active":"2024-09-16T10:21:28.010575+0000","last_peered":"2024-09-16T10:21:28.010575+0000","last_clean":"2024-09-16T10:21:28.010575+0000","last_became_active":"2024-09-16T10:18:52.834424+0000","last_became_peered":"2024-09-16T10:18:52.834424+0000","last_unstale":"2024-09-16T10:21:28.010575+0000","last_undegraded":"2024-09-16T10:21:28.010575+0000","last_fullsized":"2024-09-16T10:21:28.010575+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:15.146482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1214906392,"num_objects":18539,"num_object_clones":0,"num_object_copies":55617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18539,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236097,"num_write_kb":8148289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10118,"num_bytes_recovered":663093248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4227739648,"data_stored":4227727432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68947,"ondisk_log_size":68947,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":465,"num_read_kb":460,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":17,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1106,"ondisk_log_size":1106,"up":51,"acting":51,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234082,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735528,"kb_used_data":455764,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93009624,"statfs":{"total":95995035648,"available":95241854976,"internally_reserved":0,"allocated":466702336,"data_stored":466056191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":286390730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528422,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860188,"kb_used_data":558280,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92884964,"statfs":{"total":95995035648,"available":95114203136,"internally_reserved":0,"allocated":571678720,"data_stored":571037522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309131718},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790060,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1001500,"kb_used_data":727372,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92743652,"statfs":{"total":95995035648,"available":94969499648,"internally_reserved":0,"allocated":744828928,"data_stored":744184958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986289,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740704,"kb_used_data":466572,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93004448,"statfs":{"total":95995035648,"available":95236554752,"internally_reserved":0,"allocated":477769728,"data_stored":477124439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215222,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":838532,"kb_used_data":522608,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92906620,"statfs":{"total":95995035648,"available":95136378880,"internally_reserved":0,"allocated":535150592,"data_stored":534477933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":323353032},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826176,"kb_used_data":524080,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92918976,"statfs":{"total":95995035648,"available":95149031424,"internally_reserved":0,"allocated":536657920,"data_stored":535983872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738501,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761404,"kb_used_data":459488,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92983748,"statfs":{"total":95995035648,"available":95215357952,"internally_reserved":0,"allocated":470515712,"data_stored":469858177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446365696,"data_stored":446365696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":530120704,"data_stored":530120704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":562823168,"data_stored":562823168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":522911744,"data_stored":522911744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":465240064,"data_stored":465240064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":692457472,"data_stored":692453400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":544149504,"data_stored":544145432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":463671296,"data_stored":463667224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:21:33.407 INFO:tasks.thrashosds.thrasher:Setting 3.10 to [3, 1, 6, 5, 2, 7] 2024-09-16T10:21:33.407 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.10', '3', '1', '6', '5', '2', '7'] 2024-09-16T10:21:33.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 3.10 3 1 6 5 2 7 2024-09-16T10:21:33.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:33.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:33.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:33 smithi165 ceph-mon[32696]: 4.c scrub starts 2024-09-16T10:21:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:33 smithi165 ceph-mon[32696]: 4.c scrub ok 2024-09-16T10:21:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:33 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-16T10:21:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:33 smithi165 ceph-mon[32696]: osdmap e253: 8 total, 8 up, 7 in 2024-09-16T10:21:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:33 smithi165 ceph-mon[32696]: from='client.45512 v1:172.21.15.63:0/2503843296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:21:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:33.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:33.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:33.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:33.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:21:33.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:33.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:33.844 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:21:33.910 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:33.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:33.936 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:33.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:33.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:34.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:34.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:34.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:34.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:34.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:34.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:34.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:34.360 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:34.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:34.369 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:34 smithi063 ceph-mon[29993]: osdmap e254: 8 total, 8 up, 7 in 2024-09-16T10:21:34.369 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:34 smithi063 ceph-mon[29993]: pgmap v407: 164 pgs: 1 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail 2024-09-16T10:21:34.369 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:34 smithi063 ceph-mon[39470]: osdmap e254: 8 total, 8 up, 7 in 2024-09-16T10:21:34.370 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:34 smithi063 ceph-mon[39470]: pgmap v407: 164 pgs: 1 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail 2024-09-16T10:21:34.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:34.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:34.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:34 smithi165 ceph-mon[32696]: osdmap e254: 8 total, 8 up, 7 in 2024-09-16T10:21:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:34 smithi165 ceph-mon[32696]: pgmap v407: 164 pgs: 1 peering, 163 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail 2024-09-16T10:21:34.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:34.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:34.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:34.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:34.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:34.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:34.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:34.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:34.919 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:34.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:35.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:35.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:35.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:35.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:35.198 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:35.223 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:35.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:35.242 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:35.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:35.348 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[29993]: osdmap e255: 8 total, 8 up, 7 in 2024-09-16T10:21:35.348 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[29993]: 4.1e scrub starts 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[29993]: 4.1e scrub ok 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[29993]: 1.f scrub ok 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[39470]: osdmap e255: 8 total, 8 up, 7 in 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[39470]: 4.1e scrub starts 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[39470]: 4.1e scrub ok 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:21:35.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:21:35.350 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:21:35.350 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:35 smithi063 ceph-mon[39470]: 1.f scrub ok 2024-09-16T10:21:35.366 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:35.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:35.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:35.513 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:35.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:35 smithi165 ceph-mon[32696]: osdmap e255: 8 total, 8 up, 7 in 2024-09-16T10:21:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:35 smithi165 ceph-mon[32696]: 4.1e scrub starts 2024-09-16T10:21:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:35 smithi165 ceph-mon[32696]: 4.1e scrub ok 2024-09-16T10:21:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:35 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:21:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:35 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:21:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:35 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:21:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:35 smithi165 ceph-mon[32696]: 1.f scrub ok 2024-09-16T10:21:35.616 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:35.648 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:35.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:35.749 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:35.783 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:35.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:35.885 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:35.917 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:35.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:36.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:36.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:36.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:36.101 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:21:36.107 INFO:teuthology.orchestra.run.smithi063.stderr:set 3.10 pg_upmap_items mapping to [3->1,6->5,2->7] 2024-09-16T10:21:36.158 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:36.186 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:36.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:36.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:36.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:36.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:36.404 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: osdmap e256: 8 total, 8 up, 7 in 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: pgmap v410: 163 pgs: 1 clean+premerge+peered, 162 active+clean; 718 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1331445750' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: 6.1 scrub starts 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: 6.1 scrub ok 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3126931039' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]} : dispatch 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[29993]: from='client.25573 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]} : dispatch 2024-09-16T10:21:36.405 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: osdmap e256: 8 total, 8 up, 7 in 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: pgmap v410: 163 pgs: 1 clean+premerge+peered, 162 active+clean; 718 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1331445750' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: 6.1 scrub starts 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: 6.1 scrub ok 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3126931039' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]} : dispatch 2024-09-16T10:21:36.406 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:36 smithi063 ceph-mon[39470]: from='client.25573 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]} : dispatch 2024-09-16T10:21:36.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:36.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:36.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: osdmap e256: 8 total, 8 up, 7 in 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: pgmap v410: 163 pgs: 1 clean+premerge+peered, 162 active+clean; 718 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1331445750' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: 6.1 scrub starts 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: 6.1 scrub ok 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3126931039' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]} : dispatch 2024-09-16T10:21:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:36 smithi165 ceph-mon[32696]: from='client.25573 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]} : dispatch 2024-09-16T10:21:36.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:36.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:36.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:36.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:36.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:36.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:36.827 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:36.862 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:36.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:36.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:37.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:37.106 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:37.136 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:37.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:37.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:37.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:37.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[29993]: 4.8 scrub starts 2024-09-16T10:21:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[29993]: 4.8 scrub ok 2024-09-16T10:21:37.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1331445750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[29993]: from='client.25573 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]}]': finished 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[29993]: osdmap e257: 8 total, 8 up, 7 in 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:21:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[39470]: 4.8 scrub starts 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[39470]: 4.8 scrub ok 2024-09-16T10:21:37.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-16T10:21:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1331445750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:21:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[39470]: from='client.25573 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]}]': finished 2024-09-16T10:21:37.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:37 smithi063 ceph-mon[39470]: osdmap e257: 8 total, 8 up, 7 in 2024-09-16T10:21:37.516 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:37.547 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:37 smithi165 ceph-mon[32696]: 4.8 scrub starts 2024-09-16T10:21:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:37 smithi165 ceph-mon[32696]: 4.8 scrub ok 2024-09-16T10:21:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:37 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:37 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-16T10:21:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1331445750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:21:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:37 smithi165 ceph-mon[32696]: from='client.25573 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.10", "id": [3, 1, 6, 5, 2, 7]}]': finished 2024-09-16T10:21:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:37 smithi165 ceph-mon[32696]: osdmap e257: 8 total, 8 up, 7 in 2024-09-16T10:21:37.648 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:37.681 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:37.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:37.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:37.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:37.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:37.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:38.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:38.087 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:38.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:38.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:38.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:38.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:38.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:38.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:38.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:38.393 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:38 smithi063 ceph-mon[29993]: osdmap e258: 8 total, 8 up, 7 in 2024-09-16T10:21:38.393 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:38 smithi063 ceph-mon[29993]: pgmap v413: 163 pgs: 1 clean+premerge+peered, 162 active+clean; 718 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-09-16T10:21:38.394 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:38 smithi063 ceph-mon[39470]: osdmap e258: 8 total, 8 up, 7 in 2024-09-16T10:21:38.394 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:38 smithi063 ceph-mon[39470]: pgmap v413: 163 pgs: 1 clean+premerge+peered, 162 active+clean; 718 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-09-16T10:21:38.453 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:38.454 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:38.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:38.492 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:38.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:38.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:38 smithi165 ceph-mon[32696]: osdmap e258: 8 total, 8 up, 7 in 2024-09-16T10:21:38.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:38 smithi165 ceph-mon[32696]: pgmap v413: 163 pgs: 1 clean+premerge+peered, 162 active+clean; 718 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-09-16T10:21:38.530 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:38.530 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:38.565 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:38.565 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:38.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:38.600 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:38.601 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:38.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:38.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:38.639 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:38.679 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:38.680 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:38.682 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:38.682 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:21:38.682 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:21:38.682 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:21:38.682 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:38.713 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:38.713 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:38.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:38.805 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:38.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:38.806 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:21:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:21:38.906 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:38.946 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:38.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:39.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:39.080 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:39.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:39.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:39.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:39.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:39.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:39.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:39.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[29993]: osdmap e259: 8 total, 8 up, 7 in 2024-09-16T10:21:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1673859510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:39.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[39470]: osdmap e259: 8 total, 8 up, 7 in 2024-09-16T10:21:39.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:39.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1673859510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:39.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:39.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:39.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:39.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:39.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:39 smithi165 ceph-mon[32696]: osdmap e259: 8 total, 8 up, 7 in 2024-09-16T10:21:39.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:39.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1673859510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:39.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:39.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:39.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:39.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:39.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:39.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:39.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:39.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:39.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:39.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:40.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:40.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:40.150 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:40.182 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:40.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:40.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:40.319 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:40.421 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:40.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[29993]: osdmap e260: 8 total, 8 up, 7 in 2024-09-16T10:21:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[29993]: pgmap v416: 162 pgs: 1 peering, 161 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:21:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:21:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[39470]: osdmap e260: 8 total, 8 up, 7 in 2024-09-16T10:21:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[39470]: pgmap v416: 162 pgs: 1 peering, 161 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:21:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:40 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:21:40.454 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:40 smithi165 ceph-mon[32696]: osdmap e260: 8 total, 8 up, 7 in 2024-09-16T10:21:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:40 smithi165 ceph-mon[32696]: pgmap v416: 162 pgs: 1 peering, 161 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:40 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:21:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:40 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:21:40.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.588 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:40.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:40.689 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:40.723 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:40.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.859 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:40.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:40.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:40.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:40.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:41.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:41.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:41.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:41.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:41.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:41.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:41.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:21:41.363 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:41 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-16T10:21:41.363 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:41 smithi063 ceph-mon[29993]: osdmap e261: 8 total, 8 up, 7 in 2024-09-16T10:21:41.363 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:41 smithi063 ceph-mon[29993]: osdmap e262: 8 total, 8 up, 7 in 2024-09-16T10:21:41.363 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:41 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-16T10:21:41.363 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:41 smithi063 ceph-mon[39470]: osdmap e261: 8 total, 8 up, 7 in 2024-09-16T10:21:41.364 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:41 smithi063 ceph-mon[39470]: osdmap e262: 8 total, 8 up, 7 in 2024-09-16T10:21:41.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:41.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:41.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:41.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:41 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-16T10:21:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:41 smithi165 ceph-mon[32696]: osdmap e261: 8 total, 8 up, 7 in 2024-09-16T10:21:41.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:41 smithi165 ceph-mon[32696]: osdmap e262: 8 total, 8 up, 7 in 2024-09-16T10:21:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:41.565 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:41.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:41.614 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:21:41.666 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:41.721 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:41.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:41.755 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Removed 127317 objects 2024-09-16T10:21:41.755 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Clean up completed and total clean up time :90.2395 2024-09-16T10:21:41.777 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:41.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:41.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:41.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:41.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:41.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:41.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:42.069 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:42.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:42.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:42.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:42.197 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:42 smithi063 ceph-mon[39470]: pgmap v419: 162 pgs: 1 peering, 161 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:42.197 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:42 smithi063 ceph-mon[39470]: osdmap e263: 8 total, 8 up, 7 in 2024-09-16T10:21:42.197 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:42 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:21:42.197 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:42 smithi063 ceph-mon[29993]: pgmap v419: 162 pgs: 1 peering, 161 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:42.197 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:42 smithi063 ceph-mon[29993]: osdmap e263: 8 total, 8 up, 7 in 2024-09-16T10:21:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:42 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:21:42.220 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:42.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:42.328 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:42.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:42.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:42.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:42 smithi165 ceph-mon[32696]: pgmap v419: 162 pgs: 1 peering, 161 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-09-16T10:21:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:42 smithi165 ceph-mon[32696]: osdmap e263: 8 total, 8 up, 7 in 2024-09-16T10:21:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:42 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:21:42.573 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:42.595 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:42.697 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:42.719 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:42.820 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:21:42.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:42.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:42.872 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:21:42.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:42.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:42.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:43.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:43.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:43.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:43.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:43.215 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:43.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:43.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:43.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:43.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:43.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:21:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2958607666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2958607666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[29993]: osdmap e264: 8 total, 8 up, 7 in 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2958607666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2958607666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:43.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:43 smithi063 ceph-mon[39470]: osdmap e264: 8 total, 8 up, 7 in 2024-09-16T10:21:43.465 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:43.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:43 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:21:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:43 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:21:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:43 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:21:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2958607666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:21:43.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:43 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:43.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2958607666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:21:43.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:43 smithi165 ceph-mon[32696]: osdmap e264: 8 total, 8 up, 7 in 2024-09-16T10:21:43.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:43.585 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:40100 7.e 7:7326c3b0:::benchmark_data_smithi063_73086_object40099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 393.27965139600002, 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.113082665, 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.586 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 40100 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546898+0000", 2024-09-16T10:21:43.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546900+0000", 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2330000000000001e-06 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546927+0000", 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6721999999999999e-05 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546939+0000", 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1793999999999999e-05 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.546947+0000", 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.704e-06 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547026+0000", 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8479999999999994e-05 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547144+0000", 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011862999999999999 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.547278+0000", 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013377800000000001 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.548993+0000", 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001715261 2024-09-16T10:21:43.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.549008+0000", 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4158000000000001e-05 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659763+0000", 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.110755881 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659886+0000", 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012276600000000001 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.659980+0000", 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4258000000000005e-05 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:43495 7.e 7:7a740846:::benchmark_data_smithi063_73086_object43494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T10:21:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 391.07365378999998, 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115699248, 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 43495 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752895+0000", 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752898+0000", 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1710000000000001e-06 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752922+0000", 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4593e-05 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752933+0000", 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1083999999999999e-05 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.752941+0000", 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6920000000000002e-06 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753001+0000", 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0109000000000003e-05 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753158+0000", 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000156668 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.753289+0000", 2024-09-16T10:21:43.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013114999999999999 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.755266+0000", 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00197679 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.755404+0000", 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013795999999999999 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868497+0000", 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11309365 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868554+0000", 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6641999999999997e-05 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:11.868595+0000", 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0738999999999997e-05 2024-09-16T10:21:43.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65096 7.e 7:7bb27b3e:::benchmark_data_smithi063_73086_object65095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 375.26264553599998, 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.125924595, 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.599 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65096 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563905+0000", 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.606e-06 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563926+0000", 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1009000000000002e-05 2024-09-16T10:21:43.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563936+0000", 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6460000000000005e-06 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563942+0000", 2024-09-16T10:21:43.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1589999999999998e-06 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563991+0000", 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9279e-05 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564082+0000", 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0890000000000006e-05 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564170+0000", 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7435000000000006e-05 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.565824+0000", 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001654345 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.566103+0000", 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027876700000000002 2024-09-16T10:21:43.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689664+0000", 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123561491 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689778+0000", 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113677 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689828+0000", 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0290999999999997e-05 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70300 7.8 7:10588a9d:::benchmark_data_smithi063_73086_object70299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 370.92345610000001, 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12214414, 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.606 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70300 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903095+0000", 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9099999999999999e-06 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903119+0000", 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3856999999999999e-05 2024-09-16T10:21:43.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903132+0000", 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2873e-05 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903139+0000", 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9739999999999996e-06 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903195+0000", 2024-09-16T10:21:43.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6121e-05 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903309+0000", 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113962 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903448+0000", 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013888799999999999 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905996+0000", 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025485479999999999 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909244+0000", 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032479420000000002 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.024965+0000", 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115720986 2024-09-16T10:21:43.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025185+0000", 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220177 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025237+0000", 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1901999999999999e-05 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70303 7.8 7:176924a2:::benchmark_data_smithi063_73086_object70302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 370.921155645, 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120210123, 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70303 2024-09-16T10:21:43.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905396+0000", 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5900000000000002e-06 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905422+0000", 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6364999999999999e-05 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905438+0000", 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5795e-05 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905448+0000", 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8020000000000003e-06 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906033+0000", 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058453800000000001 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906161+0000", 2024-09-16T10:21:43.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012883799999999999 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906300+0000", 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013873499999999999 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909282+0000", 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029820070000000001 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.910327+0000", 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001044332 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025288+0000", 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11496116100000001 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025559+0000", 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027147 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025604+0000", 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4490000000000003e-05 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72689 7.e 7:7f5699d3:::benchmark_data_smithi063_73086_object72688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 368.95706187399998, 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12073505599999999, 2024-09-16T10:21:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72689 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869489+0000", 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.237e-06 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869521+0000", 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2728999999999998e-05 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869531+0000", 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8989999999999996e-06 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869539+0000", 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2209999999999997e-06 2024-09-16T10:21:43.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872936+0000", 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003396296 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873090+0000", 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015414000000000001 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873225+0000", 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013536000000000001 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878402+0000", 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0051764419999999998 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878413+0000", 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1810999999999999e-05 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.989923+0000", 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.111509897 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990168+0000", 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00024499400000000001 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990222+0000", 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4030000000000003e-05 2024-09-16T10:21:43.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72696 7.e 7:7eb0d49d:::benchmark_data_smithi063_73086_object72695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 368.95376248899998, 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120396989, 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.626 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72696 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:21:43.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872789+0000", 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.979e-06 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872820+0000", 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1522000000000001e-05 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872830+0000", 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7550000000000006e-06 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872837+0000", 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7399999999999998e-06 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875523+0000", 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026860759999999999 2024-09-16T10:21:43.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875632+0000", 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000109343 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875757+0000", 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012532799999999999 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878452+0000", 2024-09-16T10:21:43.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026941249999999999 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.880841+0000", 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023892340000000001 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993090+0000", 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.112248941 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993145+0000", 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5043999999999997e-05 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993184+0000", 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8902000000000001e-05 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72702 7.e 7:720829b1:::benchmark_data_smithi063_73086_object72701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:21:43.632 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 368.94849870299998, 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119080255, 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72702 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878052+0000", 2024-09-16T10:21:43.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3689999999999999e-06 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878078+0000", 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6140999999999999e-05 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878088+0000", 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0147e-05 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878096+0000", 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5089999999999996e-06 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878485+0000", 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00038902000000000001 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878652+0000", 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016732800000000001 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878733+0000", 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0873999999999999e-05 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.883480+0000", 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047467120000000002 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884355+0000", 2024-09-16T10:21:43.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087525899999999998 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997027+0000", 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11267184399999999 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997084+0000", 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.766e-05 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997131+0000", 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6391999999999999e-05 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72710 7.e 7:792abf92:::benchmark_data_smithi063_73086_object72709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 368.94230004899998, 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11643550799999999, 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.639 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72710 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884249+0000", 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884251+0000", 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8929999999999999e-06 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884276+0000", 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4837999999999998e-05 2024-09-16T10:21:43.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884287+0000", 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1464000000000001e-05 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884295+0000", 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2300000000000002e-06 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884389+0000", 2024-09-16T10:21:43.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4073000000000005e-05 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884555+0000", 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000166782 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884698+0000", 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000142917 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888931+0000", 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042330529999999996 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984052+0000", 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.095120447999999996 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000566+0000", 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016514217000000001 2024-09-16T10:21:43.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000630+0000", 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4459000000000001e-05 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.000685+0000", 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4134e-05 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72712 7.e 7:7d21703e:::benchmark_data_smithi063_73086_object72711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 368.941978592, 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.118747798, 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72712 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.646 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884572+0000", 2024-09-16T10:21:43.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4270000000000001e-06 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884607+0000", 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4535000000000001e-05 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884617+0000", 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0774000000000001e-05 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884625+0000", 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.644e-06 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.886978+0000", 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00235257 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887080+0000", 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010283 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887167+0000", 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6706999999999995e-05 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984059+0000", 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096891692000000001 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984099+0000", 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0516000000000003e-05 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003069+0000", 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.018970035999999999 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003200+0000", 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000130961 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003318+0000", 2024-09-16T10:21:43.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118106 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72715 7.e 7:70bb1898:::benchmark_data_smithi063_73086_object72714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 368.93834290299998, 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115258741, 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.652 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72715 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888206+0000", 2024-09-16T10:21:43.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888208+0000", 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0990000000000001e-06 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888233+0000", 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4961999999999999e-05 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888243+0000", 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3680000000000008e-06 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888248+0000", 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1479999999999997e-06 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.888960+0000", 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00071208700000000003 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889036+0000", 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6175999999999997e-05 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981822+0000", 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.092785459000000001 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984260+0000", 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002438127 2024-09-16T10:21:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.986355+0000", 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020951099999999999 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003378+0000", 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.017022926000000001 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003430+0000", 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1847e-05 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003465+0000", 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5432000000000002e-05 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:72718 7.2 e107/81 7:450c0e00:::benchmark_data_smithi063_73086_object72717:head v 107'4483, mlcod=107'4481)", 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:21:43.658 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 368.93717875099998, 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.094911379000000004, 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889370+0000", 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889372+0000", 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3349999999999999e-06 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.889396+0000", 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3961999999999999e-05 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981880+0000", 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.092484449999999996 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981892+0000", 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1661000000000001e-05 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981956+0000", 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4168e-05 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.981973+0000", 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7357999999999999e-05 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984187+0000", 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022133159999999999 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984282+0000", 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5129000000000003e-05 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90069 7.e 7:7df0bbc7:::benchmark_data_smithi063_73086_object90068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 353.30284018999998, 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123224003, 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.663 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90069 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523711+0000", 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6309999999999999e-06 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523734+0000", 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3002000000000001e-05 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523745+0000", 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1173e-05 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523752+0000", 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2579999999999998e-06 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523836+0000", 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3758999999999995e-05 2024-09-16T10:21:43.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523945+0000", 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010883999999999999 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524079+0000", 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013397600000000001 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640421+0000", 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116342404 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640488+0000", 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6561000000000002e-05 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646786+0000", 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0062988599999999999 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646881+0000", 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4291999999999998e-05 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646933+0000", 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2247000000000003e-05 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.669 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.44750.0:90069 7.e e112/81 ondisk, result = 0)", 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.526896+0000", 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 353.29965283199999, 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11359791399999999, 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526896+0000", 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526896+0000", 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526897+0000", 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8899999999999998e-07 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.526901+0000", 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5999999999999998e-06 2024-09-16T10:21:43.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640420+0000", 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.113519355 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640428+0000", 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8830000000000006e-06 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640475+0000", 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6498000000000001e-05 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640483+0000", 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0369999999999995e-06 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640494+0000", 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1652e-05 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.673 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:106093 7.e 7:79e3a9e4:::benchmark_data_smithi063_73086_object106092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:16:06.441330+0000", 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 336.38521881999998, 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.101189283, 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.674 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 106093 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441330+0000", 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441330+0000", 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441333+0000", 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7760000000000002e-06 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441358+0000", 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4862e-05 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441372+0000", 2024-09-16T10:21:43.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4235e-05 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441381+0000", 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4149999999999999e-06 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441443+0000", 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2061000000000001e-05 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441608+0000", 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016502500000000001 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.441739+0000", 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013140299999999999 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.445301+0000", 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003561678 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.446064+0000", 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00076295200000000001 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.542395+0000", 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096331128000000002 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.542463+0000", 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8294999999999996e-05 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.542520+0000", 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6453000000000003e-05 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:106096 7.e 7:7328d7e9:::benchmark_data_smithi063_73086_object106095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:16:06.447115+0000", 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 336.37943436299997, 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.099332932999999998, 2024-09-16T10:21:43.680 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 106096 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447115+0000", 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447115+0000", 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447117+0000", 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2809999999999998e-06 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447141+0000", 2024-09-16T10:21:43.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4355e-05 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447206+0000", 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4389999999999998e-05 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447214+0000", 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5830000000000007e-06 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447332+0000", 2024-09-16T10:21:43.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117148 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447469+0000", 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013717699999999999 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447611+0000", 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014180200000000001 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.455600+0000", 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079892939999999992 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.455800+0000", 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000200514 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546278+0000", 2024-09-16T10:21:43.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.090477345000000001 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546399+0000", 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121357 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546448+0000", 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8687000000000003e-05 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:106097 7.e 7:77c251f7:::benchmark_data_smithi063_73086_object106096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:16:06.447290+0000", 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 336.379259284, 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.100309696, 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 106097 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.687 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447290+0000", 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447290+0000", 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447293+0000", 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5480000000000002e-06 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447317+0000", 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3870999999999999e-05 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447346+0000", 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8588000000000001e-05 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.447358+0000", 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2554e-05 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.452036+0000", 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046777880000000004 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.452168+0000", 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131891 2024-09-16T10:21:43.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.452302+0000", 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013436700000000001 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.456017+0000", 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037144750000000001 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.458856+0000", 2024-09-16T10:21:43.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028388659999999998 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.546498+0000", 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.087642128999999999 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.547428+0000", 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00092988800000000003 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:16:06.547600+0000", 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000171731 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:200781 7.e 7:72d3e70a:::benchmark_data_smithi063_73086_object81502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:40.222619+0000", 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 242.60393028999999, 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.109932639, 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.693 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 200781 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222619+0000", 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222619+0000", 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222620+0000", 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.201e-06 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222625+0000", 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5000000000000001e-06 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222634+0000", 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.577e-06 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.222641+0000", 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4790000000000001e-06 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.227085+0000", 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044442830000000003 2024-09-16T10:21:43.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.227204+0000", 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118829 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.227971+0000", 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00076681399999999999 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.234790+0000", 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0068190100000000003 2024-09-16T10:21:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.235401+0000", 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061134600000000002 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332436+0000", 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.097035225000000003 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332505+0000", 2024-09-16T10:21:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9275000000000003e-05 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332552+0000", 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6100000000000002e-05 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:200797 7.e 7:7d66ac12:::benchmark_data_smithi063_73086_object81518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:40.236303+0000", 2024-09-16T10:21:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 242.590245713, 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096365922000000007, 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 200797 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236303+0000", 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236303+0000", 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236305+0000", 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7829999999999999e-06 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236310+0000", 2024-09-16T10:21:43.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9799999999999998e-06 2024-09-16T10:21:43.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236321+0000", 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0311e-05 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236328+0000", 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4549999999999998e-06 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236384+0000", 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6155e-05 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236489+0000", 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000104453 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.236592+0000", 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103695 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.245646+0000", 2024-09-16T10:21:43.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0090532879999999996 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.245736+0000", 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0842999999999997e-05 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332593+0000", 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.086856218999999998 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332636+0000", 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3544000000000002e-05 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:40.332669+0000", 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3195999999999999e-05 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:201853 7.8 7:1d3a7237:::benchmark_data_smithi063_73086_object82574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:17:41.386689+0000", 2024-09-16T10:21:43.706 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 241.43986031, 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10598900999999999, 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 201853 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386689+0000", 2024-09-16T10:21:43.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386689+0000", 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386691+0000", 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.829e-06 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:43.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386696+0000", 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6239999999999999e-06 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386708+0000", 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1989e-05 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386716+0000", 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8699999999999992e-06 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386776+0000", 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0087999999999998e-05 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.386908+0000", 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131321 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.387012+0000", 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010425999999999999 2024-09-16T10:21:43.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.390595+0000", 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0035829210000000002 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.390683+0000", 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8437999999999996e-05 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.492600+0000", 2024-09-16T10:21:43.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10191681900000001 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.492643+0000", 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2914000000000003e-05 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:17:41.492678+0000", 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4937000000000001e-05 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.712 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.713 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:43.713 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:43.713 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:43.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:43.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:43.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:43.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:43.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:43.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:43.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:43.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:44.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:44.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:44.187 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:44.210 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:44.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:44.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:44.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:44 smithi063 ceph-mon[29993]: pgmap v422: 161 pgs: 1 peering, 160 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-16T10:21:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:44 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"} : dispatch 2024-09-16T10:21:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:44 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"} : dispatch 2024-09-16T10:21:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:44 smithi063 ceph-mon[39470]: pgmap v422: 161 pgs: 1 peering, 160 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-16T10:21:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:44 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"} : dispatch 2024-09-16T10:21:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:44 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"} : dispatch 2024-09-16T10:21:44.458 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:44.499 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-16T10:21:44.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-16T10:21:44.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:44 smithi165 ceph-mon[32696]: pgmap v422: 161 pgs: 1 peering, 160 active+clean; 296 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail 2024-09-16T10:21:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:44 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"} : dispatch 2024-09-16T10:21:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:44 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"} : dispatch 2024-09-16T10:21:44.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:44.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:44.685 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:44.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:21:44.711 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:44.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:44.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:44.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:44.960 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:44.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:45.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:45.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:45.190 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:45.212 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:45.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:45.340 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:45.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:45 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-09-16T10:21:45.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:45 smithi063 ceph-mon[29993]: osdmap e265: 8 total, 8 up, 7 in 2024-09-16T10:21:45.397 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:45 smithi063 ceph-mon[29993]: osdmap e266: 8 total, 8 up, 7 in 2024-09-16T10:21:45.397 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:45 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-09-16T10:21:45.397 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:45 smithi063 ceph-mon[39470]: osdmap e265: 8 total, 8 up, 7 in 2024-09-16T10:21:45.397 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:45 smithi063 ceph-mon[39470]: osdmap e266: 8 total, 8 up, 7 in 2024-09-16T10:21:45.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.468 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:45.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:45.472 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:45.495 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:45 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-09-16T10:21:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:45 smithi165 ceph-mon[32696]: osdmap e265: 8 total, 8 up, 7 in 2024-09-16T10:21:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:45 smithi165 ceph-mon[32696]: osdmap e266: 8 total, 8 up, 7 in 2024-09-16T10:21:45.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:45.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:45.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:45.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:45.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:45.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:45.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:45.959 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:21:45.959 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:21:45.959 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-16T10:21:45.960 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T10:21:45.960 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T10:21:45.960 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-16T10:21:45.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:45.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:46.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:46.109 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:46.204 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:46.204 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:46.204 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:46.204 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:46.204 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:46.204 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:46.205 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:46.205 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:46.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:46.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:46.337 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:46.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:46.435 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:46 smithi063 ceph-mon[29993]: pgmap v425: 161 pgs: 161 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:21:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3201700026' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:46 smithi063 ceph-mon[39470]: pgmap v425: 161 pgs: 161 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:21:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3201700026' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:46.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:46.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:46 smithi165 ceph-mon[32696]: pgmap v425: 161 pgs: 161 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-16T10:21:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3201700026' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:46.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:46.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:46.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:46.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:46.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:46.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:46.861 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:46.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:46.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:46.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:47.087 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:47.109 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:47.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:47.166 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_1' removed 2024-09-16T10:21:47.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:47.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:47.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:47.313 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:21:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:21:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:21:47.314 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:47 smithi063 ceph-mon[39470]: osdmap e267: 8 total, 8 up, 7 in 2024-09-16T10:21:47.314 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1529394980' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:21:47.314 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:47 smithi063 ceph-mon[29993]: osdmap e267: 8 total, 8 up, 7 in 2024-09-16T10:21:47.314 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1529394980' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:21:47.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:47.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:47.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:47.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:21:47.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:47.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:47.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:47.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:47 smithi165 ceph-mon[32696]: osdmap e267: 8 total, 8 up, 7 in 2024-09-16T10:21:47.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1529394980' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:21:47.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:47.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:47.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:47.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:47.738 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:21:47.738 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:21:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:47.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:47.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:47.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:47.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:47.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:48.010 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:48.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:48.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:48.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:48.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:48.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.389 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:48.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1529394980' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:21:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[29993]: osdmap e268: 8 total, 8 up, 7 in 2024-09-16T10:21:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[29993]: pgmap v428: 144 pgs: 144 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:21:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"} : dispatch 2024-09-16T10:21:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"} : dispatch 2024-09-16T10:21:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1529394980' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:21:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[39470]: osdmap e268: 8 total, 8 up, 7 in 2024-09-16T10:21:48.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[39470]: pgmap v428: 144 pgs: 144 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:21:48.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"} : dispatch 2024-09-16T10:21:48.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:48 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"} : dispatch 2024-09-16T10:21:48.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:48.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:48.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:48.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1529394980' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:21:48.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:48 smithi165 ceph-mon[32696]: osdmap e268: 8 total, 8 up, 7 in 2024-09-16T10:21:48.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:48 smithi165 ceph-mon[32696]: pgmap v428: 144 pgs: 144 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:21:48.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:48 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"} : dispatch 2024-09-16T10:21:48.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:48 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"} : dispatch 2024-09-16T10:21:48.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:48.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:48.646 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:21:48.693 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:48.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:48.768 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:48.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:21:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:21:48.869 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:48.895 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:48.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:48.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:49.018 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:49.053 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:21:49.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:49.143 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:49.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-09-16T10:21:49.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:49 smithi063 ceph-mon[39470]: osdmap e269: 8 total, 8 up, 7 in 2024-09-16T10:21:49.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:49.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:49.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:49.372 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.396 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:49.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:49.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-09-16T10:21:49.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:49 smithi063 ceph-mon[29993]: osdmap e269: 8 total, 8 up, 7 in 2024-09-16T10:21:49.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:49.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:49.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-09-16T10:21:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:49 smithi165 ceph-mon[32696]: osdmap e269: 8 total, 8 up, 7 in 2024-09-16T10:21:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:49.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:49.651 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:49.753 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:49.779 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:49.881 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:49.909 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:49.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:50.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:50.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:50.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:50.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:50.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:50.293 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:50.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:50.422 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:50.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:50 smithi063 ceph-mon[29993]: osdmap e270: 8 total, 8 up, 7 in 2024-09-16T10:21:50.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:50 smithi063 ceph-mon[29993]: pgmap v431: 144 pgs: 2 peering, 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:21:50.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:50 smithi063 ceph-mon[39470]: osdmap e270: 8 total, 8 up, 7 in 2024-09-16T10:21:50.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:50 smithi063 ceph-mon[39470]: pgmap v431: 144 pgs: 2 peering, 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:21:50.504 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-16T10:21:50.505 INFO:tasks.radosbench:Beginning radosbench... 2024-09-16T10:21:50.506 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-16T10:21:50.506 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-16T10:21:50.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:50.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:50.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:50 smithi165 ceph-mon[32696]: osdmap e270: 8 total, 8 up, 7 in 2024-09-16T10:21:50.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:50 smithi165 ceph-mon[32696]: pgmap v431: 144 pgs: 2 peering, 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T10:21:50.568 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:50.574 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:50.576 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:50.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:50.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:50.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:50.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:50.907 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:50.930 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:50.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:51.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.057 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:51.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:51.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:51.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:51.203 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:51.203 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T10:21:51.203 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:51.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:51.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:51.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:51.438 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:51.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:51 smithi063 ceph-mon[39470]: osdmap e271: 8 total, 8 up, 7 in 2024-09-16T10:21:51.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:51 smithi063 ceph-mon[39470]: osdmap e272: 8 total, 8 up, 7 in 2024-09-16T10:21:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:51 smithi063 ceph-mon[29993]: osdmap e271: 8 total, 8 up, 7 in 2024-09-16T10:21:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:51 smithi063 ceph-mon[29993]: osdmap e272: 8 total, 8 up, 7 in 2024-09-16T10:21:51.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:51.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:51 smithi165 ceph-mon[32696]: osdmap e271: 8 total, 8 up, 7 in 2024-09-16T10:21:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:51 smithi165 ceph-mon[32696]: osdmap e272: 8 total, 8 up, 7 in 2024-09-16T10:21:51.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:51.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.692 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:51.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:51.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:51.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:51.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:51.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:51.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:52.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:52.071 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:52.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:52.197 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:52.266 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:52 smithi063 ceph-mon[29993]: pgmap v434: 143 pgs: 1 peering, 142 active+clean; 263 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:21:52.266 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:52 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:52.267 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:52 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:52.267 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:52 smithi063 ceph-mon[39470]: pgmap v434: 143 pgs: 1 peering, 142 active+clean; 263 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:21:52.267 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:52 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:52.267 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:52 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:52.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:52.322 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:52.369 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:52.369 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:52.386 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:52.386 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:52.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:52.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:52.418 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:52.418 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:52.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:52.435 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:52.435 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:52.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:52.453 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:52.453 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:52.469 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:52.469 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:52.488 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:52.489 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:52 smithi165 ceph-mon[32696]: pgmap v434: 143 pgs: 1 peering, 142 active+clean; 263 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:21:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:52 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:52 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-16T10:21:52.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.576 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:52.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:52.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:52.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:52.717 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:52.806 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:52.829 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:52.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:52.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:52.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:53.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:53.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:53.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:53.204 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:21:53.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:53.310 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:53.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:53.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[29993]: osdmap e273: 8 total, 8 up, 7 in 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/720015004' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/366282250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[29993]: from='client.45551 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[39470]: osdmap e273: 8 total, 8 up, 7 in 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/720015004' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:53.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/366282250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:53.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:53 smithi063 ceph-mon[39470]: from='client.45551 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:53.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:53 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T10:21:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:53 smithi165 ceph-mon[32696]: osdmap e273: 8 total, 8 up, 7 in 2024-09-16T10:21:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/720015004' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/366282250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:53 smithi165 ceph-mon[32696]: from='client.45551 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:21:53.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:53.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:53.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:53.695 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:53.695 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:21:53.695 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:21:53.695 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:21:53.696 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:53.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:53.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:53.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:53.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:53.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:54.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:54.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:54.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:54.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:21:54.307 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:21:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:54.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:21:54.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:54.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:54.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[29993]: pgmap v436: 143 pgs: 1 peering, 142 active+clean; 263 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-16T10:21:54.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:54.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[29993]: from='client.45551 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:21:54.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[29993]: osdmap e274: 8 total, 8 up, 7 in 2024-09-16T10:21:54.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:54.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:21:54.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:21:54.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[39470]: pgmap v436: 143 pgs: 1 peering, 142 active+clean; 263 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-16T10:21:54.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:54.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[39470]: from='client.45551 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:21:54.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[39470]: osdmap e274: 8 total, 8 up, 7 in 2024-09-16T10:21:54.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:54.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:21:54.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:54 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:21:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:54 smithi165 ceph-mon[32696]: pgmap v436: 143 pgs: 1 peering, 142 active+clean; 263 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-16T10:21:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:54 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:21:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:54 smithi165 ceph-mon[32696]: from='client.45551 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:21:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:54 smithi165 ceph-mon[32696]: osdmap e274: 8 total, 8 up, 7 in 2024-09-16T10:21:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:21:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:54 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:21:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:54 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:21:54.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:54.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:54.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:54.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:54.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:54.825 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:54.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:54.950 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:54.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:55.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:55.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:55.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:55.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.198 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:55.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:55.217 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_2' created 2024-09-16T10:21:55.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:55.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:55.356 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[29993]: 3.a scrub starts 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[29993]: 3.a scrub ok 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[29993]: osdmap e275: 8 total, 8 up, 7 in 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1420375714' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[39470]: 3.a scrub starts 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[39470]: 3.a scrub ok 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[39470]: osdmap e275: 8 total, 8 up, 7 in 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:21:55.357 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:21:55.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:55 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1420375714' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-16T10:21:55.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:55.445 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:55.496 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:21:55.546 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:55.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:55 smithi165 ceph-mon[32696]: 3.a scrub starts 2024-09-16T10:21:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:55 smithi165 ceph-mon[32696]: 3.a scrub ok 2024-09-16T10:21:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:55 smithi165 ceph-mon[32696]: osdmap e275: 8 total, 8 up, 7 in 2024-09-16T10:21:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:21:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:21:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:55 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1420375714' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-16T10:21:55.568 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:55.670 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:55.693 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:55.795 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:55.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:55.919 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:55.941 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:55.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:56.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-16T10:21:56.034 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:56.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:56.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:56.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:56.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:56.188 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:56.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:56.260 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:56 smithi063 ceph-mon[29993]: pgmap v439: 142 pgs: 1 clean+premerge+peered, 141 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 74 KiB/s, 0 objects/s recovering 2024-09-16T10:21:56.260 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1420375714' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-16T10:21:56.260 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:56 smithi063 ceph-mon[29993]: osdmap e276: 8 total, 8 up, 7 in 2024-09-16T10:21:56.261 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:56 smithi063 ceph-mon[39470]: pgmap v439: 142 pgs: 1 clean+premerge+peered, 141 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 74 KiB/s, 0 objects/s recovering 2024-09-16T10:21:56.261 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1420375714' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-16T10:21:56.261 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:56 smithi063 ceph-mon[39470]: osdmap e276: 8 total, 8 up, 7 in 2024-09-16T10:21:56.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:56.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:56.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:56.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:56.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:56.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:56.493 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:56.530 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:56.530 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:21:56.530 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:39762 7.a 7:5fb7c7c3:::benchmark_data_smithi063_73086_object39761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 407.27953710200001, 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.135043899, 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:56.531 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 39762 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247777+0000", 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247778+0000", 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.16e-07 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247809+0000", 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0239999999999998e-05 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.533 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247817+0000", 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1280000000000008e-06 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.247823+0000", 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1360000000000002e-06 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249271+0000", 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001448193 2024-09-16T10:21:56.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249391+0000", 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011986500000000001 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,5", 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.249517+0000", 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125967 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.535 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.252815+0000", 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003298282 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382711+0000", 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12989600700000001 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382722+0000", 2024-09-16T10:21:56.536 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0957e-05 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382773+0000", 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1057999999999998e-05 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:09.382821+0000", 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8250000000000001e-05 2024-09-16T10:21:56.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65064 7.2 7:47084aaf:::benchmark_data_smithi063_73086_object65063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 388.98751316699997, 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.140792484, 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.538 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65064 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539801+0000", 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539802+0000", 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1500000000000004e-07 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.540 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539887+0000", 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5189e-05 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539896+0000", 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2810000000000008e-06 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.539908+0000", 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2140999999999999e-05 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545096+0000", 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0051880040000000004 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.542 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545228+0000", 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013243200000000001 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545886+0000", 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00065781499999999996 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.554263+0000", 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0083769059999999999 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.554308+0000", 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4771000000000002e-05 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680481+0000", 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12617313499999999 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680544+0000", 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3608000000000005e-05 2024-09-16T10:21:56.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680594+0000", 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9586999999999998e-05 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65067 7.2 7:48a7aec7:::benchmark_data_smithi063_73086_object65066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:21:56.545 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 388.98634356299999, 2024-09-16T10:21:56.546 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.139936807, 2024-09-16T10:21:56.546 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.546 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.546 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.546 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.546 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:56.548 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65067 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:21:56.549 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540971+0000", 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.550 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540972+0000", 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.282e-06 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.540992+0000", 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9539999999999999e-05 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541001+0000", 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.704e-06 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541007+0000", 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4269999999999996e-06 2024-09-16T10:21:56.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548200+0000", 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071930350000000004 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.553 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548316+0000", 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116468 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.548442+0000", 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012523999999999999 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563514+0000", 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015071782000000001 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680655+0000", 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.117141604 2024-09-16T10:21:56.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680817+0000", 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016221599999999999 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.556 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680870+0000", 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2448999999999999e-05 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680908+0000", 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8059999999999998e-05 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.557 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65068 7.2 7:4e1ab327:::benchmark_data_smithi063_73086_object65067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 388.98629779700002, 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14002576999999999, 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.558 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65068 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541017+0000", 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6400000000000002e-07 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541038+0000", 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0228999999999999e-05 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.561 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541044+0000", 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3770000000000001e-06 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.541050+0000", 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.485e-06 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.550940+0000", 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0098900169999999992 2024-09-16T10:21:56.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.551033+0000", 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3523000000000006e-05 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.551110+0000", 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7131000000000004e-05 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.563 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680659+0000", 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.129549215 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680696+0000", 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6424000000000002e-05 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680964+0000", 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026829000000000002 2024-09-16T10:21:56.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681008+0000", 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3399000000000002e-05 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681043+0000", 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5015999999999998e-05 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65075 7.2 7:4c929240:::benchmark_data_smithi063_73086_object65074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 388.98135785099998, 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13568592600000001, 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65075 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.566 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545957+0000", 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545959+0000", 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8649999999999999e-06 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545980+0000", 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1696e-05 2024-09-16T10:21:56.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545991+0000", 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0359e-05 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.545999+0000", 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1370000000000002e-06 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559321+0000", 2024-09-16T10:21:56.569 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013321932 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559528+0000", 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020748899999999999 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:21:56.570 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.559651+0000", 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123013 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680667+0000", 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12101582399999999 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681281+0000", 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061402500000000003 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.571 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681562+0000", 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00028057400000000001 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681607+0000", 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5578000000000003e-05 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681643+0000", 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5434e-05 2024-09-16T10:21:56.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65082 7.a 7:552f4fed:::benchmark_data_smithi063_73086_object65081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 388.97441340500001, 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.144873693, 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.573 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65082 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552901+0000", 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552904+0000", 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4499999999999998e-06 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552927+0000", 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3240000000000001e-05 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552940+0000", 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3579e-05 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.552949+0000", 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1790000000000008e-06 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681082+0000", 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12813308800000001 2024-09-16T10:21:56.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681222+0000", 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000140464 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,5", 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.681372+0000", 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000149369 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.685517+0000", 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041449700000000004 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.685528+0000", 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1440999999999999e-05 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697681+0000", 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012153051 2024-09-16T10:21:56.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697731+0000", 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0453e-05 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697775+0000", 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3408999999999997e-05 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65095 7.2 7:414dfc36:::benchmark_data_smithi063_73086_object65094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 388.96378438699998, 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13620320899999999, 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65095 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563530+0000", 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563532+0000", 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5659999999999999e-06 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563555+0000", 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2747000000000001e-05 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680763+0000", 2024-09-16T10:21:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.117208851 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.680777+0000", 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3148e-05 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689111+0000", 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0083343150000000001 2024-09-16T10:21:56.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689240+0000", 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012911199999999999 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689823+0000", 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058294800000000002 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.695962+0000", 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0061393569999999998 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.697651+0000", 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016887930000000001 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699637+0000", 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986032 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699691+0000", 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4326000000000001e-05 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.699733+0000", 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2014000000000001e-05 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90038 7.6 e112/81 7:605b4ba6:::benchmark_data_smithi063_73086_object90037:head v 112'5732, mlcod=112'5730)", 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 367.025514746, 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14626494100000001, 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501800+0000", 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501801+0000", 2024-09-16T10:21:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.47e-07 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501838+0000", 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7397000000000001e-05 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501846+0000", 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2619999999999998e-06 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.501852+0000", 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4210000000000003e-06 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504465+0000", 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026126890000000001 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.504484+0000", 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9619999999999998e-05 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646628+0000", 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.142143565 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648065+0000", 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014370400000000001 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.592 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90048 7.6 e112/81 7:6327a166:::benchmark_data_smithi063_73086_object90047:head v 112'5733, mlcod=112'5730)", 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 367.01929655100002, 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.140110544, 2024-09-16T10:21:56.593 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.594 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.594 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.594 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:21:56.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508018+0000", 2024-09-16T10:21:56.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508020+0000", 2024-09-16T10:21:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.815e-06 2024-09-16T10:21:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508049+0000", 2024-09-16T10:21:56.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8785999999999998e-05 2024-09-16T10:21:56.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508060+0000", 2024-09-16T10:21:56.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1155000000000001e-05 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508067+0000", 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.289e-06 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508275+0000", 2024-09-16T10:21:56.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020751799999999999 2024-09-16T10:21:56.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.508293+0000", 2024-09-16T10:21:56.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7975e-05 2024-09-16T10:21:56.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648068+0000", 2024-09-16T10:21:56.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13977563800000001 2024-09-16T10:21:56.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648129+0000", 2024-09-16T10:21:56.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0368000000000002e-05 2024-09-16T10:21:56.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.601 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.602 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.44750.0:90050 7.6 e112/81 7:60112b96:::benchmark_data_smithi063_73086_object90049:head v 112'5734, mlcod=112'5730)", 2024-09-16T10:21:56.602 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:21:56.602 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 367.01747029699999, 2024-09-16T10:21:56.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13835439699999999, 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509844+0000", 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509846+0000", 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2079999999999999e-06 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509888+0000", 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.244e-05 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509897+0000", 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3610000000000007e-06 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.509904+0000", 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5420000000000002e-06 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.511017+0000", 2024-09-16T10:21:56.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0011129860000000001 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.511034+0000", 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6891000000000001e-05 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648132+0000", 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13709860700000001 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.648199+0000", 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6361999999999999e-05 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51200 8.f 8:fe221df4:::benchmark_data_smithi063_101290_object51199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.306440+0000", 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 163.22087449599999, 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99471615099999999, 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.607 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51200 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306440+0000", 2024-09-16T10:21:56.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306440+0000", 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306442+0000", 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.486e-06 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306461+0000", 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8940000000000002e-05 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306472+0000", 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1015999999999999e-05 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306477+0000", 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1630000000000004e-06 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306529+0000", 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2510999999999999e-05 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306536+0000", 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6200000000000001e-06 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.295906+0000", 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98937029099999996 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296049+0000", 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014290799999999999 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296156+0000", 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106758 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300931+0000", 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047748139999999996 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300999+0000", 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8221999999999998e-05 2024-09-16T10:21:56.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301050+0000", 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1183000000000001e-05 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301104+0000", 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3686000000000001e-05 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301156+0000", 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2553000000000003e-05 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51210 8.f 8:f185333d:::benchmark_data_smithi063_101290_object51209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.320115+0000", 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 163.20719979899999, 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98597720799999999, 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.615 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51210 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320115+0000", 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320115+0000", 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320117+0000", 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8610000000000001e-06 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320140+0000", 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3788e-05 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320155+0000", 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4246999999999999e-05 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320172+0000", 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7533e-05 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320244+0000", 2024-09-16T10:21:56.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1761000000000006e-05 2024-09-16T10:21:56.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:56.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:56 smithi165 ceph-mon[32696]: pgmap v439: 142 pgs: 1 clean+premerge+peered, 141 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 74 KiB/s, 0 objects/s recovering 2024-09-16T10:21:56.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1420375714' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-16T10:21:56.621 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:56 smithi165 ceph-mon[32696]: osdmap e276: 8 total, 8 up, 7 in 2024-09-16T10:21:56.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320255+0000", 2024-09-16T10:21:56.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1044e-05 2024-09-16T10:21:56.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.298864+0000", 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97860911399999995 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.298997+0000", 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000132562 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299145+0000", 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014797599999999999 2024-09-16T10:21:56.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303492+0000", 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043472229999999999 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303669+0000", 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017695800000000001 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305992+0000", 2024-09-16T10:21:56.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023235019999999999 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306047+0000", 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4920999999999998e-05 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306092+0000", 2024-09-16T10:21:56.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4718000000000001e-05 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51245 8.f 8:fbb4951d:::benchmark_data_smithi063_101290_object51244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.345508+0000", 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 163.18180636899999, 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96301515500000001, 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.625 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51245 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345508+0000", 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345508+0000", 2024-09-16T10:21:56.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345510+0000", 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.933e-06 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345534+0000", 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3702000000000001e-05 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345547+0000", 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3266e-05 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345555+0000", 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7610000000000007e-06 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345618+0000", 2024-09-16T10:21:56.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3330999999999994e-05 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345631+0000", 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2758999999999999e-05 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301191+0000", 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95556000500000005 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301334+0000", 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014330199999999999 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301564+0000", 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00022991799999999999 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306110+0000", 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045453710000000003 2024-09-16T10:21:56.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308381+0000", 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002271669 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308425+0000", 2024-09-16T10:21:56.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3473000000000002e-05 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308481+0000", 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6150000000000003e-05 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308523+0000", 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2515000000000002e-05 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51248 8.f 8:fa504e1d:::benchmark_data_smithi063_101290_object51247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:56.634 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.347607+0000", 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 163.17970725000001, 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96156481999999999, 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.635 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51248 2024-09-16T10:21:56.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.636 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347607+0000", 2024-09-16T10:21:56.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347607+0000", 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347609+0000", 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8300000000000001e-06 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347634+0000", 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4355e-05 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347650+0000", 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5988e-05 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347658+0000", 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9999999999999996e-06 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347736+0000", 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8182999999999993e-05 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.347748+0000", 2024-09-16T10:21:56.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2157999999999999e-05 2024-09-16T10:21:56.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303734+0000", 2024-09-16T10:21:56.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95598619699999998 2024-09-16T10:21:56.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303866+0000", 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131519 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304017+0000", 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015108600000000001 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.308562+0000", 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045456350000000001 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309060+0000", 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00049744100000000005 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309072+0000", 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2463000000000001e-05 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309130+0000", 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7555e-05 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309172+0000", 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2410000000000002e-05 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51266 8.f 8:ff3eb03d:::benchmark_data_smithi063_101290_object51265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.376408+0000", 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 163.150906835, 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93575255499999999, 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51266 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.645 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376408+0000", 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376408+0000", 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376409+0000", 2024-09-16T10:21:56.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1599999999999999e-06 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376432+0000", 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3329e-05 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376441+0000", 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1209999999999996e-06 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376446+0000", 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5050000000000001e-06 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376495+0000", 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8776999999999997e-05 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.376501+0000", 2024-09-16T10:21:56.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4999999999999996e-06 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306171+0000", 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92966961199999998 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306291+0000", 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119981 2024-09-16T10:21:56.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.306420+0000", 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000128925 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309180+0000", 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027602690000000001 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309272+0000", 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1778000000000001e-05 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312052+0000", 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027807169999999998 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312112+0000", 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9951000000000002e-05 2024-09-16T10:21:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312160+0000", 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7929999999999997e-05 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:231082 8.f 8:f501a8ef:::benchmark_data_smithi063_101290_object103762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:21:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:21:24.747373+0000", 2024-09-16T10:21:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 31.779941700999998, 2024-09-16T10:21:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13932475899999999, 2024-09-16T10:21:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 231082 2024-09-16T10:21:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747373+0000", 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747373+0000", 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747394+0000", 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0988e-05 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747412+0000", 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7813999999999999e-05 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747427+0000", 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4846e-05 2024-09-16T10:21:56.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747433+0000", 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6429999999999998e-06 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747729+0000", 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00029608999999999999 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747957+0000", 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000227316 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.748064+0000", 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000107358 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.755833+0000", 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0077685929999999999 2024-09-16T10:21:56.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.755975+0000", 2024-09-16T10:21:56.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014199699999999999 2024-09-16T10:21:56.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886603+0000", 2024-09-16T10:21:56.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13062889499999999 2024-09-16T10:21:56.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886656+0000", 2024-09-16T10:21:56.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2160000000000002e-05 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886698+0000", 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2058999999999998e-05 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:231084 8.f 8:ff95bbc7:::benchmark_data_smithi063_101290_object103764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:21:24.747668+0000", 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 31.779646392, 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13913383200000001, 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 231084 2024-09-16T10:21:56.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747668+0000", 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747668+0000", 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747669+0000", 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0899999999999999e-06 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747681+0000", 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1273e-05 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747688+0000", 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0979999999999996e-06 2024-09-16T10:21:56.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.747694+0000", 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0090000000000004e-06 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.750508+0000", 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028142240000000002 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.750606+0000", 2024-09-16T10:21:56.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8238000000000004e-05 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.750710+0000", 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103704 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.756095+0000", 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0053855919999999998 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.756110+0000", 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4171999999999999e-05 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886733+0000", 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13062299799999999 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886771+0000", 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8622999999999999e-05 2024-09-16T10:21:56.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886802+0000", 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0811e-05 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:231093 8.f 8:f1e604ab:::benchmark_data_smithi063_101290_object103773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:21:24.749475+0000", 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 31.777840016999999, 2024-09-16T10:21:56.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.137439863, 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 231093 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.749475+0000", 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.749475+0000", 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.749476+0000", 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3600000000000002e-07 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.749479+0000", 2024-09-16T10:21:56.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9809999999999997e-06 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.749489+0000", 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7699999999999996e-06 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.749496+0000", 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5740000000000001e-06 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.752708+0000", 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032126479999999998 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.752776+0000", 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7150999999999995e-05 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.752849+0000", 2024-09-16T10:21:56.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.381e-05 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.756114+0000", 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032646950000000002 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.880979+0000", 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.124864663 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886843+0000", 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058644880000000002 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886883+0000", 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9372999999999997e-05 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.886914+0000", 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1773999999999998e-05 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.679 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:231095 8.f 8:f2406ad4:::benchmark_data_smithi063_101290_object103775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:21:24.754558+0000", 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 31.772756183999999, 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.143713865, 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:21:56.680 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 231095 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.754558+0000", 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.754558+0000", 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.754560+0000", 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5659999999999999e-06 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.754565+0000", 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.259e-06 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.754576+0000", 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1026000000000001e-05 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.754584+0000", 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4000000000000003e-06 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.889987+0000", 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13540358199999999 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.890091+0000", 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000104252 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.890205+0000", 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113359 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.894940+0000", 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047355720000000004 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.897296+0000", 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023553210000000001 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.898185+0000", 2024-09-16T10:21:56.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00088893100000000003 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.898235+0000", 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9882000000000001e-05 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.898272+0000", 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7715000000000001e-05 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.15333.0:231099 8.7 e245/165 8:e06f9a0e:::benchmark_data_smithi063_101290_object103779:head v 245'14344, mlcod=245'14341)", 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:21:24.757269+0000", 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 31.770045746000001, 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13515640800000001, 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:21:56.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.757269+0000", 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.757269+0000", 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.757271+0000", 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6610000000000001e-06 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.757277+0000", 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3489999999999997e-06 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.880721+0000", 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123444493 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.880748+0000", 2024-09-16T10:21:56.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6743000000000001e-05 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.883905+0000", 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031572530000000001 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.883933+0000", 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8031e-05 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.892392+0000", 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0084582159999999993 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:21:24.892425+0000", 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3661999999999997e-05 2024-09-16T10:21:56.691 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.692 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.692 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.692 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:21:56.692 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:21:56.692 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:21:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:56.693 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:56.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:56.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:56.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:56.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:56.882 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:21:56.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:21:56.883 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-16T10:21:56.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd in 2 2024-09-16T10:21:56.919 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:56.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:57.054 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:57.077 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:57.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:57.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:21:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:21:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.203 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:57.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:57.288 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:21:57.288 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:21:57.288 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:21:57.288 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:21:57.288 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:21:57.288 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:21:57.303 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:21:57.303 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:21:57.305 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:57.320 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:21:57.320 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:21:57.327 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:57.337 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:21:57.337 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:21:57.354 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:21:57.354 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:21:57.372 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:21:57.373 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:21:57.381 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:21:57.428 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.451 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:57.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:57 smithi165 ceph-mon[32696]: 3.5 scrub starts 2024-09-16T10:21:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:57 smithi165 ceph-mon[32696]: 3.5 scrub ok 2024-09-16T10:21:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:57 smithi165 ceph-mon[32696]: osdmap e277: 8 total, 8 up, 7 in 2024-09-16T10:21:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:57 smithi165 ceph-mon[32696]: 4.14 scrub starts 2024-09-16T10:21:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:57 smithi165 ceph-mon[32696]: 4.14 scrub ok 2024-09-16T10:21:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:57 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:21:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:57 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:21:57.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:57.563 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[29993]: 3.5 scrub starts 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[29993]: 3.5 scrub ok 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[29993]: osdmap e277: 8 total, 8 up, 7 in 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[29993]: 4.14 scrub starts 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[29993]: 4.14 scrub ok 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[39470]: 3.5 scrub starts 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[39470]: 3.5 scrub ok 2024-09-16T10:21:57.564 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[39470]: osdmap e277: 8 total, 8 up, 7 in 2024-09-16T10:21:57.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[39470]: 4.14 scrub starts 2024-09-16T10:21:57.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[39470]: 4.14 scrub ok 2024-09-16T10:21:57.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:21:57.565 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:57 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:21:57.577 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:57.679 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:57.803 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:57.825 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:57.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:21:57.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:57.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:57.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:58.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:58.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:58.161 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:21:58.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.223 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:58.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:58.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.348 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:58.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:58.412 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 6.7 scrub starts 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 6.7 scrub ok 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: pgmap v442: 158 pgs: 16 unknown, 1 clean+premerge+peered, 141 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: osdmap e278: 8 total, 8 up, 7 in 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2332411795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 6.1f scrub starts 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 6.1f scrub ok 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 5.c scrub starts 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 5.c scrub ok 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 4.1b scrub starts 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 4.1b scrub ok 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:21:58.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 6.7 scrub starts 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 6.7 scrub ok 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: pgmap v442: 158 pgs: 16 unknown, 1 clean+premerge+peered, 141 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: osdmap e278: 8 total, 8 up, 7 in 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2332411795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 6.1f scrub starts 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 6.1f scrub ok 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 5.c scrub starts 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 5.c scrub ok 2024-09-16T10:21:58.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 4.1b scrub starts 2024-09-16T10:21:58.415 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 4.1b scrub ok 2024-09-16T10:21:58.415 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:21:58.415 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:58 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:21:58.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:58.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 6.7 scrub starts 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 6.7 scrub ok 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: pgmap v442: 158 pgs: 16 unknown, 1 clean+premerge+peered, 141 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: osdmap e278: 8 total, 8 up, 7 in 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2332411795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 6.1f scrub starts 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 6.1f scrub ok 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 5.c scrub starts 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 5.c scrub ok 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 4.1b scrub starts 2024-09-16T10:21:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 4.1b scrub ok 2024-09-16T10:21:58.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:21:58.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:21:58.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:21:58.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:58.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:21:58.664 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:21:58.664 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:21:58.696 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:58.719 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:58.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:21:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:21:58.820 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:58.842 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:21:58.943 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:58.966 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:58.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:59.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:59.091 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:59.192 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:59.193 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:21:59.217 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:21:59.218 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:21:59.242 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-16T10:21:59.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:59.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 3.12 scrub starts 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 3.12 scrub ok 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 4.10 scrub starts 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 4.10 scrub ok 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:59.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/318223744' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: from='client.45575 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 3.12 scrub starts 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 3.12 scrub ok 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 4.10 scrub starts 2024-09-16T10:21:59.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 4.10 scrub ok 2024-09-16T10:21:59.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:59.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:21:59.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:21:59.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/318223744' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:21:59.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: from='client.45575 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:21:59.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:21:59.360 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:21:59 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:21:59.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:21:59.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:21:59.470 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:21:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:21:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:21:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 3.12 scrub starts 2024-09-16T10:21:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 3.12 scrub ok 2024-09-16T10:21:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 4.10 scrub starts 2024-09-16T10:21:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 4.10 scrub ok 2024-09-16T10:21:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:21:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:21:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:21:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/318223744' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:21:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: from='client.45575 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:21:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:21:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:21:59 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:21:59.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:21:59.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:21:59.668 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:21:59.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:21:59.718 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:21:59.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:21:59.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:21:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:21:59.944 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:21:59.966 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:21:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:21:59.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:00.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:00.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:00.191 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:00.214 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:00.223 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:00.223 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:00.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:00.340 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:00.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[39470]: pgmap v444: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T10:22:00.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[39470]: from='client.45575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:22:00.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[39470]: osdmap e279: 8 total, 8 up, 7 in 2024-09-16T10:22:00.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:22:00.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:22:00.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[39470]: 3.d scrub starts 2024-09-16T10:22:00.433 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[39470]: 3.d scrub ok 2024-09-16T10:22:00.434 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[29993]: pgmap v444: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T10:22:00.434 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[29993]: from='client.45575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:22:00.434 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[29993]: osdmap e279: 8 total, 8 up, 7 in 2024-09-16T10:22:00.434 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:22:00.434 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:22:00.434 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[29993]: 3.d scrub starts 2024-09-16T10:22:00.434 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:00 smithi063 ceph-mon[29993]: 3.d scrub ok 2024-09-16T10:22:00.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:00.465 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:00.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:00 smithi165 ceph-mon[32696]: pgmap v444: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T10:22:00.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:00 smithi165 ceph-mon[32696]: from='client.45575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:22:00.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:00 smithi165 ceph-mon[32696]: osdmap e279: 8 total, 8 up, 7 in 2024-09-16T10:22:00.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:00 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:22:00.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:00 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:22:00.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:00 smithi165 ceph-mon[32696]: 3.d scrub starts 2024-09-16T10:22:00.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:00 smithi165 ceph-mon[32696]: 3.d scrub ok 2024-09-16T10:22:00.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:00.588 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:00.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:00.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:00.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:00.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:00.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:00.858 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:22:00.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:22:00.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:00.977 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:00.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:01.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:01.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:01.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:01.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:22:01.165 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:01.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:01.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:01.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:01.326 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:01.347 DEBUG:teuthology.orchestra.run.smithi063:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-16T10:22:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:01.351 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:01.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:01.352 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-16T10:22:01.352 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-16T10:22:01.401 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:hints = 1 2024-09-16T10:22:01.401 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-16T10:22:01.401 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object prefix: benchmark_data_smithi063_124717 2024-09-16T10:22:01.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:22:01.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:22:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:22:01.005+0000 7f17f27f5640 -1 mon.a@0(leader).osd e280 definitely_dead 0 2024-09-16T10:22:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3890788349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:01.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:22:01.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3890788349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:01.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[29993]: osdmap e280: 8 total, 8 up, 7 in 2024-09-16T10:22:01.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1631942196' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-16T10:22:01.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3890788349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:01.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:22:01.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3890788349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:01.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[39470]: osdmap e280: 8 total, 8 up, 7 in 2024-09-16T10:22:01.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1631942196' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-16T10:22:01.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:01.483 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:01.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3890788349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:01 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:22:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3890788349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:01 smithi165 ceph-mon[32696]: osdmap e280: 8 total, 8 up, 7 in 2024-09-16T10:22:01.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1631942196' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-16T10:22:01.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:01.623 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:01.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:01.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:01.769 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:01.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:01.779 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:01.779 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:01.803 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:01.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:01.880 INFO:teuthology.orchestra.run.smithi063.stderr:marked in osd.2. 2024-09-16T10:22:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:01.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:01.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:02.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:02.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:02.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:02.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:02.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:02.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: pgmap v447: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1631942196' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: osdmap e281: 8 total, 8 up, 8 in 2024-09-16T10:22:02.401 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:22:02.402 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:02 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:22:02.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 1 16 1697 1681 105.056 105.062 0.00102889 0.00483688 2024-09-16T10:22:02.407 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:02.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:22:02.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:02.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:02.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:02.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:02.677 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:02.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: pgmap v447: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1631942196' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: osdmap e281: 8 total, 8 up, 8 in 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: pgmap v447: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T10:22:02.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:22:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:22:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1631942196' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-16T10:22:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: osdmap e281: 8 total, 8 up, 8 in 2024-09-16T10:22:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:22:02.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:02 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:22:02.755 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:02.779 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:02.781 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-16T10:22:02.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:02.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:02.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:03.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:03.039 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:03.074 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:03.075 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:03.094 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:03.111 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:03.112 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:03.128 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:03.128 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:03.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:03.144 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:03.144 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:03.161 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:03.162 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:03.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:03.178 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:03.195 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:03.195 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:03.200 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:03.200 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:22:03.200 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:22:03.200 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:03.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:03.288 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:03.390 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:03.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 2 16 1842 1826 57.0587 9.0625 0.0147302 0.00967887 2024-09-16T10:22:03.412 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:03.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: 5.16 scrub starts 2024-09-16T10:22:03.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: 5.16 scrub ok 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: 5.0 scrub starts 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: 5.0 scrub ok 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: 5.19 scrub starts 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: 5.19 scrub ok 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: osdmap e282: 8 total, 8 up, 8 in 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2980324821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: 5.16 scrub starts 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: 5.16 scrub ok 2024-09-16T10:22:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: 5.0 scrub starts 2024-09-16T10:22:03.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: 5.0 scrub ok 2024-09-16T10:22:03.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: 5.19 scrub starts 2024-09-16T10:22:03.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: 5.19 scrub ok 2024-09-16T10:22:03.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:22:03.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:22:03.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: osdmap e282: 8 total, 8 up, 8 in 2024-09-16T10:22:03.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:03 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2980324821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:03.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:03.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: 5.16 scrub starts 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: 5.16 scrub ok 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: 5.0 scrub starts 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: 5.0 scrub ok 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: 5.19 scrub starts 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: 5.19 scrub ok 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: osdmap e282: 8 total, 8 up, 8 in 2024-09-16T10:22:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:03 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2980324821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:03.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:03.660 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:03.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:22:03.761 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:03.784 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:03.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:22:03.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:03.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:03.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:03.980 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:04.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:04.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:04.066 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:04.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:04.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:04.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:04.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:04.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:04.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 3 16 2571 2555 53.2252 45.5625 0.00356396 0.0187341 2024-09-16T10:22:04.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:04.472 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:04 smithi165 ceph-mon[32696]: pgmap v450: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-16T10:22:04.472 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:04 smithi165 ceph-mon[32696]: 5.8 scrub starts 2024-09-16T10:22:04.472 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:04 smithi165 ceph-mon[32696]: 5.8 scrub ok 2024-09-16T10:22:04.472 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:04 smithi165 ceph-mon[32696]: osdmap e283: 8 total, 8 up, 8 in 2024-09-16T10:22:04.509 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[29993]: pgmap v450: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-16T10:22:04.509 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[29993]: 5.8 scrub starts 2024-09-16T10:22:04.509 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[29993]: 5.8 scrub ok 2024-09-16T10:22:04.509 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[29993]: osdmap e283: 8 total, 8 up, 8 in 2024-09-16T10:22:04.509 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[39470]: pgmap v450: 158 pgs: 158 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-16T10:22:04.509 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[39470]: 5.8 scrub starts 2024-09-16T10:22:04.509 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[39470]: 5.8 scrub ok 2024-09-16T10:22:04.510 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:04 smithi063 ceph-mon[39470]: osdmap e283: 8 total, 8 up, 8 in 2024-09-16T10:22:04.528 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.574 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:04.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:04.637 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:04.637 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:22:04.637 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:22:04.637 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:22:04.637 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:04.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:04.716 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:04.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:04.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:04.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:04.988 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:04.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:05.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:05.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:05.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:05.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:05.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:05.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:22:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:05.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:05.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 4 16 4429 4413 68.9469 116.125 0.00121712 0.012895 2024-09-16T10:22:05.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:05.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:22:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 1.1 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:22:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 3.1f scrub starts 2024-09-16T10:22:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 3.1f scrub ok 2024-09-16T10:22:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:22:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:22:05.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: osdmap e284: 8 total, 8 up, 8 in 2024-09-16T10:22:05.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3999944641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:05.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 6.3 scrub starts 2024-09-16T10:22:05.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:05 smithi165 ceph-mon[32696]: 6.3 scrub ok 2024-09-16T10:22:05.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:05.603 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:05.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 1.1 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 3.1f scrub starts 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 3.1f scrub ok 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: osdmap e284: 8 total, 8 up, 8 in 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3999944641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 6.3 scrub starts 2024-09-16T10:22:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[29993]: 6.3 scrub ok 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 1.1 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 3.1f scrub starts 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 3.1f scrub ok 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: osdmap e284: 8 total, 8 up, 8 in 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3999944641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 6.3 scrub starts 2024-09-16T10:22:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:05 smithi063 ceph-mon[39470]: 6.3 scrub ok 2024-09-16T10:22:05.707 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:05.731 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:05.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:05.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:05.892 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:22:05.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:22:05.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:05.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:05.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:06.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:06.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:06.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:06.214 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:22:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:22:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:22:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90056 7.0 7:081070f4:::benchmark_data_smithi063_73086_object90055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:22:06.214 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 376.69636245700002, 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13312953299999999, 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90056 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513588+0000", 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513590+0000", 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.531e-06 2024-09-16T10:22:06.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513668+0000", 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8187000000000002e-05 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513679+0000", 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0781999999999999e-05 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513685+0000", 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.93e-06 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513734+0000", 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9539000000000001e-05 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513851+0000", 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116749 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513939+0000", 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8208999999999995e-05 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519242+0000", 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053023430000000002 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523299+0000", 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040579250000000004 2024-09-16T10:22:06.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646650+0000", 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.123350551 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646686+0000", 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6316999999999998e-05 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646718+0000", 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1470000000000002e-05 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90058 7.0 7:0216797c:::benchmark_data_smithi063_73086_object90057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 376.69616377, 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13450020500000001, 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.221 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90058 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513787+0000", 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513788+0000", 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1289999999999999e-06 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516073+0000", 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022849010000000002 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516088+0000", 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5031000000000001e-05 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516096+0000", 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9680000000000006e-06 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516691+0000", 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00059533899999999996 2024-09-16T10:22:06.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516812+0000", 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120388 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516932+0000", 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120689 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519245+0000", 2024-09-16T10:22:06.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023127080000000001 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523333+0000", 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040881149999999998 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648185+0000", 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124851428 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648246+0000", 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1354999999999998e-05 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.648287+0000", 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1153999999999999e-05 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90059 7.0 7:0d97e368:::benchmark_data_smithi063_73086_object90058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:22:06.227 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:22:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 376.69383504899997, 2024-09-16T10:22:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.1397098, 2024-09-16T10:22:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:06.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90059 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516116+0000", 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516117+0000", 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5400000000000001e-06 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516153+0000", 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5546000000000002e-05 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516161+0000", 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8539999999999997e-06 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.516166+0000", 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2310000000000004e-06 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519295+0000", 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031290430000000002 2024-09-16T10:22:06.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519373+0000", 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8521000000000003e-05 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.519458+0000", 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4351000000000006e-05 2024-09-16T10:22:06.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521937+0000", 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024794130000000002 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.523362+0000", 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001425228 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655714+0000", 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13235127199999999 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655770+0000", 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6622000000000001e-05 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.655826+0000", 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5179000000000001e-05 2024-09-16T10:22:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51212 8.4 8:27b2b5e1:::benchmark_data_smithi063_101290_object51211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.88923885200001, 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98431940500000004, 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51212 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320713+0000", 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.559e-06 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320735+0000", 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2010000000000001e-05 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320745+0000", 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6690000000000001e-06 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320750+0000", 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2939999999999996e-06 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320816+0000", 2024-09-16T10:22:06.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5090000000000002e-05 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320824+0000", 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8599999999999999e-06 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295578+0000", 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97475403100000002 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295697+0000", 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118301 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295799+0000", 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102376 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304751+0000", 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089521540000000004 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304762+0000", 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1044e-05 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304922+0000", 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015943199999999999 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304966+0000", 2024-09-16T10:22:06.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4545000000000001e-05 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305031+0000", 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5040000000000001e-05 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51220 8.4 8:2e2a097c:::benchmark_data_smithi063_101290_object51219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:06.243 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.88407092099999, 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97951371499999995, 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51220 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325882+0000", 2024-09-16T10:22:06.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.827e-06 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325937+0000", 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5121000000000003e-05 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325948+0000", 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1066999999999999e-05 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325954+0000", 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999999e-06 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326011+0000", 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6937000000000001e-05 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326023+0000", 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1344e-05 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298310+0000", 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97228777099999997 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298432+0000", 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012194200000000001 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298661+0000", 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000228497 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304876+0000", 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062151819999999997 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305227+0000", 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035094899999999998 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305288+0000", 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0804000000000003e-05 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305361+0000", 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2706999999999994e-05 2024-09-16T10:22:06.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305394+0000", 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2972999999999998e-05 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51223 8.4 8:287d1d16:::benchmark_data_smithi063_101290_object51222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.883431879, 2024-09-16T10:22:06.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98451053700000002, 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51223 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326520+0000", 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.079e-06 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326573+0000", 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2924999999999999e-05 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326581+0000", 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3659999999999992e-06 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326586+0000", 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8369999999999996e-06 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326636+0000", 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9667999999999999e-05 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326646+0000", 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0389e-05 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302523+0000", 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.975876682 2024-09-16T10:22:06.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302647+0000", 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123745 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302774+0000", 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012772799999999999 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305245+0000", 2024-09-16T10:22:06.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024711160000000002 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309074+0000", 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038286169999999999 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.310943+0000", 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018688979999999999 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.310987+0000", 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4057999999999997e-05 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311029+0000", 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2429000000000003e-05 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.258 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51230 8.4 8:23e3d41e:::benchmark_data_smithi063_101290_object51229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.87179409500001, 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97448984100000002, 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51230 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338159+0000", 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1610000000000001e-06 2024-09-16T10:22:06.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338222+0000", 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3161000000000001e-05 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338235+0000", 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3129000000000001e-05 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338243+0000", 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8420000000000005e-06 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338299+0000", 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6477999999999997e-05 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338313+0000", 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3852000000000001e-05 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305424+0000", 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96711084599999997 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305539+0000", 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115146 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305678+0000", 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013881700000000001 2024-09-16T10:22:06.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311084+0000", 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054059110000000002 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311113+0000", 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9405999999999999e-05 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312548+0000", 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001434533 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312609+0000", 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1503999999999999e-05 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312646+0000", 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7054999999999997e-05 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51243 8.4 8:253d325f:::benchmark_data_smithi063_101290_object51242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.86751510400001, 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97387738800000001, 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.266 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51243 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342437+0000", 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.505e-06 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342507+0000", 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9720000000000003e-05 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342520+0000", 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3478999999999999e-05 2024-09-16T10:22:06.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342529+0000", 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3380000000000005e-06 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342652+0000", 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012287600000000001 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342658+0000", 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5889999999999999e-06 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307308+0000", 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96465014199999999 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307405+0000", 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6510999999999998e-05 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:22:06.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307533+0000", 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012844199999999999 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312655+0000", 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051218239999999996 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.315330+0000", 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026750229999999999 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316199+0000", 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086935199999999999 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316263+0000", 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3326999999999999e-05 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316313+0000", 2024-09-16T10:22:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0260000000000003e-05 2024-09-16T10:22:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51249 8.4 8:208cf65c:::benchmark_data_smithi063_101290_object51248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:22:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.861111163, 2024-09-16T10:22:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96761772599999996, 2024-09-16T10:22:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.274 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51249 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348841+0000", 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5460000000000001e-06 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348879+0000", 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7716999999999998e-05 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348889+0000", 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0327e-05 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348896+0000", 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.742e-06 2024-09-16T10:22:06.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348953+0000", 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7028999999999999e-05 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348960+0000", 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4209999999999998e-06 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309120+0000", 2024-09-16T10:22:06.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96015961500000002 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309226+0000", 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106158 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309348+0000", 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012231699999999999 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.315758+0000", 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064091460000000001 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316153+0000", 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039516799999999999 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316354+0000", 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000201452 2024-09-16T10:22:06.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316402+0000", 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7577999999999997e-05 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316457+0000", 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5510000000000002e-05 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.281 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51270 8.d 8:b5d1ccff:::benchmark_data_smithi063_101290_object51269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.903839482, 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.001723546, 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.282 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51270 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306113+0000", 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.136e-06 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306261+0000", 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147632 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306274+0000", 2024-09-16T10:22:06.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2850000000000001e-05 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306282+0000", 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.388e-06 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306347+0000", 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4898000000000001e-05 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306359+0000", 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2149e-05 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.296824+0000", 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99046429999999996 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.296988+0000", 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164017 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.297156+0000", 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016855799999999999 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307591+0000", 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010434652000000001 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307602+0000", 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.083e-05 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307749+0000", 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014768800000000001 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307800+0000", 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0800999999999997e-05 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307835+0000", 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4647000000000002e-05 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51294 8.d 8:b3f59715:::benchmark_data_smithi063_101290_object51293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.88895119399999, 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.986972195, 2024-09-16T10:22:06.289 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51294 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:22:06.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321002+0000", 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2060000000000001e-06 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321046+0000", 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4345000000000003e-05 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321063+0000", 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6688e-05 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321071+0000", 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-16T10:22:06.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321156+0000", 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4980000000000003e-05 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321163+0000", 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9949999999999999e-06 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299344+0000", 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97818064299999996 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299480+0000", 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013635700000000001 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299684+0000", 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000203857 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307681+0000", 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079967979999999994 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307717+0000", 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6297999999999997e-05 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307904+0000", 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018723000000000001 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307943+0000", 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8831999999999997e-05 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307972+0000", 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8555999999999999e-05 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51296 8.d 8:b655e86a:::benchmark_data_smithi063_101290_object51295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.88655269099999, 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99276492999999999, 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51296 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.297 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323400+0000", 2024-09-16T10:22:06.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4789999999999999e-06 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323445+0000", 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5637999999999999e-05 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323455+0000", 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0127e-05 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323461+0000", 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.339e-06 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323504+0000", 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3461999999999998e-05 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323513+0000", 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2269999999999999e-06 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302178+0000", 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97866425099999999 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302303+0000", 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125862 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302439+0000", 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013551200000000001 2024-09-16T10:22:06.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307685+0000", 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052464349999999998 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307870+0000", 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184811 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316064+0000", 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081937899999999994 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316116+0000", 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1690999999999997e-05 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316163+0000", 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7305999999999997e-05 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.303 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51303 8.d 8:b2b2f5d7:::benchmark_data_smithi063_101290_object51302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.88115127500001, 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99301229000000002, 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51303 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328801+0000", 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2470000000000001e-06 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328903+0000", 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102128 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328928+0000", 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4760999999999999e-05 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328936+0000", 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4400000000000005e-06 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328998+0000", 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1628e-05 2024-09-16T10:22:06.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.329009+0000", 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1758e-05 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97591637799999997 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.305040+0000", 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113761 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.305141+0000", 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101047 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312025+0000", 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068839119999999998 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312061+0000", 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6918999999999997e-05 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321718+0000", 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096562799999999997 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321768+0000", 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0266000000000003e-05 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321812+0000", 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3764999999999999e-05 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51322 8.d 8:b9f45f41:::benchmark_data_smithi063_101290_object51321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.311 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.868485531, 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.980450343, 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51322 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341467+0000", 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1409999999999999e-06 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341501+0000", 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3157000000000001e-05 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341513+0000", 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2296999999999999e-05 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341520+0000", 2024-09-16T10:22:06.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5290000000000003e-06 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341603+0000", 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2180000000000003e-05 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341614+0000", 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1748e-05 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307997+0000", 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96638288500000002 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.308109+0000", 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112235 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.308233+0000", 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012397800000000001 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312071+0000", 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003837619 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.315989+0000", 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039176020000000001 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321846+0000", 2024-09-16T10:22:06.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058578290000000002 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321886+0000", 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9400999999999998e-05 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321916+0000", 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9742000000000001e-05 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51332 8.d 8:baf4d594:::benchmark_data_smithi063_101290_object51331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.85974526199999, 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97184618199999995, 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.319 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51332 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350208+0000", 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.176e-06 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350266+0000", 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7840999999999997e-05 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350279+0000", 2024-09-16T10:22:06.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3718e-05 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350287+0000", 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5240000000000003e-06 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350342+0000", 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4755e-05 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350350+0000", 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1259999999999998e-06 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.309962+0000", 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95961197799999998 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.310060+0000", 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8022000000000004e-05 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.310179+0000", 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119713 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.315951+0000", 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057717239999999998 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316034+0000", 2024-09-16T10:22:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2670000000000006e-05 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321980+0000", 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059458560000000002 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322021+0000", 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1328999999999997e-05 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322052+0000", 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0750000000000002e-05 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51340 8.d 8:be7e8c60:::benchmark_data_smithi063_101290_object51339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.85327299599999, 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96553988000000002, 2024-09-16T10:22:06.326 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51340 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356679+0000", 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356716+0000", 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6591000000000003e-05 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356726+0000", 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6560000000000004e-06 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356733+0000", 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9789999999999996e-06 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356809+0000", 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6790000000000002e-05 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356819+0000", 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8269999999999993e-06 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312094+0000", 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.955275333 2024-09-16T10:22:06.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312207+0000", 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112341 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312340+0000", 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013320399999999999 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316004+0000", 2024-09-16T10:22:06.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036634889999999998 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316206+0000", 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020242200000000001 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322131+0000", 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059253600000000002 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322183+0000", 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1372999999999998e-05 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322218+0000", 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4950000000000002e-05 2024-09-16T10:22:06.334 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51359 8.d 8:be832237:::benchmark_data_smithi063_101290_object51358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.77339560600001, 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88882225199999998, 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51359 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:22:06.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436556+0000", 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.328e-06 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436612+0000", 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5040999999999997e-05 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436622+0000", 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0491e-05 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436629+0000", 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9929999999999998e-06 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436696+0000", 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6544000000000005e-05 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436710+0000", 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4058e-05 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314089+0000", 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87737920700000005 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314198+0000", 2024-09-16T10:22:06.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109348 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314324+0000", 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012599 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321682+0000", 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073583529999999998 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322263+0000", 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058049700000000002 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325295+0000", 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030324670000000001 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325342+0000", 2024-09-16T10:22:06.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6499999999999999e-05 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325377+0000", 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5435000000000002e-05 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51370 8.d 8:b1e0f19d:::benchmark_data_smithi063_101290_object51369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.76518740099999, 2024-09-16T10:22:06.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88139332999999997, 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51370 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444765+0000", 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.412e-06 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444815+0000", 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0201999999999998e-05 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444825+0000", 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0175e-05 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444830+0000", 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1429999999999997e-06 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444910+0000", 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9300000000000003e-05 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444919+0000", 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2499999999999995e-06 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316237+0000", 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87131822199999998 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316359+0000", 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012239799999999999 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316474+0000", 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011482800000000001 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321948+0000", 2024-09-16T10:22:06.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054734090000000003 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322268+0000", 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032003100000000003 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326071+0000", 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038029539999999999 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326120+0000", 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9326999999999997e-05 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326157+0000", 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6678999999999999e-05 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.349 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51382 8.d 8:b6fd2994:::benchmark_data_smithi063_101290_object51381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.456229+0000", 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.753721767, 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87003440600000004, 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51382 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456229+0000", 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456229+0000", 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456231+0000", 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1900000000000002e-06 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456282+0000", 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0651e-05 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456296+0000", 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4289e-05 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456305+0000", 2024-09-16T10:22:06.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4179999999999997e-06 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456374+0000", 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9666999999999996e-05 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.456388+0000", 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4032000000000001e-05 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.318084+0000", 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86169573099999996 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.318200+0000", 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115563 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.318337+0000", 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013709599999999999 2024-09-16T10:22:06.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322275+0000", 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039379439999999996 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325445+0000", 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031708299999999999 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326189+0000", 2024-09-16T10:22:06.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074375799999999996 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326233+0000", 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3921000000000002e-05 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326263+0000", 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0315999999999999e-05 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.356 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:164640 8.8 8:14c99ab6:::benchmark_data_smithi063_101290_object37320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:37.595518+0000", 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 88.614432371000007, 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12907907199999999, 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:06.357 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164640 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595518+0000", 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595518+0000", 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595520+0000", 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8109999999999999e-06 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595526+0000", 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5589999999999996e-06 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595536+0000", 2024-09-16T10:22:06.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9979999999999998e-06 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595542+0000", 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5509999999999996e-06 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595608+0000", 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5627e-05 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595749+0000", 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140712 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.595832+0000", 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3505999999999996e-05 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.598157+0000", 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023244089999999999 2024-09-16T10:22:06.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.598587+0000", 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000430145 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.724500+0000", 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12591281200000001 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.724552+0000", 2024-09-16T10:22:06.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2630000000000003e-05 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:37.724597+0000", 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5312000000000003e-05 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:06.363 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: pgmap v453: 158 pgs: 3 remapped+peering, 155 active+clean; 115 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 457 op/s; 3.0 MiB/s, 55 objects/s recovering 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: 3.4 scrub starts 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: 3.4 scrub ok 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: 6.f scrub starts 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: 6.f scrub ok 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3999944641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:22:06.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:06 smithi165 ceph-mon[32696]: osdmap e285: 8 total, 8 up, 8 in 2024-09-16T10:22:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:06.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:06.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:06.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 5 16 5807 5791 72.3808 86.125 0.00302055 0.0138063 2024-09-16T10:22:06.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:06.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:06.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:06.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:06.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:06.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:06.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: pgmap v453: 158 pgs: 3 remapped+peering, 155 active+clean; 115 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 457 op/s; 3.0 MiB/s, 55 objects/s recovering 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: 3.4 scrub starts 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: 3.4 scrub ok 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: 6.f scrub starts 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: 6.f scrub ok 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3999944641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[39470]: osdmap e285: 8 total, 8 up, 8 in 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: pgmap v453: 158 pgs: 3 remapped+peering, 155 active+clean; 115 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 457 op/s; 3.0 MiB/s, 55 objects/s recovering 2024-09-16T10:22:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: 3.4 scrub starts 2024-09-16T10:22:06.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: 3.4 scrub ok 2024-09-16T10:22:06.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: 6.f scrub starts 2024-09-16T10:22:06.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: 6.f scrub ok 2024-09-16T10:22:06.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:06.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3999944641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:22:06.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:06 smithi063 ceph-mon[29993]: osdmap e285: 8 total, 8 up, 8 in 2024-09-16T10:22:06.743 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:06.789 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:06.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:06.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:06.921 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:06.938 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:06.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:07.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:07.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:07.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:07.176 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:07.200 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:07.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:07.302 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:07.340 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:07.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:07.370 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:22:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:22:07.370 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[29993]: 5.1c scrub starts 2024-09-16T10:22:07.370 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[29993]: 5.1c scrub ok 2024-09-16T10:22:07.370 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:22:07.370 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:22:07.371 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[39470]: 5.1c scrub starts 2024-09-16T10:22:07.371 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[39470]: 5.1c scrub ok 2024-09-16T10:22:07.371 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:22:07.371 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:07 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:22:07.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 6 15 8604 8589 89.4606 174.875 0.00305038 0.01117 2024-09-16T10:22:07.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:07.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:07.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:07.551 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:07.551 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:07.577 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:07.590 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:07.591 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:07.623 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:07.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:07.625 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:07.626 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:07.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:07 smithi165 ceph-mon[32696]: 5.1c scrub starts 2024-09-16T10:22:07.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:07 smithi165 ceph-mon[32696]: 5.1c scrub ok 2024-09-16T10:22:07.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:07 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:22:07.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:07 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:22:07.659 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:07.660 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:07.692 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:07.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:22:07.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:07.728 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:07.762 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:07.762 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:07.765 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:07.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:07.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:22:07.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:22:07.782 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T10:22:07.782 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T10:22:07.782 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T10:22:07.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-16T10:22:07.799 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:07.799 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:07.866 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:07.904 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:07.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:08.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:08.044 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:08.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:08.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:08.154 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:08.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:08.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:08.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:08.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:08.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:08.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 7 16 11516 11500 102.669 181.938 0.00549616 0.00973189 2024-09-16T10:22:08.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:08.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:08.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:08.472 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:08 smithi165 ceph-mon[32696]: pgmap v455: 158 pgs: 3 remapped+peering, 155 active+clean; 115 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 424 op/s; 2.7 MiB/s, 51 objects/s recovering 2024-09-16T10:22:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:08 smithi165 ceph-mon[32696]: osdmap e286: 8 total, 8 up, 8 in 2024-09-16T10:22:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/654549412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:08.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[29993]: pgmap v455: 158 pgs: 3 remapped+peering, 155 active+clean; 115 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 424 op/s; 2.7 MiB/s, 51 objects/s recovering 2024-09-16T10:22:08.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[29993]: osdmap e286: 8 total, 8 up, 8 in 2024-09-16T10:22:08.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/654549412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:08.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:08.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[39470]: pgmap v455: 158 pgs: 3 remapped+peering, 155 active+clean; 115 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 424 op/s; 2.7 MiB/s, 51 objects/s recovering 2024-09-16T10:22:08.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[39470]: osdmap e286: 8 total, 8 up, 8 in 2024-09-16T10:22:08.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/654549412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:08.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:08.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:08.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:08.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:08.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:08.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:08.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:08.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:22:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:22:08.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:22:08.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:08.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:08.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:08.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:09.031 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:09.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:09.043 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:09.043 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:09.132 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:09.178 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:09.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:09.263 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "" 2024-09-16T10:22:09.263 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:09.280 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:09.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:09.325 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:09.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:09.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 8 16 14120 14104 110.177 162.75 0.00136973 0.00904418 2024-09-16T10:22:09.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:09.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:09.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:09.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:09.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:09.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:09.706 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:09.744 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:09.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:09.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:09.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:09.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:09.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:22:09.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:10.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:10.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:10.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:10.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:10.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:10 smithi165 ceph-mon[32696]: pgmap v457: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 963 B/s rd, 117 MiB/s wr, 1.87k op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:10 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-16T10:22:10.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:10 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-16T10:22:10.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:10.402 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 9 16 16719 16703 115.982 162.438 0.00573733 0.00861637 2024-09-16T10:22:10.430 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:10.532 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:10.564 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:10.628 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:22:10.648 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:10 smithi063 ceph-mon[29993]: pgmap v457: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 963 B/s rd, 117 MiB/s wr, 1.87k op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:10.648 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:10 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-16T10:22:10.648 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:10 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-16T10:22:10.649 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:10 smithi063 ceph-mon[39470]: pgmap v457: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 963 B/s rd, 117 MiB/s wr, 1.87k op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:10.649 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:10 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-16T10:22:10.649 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:10 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]} : dispatch 2024-09-16T10:22:10.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:10.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:10.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:10.827 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:10.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:10.975 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:10.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:10.993 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:11.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:11.095 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:11.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:11.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:11.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:11.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:11.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:11.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:11.368 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:11.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:11.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 10 16 18905 18889 118.045 136.625 0.00307509 0.00846514 2024-09-16T10:22:11.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-16T10:22:11.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:11 smithi165 ceph-mon[32696]: osdmap e287: 8 total, 8 up, 8 in 2024-09-16T10:22:11.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:11 smithi165 ceph-mon[32696]: osdmap e288: 8 total, 8 up, 8 in 2024-09-16T10:22:11.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3939298044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:11.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:11.523 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:11.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:11.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-16T10:22:11.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[29993]: osdmap e287: 8 total, 8 up, 8 in 2024-09-16T10:22:11.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[29993]: osdmap e288: 8 total, 8 up, 8 in 2024-09-16T10:22:11.545 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3939298044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:11.546 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-16T10:22:11.546 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[39470]: osdmap e287: 8 total, 8 up, 8 in 2024-09-16T10:22:11.546 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[39470]: osdmap e288: 8 total, 8 up, 8 in 2024-09-16T10:22:11.546 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3939298044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:11.568 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:11.568 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:11.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:11.671 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:11.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:11.745 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:11.745 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:11.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:11.788 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:11.788 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:11.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:11.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:11.825 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:11.825 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:11.863 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:11.863 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:11.877 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:22:11.899 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:11.899 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:11.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:11.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:11.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:11.932 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:11.933 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:11.965 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:11.965 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:11.999 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:12.000 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:12.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:12.074 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:12.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:12.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:12.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:12.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:12.308 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:12.347 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:12.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:12.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 11 16 21026 21010 119.364 132.562 0.00348188 0.00837082 2024-09-16T10:22:12.449 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[29993]: pgmap v460: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 965 B/s rd, 117 MiB/s wr, 1.87k op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/408099454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3939298044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[29993]: osdmap e289: 8 total, 8 up, 8 in 2024-09-16T10:22:12.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[39470]: pgmap v460: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 965 B/s rd, 117 MiB/s wr, 1.87k op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:12.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/408099454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:12.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:12.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3939298044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:12.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:12 smithi063 ceph-mon[39470]: osdmap e289: 8 total, 8 up, 8 in 2024-09-16T10:22:12.488 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:12.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:12 smithi165 ceph-mon[32696]: pgmap v460: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 965 B/s rd, 117 MiB/s wr, 1.87k op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:12 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/408099454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:12 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:12 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3939298044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:12 smithi165 ceph-mon[32696]: osdmap e289: 8 total, 8 up, 8 in 2024-09-16T10:22:12.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:12.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:12.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:12.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:22:12.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:12.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:12.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:12.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:12.914 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:12.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:13.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:13.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:13.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:13.114 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:13.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:13.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:13.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:13.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:13.331 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:13.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:13.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 12 16 23344 23328 121.489 144.875 0.00425619 0.00822554 2024-09-16T10:22:13.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:13.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:13.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:13.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:13.606 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:13.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:13.708 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:13.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:13.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:13.847 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:13.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:13.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:13.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:14.029 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:14.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:14.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:14.173 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:14.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:14.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:14.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:14.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:14.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 13 16 25703 25687 123.484 147.438 0.00281589 0.00808786 2024-09-16T10:22:14.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:14.437 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:14.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:14.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:14 smithi165 ceph-mon[32696]: pgmap v462: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 863 B/s rd, 105 MiB/s wr, 1.68k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:22:14.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:14.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:14.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:14.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:14.701 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:14.701 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39765 7.c 7:361d2464:::benchmark_data_smithi063_73086_object39764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.44941657599998, 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13511668299999999, 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.702 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39765 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247950+0000", 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.247952+0000", 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.889e-06 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248012+0000", 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0418000000000003e-05 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248023+0000", 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0852e-05 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248030+0000", 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9489999999999999e-06 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248067+0000", 2024-09-16T10:22:14.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7120999999999999e-05 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248138+0000", 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0885999999999999e-05 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248217+0000", 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9058000000000001e-05 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250443+0000", 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022264720000000002 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250487+0000", 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3823999999999999e-05 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.382955+0000", 2024-09-16T10:22:14.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13246761100000001 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383013+0000", 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7670000000000002e-05 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383066+0000", 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3933e-05 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.708 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39767 7.c 7:3eeafd38:::benchmark_data_smithi063_73086_object39766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.44932131000002, 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13517669099999999, 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.709 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39767 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248045+0000", 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248046+0000", 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5199999999999996e-07 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248153+0000", 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106803 2024-09-16T10:22:14.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248170+0000", 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.721e-05 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.248175+0000", 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5609999999999998e-06 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249333+0000", 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011573200000000001 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249455+0000", 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012275400000000001 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.249590+0000", 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013500099999999999 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250643+0000", 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001052966 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252177+0000", 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015336340000000001 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383126+0000", 2024-09-16T10:22:14.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13094891 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383178+0000", 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2141000000000001e-05 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383222+0000", 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3739000000000003e-05 2024-09-16T10:22:14.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39768 7.c 7:3b0be1c9:::benchmark_data_smithi063_73086_object39767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.447067602, 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13302192400000001, 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39768 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250299+0000", 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250301+0000", 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2529999999999999e-06 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250409+0000", 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108145 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250635+0000", 2024-09-16T10:22:14.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225582 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250641+0000", 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.63e-06 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250684+0000", 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3006000000000001e-05 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250790+0000", 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105831 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250923+0000", 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013304199999999999 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252242+0000", 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013184189999999999 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.252758+0000", 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051628000000000004 2024-09-16T10:22:14.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383257+0000", 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13049871900000001 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383300+0000", 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3223999999999998e-05 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383321+0000", 2024-09-16T10:22:14.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0792999999999999e-05 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39770 7.7 7:ec0b9b23:::benchmark_data_smithi063_73086_object39769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.44671204399998, 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13275636499999999, 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.723 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39770 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250654+0000", 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250655+0000", 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7999999999999993e-07 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250688+0000", 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2687000000000001e-05 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250695+0000", 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0110000000000003e-06 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250700+0000", 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250739+0000", 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8665999999999999e-05 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250820+0000", 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1252000000000001e-05 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.250936+0000", 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116036 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.253100+0000", 2024-09-16T10:22:14.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002163674 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255541+0000", 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024417549999999999 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383318+0000", 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.127776905 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383372+0000", 2024-09-16T10:22:14.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3826000000000003e-05 2024-09-16T10:22:14.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.383411+0000", 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8621000000000002e-05 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39786 7.c 7:3b648db5:::benchmark_data_smithi063_73086_object39785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.43585408799999, 2024-09-16T10:22:14.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124519465, 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39786 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261512+0000", 2024-09-16T10:22:14.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.733 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:14 smithi063 ceph-mon[29993]: pgmap v462: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 863 B/s rd, 105 MiB/s wr, 1.68k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:22:14.733 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:14 smithi063 ceph-mon[39470]: pgmap v462: 158 pgs: 158 active+clean; 735 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 863 B/s rd, 105 MiB/s wr, 1.68k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261514+0000", 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3430000000000001e-06 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261577+0000", 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3646999999999996e-05 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261587+0000", 2024-09-16T10:22:14.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9499999999999996e-06 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261593+0000", 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6810000000000003e-06 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261633+0000", 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0262999999999997e-05 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261709+0000", 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.551e-05 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.261799+0000", 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0437000000000002e-05 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.262947+0000", 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001148224 2024-09-16T10:22:14.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.263126+0000", 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000178529 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.385937+0000", 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122810896 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.385988+0000", 2024-09-16T10:22:14.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0961000000000002e-05 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.386032+0000", 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4023999999999997e-05 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40098 7.7 7:e51c64ce:::benchmark_data_smithi063_73086_object40097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:22:14.738 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.15200263899999, 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118499387, 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40098 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545364+0000", 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545366+0000", 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0839999999999999e-06 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545435+0000", 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9424999999999993e-05 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545449+0000", 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3283e-05 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545456+0000", 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0650000000000001e-06 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545522+0000", 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.614e-05 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545636+0000", 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011427800000000001 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.545759+0000", 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012285799999999999 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.547169+0000", 2024-09-16T10:22:14.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001410251 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.550261+0000", 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030920029999999999 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663775+0000", 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113513832 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663832+0000", 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6771999999999998e-05 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663863+0000", 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1396000000000002e-05 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40106 7.d 7:bffd718f:::benchmark_data_smithi063_73086_object40105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:22:14.744 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.14775591799997, 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112057936, 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40106 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549610+0000", 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549612+0000", 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3289999999999999e-06 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549663+0000", 2024-09-16T10:22:14.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1014999999999999e-05 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549673+0000", 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9229999999999997e-06 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549678+0000", 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1120000000000004e-06 2024-09-16T10:22:14.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549728+0000", 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9700999999999997e-05 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549857+0000", 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012977400000000001 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.549995+0000", 2024-09-16T10:22:14.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137498 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551274+0000", 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012796140000000001 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551903+0000", 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062848199999999998 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661566+0000", 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109663317 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661615+0000", 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9193000000000001e-05 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.661668+0000", 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2978e-05 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:40108 7.7 7:e3eeca68:::benchmark_data_smithi063_73086_object40107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 425.145895037, 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12682985199999999, 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 40108 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551471+0000", 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551473+0000", 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3090000000000001e-06 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551526+0000", 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999999e-05 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551536+0000", 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6120000000000005e-06 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551541+0000", 2024-09-16T10:22:14.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8790000000000002e-06 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551580+0000", 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9727000000000001e-05 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551660+0000", 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9480999999999993e-05 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:22:14.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.551754+0000", 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4548000000000003e-05 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.552878+0000", 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011237459999999999 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.663896+0000", 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11101776200000001 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678241+0000", 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014344894 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678275+0000", 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4551000000000002e-05 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.678301+0000", 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5843e-05 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65089 7.d 7:bff27887:::benchmark_data_smithi063_73086_object65088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 407.14080574899998, 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12756952299999999, 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65089 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.757 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556561+0000", 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556562+0000", 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.686e-06 2024-09-16T10:22:14.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556698+0000", 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013535199999999999 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556711+0000", 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3067e-05 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556719+0000", 2024-09-16T10:22:14.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7919999999999993e-06 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556773+0000", 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4647e-05 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.556924+0000", 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015124 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.557015+0000", 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0681999999999997e-05 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.558770+0000", 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001754395 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.562719+0000", 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039499050000000001 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684014+0000", 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.1212949 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684082+0000", 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7417999999999997e-05 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684130+0000", 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8439000000000002e-05 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.762 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:65092 7.d 7:b5d3643d:::benchmark_data_smithi063_73086_object65091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 407.13771594500002, 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12473208600000001, 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65092 2024-09-16T10:22:14.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559650+0000", 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559652+0000", 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4109999999999999e-06 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559730+0000", 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8343000000000005e-05 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559741+0000", 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-05 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559747+0000", 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.417e-06 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559788+0000", 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1597999999999997e-05 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559877+0000", 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9018999999999996e-05 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.559960+0000", 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2101000000000006e-05 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.561615+0000", 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00165525 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.562754+0000", 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00113956 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684280+0000", 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121525444 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684345+0000", 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:27.684383+0000", 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7639999999999999e-05 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.769 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72694 7.d 7:b02443fe:::benchmark_data_smithi063_73086_object72693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 400.8271775, 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11432945999999999, 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72694 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870189+0000", 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870190+0000", 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0499999999999999e-06 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870260+0000", 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9900000000000005e-05 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870269+0000", 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9959999999999999e-06 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.870273+0000", 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6210000000000003e-06 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872430+0000", 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002156748 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872556+0000", 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125859 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.872660+0000", 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103859 2024-09-16T10:22:14.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.874597+0000", 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373229999999999 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.877583+0000", 2024-09-16T10:22:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:14.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:14.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002986247 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984443+0000", 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10685934800000001 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984481+0000", 2024-09-16T10:22:14.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7718e-05 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984518+0000", 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7790999999999998e-05 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.777 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90054 7.c 7:3cae452b:::benchmark_data_smithi063_73086_object90053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.18392918699999, 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12751799899999999, 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90054 2024-09-16T10:22:14.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:22:14.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513440+0000", 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.452e-06 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513492+0000", 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1950999999999998e-05 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513503+0000", 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1232e-05 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513509+0000", 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0680000000000002e-06 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513571+0000", 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2434999999999994e-05 2024-09-16T10:22:14.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513694+0000", 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012256499999999999 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513865+0000", 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017080599999999999 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.517203+0000", 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033382860000000002 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.518058+0000", 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085528199999999996 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640875+0000", 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12281645400000001 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640919+0000", 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.401e-05 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640955+0000", 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6458000000000002e-05 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.784 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90065 7.d 7:bffa18df:::benchmark_data_smithi063_73086_object90064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.17549575999999, 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124785905, 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.785 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90065 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521873+0000", 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.497e-06 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521903+0000", 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9407000000000002e-05 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521916+0000", 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3597999999999999e-05 2024-09-16T10:22:14.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521924+0000", 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521991+0000", 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7318999999999997e-05 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522105+0000", 2024-09-16T10:22:14.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113467 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522205+0000", 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9829999999999998e-05 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525304+0000", 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030995089999999999 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525567+0000", 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026322999999999999 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646542+0000", 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120974217 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646604+0000", 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2543000000000001e-05 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646657+0000", 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2438000000000002e-05 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174956 7.7 7:eae3f4a6:::benchmark_data_smithi063_73086_object55677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 300.80767770300002, 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12889425400000001, 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.791 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174956 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889690+0000", 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.562e-06 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889695+0000", 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8149999999999996e-06 2024-09-16T10:22:14.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889708+0000", 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2943e-05 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889715+0000", 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.272e-06 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889765+0000", 2024-09-16T10:22:14.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9412000000000001e-05 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889881+0000", 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116789 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889985+0000", 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103195 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895390+0000", 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054051689999999996 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991131+0000", 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095740960999999999 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018515+0000", 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027384196 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018547+0000", 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1973e-05 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018583+0000", 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5967000000000003e-05 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174957 7.7 7:e71673f7:::benchmark_data_smithi063_73086_object55678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:22:14.797 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 300.80501785799999, 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.126386786, 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174957 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892351+0000", 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0310000000000001e-06 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892356+0000", 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2689999999999999e-06 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892368+0000", 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1848000000000001e-05 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892376+0000", 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0649999999999999e-06 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892738+0000", 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036210000000000002 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892862+0000", 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012463099999999999 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892975+0000", 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112526 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895478+0000", 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002503211 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991158+0000", 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095680116999999995 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018680+0000", 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027522007000000001 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018705+0000", 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4802000000000001e-05 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018735+0000", 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0179e-05 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.803 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:50103 8.5 8:a9c61bbe:::benchmark_data_smithi063_101290_object50102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:11.534001+0000", 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 183.163365613, 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113298935, 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 50103 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534001+0000", 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534001+0000", 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534002+0000", 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3170000000000001e-06 2024-09-16T10:22:14.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534032+0000", 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9859999999999999e-05 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534042+0000", 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0040999999999999e-05 2024-09-16T10:22:14.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534048+0000", 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2790000000000004e-06 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.537093+0000", 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003044593 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.537248+0000", 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155484 2024-09-16T10:22:14.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.540637+0000", 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033885410000000001 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.547818+0000", 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071807579999999998 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647081+0000", 2024-09-16T10:22:14.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099263245 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647209+0000", 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000127782 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647261+0000", 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2624999999999998e-05 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647300+0000", 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.841e-05 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:130414 8.9 8:968d8679:::benchmark_data_smithi063_101290_object3094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:13.782273+0000", 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 120.91509311599999, 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115477528, 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 130414 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782273+0000", 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782273+0000", 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782275+0000", 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.733e-06 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782280+0000", 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.006e-06 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782290+0000", 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8400000000000007e-06 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782296+0000", 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782342+0000", 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5874999999999997e-05 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782443+0000", 2024-09-16T10:22:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100833 2024-09-16T10:22:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782618+0000", 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017475600000000001 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.785644+0000", 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030258749999999999 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.785923+0000", 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027986599999999998 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.897607+0000", 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111683594 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.897694+0000", 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7159999999999999e-05 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.897751+0000", 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6598000000000003e-05 2024-09-16T10:22:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:213868 8.9 8:9bfe1c7f:::benchmark_data_smithi063_101290_object86548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:13.085724+0000", 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 61.611642676999999, 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13858646499999999, 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213868 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085724+0000", 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085724+0000", 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085726+0000", 2024-09-16T10:22:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.804e-06 2024-09-16T10:22:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085731+0000", 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3329999999999996e-06 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085743+0000", 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2173e-05 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085749+0000", 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0549999999999996e-06 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085809+0000", 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0038999999999999e-05 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085897+0000", 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7892999999999993e-05 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085969+0000", 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2039000000000006e-05 2024-09-16T10:22:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.087832+0000", 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018625429999999999 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.088125+0000", 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029382799999999999 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224207+0000", 2024-09-16T10:22:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13608161899999999 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224261+0000", 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4091000000000001e-05 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224310+0000", 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9048000000000002e-05 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:213877 8.9 8:96066085:::benchmark_data_smithi063_101290_object86557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:13.089496+0000", 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 61.607869966000003, 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.134949242, 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213877 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089496+0000", 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089496+0000", 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089499+0000", 2024-09-16T10:22:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1950000000000002e-06 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089504+0000", 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5160000000000002e-06 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089518+0000", 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.398e-05 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089525+0000", 2024-09-16T10:22:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999996e-06 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089569+0000", 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3733000000000003e-05 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089647+0000", 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8411000000000005e-05 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089716+0000", 2024-09-16T10:22:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9054999999999995e-05 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.091677+0000", 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019602700000000001 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.092671+0000", 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00099401099999999994 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224358+0000", 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.131687784 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224406+0000", 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7942000000000003e-05 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224446+0000", 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9314999999999999e-05 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:231103 8.9 8:90e59f9e:::benchmark_data_smithi063_101290_object103783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:24.761238+0000", 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 49.936128160999999, 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111702601, 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 231103 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761238+0000", 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761238+0000", 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761239+0000", 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0729999999999999e-06 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761243+0000", 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.67e-06 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761249+0000", 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8810000000000001e-06 2024-09-16T10:22:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761253+0000", 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5079999999999999e-06 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761305+0000", 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2107000000000001e-05 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761419+0000", 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113292 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761540+0000", 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121296 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.763633+0000", 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020926930000000001 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.763827+0000", 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000194539 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.872878+0000", 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109050915 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.872914+0000", 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5453000000000001e-05 2024-09-16T10:22:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.872941+0000", 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7174000000000001e-05 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:14.837 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:14.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:14.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:14.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:14.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:22:15.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:15.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:15.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:15.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:15.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:15.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:15.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:15.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:15.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:15.405 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 14 16 27932 27916 124.613 139.312 0.012298 0.00801875 2024-09-16T10:22:15.412 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:15.442 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:15.455 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:15.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:15.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:22:15.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:15.601 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:15.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:15.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:15.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:15.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:15.838 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:15.880 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:15.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:15.913 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:22:15.913 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:22:15.913 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T10:22:15.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:22:15.983 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:16.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:16.127 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:16.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:16.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:16.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:16.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 15 16 29763 29747 123.934 114.438 0.00231942 0.00805255 2024-09-16T10:22:16.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:16.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:16.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:16.570 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:16.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:16.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:16 smithi063 ceph-mon[29993]: pgmap v463: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 122 MiB/s wr, 1.96k op/s; 18 B/s, 0 objects/s recovering 2024-09-16T10:22:16.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:16 smithi063 ceph-mon[29993]: osdmap e290: 8 total, 8 up, 8 in 2024-09-16T10:22:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:16 smithi063 ceph-mon[39470]: pgmap v463: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 122 MiB/s wr, 1.96k op/s; 18 B/s, 0 objects/s recovering 2024-09-16T10:22:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:16 smithi063 ceph-mon[39470]: osdmap e290: 8 total, 8 up, 8 in 2024-09-16T10:22:16.708 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:16 smithi165 ceph-mon[32696]: pgmap v463: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 122 MiB/s wr, 1.96k op/s; 18 B/s, 0 objects/s recovering 2024-09-16T10:22:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:16 smithi165 ceph-mon[32696]: osdmap e290: 8 total, 8 up, 8 in 2024-09-16T10:22:16.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:16.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:16.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:16.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:16.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:17.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:17.094 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:17.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:17.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:17.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:17.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:17.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:22:17.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:17.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:17.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:17.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 16 15 31172 31157 121.696 88.125 0.00860416 0.00821229 2024-09-16T10:22:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:22:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:22:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:22:17.477 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:17.513 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:17.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:17.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:17.651 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:17.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:17.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:17.677 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:17.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:17.779 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:17.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:17.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:17.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:17.915 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:17.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:18.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:18.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:18.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:18.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:18.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:18.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:18.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:18.291 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:32709 9.c e290/276 9:343bebbe:::benchmark_data_smithi063_124717_object32708:head v 290'1962, mlcod=290'1960)", 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:18.286602+0000", 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00155554, 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015819670000000001, 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286602+0000", 2024-09-16T10:22:18.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286602+0000", 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286604+0000", 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2160000000000001e-06 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286634+0000", 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0165999999999999e-05 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286648+0000", 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4443e-05 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:18.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286654+0000", 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286735+0000", 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0649999999999995e-05 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.286752+0000", 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7503000000000002e-05 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:18.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:32713 9.9 e290/276 9:95b6edbb:::benchmark_data_smithi063_124717_object32712:head v 290'2089, mlcod=290'2084)", 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:18.287732+0000", 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00042593699999999998, 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050522700000000002, 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:18.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287732+0000", 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287732+0000", 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287733+0000", 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1900000000000001e-07 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287843+0000", 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110609 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287871+0000", 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7724000000000001e-05 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287878+0000", 2024-09-16T10:22:18.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9349999999999997e-06 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287985+0000", 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106832 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.288002+0000", 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7399000000000001e-05 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:32712 9.a e290/276 9:50e9d45c:::benchmark_data_smithi063_124717_object32711:head v 290'2032, mlcod=290'2029)", 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:18.287980+0000", 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.000178018, 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030798700000000002, 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:18.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287980+0000", 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287980+0000", 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:18.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287980+0000", 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7999999999999995e-07 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.288012+0000", 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1155000000000002e-05 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.288019+0000", 2024-09-16T10:22:18.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5809999999999999e-06 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.288024+0000", 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2830000000000001e-06 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.288101+0000", 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6729000000000003e-05 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.288117+0000", 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5970000000000001e-05 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:32704 9.d 9:bded28b4:::benchmark_data_smithi063_124717_object32703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:18.283939+0000", 2024-09-16T10:22:18.305 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0042189829999999999, 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044099639999999997, 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32704 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:18.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.283939+0000", 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.283939+0000", 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.283941+0000", 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.306e-06 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.283975+0000", 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3797999999999997e-05 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.283986+0000", 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0784999999999999e-05 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:18.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.283993+0000", 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2749999999999998e-06 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.284050+0000", 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7120999999999997e-05 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.284327+0000", 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027698499999999999 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.284492+0000", 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016505400000000001 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.287877+0000", 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033845400000000001 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:18.288119+0000", 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024183399999999999 2024-09-16T10:22:18.310 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.311 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:18.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:18.311 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:22:18.311 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 4 2024-09-16T10:22:18.311 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:18.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:18.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:18.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 17 16 32953 32937 121.081 111.25 0.00431554 0.00825404 2024-09-16T10:22:18.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:18.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:18.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:18 smithi165 ceph-mon[32696]: pgmap v465: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 808 B/s rd, 116 MiB/s wr, 1.86k op/s; 17 B/s, 0 objects/s recovering 2024-09-16T10:22:18.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:18.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:18.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:18.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:18.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:18 smithi063 ceph-mon[29993]: pgmap v465: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 808 B/s rd, 116 MiB/s wr, 1.86k op/s; 17 B/s, 0 objects/s recovering 2024-09-16T10:22:18.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:18 smithi063 ceph-mon[39470]: pgmap v465: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 808 B/s rd, 116 MiB/s wr, 1.86k op/s; 17 B/s, 0 objects/s recovering 2024-09-16T10:22:18.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:18.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:18.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:18.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:18.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:18.866 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:22:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:22:18.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:19.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:19.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:22:19.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:19.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:19.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:19.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:19.403 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 18 16 34637 34621 120.201 105.25 0.0036452 0.00830413 2024-09-16T10:22:19.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:19.434 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:19.459 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:19.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:19.574 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:19.676 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.699 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:19.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:19.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:19.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:19.800 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:19.829 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:19.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:19.963 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:19.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:20.065 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:20.073 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:22:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:22:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:22:20.073 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:20.105 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:20.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:20.207 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:20.247 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:20.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:20.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:20.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:20.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:20.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 19 16 36140 36124 118.818 93.9375 0.00456602 0.00841051 2024-09-16T10:22:20.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:20.483 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:22:20.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:20.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:20.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:20.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:20.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:20 smithi063 ceph-mon[29993]: pgmap v466: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 167 MiB/s wr, 2.67k op/s; 13 B/s, 0 objects/s recovering 2024-09-16T10:22:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/455447192' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:22:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:20 smithi063 ceph-mon[29993]: from='client.45632 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:22:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:20 smithi063 ceph-mon[39470]: pgmap v466: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 167 MiB/s wr, 2.67k op/s; 13 B/s, 0 objects/s recovering 2024-09-16T10:22:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/455447192' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:22:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:20 smithi063 ceph-mon[39470]: from='client.45632 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:22:20.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:20.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:20.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:20.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:20 smithi165 ceph-mon[32696]: pgmap v466: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 167 MiB/s wr, 2.67k op/s; 13 B/s, 0 objects/s recovering 2024-09-16T10:22:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/455447192' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:22:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:20 smithi165 ceph-mon[32696]: from='client.45632 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:22:20.881 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:20.919 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:20.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:22:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:20.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:21.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:21.062 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:21.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:21.163 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:21.194 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:21.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:21.296 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:21.326 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:22:21.335 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:21.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:21.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:22:21.403061+0000 min lat: 0.00102889 max lat: 2.20645 avg lat: 0.00846237 2024-09-16T10:22:21.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:22:21.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 20 16 37815 37799 118.111 104.688 0.00597291 0.00846237 2024-09-16T10:22:21.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:21.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:21.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:21.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:21.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:21.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: from='client.45632 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: osdmap e291: 8 total, 8 up, 8 in 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: 5.17 scrub starts 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: 5.17 scrub ok 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: 4.2 scrub starts 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: 4.2 scrub ok 2024-09-16T10:22:21.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:21 smithi165 ceph-mon[32696]: osdmap e292: 8 total, 8 up, 8 in 2024-09-16T10:22:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: from='client.45632 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:22:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: osdmap e291: 8 total, 8 up, 8 in 2024-09-16T10:22:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: 5.17 scrub starts 2024-09-16T10:22:21.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: 5.17 scrub ok 2024-09-16T10:22:21.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: 4.2 scrub starts 2024-09-16T10:22:21.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: 4.2 scrub ok 2024-09-16T10:22:21.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[29993]: osdmap e292: 8 total, 8 up, 8 in 2024-09-16T10:22:21.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:21.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: from='client.45632 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:22:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: osdmap e291: 8 total, 8 up, 8 in 2024-09-16T10:22:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: 5.17 scrub starts 2024-09-16T10:22:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: 5.17 scrub ok 2024-09-16T10:22:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: 4.2 scrub starts 2024-09-16T10:22:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: 4.2 scrub ok 2024-09-16T10:22:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:21 smithi063 ceph-mon[39470]: osdmap e292: 8 total, 8 up, 8 in 2024-09-16T10:22:21.706 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:21.737 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:21.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:21.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:21.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:21.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:21.895 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:22:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72703 7.5 7:ace6f8ac:::benchmark_data_smithi063_73086_object72702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.878319+0000", 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 408.014173981, 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10653797499999999, 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72703 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878319+0000", 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878319+0000", 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878321+0000", 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0449999999999999e-06 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878367+0000", 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5639999999999997e-05 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878379+0000", 2024-09-16T10:22:21.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2323e-05 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878386+0000", 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7989999999999996e-06 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878441+0000", 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.469e-05 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878556+0000", 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115494 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878705+0000", 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014857800000000001 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.883091+0000", 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043867230000000004 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.886390+0000", 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032982129999999999 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984773+0000", 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098383089000000007 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984822+0000", 2024-09-16T10:22:21.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8921000000000001e-05 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984857+0000", 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5460000000000003e-05 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72706 7.5 7:affb7f46:::benchmark_data_smithi063_73086_object72705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.879155+0000", 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 408.01333826500002, 2024-09-16T10:22:21.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105803568, 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72706 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879155+0000", 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879155+0000", 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879156+0000", 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879225+0000", 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9202000000000006e-05 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879240+0000", 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4399000000000001e-05 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879246+0000", 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4729999999999997e-06 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.880777+0000", 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001531325 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.880894+0000", 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116352 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.881025+0000", 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013082400000000001 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.883350+0000", 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023250409999999999 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.886489+0000", 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031394550000000002 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984891+0000", 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098401706000000005 2024-09-16T10:22:21.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984925+0000", 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4462000000000003e-05 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984958+0000", 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3092000000000001e-05 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174952 7.5 7:a881b411:::benchmark_data_smithi063_73086_object55673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.888558+0000", 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 308.00393504900001, 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10417939399999999, 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.910 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.910 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174952 2024-09-16T10:22:21.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.910 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888558+0000", 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888558+0000", 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888561+0000", 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.571e-06 2024-09-16T10:22:21.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888568+0000", 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888584+0000", 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6036e-05 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888593+0000", 2024-09-16T10:22:21.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5999999999999996e-06 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888667+0000", 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3226000000000007e-05 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888797+0000", 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013077999999999999 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888907+0000", 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109959 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.893981+0000", 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050738579999999997 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896037+0000", 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020559530000000001 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992624+0000", 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096586728999999996 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992695+0000", 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1174999999999995e-05 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992737+0000", 2024-09-16T10:22:21.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2330999999999998e-05 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174953 7.5 7:a2672bfb:::benchmark_data_smithi063_73086_object55674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.888773+0000", 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 308.00372026899998, 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10411178, 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174953 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888773+0000", 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888773+0000", 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888774+0000", 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2799999999999995e-07 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888778+0000", 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0210000000000001e-06 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888783+0000", 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8449999999999999e-06 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888787+0000", 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9060000000000004e-06 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.891543+0000", 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027551820000000001 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.891659+0000", 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0001165 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.891779+0000", 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011995 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895997+0000", 2024-09-16T10:22:21.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042179310000000003 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896225+0000", 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022793800000000001 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992799+0000", 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096574229999999997 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992845+0000", 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6041999999999997e-05 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992885+0000", 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9406999999999997e-05 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174954 7.5 7:a9783837:::benchmark_data_smithi063_73086_object55675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:22:21.922 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.888820+0000", 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 308.00367333200001, 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112695169, 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174954 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888820+0000", 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888820+0000", 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888821+0000", 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8400000000000003e-07 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888824+0000", 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5030000000000002e-06 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888831+0000", 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2389999999999996e-06 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888837+0000", 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7250000000000002e-06 2024-09-16T10:22:21.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.894010+0000", 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051734229999999999 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.894103+0000", 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2594000000000001e-05 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.894171+0000", 2024-09-16T10:22:21.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7414000000000002e-05 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896108+0000", 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373540000000001 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896336+0000", 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000227777 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.001448+0000", 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105112742 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.001482+0000", 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4014999999999999e-05 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.001515+0000", 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499000000000002e-05 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.928 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:200795 7.5 7:a5355407:::benchmark_data_smithi063_73086_object81516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.235559+0000", 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 281.65693365300001, 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.088954024000000007, 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200795 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235559+0000", 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235559+0000", 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235561+0000", 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.356e-06 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235565+0000", 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6439999999999999e-06 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235577+0000", 2024-09-16T10:22:21.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2064e-05 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235585+0000", 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2509999999999997e-06 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.237700+0000", 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021148600000000001 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.237830+0000", 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013032899999999999 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.237962+0000", 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131977 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.244826+0000", 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068638509999999998 2024-09-16T10:22:21.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.246316+0000", 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001490571 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324433+0000", 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078116624999999995 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324476+0000", 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.282e-05 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324513+0000", 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7676000000000003e-05 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:200801 7.5 7:a8e7ac74:::benchmark_data_smithi063_73086_object81522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.240067+0000", 2024-09-16T10:22:21.935 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 281.65242643599998, 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092800275000000002, 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200801 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240067+0000", 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240067+0000", 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240069+0000", 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0999999999999998e-06 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240074+0000", 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4090000000000002e-06 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240087+0000", 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2661e-05 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240095+0000", 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8590000000000005e-06 2024-09-16T10:22:21.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240719+0000", 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062448799999999997 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240840+0000", 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012078699999999999 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240961+0000", 2024-09-16T10:22:21.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121004 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.244830+0000", 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038687109999999999 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.246349+0000", 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015198130000000001 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332752+0000", 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.086402822000000004 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332814+0000", 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1289000000000002e-05 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332867+0000", 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3331999999999997e-05 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:200804 7.5 7:a1363d7b:::benchmark_data_smithi063_73086_object81525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.241179+0000", 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 281.65131367499998, 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.091818583999999995, 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:22:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200804 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241179+0000", 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241179+0000", 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241181+0000", 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3629999999999999e-06 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241185+0000", 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.34e-06 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241195+0000", 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0254999999999999e-05 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241202+0000", 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4420000000000003e-06 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.242942+0000", 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017397510000000001 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.243058+0000", 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116129 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.243163+0000", 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104888 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.244949+0000", 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001786953 2024-09-16T10:22:21.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.246401+0000", 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014510980000000001 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332921+0000", 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.086520606 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332966+0000", 2024-09-16T10:22:21.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4706000000000002e-05 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332998+0000", 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2053e-05 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:28376 9.d 9:b64251d8:::benchmark_data_smithi063_124717_object28375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:15.605145+0000", 2024-09-16T10:22:21.948 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.2873477910000002, 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092685612000000001, 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 28376 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.949 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605145+0000", 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605145+0000", 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605146+0000", 2024-09-16T10:22:21.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0800000000000004e-07 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605183+0000", 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7338000000000001e-05 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605193+0000", 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0032e-05 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605200+0000", 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8580000000000002e-06 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.607011+0000", 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001811177 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.607134+0000", 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122776 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:22:21.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.607219+0000", 2024-09-16T10:22:21.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.488e-05 2024-09-16T10:22:21.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.608041+0000", 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082233800000000004 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.608276+0000", 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000234837 2024-09-16T10:22:21.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.697740+0000", 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089463903999999997 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.697775+0000", 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4635000000000003e-05 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.697831+0000", 2024-09-16T10:22:21.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6029000000000002e-05 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34105 9.e 9:7031eac9:::benchmark_data_smithi063_124717_object34104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.010625+0000", 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.881867744, 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11631763000000001, 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34105 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.010625+0000", 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.010625+0000", 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.010627+0000", 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2619999999999999e-06 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.011187+0000", 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056009600000000005 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.011197+0000", 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0321e-05 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.011204+0000", 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6370000000000002e-06 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.012054+0000", 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085072100000000003 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.012304+0000", 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024928 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.012420+0000", 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116605 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.015009+0000", 2024-09-16T10:22:21.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025885779999999998 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.015053+0000", 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3781e-05 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126825+0000", 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11177234599999999 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126888+0000", 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3040999999999995e-05 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126943+0000", 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4962e-05 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34123 9.e 9:700d37e5:::benchmark_data_smithi063_124717_object34122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.019216+0000", 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.8732770580000002, 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11142177, 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34123 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019216+0000", 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019216+0000", 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019218+0000", 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9039999999999999e-06 2024-09-16T10:22:21.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019248+0000", 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9603000000000001e-05 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019260+0000", 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2011000000000001e-05 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019265+0000", 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1529999999999996e-06 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019313+0000", 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8605e-05 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019556+0000", 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024223199999999999 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019687+0000", 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013166600000000001 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.021167+0000", 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014795399999999999 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.021868+0000", 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070079799999999998 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130568+0000", 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10870049900000001 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130602+0000", 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3524e-05 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130638+0000", 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6235000000000001e-05 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34126 9.3 9:c6eaf356:::benchmark_data_smithi063_124717_object34125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:21.969 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.019477+0000", 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.8730159180000001, 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111048233, 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34126 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019477+0000", 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019477+0000", 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019478+0000", 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6600000000000005e-07 2024-09-16T10:22:21.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019547+0000", 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8853e-05 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019555+0000", 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8769999999999993e-06 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019559+0000", 2024-09-16T10:22:21.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1930000000000003e-06 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019610+0000", 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1343e-05 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019887+0000", 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027674900000000002 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020069+0000", 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000181509 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.022182+0000", 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002113541 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126572+0000", 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104389754 2024-09-16T10:22:21.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130383+0000", 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003811156 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130476+0000", 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2854000000000002e-05 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130525+0000", 2024-09-16T10:22:21.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9537999999999999e-05 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34128 9.d 9:b9451fa2:::benchmark_data_smithi063_124717_object34127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.020108+0000", 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.872384753, 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110568416, 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34128 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020108+0000", 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020108+0000", 2024-09-16T10:22:21.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020110+0000", 2024-09-16T10:22:21.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3200000000000001e-06 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020282+0000", 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017282899999999999 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020299+0000", 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7045e-05 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020306+0000", 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8159999999999996e-06 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020348+0000", 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2012999999999999e-05 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020456+0000", 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107539 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020569+0000", 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011336900000000001 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.023688+0000", 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031182919999999999 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.023701+0000", 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3773e-05 2024-09-16T10:22:21.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130591+0000", 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10688929899999999 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130641+0000", 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0253999999999997e-05 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130677+0000", 2024-09-16T10:22:21.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5867000000000001e-05 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34135 9.e 9:7f486d23:::benchmark_data_smithi063_124717_object34134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.024112+0000", 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.8683815510000001, 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106646003, 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.984 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34135 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024112+0000", 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024112+0000", 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024113+0000", 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.885e-06 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024216+0000", 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102953 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024227+0000", 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0966e-05 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024233+0000", 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5339999999999999e-06 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024283+0000", 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9676999999999999e-05 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024367+0000", 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4297999999999998e-05 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024472+0000", 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010469200000000001 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.025659+0000", 2024-09-16T10:22:21.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00118731 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.026080+0000", 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042150899999999998 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130689+0000", 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104608439 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130725+0000", 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6517000000000003e-05 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130758+0000", 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2223e-05 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35062 9.3 9:c7e49ef2:::benchmark_data_smithi063_124717_object35061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.657982+0000", 2024-09-16T10:22:21.990 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.2345111179999999, 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098016333999999997, 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35062 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.657982+0000", 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.657982+0000", 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.657983+0000", 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.519e-06 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658015+0000", 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1820999999999999e-05 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658031+0000", 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877999999999999e-05 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658038+0000", 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7619999999999998e-06 2024-09-16T10:22:21.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658103+0000", 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5547000000000001e-05 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658224+0000", 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120492 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658343+0000", 2024-09-16T10:22:21.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118836 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.659790+0000", 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014468860000000001 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.661285+0000", 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001495106 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.755758+0000", 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.094472798999999996 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.755824+0000", 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6662000000000007e-05 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.755998+0000", 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174026 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35071 9.e 9:7ced6e27:::benchmark_data_smithi063_124717_object35070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.663307+0000", 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.2291858680000001, 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092814160000000007, 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35071 2024-09-16T10:22:21.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663307+0000", 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663307+0000", 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663309+0000", 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.381e-06 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663380+0000", 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1410999999999998e-05 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663389+0000", 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.397e-06 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:21.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663395+0000", 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5310000000000001e-06 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663442+0000", 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7074999999999999e-05 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663550+0000", 2024-09-16T10:22:22.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107712 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663665+0000", 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115765 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666160+0000", 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024947010000000002 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666207+0000", 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6604999999999998e-05 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.756015+0000", 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089808288 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.756074+0000", 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9097e-05 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.756121+0000", 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7197000000000002e-05 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35072 9.e 9:7d9eb204:::benchmark_data_smithi063_124717_object35071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.664392+0000", 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.2281011230000001, 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097734269999999998, 2024-09-16T10:22:22.003 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35072 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664392+0000", 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664392+0000", 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664393+0000", 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.548e-06 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664514+0000", 2024-09-16T10:22:22.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012095900000000001 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664528+0000", 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3368e-05 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664535+0000", 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.007e-06 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:22.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665039+0000", 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050449100000000003 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665152+0000", 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011252999999999999 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665321+0000", 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016960599999999999 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666269+0000", 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00094797700000000002 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666889+0000", 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062005199999999995 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.762030+0000", 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095141030000000001 2024-09-16T10:22:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.762084+0000", 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3996000000000003e-05 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.762126+0000", 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1705999999999997e-05 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35073 9.d 9:b3b70288:::benchmark_data_smithi063_124717_object35072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.664788+0000", 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.227704959, 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095865565999999999, 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35073 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.010 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664788+0000", 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664788+0000", 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664789+0000", 2024-09-16T10:22:22.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-16T10:22:22.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:22.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664859+0000", 2024-09-16T10:22:22.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9968999999999994e-05 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664873+0000", 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3570999999999999e-05 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664880+0000", 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2810000000000003e-06 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664925+0000", 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4685000000000003e-05 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665060+0000", 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013474400000000001 2024-09-16T10:22:22.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665196+0000", 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013607199999999999 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666424+0000", 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012285359999999999 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666735+0000", 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000310833 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.760561+0000", 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093826309999999996 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.760614+0000", 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2232999999999999e-05 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.760654+0000", 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0052000000000002e-05 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:37532 9.3 9:cf32c78d:::benchmark_data_smithi063_124717_object37531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:21.177962+0000", 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.71453079100000005, 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.091552389999999997, 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:22.017 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 37532 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177962+0000", 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177962+0000", 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177963+0000", 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6600000000000005e-07 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177993+0000", 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9892000000000001e-05 2024-09-16T10:22:22.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177999+0000", 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-06 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178004+0000", 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6609999999999999e-06 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178045+0000", 2024-09-16T10:22:22.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1409999999999998e-05 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178151+0000", 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106105 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178269+0000", 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118021 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:22.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.181252+0000", 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00298307 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.183294+0000", 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020413699999999998 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.269406+0000", 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.086112094 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.269475+0000", 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9246999999999996e-05 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.269515+0000", 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9453999999999999e-05 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.023 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:37541 9.3 9:c2ca3a24:::benchmark_data_smithi063_124717_object37540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:21.186221+0000", 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.70627219900000004, 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092424884999999998, 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 37541 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:22:22.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186221+0000", 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186221+0000", 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186222+0000", 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6249999999999999e-06 2024-09-16T10:22:22.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186275+0000", 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2549000000000001e-05 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186285+0000", 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7650000000000005e-06 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:22:22.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186290+0000", 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9219999999999996e-06 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186329+0000", 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9509999999999999e-05 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186417+0000", 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7609999999999996e-05 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186501+0000", 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3737000000000001e-05 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.187593+0000", 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010920649999999999 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.266582+0000", 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078989689000000002 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.278548+0000", 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011965748 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.278600+0000", 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1845000000000003e-05 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.278646+0000", 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5819999999999998e-05 2024-09-16T10:22:22.029 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.030 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:22.030 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.030 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:22:22.030 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:22:22.030 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:22:22.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:22.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:22.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:22.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:22.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:22.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:22.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:22.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 21 16 39203 39187 116.617 86.75 0.0113221 0.00856963 2024-09-16T10:22:22.439 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:22.480 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:22.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:22 smithi165 ceph-mon[32696]: 3.9 scrub starts 2024-09-16T10:22:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:22 smithi165 ceph-mon[32696]: 3.9 scrub ok 2024-09-16T10:22:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:22 smithi165 ceph-mon[32696]: pgmap v469: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-16T10:22:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:22 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:22:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:22 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:22:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:22 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:22:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:22 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:22:22.582 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:22.620 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:22.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[29993]: 3.9 scrub starts 2024-09-16T10:22:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[29993]: 3.9 scrub ok 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[29993]: pgmap v469: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[39470]: 3.9 scrub starts 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[39470]: 3.9 scrub ok 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[39470]: pgmap v469: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:22:22.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:22:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:22:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:22 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:22:22.722 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:22.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:22.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:22.828 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:22:22.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:22.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:22.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:23.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:23.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:23.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:23.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:23.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:23.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:23.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:23.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:23.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:23.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 22 16 40922 40906 116.199 107.438 0.00710987 0.00860081 2024-09-16T10:22:23.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:23.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:23 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:22:23.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:23 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:22:23.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:23.413 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:23 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:22:23.413 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:23 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:22:23.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:23.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:23.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:23.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:23.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:23.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:23.592 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:23.613 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:23.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:23.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:23.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:23 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:22:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:23 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:22:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:23.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:23.857 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:23.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:23.954 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:22:23.955 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:22:23.955 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 293, 2024-09-16T10:22:23.955 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:22:23.955 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:22:23.343288+0000", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:22:01.866838+0000", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:22:23.956 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 9, 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:22:23.957 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:22:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 11, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 11, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 11, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 11, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 11, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.b", 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 275, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 274, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 274, 2024-09-16T10:22:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "275", 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "275", 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "273", 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:22:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:22:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:22:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.1800000667572021, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.1800000667572021, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-16T10:22:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:22:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:22:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:22:23.966 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:22:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:22:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:22:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:22:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:22:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:22:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:22:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:22:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:22:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:22:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:22:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:22:23.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:22:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:22:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:22:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:22:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:22:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:22:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:22:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:22:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:22:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:22:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:22:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:22:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:22:23.982 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.25, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.25, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.25, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.25, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:22:23.984 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:22:23.985 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:22:23.986 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:22:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:22:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.25, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.25, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.25, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.25, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:22:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:22:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:22:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:22:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:22:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:22:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:22:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 9, 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_2", 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:21:54.923489+0000", 2024-09-16T10:22:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:22:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "279", 2024-09-16T10:22:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:22:23.999 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:22:24.000 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:22:24.001 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:22:24.002 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.003 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 281, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:22:24.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.005 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:22:24.006 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.008 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:22:24.008 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:22:24.008 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.008 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.008 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.009 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.009 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.009 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 281, 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.010 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.011 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.012 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 281, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.014 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.015 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:22:24.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:22:24.018 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 284, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:22:24.020 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.021 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.022 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:22:24.023 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 287, 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.025 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:22:24.026 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.027 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 284, 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.029 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:22:24.030 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 263, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:22:24.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.034 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.035 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:22:24.036 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 281, 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:22:24.037 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:22:24.038 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:22:24.039 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.040 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:21:29.995072+0000", 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:21:54.296257+0000", 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.041 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:06.859608+0000", 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.042 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:21:23.881456+0000", 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:02.701780+0000", 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:21:23.824687+0000", 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:22:24.044 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:21:24.136023+0000", 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:22:24.045 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:23.134000+0000", 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [], 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.10", 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 6, 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:22:24.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.13", 2024-09-16T10:22:24.047 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:22:24.048 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:22:24.049 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:22:24.050 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:22:24.051 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:22:24.052 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:22:24.053 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:22:24.053 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:22:24.053 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:22:24.053 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:22:24.053 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:24.053 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:22:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:24.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:24.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:24.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:24.193 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:24.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:24.295 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:24.322 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:24.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:24.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 23 16 42552 42536 115.576 101.875 0.0129739 0.00864717 2024-09-16T10:22:24.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:24.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:24.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:24.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:24.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:24.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:24.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:22:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:22:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[29993]: pgmap v470: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 698 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-16T10:22:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[29993]: 6.18 scrub starts 2024-09-16T10:22:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[29993]: 6.18 scrub ok 2024-09-16T10:22:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[29993]: osdmap e293: 8 total, 8 up, 8 in 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2076979671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[39470]: pgmap v470: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 698 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[39470]: 6.18 scrub starts 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[39470]: 6.18 scrub ok 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[39470]: osdmap e293: 8 total, 8 up, 8 in 2024-09-16T10:22:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2076979671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:22:24.707 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:24.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:24.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:24 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:22:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:24 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:22:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:24 smithi165 ceph-mon[32696]: pgmap v470: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 698 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-16T10:22:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:24 smithi165 ceph-mon[32696]: 6.18 scrub starts 2024-09-16T10:22:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:24 smithi165 ceph-mon[32696]: 6.18 scrub ok 2024-09-16T10:22:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:24 smithi165 ceph-mon[32696]: osdmap e293: 8 total, 8 up, 8 in 2024-09-16T10:22:24.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2076979671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:22:24.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:24.941 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:24.941 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:24.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:24.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:24.980 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:24.980 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:24.986 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:22:25.013 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:25.014 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:25.043 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:25.043 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:25.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:25.078 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:25.078 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:25.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:25.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:25.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:25.115 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:25.150 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:25.150 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:25.183 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:25.183 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:25.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:25.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:25.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:25.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:25.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:25.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:25.405 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 24 16 44298 44282 115.305 109.125 0.00620014 0.00866736 2024-09-16T10:22:25.448 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:25.492 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:25.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:25.568 INFO:tasks.thrashosds.thrasher:j is {'epoch': 293, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:22:23.343288+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:22:01.866838+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 275, 'last_epoch_started': 274, 'last_epoch_clean': 274, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '275', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '275', 'last_force_op_resend_preluminous': '273', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-16T10:21:54.923489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 284, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 287, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 284, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 263, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:21:29.995072+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:21:54.296257+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:06.859608+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:21:23.881456+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:02.701780+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:21:23.824687+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:21:24.136023+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:23.134000+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.10', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:22:25.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:22:25.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:25.623 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:25.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 5.d scrub starts 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 5.d scrub ok 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2753788658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:22:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 5.d scrub starts 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 5.d scrub ok 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2753788658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:22:25.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:25 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:22:25.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:25.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 5.d scrub starts 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 5.d scrub ok 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2753788658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:22:25.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:25 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:22:25.803 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:25.803 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:25.866 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:25.920 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:25.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:26.021 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:26.064 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:26.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:26.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:26.203 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:26.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:26.305 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:26.341 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:26.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:26.404 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 25 16 46056 46040 115.087 109.875 0.0134998 0.00868393 2024-09-16T10:22:26.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:26.489 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:26.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:26.513 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:26 smithi063 ceph-mon[29993]: pgmap v472: 158 pgs: 158 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-16T10:22:26.514 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:26 smithi063 ceph-mon[29993]: osdmap e294: 8 total, 8 up, 8 in 2024-09-16T10:22:26.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:26 smithi063 ceph-mon[39470]: pgmap v472: 158 pgs: 158 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-16T10:22:26.514 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:26 smithi063 ceph-mon[39470]: osdmap e294: 8 total, 8 up, 8 in 2024-09-16T10:22:26.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:26.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:26.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:26.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:26.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:26.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:26.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:26 smithi165 ceph-mon[32696]: pgmap v472: 158 pgs: 158 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-16T10:22:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:26 smithi165 ceph-mon[32696]: osdmap e294: 8 total, 8 up, 8 in 2024-09-16T10:22:26.860 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:26.905 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:26.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:27.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:27.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:27.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:27.196 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:22:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:22:27.245 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:27.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:22:27.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:27.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:27.407 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 26 16 47826 47810 114.915 110.625 0.0174961 0.00869628 2024-09-16T10:22:27.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:27.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:27.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:22:27.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 1.5 scrub ok 2024-09-16T10:22:27.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 4.a scrub starts 2024-09-16T10:22:27.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 4.a scrub ok 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 1.5 scrub ok 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 4.a scrub starts 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 4.a scrub ok 2024-09-16T10:22:27.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:22:27.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:22:27.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:22:27.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:22:27.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:22:27.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:27 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:22:27.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:27.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:27.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:27.709 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 1.5 scrub ok 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 4.a scrub starts 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 4.a scrub ok 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:22:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:22:27.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:22:27.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:27 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:22:27.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:27.849 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:27.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:27.986 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:27.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:28.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:28.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:28.213 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:28.259 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:28.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:28.319 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:28.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:28.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:28.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:28.405 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 27 16 49338 49322 114.159 94.5 0.0229879 0.00875316 2024-09-16T10:22:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[29993]: pgmap v474: 158 pgs: 158 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 808 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:22:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[29993]: 3.1b scrub starts 2024-09-16T10:22:28.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[29993]: 3.1b scrub ok 2024-09-16T10:22:28.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[29993]: osdmap e295: 8 total, 8 up, 8 in 2024-09-16T10:22:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[39470]: pgmap v474: 158 pgs: 158 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 808 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:22:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[39470]: 3.1b scrub starts 2024-09-16T10:22:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[39470]: 3.1b scrub ok 2024-09-16T10:22:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:28 smithi063 ceph-mon[39470]: osdmap e295: 8 total, 8 up, 8 in 2024-09-16T10:22:28.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:28.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:28.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:28.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:22:28.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:28.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:28.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:28 smithi165 ceph-mon[32696]: pgmap v474: 158 pgs: 158 active+clean; 2.5 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 808 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:22:28.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:28 smithi165 ceph-mon[32696]: 3.1b scrub starts 2024-09-16T10:22:28.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:28 smithi165 ceph-mon[32696]: 3.1b scrub ok 2024-09-16T10:22:28.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:28 smithi165 ceph-mon[32696]: osdmap e295: 8 total, 8 up, 8 in 2024-09-16T10:22:28.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:22:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:22:28.741 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:22:28.744 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:22:28.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:28.802 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:28.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:28.903 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:28.964 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:28.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:29.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:29.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:29.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:29.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:29.246 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:29.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:29.348 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:29.390 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:29.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:29.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:29.405 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 28 16 50809 50793 113.365 91.9375 0.00976777 0.00881504 2024-09-16T10:22:29.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:29.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:29.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:29.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:29.665 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:29.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[29993]: from='client.25696 v1:172.21.15.63:0/1412968237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:22:29.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:22:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[39470]: from='client.25696 v1:172.21.15.63:0/1412968237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:22:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:22:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:29 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:22:29.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:29 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:22:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:29 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:22:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:29 smithi165 ceph-mon[32696]: from='client.25696 v1:172.21.15.63:0/1412968237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:22:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:29 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:22:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:29 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:22:29.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:29.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:29.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:29.921 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:29.962 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:29.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:29.995 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-16T10:22:27.408167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2693667355,"num_objects":41292,"num_object_clones":0,"num_object_copies":123876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41292,"num_whiteouts":0,"num_read":1241,"num_read_kb":993,"num_write":41529,"num_write_kb":2629997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":19861500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41809,"ondisk_log_size":41809,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10159036,"kb_used_data":7754920,"kb_used_omap":12,"kb_used_meta":2403955,"kb_avail":739802180,"statfs":{"total":767960285184,"available":757557432320,"internally_reserved":0,"allocated":7941038080,"data_stored":7934030274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2461650481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,6,9,7,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":521928704,"num_objects":7964,"num_object_clones":0,"num_object_copies":23892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7964,"num_write_kb":509696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.207761"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":529,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371015+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:22:23.371015+0000","last_peered":"2024-09-16T10:22:23.371015+0000","last_clean":"2024-09-16T10:22:23.371015+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:22:23.371015+0000","last_undegraded":"2024-09-16T10:22:23.371015+0000","last_fullsized":"2024-09-16T10:22:23.371015+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.795446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":558,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872078+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:22:20.872078+0000","last_peered":"2024-09-16T10:22:20.872078+0000","last_clean":"2024-09-16T10:22:20.872078+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:22:20.872078+0000","last_undegraded":"2024-09-16T10:22:20.872078+0000","last_fullsized":"2024-09-16T10:22:20.872078+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.680873+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":507,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.365299+0000","last_change":"2024-09-16T10:22:02.564509+0000","last_active":"2024-09-16T10:22:23.365299+0000","last_peered":"2024-09-16T10:22:23.365299+0000","last_clean":"2024-09-16T10:22:23.365299+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:22:23.365299+0000","last_undegraded":"2024-09-16T10:22:23.365299+0000","last_fullsized":"2024-09-16T10:22:23.365299+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.564458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.564458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.322235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":492,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872294+0000","last_change":"2024-09-16T10:21:25.880760+0000","last_active":"2024-09-16T10:22:20.872294+0000","last_peered":"2024-09-16T10:22:20.872294+0000","last_clean":"2024-09-16T10:22:20.872294+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:22:20.872294+0000","last_undegraded":"2024-09-16T10:22:20.872294+0000","last_fullsized":"2024-09-16T10:22:20.872294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:25.880690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:25.880690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:35.656146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369453+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:22:23.369453+0000","last_peered":"2024-09-16T10:22:23.369453+0000","last_clean":"2024-09-16T10:22:23.369453+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:22:23.369453+0000","last_undegraded":"2024-09-16T10:22:23.369453+0000","last_fullsized":"2024-09-16T10:22:23.369453+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.317265+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":521,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369196+0000","last_change":"2024-09-16T10:22:02.905272+0000","last_active":"2024-09-16T10:22:23.369196+0000","last_peered":"2024-09-16T10:22:23.369196+0000","last_clean":"2024-09-16T10:22:23.369196+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:22:23.369196+0000","last_undegraded":"2024-09-16T10:22:23.369196+0000","last_fullsized":"2024-09-16T10:22:23.369196+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:33.256934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:33.256934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":573,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368737+0000","last_change":"2024-09-16T10:22:02.945083+0000","last_active":"2024-09-16T10:22:23.368737+0000","last_peered":"2024-09-16T10:22:23.368737+0000","last_clean":"2024-09-16T10:22:23.368737+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:22:23.368737+0000","last_undegraded":"2024-09-16T10:22:23.368737+0000","last_fullsized":"2024-09-16T10:22:23.368737+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:20:31.270548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.270548+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.423393+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":496,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.871990+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:22:20.871990+0000","last_peered":"2024-09-16T10:22:20.871990+0000","last_clean":"2024-09-16T10:22:20.871990+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:22:20.871990+0000","last_undegraded":"2024-09-16T10:22:20.871990+0000","last_fullsized":"2024-09-16T10:22:20.871990+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.803939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369199+0000","last_change":"2024-09-16T10:21:19.427397+0000","last_active":"2024-09-16T10:22:23.369199+0000","last_peered":"2024-09-16T10:22:23.369199+0000","last_clean":"2024-09-16T10:22:23.369199+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:22:23.369199+0000","last_undegraded":"2024-09-16T10:22:23.369199+0000","last_fullsized":"2024-09-16T10:22:23.369199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:19.427214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.427214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:35.414719+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":524,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872582+0000","last_change":"2024-09-16T10:21:58.648495+0000","last_active":"2024-09-16T10:22:20.872582+0000","last_peered":"2024-09-16T10:22:20.872582+0000","last_clean":"2024-09-16T10:22:20.872582+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:22:20.872582+0000","last_undegraded":"2024-09-16T10:22:20.872582+0000","last_fullsized":"2024-09-16T10:22:20.872582+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:21:58.648414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.648414+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.343610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368633+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:22:23.368633+0000","last_peered":"2024-09-16T10:22:23.368633+0000","last_clean":"2024-09-16T10:22:23.368633+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:22:23.368633+0000","last_undegraded":"2024-09-16T10:22:23.368633+0000","last_fullsized":"2024-09-16T10:22:23.368633+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.944396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":502,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369662+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:22:23.369662+0000","last_peered":"2024-09-16T10:22:23.369662+0000","last_clean":"2024-09-16T10:22:23.369662+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:22:23.369662+0000","last_undegraded":"2024-09-16T10:22:23.369662+0000","last_fullsized":"2024-09-16T10:22:23.369662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.729203+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368710+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:22:23.368710+0000","last_peered":"2024-09-16T10:22:23.368710+0000","last_clean":"2024-09-16T10:22:23.368710+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:22:23.368710+0000","last_undegraded":"2024-09-16T10:22:23.368710+0000","last_fullsized":"2024-09-16T10:22:23.368710+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:43.695976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":518,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.871870+0000","last_change":"2024-09-16T10:21:57.684221+0000","last_active":"2024-09-16T10:22:20.871870+0000","last_peered":"2024-09-16T10:22:20.871870+0000","last_clean":"2024-09-16T10:22:20.871870+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:22:20.871870+0000","last_undegraded":"2024-09-16T10:22:20.871870+0000","last_fullsized":"2024-09-16T10:22:20.871870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:21:57.684172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.684172+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.186220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":497,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:22.118256+0000","last_change":"2024-09-16T10:22:22.118256+0000","last_active":"2024-09-16T10:22:22.118256+0000","last_peered":"2024-09-16T10:22:22.118256+0000","last_clean":"2024-09-16T10:22:22.118256+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:22:22.118256+0000","last_undegraded":"2024-09-16T10:22:22.118256+0000","last_fullsized":"2024-09-16T10:22:22.118256+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.118194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.118194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:26.327315+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":510,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368424+0000","last_change":"2024-09-16T10:21:20.402146+0000","last_active":"2024-09-16T10:22:23.368424+0000","last_peered":"2024-09-16T10:22:23.368424+0000","last_clean":"2024-09-16T10:22:23.368424+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:22:23.368424+0000","last_undegraded":"2024-09-16T10:22:23.368424+0000","last_fullsized":"2024-09-16T10:22:23.368424+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.402080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.402080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:34.120569+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":493,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873512+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:22:20.873512+0000","last_peered":"2024-09-16T10:22:20.873512+0000","last_clean":"2024-09-16T10:22:20.873512+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:22:20.873512+0000","last_undegraded":"2024-09-16T10:22:20.873512+0000","last_fullsized":"2024-09-16T10:22:20.873512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:46.754500+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":498,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371695+0000","last_change":"2024-09-16T10:20:57.263993+0000","last_active":"2024-09-16T10:22:23.371695+0000","last_peered":"2024-09-16T10:22:23.371695+0000","last_clean":"2024-09-16T10:22:23.371695+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:22:23.371695+0000","last_undegraded":"2024-09-16T10:22:23.371695+0000","last_fullsized":"2024-09-16T10:22:23.371695+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.263916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.263916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:25.361547+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":50,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.373783+0000","last_change":"2024-09-16T10:22:02.948234+0000","last_active":"2024-09-16T10:22:23.373783+0000","last_peered":"2024-09-16T10:22:23.373783+0000","last_clean":"2024-09-16T10:22:23.373783+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:22:23.373783+0000","last_undegraded":"2024-09-16T10:22:23.373783+0000","last_fullsized":"2024-09-16T10:22:23.373783+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.703324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.365551+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:22:23.365551+0000","last_peered":"2024-09-16T10:22:23.365551+0000","last_clean":"2024-09-16T10:22:23.365551+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:22:23.365551+0000","last_undegraded":"2024-09-16T10:22:23.365551+0000","last_fullsized":"2024-09-16T10:22:23.365551+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.535156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":451,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368797+0000","last_change":"2024-09-16T10:21:57.358479+0000","last_active":"2024-09-16T10:22:23.368797+0000","last_peered":"2024-09-16T10:22:23.368797+0000","last_clean":"2024-09-16T10:22:23.368797+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:22:23.368797+0000","last_undegraded":"2024-09-16T10:22:23.368797+0000","last_fullsized":"2024-09-16T10:22:23.368797+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.358418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.358418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.586679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":502,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872706+0000","last_change":"2024-09-16T10:22:02.899574+0000","last_active":"2024-09-16T10:22:20.872706+0000","last_peered":"2024-09-16T10:22:20.872706+0000","last_clean":"2024-09-16T10:22:20.872706+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:22:20.872706+0000","last_undegraded":"2024-09-16T10:22:20.872706+0000","last_fullsized":"2024-09-16T10:22:20.872706+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.764435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":552,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368568+0000","last_change":"2024-09-16T10:20:57.423064+0000","last_active":"2024-09-16T10:22:23.368568+0000","last_peered":"2024-09-16T10:22:23.368568+0000","last_clean":"2024-09-16T10:22:23.368568+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:22:23.368568+0000","last_undegraded":"2024-09-16T10:22:23.368568+0000","last_fullsized":"2024-09-16T10:22:23.368568+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:20:57.422968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.422968+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":518,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.874008+0000","last_change":"2024-09-16T10:22:05.779269+0000","last_active":"2024-09-16T10:22:20.874008+0000","last_peered":"2024-09-16T10:22:20.874008+0000","last_clean":"2024-09-16T10:22:20.874008+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:22:20.874008+0000","last_undegraded":"2024-09-16T10:22:20.874008+0000","last_fullsized":"2024-09-16T10:22:20.874008+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.779225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.779225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.069689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":500,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.884007+0000","last_change":"2024-09-16T10:22:03.031843+0000","last_active":"2024-09-16T10:22:20.884007+0000","last_peered":"2024-09-16T10:22:20.884007+0000","last_clean":"2024-09-16T10:22:20.884007+0000","last_became_active":"2024-09-16T10:22:02.905797+0000","last_became_peered":"2024-09-16T10:22:02.905797+0000","last_unstale":"2024-09-16T10:22:20.884007+0000","last_undegraded":"2024-09-16T10:22:20.884007+0000","last_fullsized":"2024-09-16T10:22:20.884007+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:06.022236+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":506,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.365402+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:22:23.365402+0000","last_peered":"2024-09-16T10:22:23.365402+0000","last_clean":"2024-09-16T10:22:23.365402+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:22:23.365402+0000","last_undegraded":"2024-09-16T10:22:23.365402+0000","last_fullsized":"2024-09-16T10:22:23.365402+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:41.635078+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":526,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370153+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:22:23.370153+0000","last_peered":"2024-09-16T10:22:23.370153+0000","last_clean":"2024-09-16T10:22:23.370153+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:22:23.370153+0000","last_undegraded":"2024-09-16T10:22:23.370153+0000","last_fullsized":"2024-09-16T10:22:23.370153+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.284945+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":614,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876800+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:22:20.876800+0000","last_peered":"2024-09-16T10:22:20.876800+0000","last_clean":"2024-09-16T10:22:20.876800+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:22:20.876800+0000","last_undegraded":"2024-09-16T10:22:20.876800+0000","last_fullsized":"2024-09-16T10:22:20.876800+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:43.822126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"293'2527","reported_seq":2793,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.624876+0000","last_change":"2024-09-16T10:22:05.902394+0000","last_active":"2024-09-16T10:22:23.624876+0000","last_peered":"2024-09-16T10:22:23.624876+0000","last_clean":"2024-09-16T10:22:23.624876+0000","last_became_active":"2024-09-16T10:22:05.900726+0000","last_became_peered":"2024-09-16T10:22:05.900726+0000","last_unstale":"2024-09-16T10:22:23.624876+0000","last_undegraded":"2024-09-16T10:22:23.624876+0000","last_fullsized":"2024-09-16T10:22:23.624876+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2527,"log_dups_size":0,"ondisk_log_size":2527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.887079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165609472,"num_objects":2527,"num_object_clones":0,"num_object_copies":7581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2527,"num_write_kb":161728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":348,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368715+0000","last_change":"2024-09-16T10:22:02.898249+0000","last_active":"2024-09-16T10:22:23.368715+0000","last_peered":"2024-09-16T10:22:23.368715+0000","last_clean":"2024-09-16T10:22:23.368715+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:22:23.368715+0000","last_undegraded":"2024-09-16T10:22:23.368715+0000","last_fullsized":"2024-09-16T10:22:23.368715+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.514870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.514870+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.680773+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":520,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.366414+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:22:23.366414+0000","last_peered":"2024-09-16T10:22:23.366414+0000","last_clean":"2024-09-16T10:22:23.366414+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:22:23.366414+0000","last_undegraded":"2024-09-16T10:22:23.366414+0000","last_fullsized":"2024-09-16T10:22:23.366414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:42.601565+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":507,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370561+0000","last_change":"2024-09-16T10:21:29.412290+0000","last_active":"2024-09-16T10:22:23.370561+0000","last_peered":"2024-09-16T10:22:23.370561+0000","last_clean":"2024-09-16T10:22:23.370561+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:22:23.370561+0000","last_undegraded":"2024-09-16T10:22:23.370561+0000","last_fullsized":"2024-09-16T10:22:23.370561+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:29.412158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:29.412158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:33.021405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":602,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.874271+0000","last_change":"2024-09-16T10:21:22.928090+0000","last_active":"2024-09-16T10:22:20.874271+0000","last_peered":"2024-09-16T10:22:20.874271+0000","last_clean":"2024-09-16T10:22:20.874271+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:22:20.874271+0000","last_undegraded":"2024-09-16T10:22:20.874271+0000","last_fullsized":"2024-09-16T10:22:20.874271+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:21:22.928016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:22.928016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:40.025296+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":487,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368802+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:22:23.368802+0000","last_peered":"2024-09-16T10:22:23.368802+0000","last_clean":"2024-09-16T10:22:23.368802+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:22:23.368802+0000","last_undegraded":"2024-09-16T10:22:23.368802+0000","last_fullsized":"2024-09-16T10:22:23.368802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:06.054264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"292'2526","reported_seq":2569,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.214916+0000","last_change":"2024-09-16T10:21:56.237699+0000","last_active":"2024-09-16T10:22:23.214916+0000","last_peered":"2024-09-16T10:22:23.214916+0000","last_clean":"2024-09-16T10:22:23.214916+0000","last_became_active":"2024-09-16T10:21:56.237309+0000","last_became_peered":"2024-09-16T10:21:56.237309+0000","last_unstale":"2024-09-16T10:22:23.214916+0000","last_undegraded":"2024-09-16T10:22:23.214916+0000","last_fullsized":"2024-09-16T10:22:23.214916+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2526,"log_dups_size":0,"ondisk_log_size":2526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:22.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165543936,"num_objects":2526,"num_object_clones":0,"num_object_copies":7578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":161664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":373,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.874102+0000","last_change":"2024-09-16T10:22:05.897992+0000","last_active":"2024-09-16T10:22:20.874102+0000","last_peered":"2024-09-16T10:22:20.874102+0000","last_clean":"2024-09-16T10:22:20.874102+0000","last_became_active":"2024-09-16T10:22:05.897648+0000","last_became_peered":"2024-09-16T10:22:05.897648+0000","last_unstale":"2024-09-16T10:22:20.874102+0000","last_undegraded":"2024-09-16T10:22:20.874102+0000","last_fullsized":"2024-09-16T10:22:20.874102+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.845331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.845331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":527,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370237+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:22:23.370237+0000","last_peered":"2024-09-16T10:22:23.370237+0000","last_clean":"2024-09-16T10:22:23.370237+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:22:23.370237+0000","last_undegraded":"2024-09-16T10:22:23.370237+0000","last_fullsized":"2024-09-16T10:22:23.370237+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:08.375110+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":471,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369339+0000","last_change":"2024-09-16T10:21:59.329899+0000","last_active":"2024-09-16T10:22:23.369339+0000","last_peered":"2024-09-16T10:22:23.369339+0000","last_clean":"2024-09-16T10:22:23.369339+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:22:23.369339+0000","last_undegraded":"2024-09-16T10:22:23.369339+0000","last_fullsized":"2024-09-16T10:22:23.369339+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.329856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.329856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:59.820276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":243,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876341+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:22:20.876341+0000","last_peered":"2024-09-16T10:22:20.876341+0000","last_clean":"2024-09-16T10:22:20.876341+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:22:20.876341+0000","last_undegraded":"2024-09-16T10:22:20.876341+0000","last_fullsized":"2024-09-16T10:22:20.876341+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:37.717939+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":515,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872505+0000","last_change":"2024-09-16T10:22:02.899911+0000","last_active":"2024-09-16T10:22:20.872505+0000","last_peered":"2024-09-16T10:22:20.872505+0000","last_clean":"2024-09-16T10:22:20.872505+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:22:20.872505+0000","last_undegraded":"2024-09-16T10:22:20.872505+0000","last_fullsized":"2024-09-16T10:22:20.872505+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.772413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"293'2529","reported_seq":2574,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.649147+0000","last_change":"2024-09-16T10:21:56.237148+0000","last_active":"2024-09-16T10:22:23.649147+0000","last_peered":"2024-09-16T10:22:23.649147+0000","last_clean":"2024-09-16T10:22:23.649147+0000","last_became_active":"2024-09-16T10:21:56.236822+0000","last_became_peered":"2024-09-16T10:21:56.236822+0000","last_unstale":"2024-09-16T10:22:23.649147+0000","last_undegraded":"2024-09-16T10:22:23.649147+0000","last_fullsized":"2024-09-16T10:22:23.649147+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2529,"log_dups_size":0,"ondisk_log_size":2529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.079994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":7587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":161856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":375,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.372196+0000","last_change":"2024-09-16T10:21:50.864307+0000","last_active":"2024-09-16T10:22:23.372196+0000","last_peered":"2024-09-16T10:22:23.372196+0000","last_clean":"2024-09-16T10:22:23.372196+0000","last_became_active":"2024-09-16T10:21:50.864098+0000","last_became_peered":"2024-09-16T10:21:50.864098+0000","last_unstale":"2024-09-16T10:22:23.372196+0000","last_undegraded":"2024-09-16T10:22:23.372196+0000","last_fullsized":"2024-09-16T10:22:23.372196+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:44.027332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":530,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872916+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:22:20.872916+0000","last_peered":"2024-09-16T10:22:20.872916+0000","last_clean":"2024-09-16T10:22:20.872916+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:22:20.872916+0000","last_undegraded":"2024-09-16T10:22:20.872916+0000","last_fullsized":"2024-09-16T10:22:20.872916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:49.638461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371220+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:22:23.371220+0000","last_peered":"2024-09-16T10:22:23.371220+0000","last_clean":"2024-09-16T10:22:23.371220+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:22:23.371220+0000","last_undegraded":"2024-09-16T10:22:23.371220+0000","last_fullsized":"2024-09-16T10:22:23.371220+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:42.132239+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":616,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370073+0000","last_change":"2024-09-16T10:22:02.357682+0000","last_active":"2024-09-16T10:22:23.370073+0000","last_peered":"2024-09-16T10:22:23.370073+0000","last_clean":"2024-09-16T10:22:23.370073+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:22:23.370073+0000","last_undegraded":"2024-09-16T10:22:23.370073+0000","last_fullsized":"2024-09-16T10:22:23.370073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:22:02.357616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.357616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.613433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":500,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.882945+0000","last_change":"2024-09-16T10:22:05.051429+0000","last_active":"2024-09-16T10:22:20.882945+0000","last_peered":"2024-09-16T10:22:20.882945+0000","last_clean":"2024-09-16T10:22:20.882945+0000","last_became_active":"2024-09-16T10:22:02.890865+0000","last_became_peered":"2024-09-16T10:22:02.890865+0000","last_unstale":"2024-09-16T10:22:20.882945+0000","last_undegraded":"2024-09-16T10:22:20.882945+0000","last_fullsized":"2024-09-16T10:22:20.882945+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:29.018103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"293'2513","reported_seq":2558,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.390481+0000","last_change":"2024-09-16T10:21:56.239042+0000","last_active":"2024-09-16T10:22:23.390481+0000","last_peered":"2024-09-16T10:22:23.390481+0000","last_clean":"2024-09-16T10:22:23.390481+0000","last_became_active":"2024-09-16T10:21:56.238825+0000","last_became_peered":"2024-09-16T10:21:56.238825+0000","last_unstale":"2024-09-16T10:22:23.390481+0000","last_undegraded":"2024-09-16T10:22:23.390481+0000","last_fullsized":"2024-09-16T10:22:23.390481+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2513,"log_dups_size":0,"ondisk_log_size":2513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":7539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2513,"num_write_kb":160832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":424,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876569+0000","last_change":"2024-09-16T10:21:39.142049+0000","last_active":"2024-09-16T10:22:20.876569+0000","last_peered":"2024-09-16T10:22:20.876569+0000","last_clean":"2024-09-16T10:22:20.876569+0000","last_became_active":"2024-09-16T10:21:39.141585+0000","last_became_peered":"2024-09-16T10:21:39.141585+0000","last_unstale":"2024-09-16T10:22:20.876569+0000","last_undegraded":"2024-09-16T10:22:20.876569+0000","last_fullsized":"2024-09-16T10:22:20.876569+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.164536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":535,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.366550+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:22:23.366550+0000","last_peered":"2024-09-16T10:22:23.366550+0000","last_clean":"2024-09-16T10:22:23.366550+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:22:23.366550+0000","last_undegraded":"2024-09-16T10:22:23.366550+0000","last_fullsized":"2024-09-16T10:22:23.366550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.712019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":522,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369286+0000","last_change":"2024-09-16T10:22:02.898141+0000","last_active":"2024-09-16T10:22:23.369286+0000","last_peered":"2024-09-16T10:22:23.369286+0000","last_clean":"2024-09-16T10:22:23.369286+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:22:23.369286+0000","last_undegraded":"2024-09-16T10:22:23.369286+0000","last_fullsized":"2024-09-16T10:22:23.369286+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.918613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.918613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.461798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":605,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371004+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:22:23.371004+0000","last_peered":"2024-09-16T10:22:23.371004+0000","last_clean":"2024-09-16T10:22:23.371004+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:22:23.371004+0000","last_undegraded":"2024-09-16T10:22:23.371004+0000","last_fullsized":"2024-09-16T10:22:23.371004+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.301996+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":517,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369904+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:22:23.369904+0000","last_peered":"2024-09-16T10:22:23.369904+0000","last_clean":"2024-09-16T10:22:23.369904+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:22:23.369904+0000","last_undegraded":"2024-09-16T10:22:23.369904+0000","last_fullsized":"2024-09-16T10:22:23.369904+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.592047+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"292'2628","reported_seq":2671,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.207174+0000","last_change":"2024-09-16T10:21:56.236599+0000","last_active":"2024-09-16T10:22:23.207174+0000","last_peered":"2024-09-16T10:22:23.207174+0000","last_clean":"2024-09-16T10:22:23.207174+0000","last_became_active":"2024-09-16T10:21:56.236217+0000","last_became_peered":"2024-09-16T10:21:56.236217+0000","last_unstale":"2024-09-16T10:22:23.207174+0000","last_undegraded":"2024-09-16T10:22:23.207174+0000","last_fullsized":"2024-09-16T10:22:23.207174+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2628,"log_dups_size":0,"ondisk_log_size":2628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:08.065697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172228608,"num_objects":2628,"num_object_clones":0,"num_object_copies":7884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2628,"num_write_kb":168192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":426,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876324+0000","last_change":"2024-09-16T10:21:42.939180+0000","last_active":"2024-09-16T10:22:20.876324+0000","last_peered":"2024-09-16T10:22:20.876324+0000","last_clean":"2024-09-16T10:22:20.876324+0000","last_became_active":"2024-09-16T10:21:42.938973+0000","last_became_peered":"2024-09-16T10:21:42.938973+0000","last_unstale":"2024-09-16T10:22:20.876324+0000","last_undegraded":"2024-09-16T10:22:20.876324+0000","last_fullsized":"2024-09-16T10:22:20.876324+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.732986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.732986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:38.821208+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":547,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371266+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:22:23.371266+0000","last_peered":"2024-09-16T10:22:23.371266+0000","last_clean":"2024-09-16T10:22:23.371266+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:22:23.371266+0000","last_undegraded":"2024-09-16T10:22:23.371266+0000","last_fullsized":"2024-09-16T10:22:23.371266+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:38.989185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":530,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.372344+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:22:23.372344+0000","last_peered":"2024-09-16T10:22:23.372344+0000","last_clean":"2024-09-16T10:22:23.372344+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:22:23.372344+0000","last_undegraded":"2024-09-16T10:22:23.372344+0000","last_fullsized":"2024-09-16T10:22:23.372344+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:27.439600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.875843+0000","last_change":"2024-09-16T10:20:58.707720+0000","last_active":"2024-09-16T10:22:20.875843+0000","last_peered":"2024-09-16T10:22:20.875843+0000","last_clean":"2024-09-16T10:22:20.875843+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:22:20.875843+0000","last_undegraded":"2024-09-16T10:22:20.875843+0000","last_fullsized":"2024-09-16T10:22:20.875843+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.707654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.707654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":510,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.365207+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:22:23.365207+0000","last_peered":"2024-09-16T10:22:23.365207+0000","last_clean":"2024-09-16T10:22:23.365207+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:22:23.365207+0000","last_undegraded":"2024-09-16T10:22:23.365207+0000","last_fullsized":"2024-09-16T10:22:23.365207+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:39.090159+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"293'2626","reported_seq":2671,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.666392+0000","last_change":"2024-09-16T10:21:56.233704+0000","last_active":"2024-09-16T10:22:23.666392+0000","last_peered":"2024-09-16T10:22:23.666392+0000","last_clean":"2024-09-16T10:22:23.666392+0000","last_became_active":"2024-09-16T10:21:56.233167+0000","last_became_peered":"2024-09-16T10:21:56.233167+0000","last_unstale":"2024-09-16T10:22:23.666392+0000","last_undegraded":"2024-09-16T10:22:23.666392+0000","last_fullsized":"2024-09-16T10:22:23.666392+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.678798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172097536,"num_objects":2626,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2626,"num_write_kb":168064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":375,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368334+0000","last_change":"2024-09-16T10:22:05.898575+0000","last_active":"2024-09-16T10:22:23.368334+0000","last_peered":"2024-09-16T10:22:23.368334+0000","last_clean":"2024-09-16T10:22:23.368334+0000","last_became_active":"2024-09-16T10:22:05.898258+0000","last_became_peered":"2024-09-16T10:22:05.898258+0000","last_unstale":"2024-09-16T10:22:23.368334+0000","last_undegraded":"2024-09-16T10:22:23.368334+0000","last_fullsized":"2024-09-16T10:22:23.368334+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.941141+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"282'30","reported_seq":732,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.372211+0000","last_change":"2024-09-16T10:22:04.554377+0000","last_active":"2024-09-16T10:22:23.372211+0000","last_peered":"2024-09-16T10:22:23.372211+0000","last_clean":"2024-09-16T10:22:23.372211+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:22:23.372211+0000","last_undegraded":"2024-09-16T10:22:23.372211+0000","last_fullsized":"2024-09-16T10:22:23.372211+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'30","last_scrub_stamp":"2024-09-16T10:22:04.554260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.554260+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.708122+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872909+0000","last_change":"2024-09-16T10:20:56.965996+0000","last_active":"2024-09-16T10:22:20.872909+0000","last_peered":"2024-09-16T10:22:20.872909+0000","last_clean":"2024-09-16T10:22:20.872909+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:22:20.872909+0000","last_undegraded":"2024-09-16T10:22:20.872909+0000","last_fullsized":"2024-09-16T10:22:20.872909+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:56.965933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:56.965933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":510,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369225+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:22:23.369225+0000","last_peered":"2024-09-16T10:22:23.369225+0000","last_clean":"2024-09-16T10:22:23.369225+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:22:23.369225+0000","last_undegraded":"2024-09-16T10:22:23.369225+0000","last_fullsized":"2024-09-16T10:22:23.369225+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:50.811766+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":491,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370908+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:22:23.370908+0000","last_peered":"2024-09-16T10:22:23.370908+0000","last_clean":"2024-09-16T10:22:23.370908+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:22:23.370908+0000","last_undegraded":"2024-09-16T10:22:23.370908+0000","last_fullsized":"2024-09-16T10:22:23.370908+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.265494+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":511,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369003+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:22:23.369003+0000","last_peered":"2024-09-16T10:22:23.369003+0000","last_clean":"2024-09-16T10:22:23.369003+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:22:23.369003+0000","last_undegraded":"2024-09-16T10:22:23.369003+0000","last_fullsized":"2024-09-16T10:22:23.369003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.835165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"293'2547","reported_seq":2592,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.405934+0000","last_change":"2024-09-16T10:21:56.236854+0000","last_active":"2024-09-16T10:22:23.405934+0000","last_peered":"2024-09-16T10:22:23.405934+0000","last_clean":"2024-09-16T10:22:23.405934+0000","last_became_active":"2024-09-16T10:21:56.236183+0000","last_became_peered":"2024-09-16T10:21:56.236183+0000","last_unstale":"2024-09-16T10:22:23.405934+0000","last_undegraded":"2024-09-16T10:22:23.405934+0000","last_fullsized":"2024-09-16T10:22:23.405934+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2547,"log_dups_size":0,"ondisk_log_size":2547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.259343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":7641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":163008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":624,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883005+0000","last_change":"2024-09-16T10:21:27.006377+0000","last_active":"2024-09-16T10:22:20.883005+0000","last_peered":"2024-09-16T10:22:20.883005+0000","last_clean":"2024-09-16T10:22:20.883005+0000","last_became_active":"2024-09-16T10:21:27.006016+0000","last_became_peered":"2024-09-16T10:21:27.006016+0000","last_unstale":"2024-09-16T10:22:20.883005+0000","last_undegraded":"2024-09-16T10:22:20.883005+0000","last_fullsized":"2024-09-16T10:22:20.883005+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.064404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.064404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.366544+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:22:23.366544+0000","last_peered":"2024-09-16T10:22:23.366544+0000","last_clean":"2024-09-16T10:22:23.366544+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:22:23.366544+0000","last_undegraded":"2024-09-16T10:22:23.366544+0000","last_fullsized":"2024-09-16T10:22:23.366544+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:44.556505+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":514,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.366153+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:22:23.366153+0000","last_peered":"2024-09-16T10:22:23.366153+0000","last_clean":"2024-09-16T10:22:23.366153+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:22:23.366153+0000","last_undegraded":"2024-09-16T10:22:23.366153+0000","last_fullsized":"2024-09-16T10:22:23.366153+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.551901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":38,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.374319+0000","last_change":"2024-09-16T10:22:02.917475+0000","last_active":"2024-09-16T10:22:23.374319+0000","last_peered":"2024-09-16T10:22:23.374319+0000","last_clean":"2024-09-16T10:22:23.374319+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:22:23.374319+0000","last_undegraded":"2024-09-16T10:22:23.374319+0000","last_fullsized":"2024-09-16T10:22:23.374319+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.736419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":34,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.373721+0000","last_change":"2024-09-16T10:22:02.901488+0000","last_active":"2024-09-16T10:22:23.373721+0000","last_peered":"2024-09-16T10:22:23.373721+0000","last_clean":"2024-09-16T10:22:23.373721+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:22:23.373721+0000","last_undegraded":"2024-09-16T10:22:23.373721+0000","last_fullsized":"2024-09-16T10:22:23.373721+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.393147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.393147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:26.749600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":502,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883046+0000","last_change":"2024-09-16T10:20:52.145844+0000","last_active":"2024-09-16T10:22:20.883046+0000","last_peered":"2024-09-16T10:22:20.883046+0000","last_clean":"2024-09-16T10:22:20.883046+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:22:20.883046+0000","last_undegraded":"2024-09-16T10:22:20.883046+0000","last_fullsized":"2024-09-16T10:22:20.883046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:52.145765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.145765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"293'2593","reported_seq":2637,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.677725+0000","last_change":"2024-09-16T10:21:56.914244+0000","last_active":"2024-09-16T10:22:23.677725+0000","last_peered":"2024-09-16T10:22:23.677725+0000","last_clean":"2024-09-16T10:22:23.677725+0000","last_became_active":"2024-09-16T10:21:56.913269+0000","last_became_peered":"2024-09-16T10:21:56.913269+0000","last_unstale":"2024-09-16T10:22:23.677725+0000","last_undegraded":"2024-09-16T10:22:23.677725+0000","last_fullsized":"2024-09-16T10:22:23.677725+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.810191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169934848,"num_objects":2593,"num_object_clones":0,"num_object_copies":7779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":165952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":363,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.365662+0000","last_change":"2024-09-16T10:21:55.226523+0000","last_active":"2024-09-16T10:22:23.365662+0000","last_peered":"2024-09-16T10:22:23.365662+0000","last_clean":"2024-09-16T10:22:23.365662+0000","last_became_active":"2024-09-16T10:21:55.226200+0000","last_became_peered":"2024-09-16T10:21:55.226200+0000","last_unstale":"2024-09-16T10:22:23.365662+0000","last_undegraded":"2024-09-16T10:22:23.365662+0000","last_fullsized":"2024-09-16T10:22:23.365662+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:09.610254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":579,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370240+0000","last_change":"2024-09-16T10:22:02.930302+0000","last_active":"2024-09-16T10:22:23.370240+0000","last_peered":"2024-09-16T10:22:23.370240+0000","last_clean":"2024-09-16T10:22:23.370240+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:22:23.370240+0000","last_undegraded":"2024-09-16T10:22:23.370240+0000","last_fullsized":"2024-09-16T10:22:23.370240+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:20:55.304837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.304837+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":523,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370019+0000","last_change":"2024-09-16T10:20:52.288156+0000","last_active":"2024-09-16T10:22:23.370019+0000","last_peered":"2024-09-16T10:22:23.370019+0000","last_clean":"2024-09-16T10:22:23.370019+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:22:23.370019+0000","last_undegraded":"2024-09-16T10:22:23.370019+0000","last_fullsized":"2024-09-16T10:22:23.370019+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:52.288090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:52.288090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1045,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:21.797254+0000","last_change":"2024-09-16T10:21:58.009040+0000","last_active":"2024-09-16T10:22:21.797254+0000","last_peered":"2024-09-16T10:22:21.797254+0000","last_clean":"2024-09-16T10:22:21.797254+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:22:21.797254+0000","last_undegraded":"2024-09-16T10:22:21.797254+0000","last_fullsized":"2024-09-16T10:22:21.797254+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:21:58.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.008984+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.997371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":520,"num_read_kb":515,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369187+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:22:23.369187+0000","last_peered":"2024-09-16T10:22:23.369187+0000","last_clean":"2024-09-16T10:22:23.369187+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:22:23.369187+0000","last_undegraded":"2024-09-16T10:22:23.369187+0000","last_fullsized":"2024-09-16T10:22:23.369187+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.025944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":512,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371800+0000","last_change":"2024-09-16T10:21:58.547682+0000","last_active":"2024-09-16T10:22:23.371800+0000","last_peered":"2024-09-16T10:22:23.371800+0000","last_clean":"2024-09-16T10:22:23.371800+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:22:23.371800+0000","last_undegraded":"2024-09-16T10:22:23.371800+0000","last_fullsized":"2024-09-16T10:22:23.371800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.547634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.547634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.050150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"293'2592","reported_seq":2637,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.649384+0000","last_change":"2024-09-16T10:21:56.236351+0000","last_active":"2024-09-16T10:22:23.649384+0000","last_peered":"2024-09-16T10:22:23.649384+0000","last_clean":"2024-09-16T10:22:23.649384+0000","last_became_active":"2024-09-16T10:21:56.236070+0000","last_became_peered":"2024-09-16T10:21:56.236070+0000","last_unstale":"2024-09-16T10:22:23.649384+0000","last_undegraded":"2024-09-16T10:22:23.649384+0000","last_fullsized":"2024-09-16T10:22:23.649384+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.402455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169869312,"num_objects":2592,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":621,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876123+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:22:20.876123+0000","last_peered":"2024-09-16T10:22:20.876123+0000","last_clean":"2024-09-16T10:22:20.876123+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:22:20.876123+0000","last_undegraded":"2024-09-16T10:22:20.876123+0000","last_fullsized":"2024-09-16T10:22:20.876123+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:45.859035+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368635+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:22:23.368635+0000","last_peered":"2024-09-16T10:22:23.368635+0000","last_clean":"2024-09-16T10:22:23.368635+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:22:23.368635+0000","last_undegraded":"2024-09-16T10:22:23.368635+0000","last_fullsized":"2024-09-16T10:22:23.368635+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:57.819113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":612,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.882600+0000","last_change":"2024-09-16T10:21:57.046814+0000","last_active":"2024-09-16T10:22:20.882600+0000","last_peered":"2024-09-16T10:22:20.882600+0000","last_clean":"2024-09-16T10:22:20.882600+0000","last_became_active":"2024-09-16T10:21:35.111405+0000","last_became_peered":"2024-09-16T10:21:35.111405+0000","last_unstale":"2024-09-16T10:22:20.882600+0000","last_undegraded":"2024-09-16T10:22:20.882600+0000","last_fullsized":"2024-09-16T10:22:20.882600+0000","mapping_epoch":55,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:57.046749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.046749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:03.768702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":504,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.884168+0000","last_change":"2024-09-16T10:22:02.906870+0000","last_active":"2024-09-16T10:22:20.884168+0000","last_peered":"2024-09-16T10:22:20.884168+0000","last_clean":"2024-09-16T10:22:20.884168+0000","last_became_active":"2024-09-16T10:22:02.906315+0000","last_became_peered":"2024-09-16T10:22:02.906315+0000","last_unstale":"2024-09-16T10:22:20.884168+0000","last_undegraded":"2024-09-16T10:22:20.884168+0000","last_fullsized":"2024-09-16T10:22:20.884168+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.151113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":496,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369145+0000","last_change":"2024-09-16T10:21:56.923534+0000","last_active":"2024-09-16T10:22:23.369145+0000","last_peered":"2024-09-16T10:22:23.369145+0000","last_clean":"2024-09-16T10:22:23.369145+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:22:23.369145+0000","last_undegraded":"2024-09-16T10:22:23.369145+0000","last_fullsized":"2024-09-16T10:22:23.369145+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:56.923485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.923485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:59.840767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"292'2561","reported_seq":2603,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.208061+0000","last_change":"2024-09-16T10:21:56.234554+0000","last_active":"2024-09-16T10:22:23.208061+0000","last_peered":"2024-09-16T10:22:23.208061+0000","last_clean":"2024-09-16T10:22:23.208061+0000","last_became_active":"2024-09-16T10:21:56.234289+0000","last_became_peered":"2024-09-16T10:21:56.234289+0000","last_unstale":"2024-09-16T10:22:23.208061+0000","last_undegraded":"2024-09-16T10:22:23.208061+0000","last_fullsized":"2024-09-16T10:22:23.208061+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":0,"ondisk_log_size":2561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.348123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167837696,"num_objects":2561,"num_object_clones":0,"num_object_copies":7683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":163904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":538,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873908+0000","last_change":"2024-09-16T10:20:31.013510+0000","last_active":"2024-09-16T10:22:20.873908+0000","last_peered":"2024-09-16T10:22:20.873908+0000","last_clean":"2024-09-16T10:22:20.873908+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:22:20.873908+0000","last_undegraded":"2024-09-16T10:22:20.873908+0000","last_fullsized":"2024-09-16T10:22:20.873908+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:20:31.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.013316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.790395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":519,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.877544+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:22:20.877544+0000","last_peered":"2024-09-16T10:22:20.877544+0000","last_clean":"2024-09-16T10:22:20.877544+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:22:20.877544+0000","last_undegraded":"2024-09-16T10:22:20.877544+0000","last_fullsized":"2024-09-16T10:22:20.877544+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:31.365218+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":507,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368862+0000","last_change":"2024-09-16T10:22:22.987395+0000","last_active":"2024-09-16T10:22:23.368862+0000","last_peered":"2024-09-16T10:22:23.368862+0000","last_clean":"2024-09-16T10:22:23.368862+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:22:23.368862+0000","last_undegraded":"2024-09-16T10:22:23.368862+0000","last_fullsized":"2024-09-16T10:22:23.368862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.987351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.987351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.275819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":451,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.372105+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:22:23.372105+0000","last_peered":"2024-09-16T10:22:23.372105+0000","last_clean":"2024-09-16T10:22:23.372105+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:22:23.372105+0000","last_undegraded":"2024-09-16T10:22:23.372105+0000","last_fullsized":"2024-09-16T10:22:23.372105+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:37.449108+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"293'2599","reported_seq":2842,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.542858+0000","last_change":"2024-09-16T10:22:03.098658+0000","last_active":"2024-09-16T10:22:23.542858+0000","last_peered":"2024-09-16T10:22:23.542858+0000","last_clean":"2024-09-16T10:22:23.542858+0000","last_became_active":"2024-09-16T10:22:02.896249+0000","last_became_peered":"2024-09-16T10:22:02.896249+0000","last_unstale":"2024-09-16T10:22:23.542858+0000","last_undegraded":"2024-09-16T10:22:23.542858+0000","last_fullsized":"2024-09-16T10:22:23.542858+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2599,"log_dups_size":0,"ondisk_log_size":2599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.823024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170328064,"num_objects":2599,"num_object_clones":0,"num_object_copies":7797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":166336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":355,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368246+0000","last_change":"2024-09-16T10:22:03.029868+0000","last_active":"2024-09-16T10:22:23.368246+0000","last_peered":"2024-09-16T10:22:23.368246+0000","last_clean":"2024-09-16T10:22:23.368246+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:22:23.368246+0000","last_undegraded":"2024-09-16T10:22:23.368246+0000","last_fullsized":"2024-09-16T10:22:23.368246+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:27.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:27.405599+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:08.103053+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":578,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873965+0000","last_change":"2024-09-16T10:22:03.026909+0000","last_active":"2024-09-16T10:22:20.873965+0000","last_peered":"2024-09-16T10:22:20.873965+0000","last_clean":"2024-09-16T10:22:20.873965+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:22:20.873965+0000","last_undegraded":"2024-09-16T10:22:20.873965+0000","last_fullsized":"2024-09-16T10:22:20.873965+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:21:53.705644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.705644+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.296962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876211+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:22:20.876211+0000","last_peered":"2024-09-16T10:22:20.876211+0000","last_clean":"2024-09-16T10:22:20.876211+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:22:20.876211+0000","last_undegraded":"2024-09-16T10:22:20.876211+0000","last_fullsized":"2024-09-16T10:22:20.876211+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:00.342008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371914+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:22:23.371914+0000","last_peered":"2024-09-16T10:22:23.371914+0000","last_clean":"2024-09-16T10:22:23.371914+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:22:23.371914+0000","last_undegraded":"2024-09-16T10:22:23.371914+0000","last_fullsized":"2024-09-16T10:22:23.371914+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:01.268083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.373336+0000","last_change":"2024-09-16T10:22:05.694026+0000","last_active":"2024-09-16T10:22:23.373336+0000","last_peered":"2024-09-16T10:22:23.373336+0000","last_clean":"2024-09-16T10:22:23.373336+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:22:23.373336+0000","last_undegraded":"2024-09-16T10:22:23.373336+0000","last_fullsized":"2024-09-16T10:22:23.373336+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.693972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.693972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.767050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"293'2545","reported_seq":2590,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.632936+0000","last_change":"2024-09-16T10:21:56.913566+0000","last_active":"2024-09-16T10:22:23.632936+0000","last_peered":"2024-09-16T10:22:23.632936+0000","last_clean":"2024-09-16T10:22:23.632936+0000","last_became_active":"2024-09-16T10:21:56.913186+0000","last_became_peered":"2024-09-16T10:21:56.913186+0000","last_unstale":"2024-09-16T10:22:23.632936+0000","last_undegraded":"2024-09-16T10:22:23.632936+0000","last_fullsized":"2024-09-16T10:22:23.632936+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2545,"log_dups_size":0,"ondisk_log_size":2545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:58.492599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166789120,"num_objects":2545,"num_object_clones":0,"num_object_copies":7635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2545,"num_write_kb":162880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":613,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.882934+0000","last_change":"2024-09-16T10:22:02.905855+0000","last_active":"2024-09-16T10:22:20.882934+0000","last_peered":"2024-09-16T10:22:20.882934+0000","last_clean":"2024-09-16T10:22:20.882934+0000","last_became_active":"2024-09-16T10:22:02.905393+0000","last_became_peered":"2024-09-16T10:22:02.905393+0000","last_unstale":"2024-09-16T10:22:20.882934+0000","last_undegraded":"2024-09-16T10:22:20.882934+0000","last_fullsized":"2024-09-16T10:22:20.882934+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:21.240387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:21.240387+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:39.612488+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":507,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873402+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:22:20.873402+0000","last_peered":"2024-09-16T10:22:20.873402+0000","last_clean":"2024-09-16T10:22:20.873402+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:22:20.873402+0000","last_undegraded":"2024-09-16T10:22:20.873402+0000","last_fullsized":"2024-09-16T10:22:20.873402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:39.233985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":490,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369855+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:22:23.369855+0000","last_peered":"2024-09-16T10:22:23.369855+0000","last_clean":"2024-09-16T10:22:23.369855+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:22:23.369855+0000","last_undegraded":"2024-09-16T10:22:23.369855+0000","last_fullsized":"2024-09-16T10:22:23.369855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.375612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":43,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.374115+0000","last_change":"2024-09-16T10:22:02.914063+0000","last_active":"2024-09-16T10:22:23.374115+0000","last_peered":"2024-09-16T10:22:23.374115+0000","last_clean":"2024-09-16T10:22:23.374115+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:22:23.374115+0000","last_undegraded":"2024-09-16T10:22:23.374115+0000","last_fullsized":"2024-09-16T10:22:23.374115+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:20:55.177089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.177089+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":506,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873352+0000","last_change":"2024-09-16T10:22:02.899417+0000","last_active":"2024-09-16T10:22:20.873352+0000","last_peered":"2024-09-16T10:22:20.873352+0000","last_clean":"2024-09-16T10:22:20.873352+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:22:20.873352+0000","last_undegraded":"2024-09-16T10:22:20.873352+0000","last_fullsized":"2024-09-16T10:22:20.873352+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.873833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.873833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"293'2513","reported_seq":2746,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.409860+0000","last_change":"2024-09-16T10:22:03.102043+0000","last_active":"2024-09-16T10:22:23.409860+0000","last_peered":"2024-09-16T10:22:23.409860+0000","last_clean":"2024-09-16T10:22:23.409860+0000","last_became_active":"2024-09-16T10:22:02.898446+0000","last_became_peered":"2024-09-16T10:22:02.898446+0000","last_unstale":"2024-09-16T10:22:23.409860+0000","last_undegraded":"2024-09-16T10:22:23.409860+0000","last_fullsized":"2024-09-16T10:22:23.409860+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2513,"log_dups_size":0,"ondisk_log_size":2513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.816221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":7539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2513,"num_write_kb":160832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":510,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371164+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:22:23.371164+0000","last_peered":"2024-09-16T10:22:23.371164+0000","last_clean":"2024-09-16T10:22:23.371164+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:22:23.371164+0000","last_undegraded":"2024-09-16T10:22:23.371164+0000","last_fullsized":"2024-09-16T10:22:23.371164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:36.121507+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368804+0000","last_change":"2024-09-16T10:21:57.915828+0000","last_active":"2024-09-16T10:22:23.368804+0000","last_peered":"2024-09-16T10:22:23.368804+0000","last_clean":"2024-09-16T10:22:23.368804+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:22:23.368804+0000","last_undegraded":"2024-09-16T10:22:23.368804+0000","last_fullsized":"2024-09-16T10:22:23.368804+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.915773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.915773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.518442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.373593+0000","last_change":"2024-09-16T10:22:02.896967+0000","last_active":"2024-09-16T10:22:23.373593+0000","last_peered":"2024-09-16T10:22:23.373593+0000","last_clean":"2024-09-16T10:22:23.373593+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:22:23.373593+0000","last_undegraded":"2024-09-16T10:22:23.373593+0000","last_fullsized":"2024-09-16T10:22:23.373593+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.735258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":531,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369889+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:22:23.369889+0000","last_peered":"2024-09-16T10:22:23.369889+0000","last_clean":"2024-09-16T10:22:23.369889+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:22:23.369889+0000","last_undegraded":"2024-09-16T10:22:23.369889+0000","last_fullsized":"2024-09-16T10:22:23.369889+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.144396+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"293'2573","reported_seq":2618,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.404083+0000","last_change":"2024-09-16T10:21:56.234684+0000","last_active":"2024-09-16T10:22:23.404083+0000","last_peered":"2024-09-16T10:22:23.404083+0000","last_clean":"2024-09-16T10:22:23.404083+0000","last_became_active":"2024-09-16T10:21:56.234401+0000","last_became_peered":"2024-09-16T10:21:56.234401+0000","last_unstale":"2024-09-16T10:22:23.404083+0000","last_undegraded":"2024-09-16T10:22:23.404083+0000","last_fullsized":"2024-09-16T10:22:23.404083+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2573,"log_dups_size":0,"ondisk_log_size":2573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.262243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168624128,"num_objects":2573,"num_object_clones":0,"num_object_copies":7719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":164672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":540,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.875973+0000","last_change":"2024-09-16T10:20:51.673782+0000","last_active":"2024-09-16T10:22:20.875973+0000","last_peered":"2024-09-16T10:22:20.875973+0000","last_clean":"2024-09-16T10:22:20.875973+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:22:20.875973+0000","last_undegraded":"2024-09-16T10:22:20.875973+0000","last_fullsized":"2024-09-16T10:22:20.875973+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:20:51.673686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.673686+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883632+0000","last_change":"2024-09-16T10:21:59.998410+0000","last_active":"2024-09-16T10:22:20.883632+0000","last_peered":"2024-09-16T10:22:20.883632+0000","last_clean":"2024-09-16T10:22:20.883632+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:22:20.883632+0000","last_undegraded":"2024-09-16T10:22:20.883632+0000","last_fullsized":"2024-09-16T10:22:20.883632+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.998357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.998357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.630364+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.373321+0000","last_change":"2024-09-16T10:22:02.898200+0000","last_active":"2024-09-16T10:22:23.373321+0000","last_peered":"2024-09-16T10:22:23.373321+0000","last_clean":"2024-09-16T10:22:23.373321+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:22:23.373321+0000","last_undegraded":"2024-09-16T10:22:23.373321+0000","last_fullsized":"2024-09-16T10:22:23.373321+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.330503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.330503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:23.925809+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":508,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.882537+0000","last_change":"2024-09-16T10:22:02.906954+0000","last_active":"2024-09-16T10:22:20.882537+0000","last_peered":"2024-09-16T10:22:20.882537+0000","last_clean":"2024-09-16T10:22:20.882537+0000","last_became_active":"2024-09-16T10:22:02.906524+0000","last_became_peered":"2024-09-16T10:22:02.906524+0000","last_unstale":"2024-09-16T10:22:20.882537+0000","last_undegraded":"2024-09-16T10:22:20.882537+0000","last_fullsized":"2024-09-16T10:22:20.882537+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.105240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.105240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"293'2615","reported_seq":2660,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.532528+0000","last_change":"2024-09-16T10:21:56.235273+0000","last_active":"2024-09-16T10:22:23.532528+0000","last_peered":"2024-09-16T10:22:23.532528+0000","last_clean":"2024-09-16T10:22:23.532528+0000","last_became_active":"2024-09-16T10:21:56.234987+0000","last_became_peered":"2024-09-16T10:21:56.234987+0000","last_unstale":"2024-09-16T10:22:23.532528+0000","last_undegraded":"2024-09-16T10:22:23.532528+0000","last_fullsized":"2024-09-16T10:22:23.532528+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2615,"log_dups_size":0,"ondisk_log_size":2615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:06.384245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171376640,"num_objects":2615,"num_object_clones":0,"num_object_copies":7845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":167360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":516,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873119+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:22:20.873119+0000","last_peered":"2024-09-16T10:22:20.873119+0000","last_clean":"2024-09-16T10:22:20.873119+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:22:20.873119+0000","last_undegraded":"2024-09-16T10:22:20.873119+0000","last_fullsized":"2024-09-16T10:22:20.873119+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:40.844596+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.882567+0000","last_change":"2024-09-16T10:20:57.246050+0000","last_active":"2024-09-16T10:22:20.882567+0000","last_peered":"2024-09-16T10:22:20.882567+0000","last_clean":"2024-09-16T10:22:20.882567+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:22:20.882567+0000","last_undegraded":"2024-09-16T10:22:20.882567+0000","last_fullsized":"2024-09-16T10:22:20.882567+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.245939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.245939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.373180+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:22:23.373180+0000","last_peered":"2024-09-16T10:22:23.373180+0000","last_clean":"2024-09-16T10:22:23.373180+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:22:23.373180+0000","last_undegraded":"2024-09-16T10:22:23.373180+0000","last_fullsized":"2024-09-16T10:22:23.373180+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.537291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369535+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:22:23.369535+0000","last_peered":"2024-09-16T10:22:23.369535+0000","last_clean":"2024-09-16T10:22:23.369535+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:22:23.369535+0000","last_undegraded":"2024-09-16T10:22:23.369535+0000","last_fullsized":"2024-09-16T10:22:23.369535+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.386329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"293'2606","reported_seq":2651,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.673012+0000","last_change":"2024-09-16T10:21:56.236925+0000","last_active":"2024-09-16T10:22:23.673012+0000","last_peered":"2024-09-16T10:22:23.673012+0000","last_clean":"2024-09-16T10:22:23.673012+0000","last_became_active":"2024-09-16T10:21:56.236619+0000","last_became_peered":"2024-09-16T10:21:56.236619+0000","last_unstale":"2024-09-16T10:22:23.673012+0000","last_undegraded":"2024-09-16T10:22:23.673012+0000","last_fullsized":"2024-09-16T10:22:23.673012+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":0,"ondisk_log_size":2606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:56.551656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":7818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":166784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":513,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368556+0000","last_change":"2024-09-16T10:21:35.618881+0000","last_active":"2024-09-16T10:22:23.368556+0000","last_peered":"2024-09-16T10:22:23.368556+0000","last_clean":"2024-09-16T10:22:23.368556+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:22:23.368556+0000","last_undegraded":"2024-09-16T10:22:23.368556+0000","last_fullsized":"2024-09-16T10:22:23.368556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:21:35.618806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.618806+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:02.397177+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":509,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883755+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:22:20.883755+0000","last_peered":"2024-09-16T10:22:20.883755+0000","last_clean":"2024-09-16T10:22:20.883755+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:22:20.883755+0000","last_undegraded":"2024-09-16T10:22:20.883755+0000","last_fullsized":"2024-09-16T10:22:20.883755+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.060231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":604,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:22.268541+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:22:22.268541+0000","last_peered":"2024-09-16T10:22:22.268541+0000","last_clean":"2024-09-16T10:22:22.268541+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:22:22.268541+0000","last_undegraded":"2024-09-16T10:22:22.268541+0000","last_fullsized":"2024-09-16T10:22:22.268541+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:54.008607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":503,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368927+0000","last_change":"2024-09-16T10:22:00.982299+0000","last_active":"2024-09-16T10:22:23.368927+0000","last_peered":"2024-09-16T10:22:23.368927+0000","last_clean":"2024-09-16T10:22:23.368927+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:22:23.368927+0000","last_undegraded":"2024-09-16T10:22:23.368927+0000","last_fullsized":"2024-09-16T10:22:23.368927+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:00.982240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:00.982240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:01.375176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":496,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.877735+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:22:20.877735+0000","last_peered":"2024-09-16T10:22:20.877735+0000","last_clean":"2024-09-16T10:22:20.877735+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:22:20.877735+0000","last_undegraded":"2024-09-16T10:22:20.877735+0000","last_fullsized":"2024-09-16T10:22:20.877735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.931989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":511,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369149+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:22:23.369149+0000","last_peered":"2024-09-16T10:22:23.369149+0000","last_clean":"2024-09-16T10:22:23.369149+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:22:23.369149+0000","last_undegraded":"2024-09-16T10:22:23.369149+0000","last_fullsized":"2024-09-16T10:22:23.369149+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:44.298147+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368919+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:22:23.368919+0000","last_peered":"2024-09-16T10:22:23.368919+0000","last_clean":"2024-09-16T10:22:23.368919+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:22:23.368919+0000","last_undegraded":"2024-09-16T10:22:23.368919+0000","last_fullsized":"2024-09-16T10:22:23.368919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.628323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":477,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.882959+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:22:20.882959+0000","last_peered":"2024-09-16T10:22:20.882959+0000","last_clean":"2024-09-16T10:22:20.882959+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:22:20.882959+0000","last_undegraded":"2024-09-16T10:22:20.882959+0000","last_fullsized":"2024-09-16T10:22:20.882959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.049994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":518,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370324+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:22:23.370324+0000","last_peered":"2024-09-16T10:22:23.370324+0000","last_clean":"2024-09-16T10:22:23.370324+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:22:23.370324+0000","last_undegraded":"2024-09-16T10:22:23.370324+0000","last_fullsized":"2024-09-16T10:22:23.370324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.198247+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883869+0000","last_change":"2024-09-16T10:22:02.040477+0000","last_active":"2024-09-16T10:22:20.883869+0000","last_peered":"2024-09-16T10:22:20.883869+0000","last_clean":"2024-09-16T10:22:20.883869+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:22:20.883869+0000","last_undegraded":"2024-09-16T10:22:20.883869+0000","last_fullsized":"2024-09-16T10:22:20.883869+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.040428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.040428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.314331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":498,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368900+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:22:23.368900+0000","last_peered":"2024-09-16T10:22:23.368900+0000","last_clean":"2024-09-16T10:22:23.368900+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:22:23.368900+0000","last_undegraded":"2024-09-16T10:22:23.368900+0000","last_fullsized":"2024-09-16T10:22:23.368900+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.194439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.373876+0000","last_change":"2024-09-16T10:22:02.903139+0000","last_active":"2024-09-16T10:22:23.373876+0000","last_peered":"2024-09-16T10:22:23.373876+0000","last_clean":"2024-09-16T10:22:23.373876+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:22:23.373876+0000","last_undegraded":"2024-09-16T10:22:23.373876+0000","last_fullsized":"2024-09-16T10:22:23.373876+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.701346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":511,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369145+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:22:23.369145+0000","last_peered":"2024-09-16T10:22:23.369145+0000","last_clean":"2024-09-16T10:22:23.369145+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:22:23.369145+0000","last_undegraded":"2024-09-16T10:22:23.369145+0000","last_fullsized":"2024-09-16T10:22:23.369145+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:48.765027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":514,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370720+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:22:23.370720+0000","last_peered":"2024-09-16T10:22:23.370720+0000","last_clean":"2024-09-16T10:22:23.370720+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:22:23.370720+0000","last_undegraded":"2024-09-16T10:22:23.370720+0000","last_fullsized":"2024-09-16T10:22:23.370720+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:58.559783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":642,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.372062+0000","last_change":"2024-09-16T10:22:03.032565+0000","last_active":"2024-09-16T10:22:23.372062+0000","last_peered":"2024-09-16T10:22:23.372062+0000","last_clean":"2024-09-16T10:22:23.372062+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:22:23.372062+0000","last_undegraded":"2024-09-16T10:22:23.372062+0000","last_fullsized":"2024-09-16T10:22:23.372062+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'9","last_scrub_stamp":"2024-09-16T10:20:25.284462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:25.284462+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":506,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873452+0000","last_change":"2024-09-16T10:22:02.898309+0000","last_active":"2024-09-16T10:22:20.873452+0000","last_peered":"2024-09-16T10:22:20.873452+0000","last_clean":"2024-09-16T10:22:20.873452+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:22:20.873452+0000","last_undegraded":"2024-09-16T10:22:20.873452+0000","last_fullsized":"2024-09-16T10:22:20.873452+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:57.931714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.931714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":525,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371587+0000","last_change":"2024-09-16T10:21:56.344549+0000","last_active":"2024-09-16T10:22:23.371587+0000","last_peered":"2024-09-16T10:22:23.371587+0000","last_clean":"2024-09-16T10:22:23.371587+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:22:23.371587+0000","last_undegraded":"2024-09-16T10:22:23.371587+0000","last_fullsized":"2024-09-16T10:22:23.371587+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:56.344472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.344472+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.689637+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":511,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883324+0000","last_change":"2024-09-16T10:22:02.891069+0000","last_active":"2024-09-16T10:22:20.883324+0000","last_peered":"2024-09-16T10:22:20.883324+0000","last_clean":"2024-09-16T10:22:20.883324+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:22:20.883324+0000","last_undegraded":"2024-09-16T10:22:20.883324+0000","last_fullsized":"2024-09-16T10:22:20.883324+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.101314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":613,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368906+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:22:23.368906+0000","last_peered":"2024-09-16T10:22:23.368906+0000","last_clean":"2024-09-16T10:22:23.368906+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:22:23.368906+0000","last_undegraded":"2024-09-16T10:22:23.368906+0000","last_fullsized":"2024-09-16T10:22:23.368906+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.951237+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369910+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:22:23.369910+0000","last_peered":"2024-09-16T10:22:23.369910+0000","last_clean":"2024-09-16T10:22:23.369910+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:22:23.369910+0000","last_undegraded":"2024-09-16T10:22:23.369910+0000","last_fullsized":"2024-09-16T10:22:23.369910+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:41.183832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":522,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.872552+0000","last_change":"2024-09-16T10:22:10.878557+0000","last_active":"2024-09-16T10:22:20.872552+0000","last_peered":"2024-09-16T10:22:20.872552+0000","last_clean":"2024-09-16T10:22:20.872552+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:22:20.872552+0000","last_undegraded":"2024-09-16T10:22:20.872552+0000","last_fullsized":"2024-09-16T10:22:20.872552+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.735967+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":496,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.873761+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:22:20.873761+0000","last_peered":"2024-09-16T10:22:20.873761+0000","last_clean":"2024-09-16T10:22:20.873761+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:22:20.873761+0000","last_undegraded":"2024-09-16T10:22:20.873761+0000","last_fullsized":"2024-09-16T10:22:20.873761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:38.833623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.366359+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:22:23.366359+0000","last_peered":"2024-09-16T10:22:23.366359+0000","last_clean":"2024-09-16T10:22:23.366359+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:22:23.366359+0000","last_undegraded":"2024-09-16T10:22:23.366359+0000","last_fullsized":"2024-09-16T10:22:23.366359+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.522521+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":486,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368143+0000","last_change":"2024-09-16T10:20:54.392034+0000","last_active":"2024-09-16T10:22:23.368143+0000","last_peered":"2024-09-16T10:22:23.368143+0000","last_clean":"2024-09-16T10:22:23.368143+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:22:23.368143+0000","last_undegraded":"2024-09-16T10:22:23.368143+0000","last_fullsized":"2024-09-16T10:22:23.368143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.391970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.391970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":553,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.366639+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:22:23.366639+0000","last_peered":"2024-09-16T10:22:23.366639+0000","last_clean":"2024-09-16T10:22:23.366639+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:22:23.366639+0000","last_undegraded":"2024-09-16T10:22:23.366639+0000","last_fullsized":"2024-09-16T10:22:23.366639+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.584296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883486+0000","last_change":"2024-09-16T10:19:47.608315+0000","last_active":"2024-09-16T10:22:20.883486+0000","last_peered":"2024-09-16T10:22:20.883486+0000","last_clean":"2024-09-16T10:22:20.883486+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:22:20.883486+0000","last_undegraded":"2024-09-16T10:22:20.883486+0000","last_fullsized":"2024-09-16T10:22:20.883486+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.608243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.608243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":455,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369608+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:22:23.369608+0000","last_peered":"2024-09-16T10:22:23.369608+0000","last_clean":"2024-09-16T10:22:23.369608+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:22:23.369608+0000","last_undegraded":"2024-09-16T10:22:23.369608+0000","last_fullsized":"2024-09-16T10:22:23.369608+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:43.285375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":478,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369489+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:22:23.369489+0000","last_peered":"2024-09-16T10:22:23.369489+0000","last_clean":"2024-09-16T10:22:23.369489+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:22:23.369489+0000","last_undegraded":"2024-09-16T10:22:23.369489+0000","last_fullsized":"2024-09-16T10:22:23.369489+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.132782+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":505,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368449+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:22:23.368449+0000","last_peered":"2024-09-16T10:22:23.368449+0000","last_clean":"2024-09-16T10:22:23.368449+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:22:23.368449+0000","last_undegraded":"2024-09-16T10:22:23.368449+0000","last_fullsized":"2024-09-16T10:22:23.368449+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.285825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":507,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.368945+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:22:23.368945+0000","last_peered":"2024-09-16T10:22:23.368945+0000","last_clean":"2024-09-16T10:22:23.368945+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:22:23.368945+0000","last_undegraded":"2024-09-16T10:22:23.368945+0000","last_fullsized":"2024-09-16T10:22:23.368945+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.996569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":505,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.881965+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:22:20.881965+0000","last_peered":"2024-09-16T10:22:20.881965+0000","last_clean":"2024-09-16T10:22:20.881965+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:22:20.881965+0000","last_undegraded":"2024-09-16T10:22:20.881965+0000","last_fullsized":"2024-09-16T10:22:20.881965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":502,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.369729+0000","last_change":"2024-09-16T10:21:31.395288+0000","last_active":"2024-09-16T10:22:23.369729+0000","last_peered":"2024-09-16T10:22:23.369729+0000","last_clean":"2024-09-16T10:22:23.369729+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:22:23.369729+0000","last_undegraded":"2024-09-16T10:22:23.369729+0000","last_fullsized":"2024-09-16T10:22:23.369729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.395222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.395222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.580946+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":534,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370155+0000","last_change":"2024-09-16T10:21:58.334088+0000","last_active":"2024-09-16T10:22:23.370155+0000","last_peered":"2024-09-16T10:22:23.370155+0000","last_clean":"2024-09-16T10:22:23.370155+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:22:23.370155+0000","last_undegraded":"2024-09-16T10:22:23.370155+0000","last_fullsized":"2024-09-16T10:22:23.370155+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:21:58.334025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.334025+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.713313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":504,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.371089+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:22:23.371089+0000","last_peered":"2024-09-16T10:22:23.371089+0000","last_clean":"2024-09-16T10:22:23.371089+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:22:23.371089+0000","last_undegraded":"2024-09-16T10:22:23.371089+0000","last_fullsized":"2024-09-16T10:22:23.371089+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:51.302753+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":496,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876573+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:22:20.876573+0000","last_peered":"2024-09-16T10:22:20.876573+0000","last_clean":"2024-09-16T10:22:20.876573+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:22:20.876573+0000","last_undegraded":"2024-09-16T10:22:20.876573+0000","last_fullsized":"2024-09-16T10:22:20.876573+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:41.864975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":262,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.883874+0000","last_change":"2024-09-16T10:22:02.898982+0000","last_active":"2024-09-16T10:22:20.883874+0000","last_peered":"2024-09-16T10:22:20.883874+0000","last_clean":"2024-09-16T10:22:20.883874+0000","last_became_active":"2024-09-16T10:22:02.898630+0000","last_became_peered":"2024-09-16T10:22:02.898630+0000","last_unstale":"2024-09-16T10:22:20.883874+0000","last_undegraded":"2024-09-16T10:22:20.883874+0000","last_fullsized":"2024-09-16T10:22:20.883874+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:20.058989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":495,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.365298+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:22:23.365298+0000","last_peered":"2024-09-16T10:22:23.365298+0000","last_clean":"2024-09-16T10:22:23.365298+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:22:23.365298+0000","last_undegraded":"2024-09-16T10:22:23.365298+0000","last_fullsized":"2024-09-16T10:22:23.365298+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.593004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":506,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370477+0000","last_change":"2024-09-16T10:20:58.426297+0000","last_active":"2024-09-16T10:22:23.370477+0000","last_peered":"2024-09-16T10:22:23.370477+0000","last_clean":"2024-09-16T10:22:23.370477+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:22:23.370477+0000","last_undegraded":"2024-09-16T10:22:23.370477+0000","last_fullsized":"2024-09-16T10:22:23.370477+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.426235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.426235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":528,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-16T10:22:20.876844+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:22:20.876844+0000","last_peered":"2024-09-16T10:22:20.876844+0000","last_clean":"2024-09-16T10:22:20.876844+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:22:20.876844+0000","last_undegraded":"2024-09-16T10:22:20.876844+0000","last_fullsized":"2024-09-16T10:22:20.876844+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.159732+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":513,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-16T10:22:23.370069+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:22:23.370069+0000","last_peered":"2024-09-16T10:22:23.370069+0000","last_clean":"2024-09-16T10:22:23.370069+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:22:23.370069+0000","last_undegraded":"2024-09-16T10:22:23.370069+0000","last_fullsized":"2024-09-16T10:22:23.370069+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.158658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2693070848,"num_objects":41093,"num_object_clones":0,"num_object_copies":123279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41093,"num_write_kb":2629952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":19005440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8058044416,"data_stored":8058044416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41093,"ondisk_log_size":41093,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":520,"num_read_kb":515,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234093,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785640,"kb_used_data":1461528,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91959512,"statfs":{"total":95995035648,"available":94166540288,"internally_reserved":0,"allocated":1496604672,"data_stored":1495732587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":331872714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528433,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801940,"kb_used_data":486144,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92943212,"statfs":{"total":95995035648,"available":95173849088,"internally_reserved":0,"allocated":497811456,"data_stored":496962849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790071,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1757096,"kb_used_data":1446872,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91988056,"statfs":{"total":95995035648,"available":94195769344,"internally_reserved":0,"allocated":1481596928,"data_stored":1480725971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986300,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":752052,"kb_used_data":486240,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92993100,"statfs":{"total":95995035648,"available":95224934400,"internally_reserved":0,"allocated":497909760,"data_stored":497029426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215233,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1700048,"kb_used_data":1313468,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":92045104,"statfs":{"total":95995035648,"available":94254186496,"internally_reserved":0,"allocated":1344991232,"data_stored":1344091042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":395835848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411463,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":616408,"kb_used_data":476936,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93128744,"statfs":{"total":95995035648,"available":95363833856,"internally_reserved":0,"allocated":488382464,"data_stored":487526480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1765132,"kb_used_data":1427124,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91980020,"statfs":{"total":95995035648,"available":94187540480,"internally_reserved":0,"allocated":1461374976,"data_stored":1460474902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":980720,"kb_used_data":656608,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92764432,"statfs":{"total":95995035648,"available":94990778368,"internally_reserved":0,"allocated":672366592,"data_stored":671487017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":682754048,"data_stored":682754048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1516765184,"data_stored":1516765184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":497418240,"data_stored":497418240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346764800,"data_stored":1346764800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":504037376,"data_stored":504037376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1507262464,"data_stored":1507262464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":501284864,"data_stored":501284864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1501757440,"data_stored":1501757440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:30.002 INFO:tasks.thrashosds.thrasher:Setting 9.b to [2, 1, 6] 2024-09-16T10:22:30.002 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '9.b', '2', '1', '6'] 2024-09-16T10:22:30.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 9.b 2 1 6 2024-09-16T10:22:30.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:30.139 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:30.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:30.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:30.245 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:30.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:30.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:30.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:30.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 29 16 52353 52337 112.78 96.5 0.0100157 0.00886066 2024-09-16T10:22:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:30.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:30.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:30.445 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:22:30.503 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:30 smithi063 ceph-mon[29993]: pgmap v476: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 MiB/s wr, 2.73k op/s 2024-09-16T10:22:30.504 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2289253476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:30.504 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:30 smithi063 ceph-mon[39470]: pgmap v476: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 MiB/s wr, 2.73k op/s 2024-09-16T10:22:30.504 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2289253476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:30.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:30.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:30.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:30.568 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:30.568 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:30.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:30.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:30.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:30.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:30 smithi165 ceph-mon[32696]: pgmap v476: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 MiB/s wr, 2.73k op/s 2024-09-16T10:22:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2289253476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:30.829 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:30.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:30.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:30.966 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:30.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:31.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:31.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:31.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:31.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:31.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:31.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:31.279 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:31.280 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:31.319 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:31.319 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:31.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:31.351 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:31.352 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:31.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:31.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:31.392 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:31.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 30 16 53885 53869 112.212 95.75 0.00568164 0.00890607 2024-09-16T10:22:31.423 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:31.423 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: 5.6 scrub starts 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: 5.6 scrub ok 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2289253476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: osdmap e296: 8 total, 8 up, 8 in 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: 3.1e scrub starts 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: 3.1e scrub ok 2024-09-16T10:22:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3850318089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: 5.6 scrub starts 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: 5.6 scrub ok 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2289253476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: osdmap e296: 8 total, 8 up, 8 in 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: 3.1e scrub starts 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: 3.1e scrub ok 2024-09-16T10:22:31.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3850318089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:31.465 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:31.465 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:31.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:31.498 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:31.498 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:31.515 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:31.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:31.531 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:31.531 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:31.617 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:31.655 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:31.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:31.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:31.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:31.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: 5.6 scrub starts 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: 5.6 scrub ok 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2289253476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: osdmap e296: 8 total, 8 up, 8 in 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: 3.1e scrub starts 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: 3.1e scrub ok 2024-09-16T10:22:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3850318089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:31.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:31.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:31.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:32.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:32.070 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:32.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:32.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:32.215 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:32.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:32.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:22:32.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:32.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:32.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:32.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 31 16 55466 55450 111.78 98.8125 0.0104223 0.00894117 2024-09-16T10:22:32.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:32.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:32.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:32.588 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:32.622 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:32 smithi063 ceph-mon[39470]: pgmap v478: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-16T10:22:32.622 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:32 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:22:32.622 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:32 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:22:32.623 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:32 smithi063 ceph-mon[29993]: pgmap v478: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-16T10:22:32.623 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:32 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:22:32.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:32 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:22:32.632 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:32.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:32.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:32.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:32.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:32 smithi165 ceph-mon[32696]: pgmap v478: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-16T10:22:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:32 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:22:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:32 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:22:32.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:32.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:32.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:33.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:33.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:33.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:33.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:33.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:33.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:33.271 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:33.311 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:33.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:33.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 32 16 57033 57017 111.347 97.9375 0.00640072 0.00897595 2024-09-16T10:22:33.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:33.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:33.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:33.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:33.586 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:33.612 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:33.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:33.648 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:33.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:33 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:22:33.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:33 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:22:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:33 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:22:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:33 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:22:33.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:33.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:33.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:33 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:22:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:33 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:22:33.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:33.881 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:33.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:33.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:34.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:34.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:34.110 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:34.146 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:34.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:34.247 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:34.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:34.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:34.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 33 16 58543 58527 110.832 94.375 0.0181406 0.00901726 2024-09-16T10:22:34.442 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:34.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:34.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:34.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:34.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:34.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[29993]: 3.3 scrub starts 2024-09-16T10:22:34.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[29993]: 3.3 scrub ok 2024-09-16T10:22:34.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[29993]: pgmap v479: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 698 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-16T10:22:34.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[29993]: 3.e scrub starts 2024-09-16T10:22:34.575 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[29993]: 3.e scrub ok 2024-09-16T10:22:34.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[39470]: 3.3 scrub starts 2024-09-16T10:22:34.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[39470]: 3.3 scrub ok 2024-09-16T10:22:34.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[39470]: pgmap v479: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 698 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-16T10:22:34.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[39470]: 3.e scrub starts 2024-09-16T10:22:34.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:34 smithi063 ceph-mon[39470]: 3.e scrub ok 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51238 8.e 8:7236be56:::benchmark_data_smithi063_101290_object51237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 201.22910874999999, 2024-09-16T10:22:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96343729, 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51238 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:22:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341637+0000", 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.621e-06 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341663+0000", 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6026000000000001e-05 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341674+0000", 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0871e-05 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341682+0000", 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8390000000000007e-06 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341780+0000", 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8233999999999995e-05 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341791+0000", 2024-09-16T10:22:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1426999999999999e-05 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296465+0000", 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95467394000000005 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296604+0000", 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013824400000000001 2024-09-16T10:22:34.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296727+0000", 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012366699999999999 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304672+0000", 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079445339999999996 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304896+0000", 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000224117 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304941+0000", 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5374999999999998e-05 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305028+0000", 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6447999999999996e-05 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305073+0000", 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4947000000000002e-05 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51239 8.e 8:7166e593:::benchmark_data_smithi063_101290_object51238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 201.22869057299999, 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96520916099999998, 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:34.585 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51239 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:22:34.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342055+0000", 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.051e-06 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342073+0000", 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8325000000000002e-05 2024-09-16T10:22:34.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342083+0000", 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9850000000000001e-06 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342090+0000", 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2119999999999997e-06 2024-09-16T10:22:34.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342196+0000", 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010593400000000001 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342244+0000", 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8272000000000001e-05 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299084+0000", 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95683958300000005 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299216+0000", 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131595 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299311+0000", 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5445000000000005e-05 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305117+0000", 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058058169999999996 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305157+0000", 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9836999999999998e-05 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307167+0000", 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020101390000000002 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307221+0000", 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4317000000000002e-05 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307263+0000", 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1649000000000001e-05 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51244 8.e 8:7a51d24a:::benchmark_data_smithi063_101290_object51243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 201.22546944199999, 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96214958799999994, 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51244 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345277+0000", 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9810000000000002e-06 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345305+0000", 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7858999999999999e-05 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345320+0000", 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5268e-05 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345328+0000", 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2900000000000002e-06 2024-09-16T10:22:34.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345404+0000", 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6241000000000004e-05 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345415+0000", 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0672e-05 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300950+0000", 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95553457900000005 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301082+0000", 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013229299999999999 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301242+0000", 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015961099999999999 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307271+0000", 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060297889999999998 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307310+0000", 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8288999999999999e-05 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307349+0000", 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9499999999999998e-05 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307391+0000", 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1779000000000001e-05 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307424+0000", 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3436999999999999e-05 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51256 8.e 8:745a54ba:::benchmark_data_smithi063_101290_object51255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 201.21363096300001, 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95048443400000004, 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.603 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51256 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357113+0000", 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357116+0000", 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8820000000000001e-06 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357147+0000", 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1263999999999997e-05 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357167+0000", 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9164000000000002e-05 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357176+0000", 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8309999999999996e-06 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357255+0000", 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8213999999999993e-05 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.357267+0000", 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2729e-05 2024-09-16T10:22:34.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303081+0000", 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94581367599999999 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303176+0000", 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5471999999999997e-05 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.303272+0000", 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5874999999999999e-05 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307320+0000", 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040474079999999997 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307459+0000", 2024-09-16T10:22:34.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013938500000000001 2024-09-16T10:22:34.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307514+0000", 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4948999999999999e-05 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307560+0000", 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5803999999999997e-05 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307598+0000", 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7781000000000003e-05 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51260 8.e 8:7df47e01:::benchmark_data_smithi063_101290_object51259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 201.20850852500001, 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95708149499999995, 2024-09-16T10:22:34.612 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51260 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362238+0000", 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4140000000000001e-06 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362265+0000", 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7183999999999999e-05 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362279+0000", 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3832000000000001e-05 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362287+0000", 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6990000000000003e-06 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362347+0000", 2024-09-16T10:22:34.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0560000000000003e-05 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362356+0000", 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1990000000000006e-06 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305199+0000", 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94284296700000003 2024-09-16T10:22:34.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305311+0000", 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011281900000000001 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305449+0000", 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000137708 2024-09-16T10:22:34.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309902+0000", 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044530100000000003 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312979+0000", 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030772859999999998 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317533+0000", 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045533479999999996 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319259+0000", 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017263129999999999 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319317+0000", 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8155999999999997e-05 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:30420 9.5 9:a6eb9cf7:::benchmark_data_smithi063_124717_object30419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.765298+0000", 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.805446367000002, 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.235547703, 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 30420 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.765298+0000", 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.765298+0000", 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.765299+0000", 2024-09-16T10:22:34.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.491e-06 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.765317+0000", 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7714999999999999e-05 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.765326+0000", 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5909999999999996e-06 2024-09-16T10:22:34.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.765331+0000", 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9599999999999999e-06 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766545+0000", 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012147740000000001 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766657+0000", 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111247 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766786+0000", 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000129309 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.769914+0000", 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031279839999999999 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772262+0000", 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023484170000000002 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000736+0000", 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22847333 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000789+0000", 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3043e-05 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000846+0000", 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6842000000000002e-05 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:30423 9.5 9:af58dab2:::benchmark_data_smithi063_124717_object30422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.766088+0000", 2024-09-16T10:22:34.629 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.804656177999998, 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23486204099999999, 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 30423 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766088+0000", 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766088+0000", 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766089+0000", 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6400000000000001e-07 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766112+0000", 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3264999999999998e-05 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766123+0000", 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0645999999999999e-05 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.766130+0000", 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.525e-06 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768214+0000", 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020839669999999999 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768349+0000", 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013456000000000001 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768636+0000", 2024-09-16T10:22:34.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00028724700000000002 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772220+0000", 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003583559 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772299+0000", 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9427999999999999e-05 2024-09-16T10:22:34.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000881+0000", 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22858229999999999 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000918+0000", 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7027000000000003e-05 2024-09-16T10:22:34.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000950+0000", 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1652999999999997e-05 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:30426 9.5 9:acdd8648:::benchmark_data_smithi063_124717_object30425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:34.637 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.768004+0000", 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.802740142000001, 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.233056443, 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 30426 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768004+0000", 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768004+0000", 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768006+0000", 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1900000000000002e-06 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768028+0000", 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1769000000000001e-05 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768039+0000", 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0869999999999999e-05 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768046+0000", 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5739999999999998e-06 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.769941+0000", 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018946480000000001 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770050+0000", 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108416 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770186+0000", 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013602800000000001 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772632+0000", 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024467500000000001 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772645+0000", 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2719e-05 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000989+0000", 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22834398 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.001030+0000", 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1369999999999999e-05 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.001060+0000", 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0128999999999999e-05 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:30427 9.b 9:d716876e:::benchmark_data_smithi063_124717_object30426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.768480+0000", 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.802264524000002, 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.2347088, 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 30427 2024-09-16T10:22:34.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768480+0000", 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768480+0000", 2024-09-16T10:22:34.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768481+0000", 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1799999999999999e-06 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768503+0000", 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2631000000000001e-05 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768513+0000", 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4339999999999998e-06 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768520+0000", 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.345e-06 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768816+0000", 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000295319 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.768928+0000", 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112293 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.769063+0000", 2024-09-16T10:22:34.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000135198 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.001175+0000", 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23211178700000001 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003023+0000", 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001848452 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003062+0000", 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9119999999999998e-05 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003146+0000", 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3651000000000003e-05 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003188+0000", 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2389999999999999e-05 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:30432 9.b 9:dd028e3a:::benchmark_data_smithi063_124717_object30431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.770758+0000", 2024-09-16T10:22:34.653 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.799985711000001, 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.232606173, 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 30432 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770758+0000", 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770758+0000", 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770760+0000", 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.624e-06 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770784+0000", 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3629e-05 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770794+0000", 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0244000000000001e-05 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770801+0000", 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9500000000000004e-06 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.770868+0000", 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6835999999999995e-05 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.771007+0000", 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000139622 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:22:34.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.771131+0000", 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012350100000000001 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.002985+0000", 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.231854588 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003229+0000", 2024-09-16T10:22:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000243596 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003284+0000", 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4991000000000003e-05 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003331+0000", 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6833999999999999e-05 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003365+0000", 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3757999999999998e-05 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:30435 9.b 9:db954a65:::benchmark_data_smithi063_124717_object30434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.772053+0000", 2024-09-16T10:22:34.661 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.798691293000001, 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.235335304, 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 30435 2024-09-16T10:22:34.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.663 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772053+0000", 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772053+0000", 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772054+0000", 2024-09-16T10:22:34.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3880000000000001e-06 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772142+0000", 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8189999999999994e-05 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772165+0000", 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2569e-05 2024-09-16T10:22:34.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772173+0000", 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6029999999999999e-06 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.001208+0000", 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22903559000000001 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.001346+0000", 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000137952 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.001488+0000", 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014196599999999999 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003239+0000", 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001750466 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.007200+0000", 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0039611469999999999 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.007252+0000", 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2444000000000001e-05 2024-09-16T10:22:34.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.007348+0000", 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5304999999999996e-05 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.007388+0000", 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0683999999999999e-05 2024-09-16T10:22:34.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.45596.0:30427 9.b e290/276 ondisk, result = 0)", 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.772831+0000", 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.797912735000001, 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23019962999999999, 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.670 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772831+0000", 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772831+0000", 2024-09-16T10:22:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772832+0000", 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0189999999999999e-06 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.772836+0000", 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.574e-06 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000546+0000", 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22770959499999999 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000558+0000", 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2130000000000001e-05 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003009+0000", 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024515589999999999 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003018+0000", 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4959999999999996e-06 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.003031+0000", 2024-09-16T10:22:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3257e-05 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.45596.0:30420 9.5 e290/276 ondisk, result = 0)", 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:16.773746+0000", 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.796998301999999, 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22710665399999999, 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.675 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.773746+0000", 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.773746+0000", 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.773747+0000", 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0920000000000001e-06 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:16.773752+0000", 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1200000000000001e-06 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000534+0000", 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22678231900000001 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000548+0000", 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3743999999999999e-05 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000719+0000", 2024-09-16T10:22:34.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017083800000000001 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000727+0000", 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3760000000000008e-06 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:17.000853+0000", 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125165 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.679 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:38207 9.5 9:aba845bc:::benchmark_data_smithi063_124717_object38206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:21.671876+0000", 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.898868504999999, 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23245547599999999, 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.680 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 38207 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.671876+0000", 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.671876+0000", 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.671878+0000", 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8309999999999999e-06 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.671900+0000", 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2741000000000001e-05 2024-09-16T10:22:34.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.671911+0000", 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1133e-05 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.671919+0000", 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4800000000000004e-06 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.671972+0000", 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3572999999999997e-05 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.672086+0000", 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011370999999999999 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.672208+0000", 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121502 2024-09-16T10:22:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.673956+0000", 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017478839999999999 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.706570+0000", 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032614087 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.904254+0000", 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.19768391799999999 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.904295+0000", 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1647000000000003e-05 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.904331+0000", 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5970000000000003e-05 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:38210 9.9 9:9f50fbbc:::benchmark_data_smithi063_124717_object38209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:21.684090+0000", 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.886654436000001, 2024-09-16T10:22:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22639194600000001, 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 38210 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684090+0000", 2024-09-16T10:22:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684090+0000", 2024-09-16T10:22:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684092+0000", 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0870000000000001e-06 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684113+0000", 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0920999999999998e-05 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684126+0000", 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2807e-05 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684132+0000", 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4999999999999996e-06 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684187+0000", 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5352000000000001e-05 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684320+0000", 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000132502 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.684464+0000", 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014441899999999999 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.687569+0000", 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003104192 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.709195+0000", 2024-09-16T10:22:34.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.021626520999999999 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910392+0000", 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.201196767 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910443+0000", 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1218000000000003e-05 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910482+0000", 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8659999999999999e-05 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:38211 9.9 9:99e72e22:::benchmark_data_smithi063_124717_object38210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:21.685416+0000", 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.885327797, 2024-09-16T10:22:34.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22519320700000001, 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 38211 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685416+0000", 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685416+0000", 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685418+0000", 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.207e-06 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685437+0000", 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9419000000000002e-05 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685448+0000", 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0669999999999999e-05 2024-09-16T10:22:34.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685453+0000", 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4040000000000003e-06 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685827+0000", 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037434700000000003 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685967+0000", 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000139154 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.686131+0000", 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000164179 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.689300+0000", 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031693450000000001 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.709237+0000", 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.019936426 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910529+0000", 2024-09-16T10:22:34.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.20129217699999999 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910574+0000", 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5056999999999999e-05 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910610+0000", 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5821999999999997e-05 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.705 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:38213 9.b 9:d1e9eca5:::benchmark_data_smithi063_124717_object38212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:21.685905+0000", 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.884839212999999, 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22041852100000001, 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.706 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 38213 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685905+0000", 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685905+0000", 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685906+0000", 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2359999999999999e-06 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685928+0000", 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1654999999999998e-05 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685937+0000", 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2150000000000007e-06 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685944+0000", 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2840000000000001e-06 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.686003+0000", 2024-09-16T10:22:34.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8168999999999998e-05 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.686124+0000", 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121117 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.686224+0000", 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010002500000000001 2024-09-16T10:22:34.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.688709+0000", 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002485826 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.688973+0000", 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026326000000000002 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.906226+0000", 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.217253433 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.906284+0000", 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7587999999999998e-05 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.906323+0000", 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9712999999999997e-05 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:38214 9.9 9:9864eca2:::benchmark_data_smithi063_124717_object38213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:21.685964+0000", 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.884780292, 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22477897199999999, 2024-09-16T10:22:34.713 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 38214 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685964+0000", 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685964+0000", 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685965+0000", 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9599999999999998e-07 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685986+0000", 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1084e-05 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685993+0000", 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.046e-06 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.685999+0000", 2024-09-16T10:22:34.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.489e-06 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.687649+0000", 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001649972 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.687778+0000", 2024-09-16T10:22:34.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012915 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.687925+0000", 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014688799999999999 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.689595+0000", 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016695169999999999 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.710114+0000", 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.020519631 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910656+0000", 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.20054186099999999 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910707+0000", 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0914000000000001e-05 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910743+0000", 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5624000000000003e-05 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:38223 9.5 9:acf4a7bf:::benchmark_data_smithi063_124717_object38222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:21.690865+0000", 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.879878915000001, 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21462763000000001, 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.721 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 38223 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.690865+0000", 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.690865+0000", 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.690867+0000", 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.798e-06 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.690887+0000", 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9534999999999998e-05 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.690898+0000", 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1729000000000001e-05 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.690904+0000", 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.626e-06 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.690957+0000", 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2904e-05 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.691083+0000", 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012609700000000001 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.691233+0000", 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000149555 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.692467+0000", 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001234305 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.706611+0000", 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0141441 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.905372+0000", 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.19876097300000001 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.905445+0000", 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3285000000000001e-05 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.905493+0000", 2024-09-16T10:22:34.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7722999999999997e-05 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:38231 9.9 9:9e06c7c1:::benchmark_data_smithi063_124717_object38230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:21.707249+0000", 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 12.863494981000001, 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.20364188799999999, 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:34.729 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 38231 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707249+0000", 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707249+0000", 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707251+0000", 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.216e-06 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707277+0000", 2024-09-16T10:22:34.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5167000000000001e-05 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707293+0000", 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6543000000000001e-05 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707301+0000", 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9019999999999999e-06 2024-09-16T10:22:34.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707375+0000", 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3998999999999994e-05 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:34.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707513+0000", 2024-09-16T10:22:34.733 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:34.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:22:34.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013798900000000001 2024-09-16T10:22:34.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,7", 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.707675+0000", 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016208800000000001 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.709664+0000", 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001988854 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.710237+0000", 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00057290800000000001 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910787+0000", 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.20055007499999999 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:34.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910853+0000", 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6478999999999998e-05 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:21.910891+0000", 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7667999999999999e-05 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:34.738 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:22:34.780 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:34.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:34.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:34 smithi165 ceph-mon[32696]: 3.3 scrub starts 2024-09-16T10:22:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:34 smithi165 ceph-mon[32696]: 3.3 scrub ok 2024-09-16T10:22:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:34 smithi165 ceph-mon[32696]: pgmap v479: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 698 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-16T10:22:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:34 smithi165 ceph-mon[32696]: 3.e scrub starts 2024-09-16T10:22:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:34 smithi165 ceph-mon[32696]: 3.e scrub ok 2024-09-16T10:22:34.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:34.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:34.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:35.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:35.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:35.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:35.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:35.187 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:35.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:35.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:35.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:35.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:35.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 34 16 59927 59911 110.117 86.5 0.00499854 0.00907613 2024-09-16T10:22:35.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:35.458 INFO:teuthology.orchestra.run.smithi063.stderr:set 9.b pg_upmap mapping to [2,1,6] 2024-09-16T10:22:35.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:35.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:35.485 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4101166760' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]} : dispatch 2024-09-16T10:22:35.486 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:35 smithi063 ceph-mon[29993]: from='client.25711 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]} : dispatch 2024-09-16T10:22:35.486 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4101166760' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]} : dispatch 2024-09-16T10:22:35.486 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:35 smithi063 ceph-mon[39470]: from='client.25711 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]} : dispatch 2024-09-16T10:22:35.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:22:35.573 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:35.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:35.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:35.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:35.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:35.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4101166760' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]} : dispatch 2024-09-16T10:22:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:35 smithi165 ceph-mon[32696]: from='client.25711 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]} : dispatch 2024-09-16T10:22:35.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:35.901 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:35.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:36.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:36.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:36.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:22:36.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:36.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:36.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:36.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:36.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:36.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 35 16 60452 60436 107.908 32.8125 0.00274563 0.00906956 2024-09-16T10:22:36.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:36.522 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:36.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:36.646 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:36.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:36.753 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:22:36.753 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:22:36.753 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[29993]: pgmap v480: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.01k op/s 2024-09-16T10:22:36.753 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[29993]: from='client.25711 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]}]': finished 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[29993]: osdmap e297: 8 total, 8 up, 8 in 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[29993]: osdmap e298: 8 total, 8 up, 8 in 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[39470]: pgmap v480: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.01k op/s 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[39470]: from='client.25711 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]}]': finished 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[39470]: osdmap e297: 8 total, 8 up, 8 in 2024-09-16T10:22:36.754 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:36 smithi063 ceph-mon[39470]: osdmap e298: 8 total, 8 up, 8 in 2024-09-16T10:22:36.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:36.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:36 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:22:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:36 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:22:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:36 smithi165 ceph-mon[32696]: pgmap v480: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.01k op/s 2024-09-16T10:22:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:36 smithi165 ceph-mon[32696]: from='client.25711 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.b", "id": [2, 1, 6]}]': finished 2024-09-16T10:22:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:36 smithi165 ceph-mon[32696]: osdmap e297: 8 total, 8 up, 8 in 2024-09-16T10:22:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:36 smithi165 ceph-mon[32696]: osdmap e298: 8 total, 8 up, 8 in 2024-09-16T10:22:36.821 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:36.861 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:36.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:36.901 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:36.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:37.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:37.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:37.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:37.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:37.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:37.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:37.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:37.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:37.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:37.323 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:22:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:22:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:22:37.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 36 16 60839 60823 105.582 24.1875 0.00607626 0.00946536 2024-09-16T10:22:37.421 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:37.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:37.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:37.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:37.604 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:37.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:37 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:22:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:37 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:22:37.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:37 smithi063 ceph-mon[29993]: osdmap e299: 8 total, 8 up, 8 in 2024-09-16T10:22:37.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:37 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:22:37.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:37 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:22:37.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:37 smithi063 ceph-mon[39470]: osdmap e299: 8 total, 8 up, 8 in 2024-09-16T10:22:37.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:37.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:37.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:37.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:37 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:22:37.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:37 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:22:37.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:37 smithi165 ceph-mon[32696]: osdmap e299: 8 total, 8 up, 8 in 2024-09-16T10:22:37.837 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:37.890 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:37.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:37.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:38.014 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:38.059 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:38.060 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:38.100 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:38.100 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:38.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:38.142 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:38.142 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:38.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:38.178 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:38.211 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:38.212 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:38.217 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:38.257 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:38.257 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:38.259 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:38.291 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.303 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:38.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:38.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:38.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 37 16 61928 61912 104.568 68.0625 0.00965694 0.00955611 2024-09-16T10:22:38.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:38.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:38.576 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:38.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:38.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:38.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:22:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:22:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:22:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-mon[32696]: pgmap v484: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-16T10:22:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-mon[32696]: osdmap e300: 8 total, 8 up, 8 in 2024-09-16T10:22:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1874811764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:38.807 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:38.853 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:38.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:38.914 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:22:38.914 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:22:38.914 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[29993]: pgmap v484: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-16T10:22:38.914 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[29993]: osdmap e300: 8 total, 8 up, 8 in 2024-09-16T10:22:38.914 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1874811764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:38.914 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:38.914 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:38.915 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:22:38.915 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:22:38.915 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[39470]: pgmap v484: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-16T10:22:38.915 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[39470]: osdmap e300: 8 total, 8 up, 8 in 2024-09-16T10:22:38.915 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1874811764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:38.915 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:38.915 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:38.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:38.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:38.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:39.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:39.127 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:39.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:39.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:39.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:39.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:39.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:39.398 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:39.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:39.406 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 38 16 63081 63065 103.713 72.0625 0.0153486 0.00963585 2024-09-16T10:22:39.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:22:39.488 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:22:39.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:39.581 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:39.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:39.683 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:39.696 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:39.697 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3933697235' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:39.697 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:39 smithi063 ceph-mon[39470]: from='client.25720 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:39.697 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:39.697 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3933697235' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:39.697 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:39 smithi063 ceph-mon[29993]: from='client.25720 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:39.730 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:39.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3933697235' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:39 smithi165 ceph-mon[32696]: from='client.25720 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:22:39.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:39.880 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:39.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:39.981 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.022 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:40.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:40.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:40.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:40.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:40.279 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:40.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:40.381 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:40.408 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 39 16 64264 64248 102.945 73.9375 0.0124237 0.00970811 2024-09-16T10:22:40.420 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:40.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:40.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:40.653 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:40.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:40.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[29993]: pgmap v486: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 384 KiB/s, 5 objects/s recovering 2024-09-16T10:22:40.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[29993]: Health check failed: Degraded data redundancy: 1/186768 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[29993]: from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[29993]: osdmap e301: 8 total, 8 up, 8 in 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[39470]: pgmap v486: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 384 KiB/s, 5 objects/s recovering 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[39470]: Health check failed: Degraded data redundancy: 1/186768 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[39470]: from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:22:40.775 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:40 smithi063 ceph-mon[39470]: osdmap e301: 8 total, 8 up, 8 in 2024-09-16T10:22:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:40 smithi165 ceph-mon[32696]: pgmap v486: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 384 KiB/s, 5 objects/s recovering 2024-09-16T10:22:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:40 smithi165 ceph-mon[32696]: Health check failed: Degraded data redundancy: 1/186768 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:40 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:40 smithi165 ceph-mon[32696]: from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:22:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:40 smithi165 ceph-mon[32696]: osdmap e301: 8 total, 8 up, 8 in 2024-09-16T10:22:40.803 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.854 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:40.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:40.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:40.995 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:40.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:41.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:41.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:41.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:41.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:41.236 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:41.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:41.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:41.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:41.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:41.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:41.408 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:22:41.407382+0000 min lat: 0.00102889 max lat: 2.20645 avg lat: 0.00978474 2024-09-16T10:22:41.408 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:22:41.408 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 40 16 65389 65373 102.129 70.3125 0.00759263 0.00978474 2024-09-16T10:22:41.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:41.536 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:41.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:41.638 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:41.661 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:41.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:41.762 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:41.783 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:22:41.783 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:22:41.783 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T10:22:41.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:22:41.805 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:41.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:41.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:41.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:41.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:42.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:42.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:42.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:42.160 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:42.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:41 smithi063 ceph-mon[29993]: osdmap e302: 8 total, 8 up, 8 in 2024-09-16T10:22:42.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:41 smithi063 ceph-mon[29993]: pgmap v489: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 354 KiB/s, 5 objects/s recovering 2024-09-16T10:22:42.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:41 smithi063 ceph-mon[39470]: osdmap e302: 8 total, 8 up, 8 in 2024-09-16T10:22:42.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:41 smithi063 ceph-mon[39470]: pgmap v489: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 354 KiB/s, 5 objects/s recovering 2024-09-16T10:22:42.204 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:42.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:41 smithi165 ceph-mon[32696]: osdmap e302: 8 total, 8 up, 8 in 2024-09-16T10:22:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:41 smithi165 ceph-mon[32696]: pgmap v489: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 354 KiB/s, 5 objects/s recovering 2024-09-16T10:22:42.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:42.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:42.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:42.408 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 41 16 66317 66301 101.053 58 0.00710801 0.00988879 2024-09-16T10:22:42.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:42.467 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:42.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:42.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:42.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:42.658 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:22:42.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:42.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:42.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:42.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:42.893 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:42.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:42.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:43.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:43.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:43.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:43.138 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:22:43.194 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:43.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:43.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:43.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:43.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:43.413 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 42 16 67142 67126 99.8717 51.5625 0.0136583 0.0100072 2024-09-16T10:22:43.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:43.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:43 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:22:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:43 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:22:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:43.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:43 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:22:43.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:43.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:43.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:43.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:43.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:43.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:43.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:43.857 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:43.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:43.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:44.000 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:44.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:44.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:44.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:44.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:44.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:44 smithi063 ceph-mon[29993]: pgmap v490: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 813 op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 256 KiB/s, 3 objects/s recovering 2024-09-16T10:22:44.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:44 smithi063 ceph-mon[39470]: pgmap v490: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 813 op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 256 KiB/s, 3 objects/s recovering 2024-09-16T10:22:44.361 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:44.409 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 43 16 68242 68226 99.1479 68.75 0.0128322 0.0100799 2024-09-16T10:22:44.417 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:44.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:44.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:44.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:44 smithi165 ceph-mon[32696]: pgmap v490: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 813 op/s; 1/186768 objects degraded (0.001%); 7503/186768 objects misplaced (4.017%); 256 KiB/s, 3 objects/s recovering 2024-09-16T10:22:44.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:44.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:44.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:44.810 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:44.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:44.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:44.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:45.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:45.098 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:45.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:45.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:45.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:45.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:45.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:45.400 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:45.401 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:45.409 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 44 16 69323 69307 98.43 67.5625 0.00426889 0.010152 2024-09-16T10:22:45.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:45.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:45.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:45.574 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:45.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:45.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:45.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:45 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/202461 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:45.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:45 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/202461 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:45.717 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:45.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:45.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:45 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/202461 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:45.818 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:45.861 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:45.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:45.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:45.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:45.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:46.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:46.149 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:46.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:46.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:46.298 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:46.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:46.400 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:46.410 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 45 16 70352 70336 97.6714 64.3125 0.010623 0.0102311 2024-09-16T10:22:46.439 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:46.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:46.519 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:22:46.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:46.575 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:46.594 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:46.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[29993]: pgmap v491: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 1/202461 objects degraded (0.000%); 7340/202461 objects misplaced (3.625%); 918 KiB/s, 14 objects/s recovering 2024-09-16T10:22:46.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:22:46.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:22:46.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:46.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[39470]: pgmap v491: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 1/202461 objects degraded (0.000%); 7340/202461 objects misplaced (3.625%); 918 KiB/s, 14 objects/s recovering 2024-09-16T10:22:46.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:22:46.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:22:46.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:46 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:46.624 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:46.694 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:46.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:46.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:46.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:46 smithi165 ceph-mon[32696]: pgmap v491: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 1/202461 objects degraded (0.000%); 7340/202461 objects misplaced (3.625%); 918 KiB/s, 14 objects/s recovering 2024-09-16T10:22:46.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:46 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:22:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:46 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:22:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:46 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:46.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:46.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:46.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:46.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:46.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:47.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:47.083 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:47.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:47.130 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:22:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:22:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:22:47.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:47.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:47.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:47.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:47.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:47.410 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 46 15 71214 71199 96.7207 53.9375 0.0172912 0.0103327 2024-09-16T10:22:47.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:47.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:47.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:47.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:47.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:47.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:47.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:47.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:47.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:47.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:22:47.872 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:22:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:47.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:47.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:47.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:48.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:48.063 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:48.064 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:48.099 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:48.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:48.134 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:48.134 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:48.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:48.167 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:48.168 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:48.203 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:48.203 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:48.242 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:48.242 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:48.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:48.275 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:48.275 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:48.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:48.313 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:48.313 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:48.361 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:22:48.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:48.412 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 47 16 72174 72158 95.9332 59.9375 0.018014 0.0104186 2024-09-16T10:22:48.430 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:48.532 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:48.572 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:48.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:48 smithi165 ceph-mon[32696]: pgmap v492: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 654 op/s; 1/202461 objects degraded (0.000%); 7340/202461 objects misplaced (3.625%); 648 KiB/s, 10 objects/s recovering 2024-09-16T10:22:48.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/408586377' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:48.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:48 smithi165 ceph-mon[32696]: from='client.25726 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:48.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4037859171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:48.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:48.691 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[29993]: pgmap v492: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 654 op/s; 1/202461 objects degraded (0.000%); 7340/202461 objects misplaced (3.625%); 648 KiB/s, 10 objects/s recovering 2024-09-16T10:22:48.691 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/408586377' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:48.691 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[29993]: from='client.25726 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:48.692 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4037859171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:48.692 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[39470]: pgmap v492: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 654 op/s; 1/202461 objects degraded (0.000%); 7340/202461 objects misplaced (3.625%); 648 KiB/s, 10 objects/s recovering 2024-09-16T10:22:48.692 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/408586377' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:48.692 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[39470]: from='client.25726 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:22:48.692 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4037859171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:48.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:48.755 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T10:22:47.212849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4410382875,"num_objects":67487,"num_object_clones":0,"num_object_copies":202461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":7340,"num_objects_unfound":0,"num_objects_dirty":67487,"num_whiteouts":0,"num_read":1261,"num_read_kb":1013,"num_write":67724,"num_write_kb":4306477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":26742780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68004,"ondisk_log_size":68004,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":476,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15389220,"kb_used_data":12871260,"kb_used_omap":12,"kb_used_meta":2517811,"kb_avail":734571996,"statfs":{"total":767960285184,"available":752201723904,"internally_reserved":0,"allocated":13180170240,"data_stored":13172744697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2578239025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":342818816,"num_objects":5231,"num_object_clones":0,"num_object_copies":15693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-163,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5231,"num_write_kb":334784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5308416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005502"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":547,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.891481+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:22:40.891481+0000","last_peered":"2024-09-16T10:22:40.891481+0000","last_clean":"2024-09-16T10:22:40.891481+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:22:40.891481+0000","last_undegraded":"2024-09-16T10:22:40.891481+0000","last_fullsized":"2024-09-16T10:22:40.891481+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.795446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":578,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876122+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:22:40.876122+0000","last_peered":"2024-09-16T10:22:40.876122+0000","last_clean":"2024-09-16T10:22:40.876122+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:22:40.876122+0000","last_undegraded":"2024-09-16T10:22:40.876122+0000","last_fullsized":"2024-09-16T10:22:40.876122+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.680873+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":525,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.878905+0000","last_change":"2024-09-16T10:22:02.564509+0000","last_active":"2024-09-16T10:22:40.878905+0000","last_peered":"2024-09-16T10:22:40.878905+0000","last_clean":"2024-09-16T10:22:40.878905+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:22:40.878905+0000","last_undegraded":"2024-09-16T10:22:40.878905+0000","last_fullsized":"2024-09-16T10:22:40.878905+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.564458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.564458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.322235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":520,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876288+0000","last_change":"2024-09-16T10:22:35.847447+0000","last_active":"2024-09-16T10:22:40.876288+0000","last_peered":"2024-09-16T10:22:40.876288+0000","last_clean":"2024-09-16T10:22:40.876288+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:22:40.876288+0000","last_undegraded":"2024-09-16T10:22:40.876288+0000","last_fullsized":"2024-09-16T10:22:40.876288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:35.847382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:35.847382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.635415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":512,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887211+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:22:40.887211+0000","last_peered":"2024-09-16T10:22:40.887211+0000","last_clean":"2024-09-16T10:22:40.887211+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:22:40.887211+0000","last_undegraded":"2024-09-16T10:22:40.887211+0000","last_fullsized":"2024-09-16T10:22:40.887211+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.317265+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":547,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886353+0000","last_change":"2024-09-16T10:22:30.653658+0000","last_active":"2024-09-16T10:22:40.886353+0000","last_peered":"2024-09-16T10:22:40.886353+0000","last_clean":"2024-09-16T10:22:40.886353+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:22:40.886353+0000","last_undegraded":"2024-09-16T10:22:40.886353+0000","last_fullsized":"2024-09-16T10:22:40.886353+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:30.653580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:30.653580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:40.872677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":591,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883743+0000","last_change":"2024-09-16T10:22:02.945083+0000","last_active":"2024-09-16T10:22:40.883743+0000","last_peered":"2024-09-16T10:22:40.883743+0000","last_clean":"2024-09-16T10:22:40.883743+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:22:40.883743+0000","last_undegraded":"2024-09-16T10:22:40.883743+0000","last_fullsized":"2024-09-16T10:22:40.883743+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:20:31.270548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.270548+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.423393+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":516,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876052+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:22:40.876052+0000","last_peered":"2024-09-16T10:22:40.876052+0000","last_clean":"2024-09-16T10:22:40.876052+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:22:40.876052+0000","last_undegraded":"2024-09-16T10:22:40.876052+0000","last_fullsized":"2024-09-16T10:22:40.876052+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.803939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":533,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879820+0000","last_change":"2024-09-16T10:21:19.427397+0000","last_active":"2024-09-16T10:22:40.879820+0000","last_peered":"2024-09-16T10:22:40.879820+0000","last_clean":"2024-09-16T10:22:40.879820+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:22:40.879820+0000","last_undegraded":"2024-09-16T10:22:40.879820+0000","last_fullsized":"2024-09-16T10:22:40.879820+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:19.427214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.427214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":544,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876444+0000","last_change":"2024-09-16T10:21:58.648495+0000","last_active":"2024-09-16T10:22:40.876444+0000","last_peered":"2024-09-16T10:22:40.876444+0000","last_clean":"2024-09-16T10:22:40.876444+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:22:40.876444+0000","last_undegraded":"2024-09-16T10:22:40.876444+0000","last_fullsized":"2024-09-16T10:22:40.876444+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:21:58.648414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.648414+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.343610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":517,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879347+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:22:40.879347+0000","last_peered":"2024-09-16T10:22:40.879347+0000","last_clean":"2024-09-16T10:22:40.879347+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:22:40.879347+0000","last_undegraded":"2024-09-16T10:22:40.879347+0000","last_fullsized":"2024-09-16T10:22:40.879347+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.944396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":520,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890329+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:22:40.890329+0000","last_peered":"2024-09-16T10:22:40.890329+0000","last_clean":"2024-09-16T10:22:40.890329+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:22:40.890329+0000","last_undegraded":"2024-09-16T10:22:40.890329+0000","last_fullsized":"2024-09-16T10:22:40.890329+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.729203+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":501,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879553+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:22:40.879553+0000","last_peered":"2024-09-16T10:22:40.879553+0000","last_clean":"2024-09-16T10:22:40.879553+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:22:40.879553+0000","last_undegraded":"2024-09-16T10:22:40.879553+0000","last_fullsized":"2024-09-16T10:22:40.879553+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:43.695976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":538,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876291+0000","last_change":"2024-09-16T10:21:57.684221+0000","last_active":"2024-09-16T10:22:40.876291+0000","last_peered":"2024-09-16T10:22:40.876291+0000","last_clean":"2024-09-16T10:22:40.876291+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:22:40.876291+0000","last_undegraded":"2024-09-16T10:22:40.876291+0000","last_fullsized":"2024-09-16T10:22:40.876291+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:21:57.684172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.684172+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.186220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":517,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.882863+0000","last_change":"2024-09-16T10:22:22.118256+0000","last_active":"2024-09-16T10:22:40.882863+0000","last_peered":"2024-09-16T10:22:40.882863+0000","last_clean":"2024-09-16T10:22:40.882863+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:22:40.882863+0000","last_undegraded":"2024-09-16T10:22:40.882863+0000","last_fullsized":"2024-09-16T10:22:40.882863+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.118194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.118194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:26.327315+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":528,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879264+0000","last_change":"2024-09-16T10:21:20.402146+0000","last_active":"2024-09-16T10:22:40.879264+0000","last_peered":"2024-09-16T10:22:40.879264+0000","last_clean":"2024-09-16T10:22:40.879264+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:22:40.879264+0000","last_undegraded":"2024-09-16T10:22:40.879264+0000","last_fullsized":"2024-09-16T10:22:40.879264+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.402080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.402080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":513,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876753+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:22:40.876753+0000","last_peered":"2024-09-16T10:22:40.876753+0000","last_clean":"2024-09-16T10:22:40.876753+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:22:40.876753+0000","last_undegraded":"2024-09-16T10:22:40.876753+0000","last_fullsized":"2024-09-16T10:22:40.876753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:46.754500+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":524,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.894179+0000","last_change":"2024-09-16T10:22:27.210541+0000","last_active":"2024-09-16T10:22:40.894179+0000","last_peered":"2024-09-16T10:22:40.894179+0000","last_clean":"2024-09-16T10:22:40.894179+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:22:40.894179+0000","last_undegraded":"2024-09-16T10:22:40.894179+0000","last_fullsized":"2024-09-16T10:22:40.894179+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:34.232050+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":68,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884775+0000","last_change":"2024-09-16T10:22:02.948234+0000","last_active":"2024-09-16T10:22:40.884775+0000","last_peered":"2024-09-16T10:22:40.884775+0000","last_clean":"2024-09-16T10:22:40.884775+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:22:40.884775+0000","last_undegraded":"2024-09-16T10:22:40.884775+0000","last_fullsized":"2024-09-16T10:22:40.884775+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.703324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":517,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.880785+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:22:40.880785+0000","last_peered":"2024-09-16T10:22:40.880785+0000","last_clean":"2024-09-16T10:22:40.880785+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:22:40.880785+0000","last_undegraded":"2024-09-16T10:22:40.880785+0000","last_fullsized":"2024-09-16T10:22:40.880785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.535156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":470,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886644+0000","last_change":"2024-09-16T10:21:57.358479+0000","last_active":"2024-09-16T10:22:40.886644+0000","last_peered":"2024-09-16T10:22:40.886644+0000","last_clean":"2024-09-16T10:22:40.886644+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:22:40.886644+0000","last_undegraded":"2024-09-16T10:22:40.886644+0000","last_fullsized":"2024-09-16T10:22:40.886644+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.358418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.358418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.586679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":522,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876538+0000","last_change":"2024-09-16T10:22:02.899574+0000","last_active":"2024-09-16T10:22:40.876538+0000","last_peered":"2024-09-16T10:22:40.876538+0000","last_clean":"2024-09-16T10:22:40.876538+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:22:40.876538+0000","last_undegraded":"2024-09-16T10:22:40.876538+0000","last_fullsized":"2024-09-16T10:22:40.876538+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.764435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":570,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883231+0000","last_change":"2024-09-16T10:20:57.423064+0000","last_active":"2024-09-16T10:22:40.883231+0000","last_peered":"2024-09-16T10:22:40.883231+0000","last_clean":"2024-09-16T10:22:40.883231+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:22:40.883231+0000","last_undegraded":"2024-09-16T10:22:40.883231+0000","last_fullsized":"2024-09-16T10:22:40.883231+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:20:57.422968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:57.422968+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":538,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876639+0000","last_change":"2024-09-16T10:22:05.779269+0000","last_active":"2024-09-16T10:22:40.876639+0000","last_peered":"2024-09-16T10:22:40.876639+0000","last_clean":"2024-09-16T10:22:40.876639+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:22:40.876639+0000","last_undegraded":"2024-09-16T10:22:40.876639+0000","last_fullsized":"2024-09-16T10:22:40.876639+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.779225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.779225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.069689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":520,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883944+0000","last_change":"2024-09-16T10:22:03.031843+0000","last_active":"2024-09-16T10:22:40.883944+0000","last_peered":"2024-09-16T10:22:40.883944+0000","last_clean":"2024-09-16T10:22:40.883944+0000","last_became_active":"2024-09-16T10:22:02.905797+0000","last_became_peered":"2024-09-16T10:22:02.905797+0000","last_unstale":"2024-09-16T10:22:40.883944+0000","last_undegraded":"2024-09-16T10:22:40.883944+0000","last_fullsized":"2024-09-16T10:22:40.883944+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:06.022236+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":524,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.878815+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:22:40.878815+0000","last_peered":"2024-09-16T10:22:40.878815+0000","last_clean":"2024-09-16T10:22:40.878815+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:22:40.878815+0000","last_undegraded":"2024-09-16T10:22:40.878815+0000","last_fullsized":"2024-09-16T10:22:40.878815+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:41.635078+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":544,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890935+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:22:40.890935+0000","last_peered":"2024-09-16T10:22:40.890935+0000","last_clean":"2024-09-16T10:22:40.890935+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:22:40.890935+0000","last_undegraded":"2024-09-16T10:22:40.890935+0000","last_fullsized":"2024-09-16T10:22:40.890935+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.284945+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":638,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887666+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:22:40.887666+0000","last_peered":"2024-09-16T10:22:40.887666+0000","last_clean":"2024-09-16T10:22:40.887666+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:22:40.887666+0000","last_undegraded":"2024-09-16T10:22:40.887666+0000","last_fullsized":"2024-09-16T10:22:40.887666+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:43.822126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"302'4073","reported_seq":4358,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.651331+0000","last_change":"2024-09-16T10:22:05.902394+0000","last_active":"2024-09-16T10:22:43.651331+0000","last_peered":"2024-09-16T10:22:43.651331+0000","last_clean":"2024-09-16T10:22:43.651331+0000","last_became_active":"2024-09-16T10:22:05.900726+0000","last_became_peered":"2024-09-16T10:22:05.900726+0000","last_unstale":"2024-09-16T10:22:43.651331+0000","last_undegraded":"2024-09-16T10:22:43.651331+0000","last_fullsized":"2024-09-16T10:22:43.651331+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.887079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266928128,"num_objects":4073,"num_object_clones":0,"num_object_copies":12219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":260672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":367,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879219+0000","last_change":"2024-09-16T10:22:02.898249+0000","last_active":"2024-09-16T10:22:40.879219+0000","last_peered":"2024-09-16T10:22:40.879219+0000","last_clean":"2024-09-16T10:22:40.879219+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:22:40.879219+0000","last_undegraded":"2024-09-16T10:22:40.879219+0000","last_fullsized":"2024-09-16T10:22:40.879219+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.514870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.514870+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.680773+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":539,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.878975+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:22:40.878975+0000","last_peered":"2024-09-16T10:22:40.878975+0000","last_clean":"2024-09-16T10:22:40.878975+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:22:40.878975+0000","last_undegraded":"2024-09-16T10:22:40.878975+0000","last_fullsized":"2024-09-16T10:22:40.878975+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":525,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883115+0000","last_change":"2024-09-16T10:21:29.412290+0000","last_active":"2024-09-16T10:22:40.883115+0000","last_peered":"2024-09-16T10:22:40.883115+0000","last_clean":"2024-09-16T10:22:40.883115+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:22:40.883115+0000","last_undegraded":"2024-09-16T10:22:40.883115+0000","last_fullsized":"2024-09-16T10:22:40.883115+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:29.412158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:29.412158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":626,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876824+0000","last_change":"2024-09-16T10:21:22.928090+0000","last_active":"2024-09-16T10:22:40.876824+0000","last_peered":"2024-09-16T10:22:40.876824+0000","last_clean":"2024-09-16T10:22:40.876824+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:22:40.876824+0000","last_undegraded":"2024-09-16T10:22:40.876824+0000","last_fullsized":"2024-09-16T10:22:40.876824+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:21:22.928016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:22.928016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":505,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879548+0000","last_change":"2024-09-16T10:19:18.923605+0000","last_active":"2024-09-16T10:22:40.879548+0000","last_peered":"2024-09-16T10:22:40.879548+0000","last_clean":"2024-09-16T10:22:40.879548+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:22:40.879548+0000","last_undegraded":"2024-09-16T10:22:40.879548+0000","last_fullsized":"2024-09-16T10:22:40.879548+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.923371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.923371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:06.054264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"302'4226","reported_seq":4289,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.205693+0000","last_change":"2024-09-16T10:21:56.237699+0000","last_active":"2024-09-16T10:22:43.205693+0000","last_peered":"2024-09-16T10:22:43.205693+0000","last_clean":"2024-09-16T10:22:43.205693+0000","last_became_active":"2024-09-16T10:21:56.237309+0000","last_became_peered":"2024-09-16T10:21:56.237309+0000","last_unstale":"2024-09-16T10:22:43.205693+0000","last_undegraded":"2024-09-16T10:22:43.205693+0000","last_fullsized":"2024-09-16T10:22:43.205693+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:22.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276955136,"num_objects":4226,"num_object_clones":0,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":270464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":401,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.877207+0000","last_change":"2024-09-16T10:22:25.837672+0000","last_active":"2024-09-16T10:22:40.877207+0000","last_peered":"2024-09-16T10:22:40.877207+0000","last_clean":"2024-09-16T10:22:40.877207+0000","last_became_active":"2024-09-16T10:22:05.897648+0000","last_became_peered":"2024-09-16T10:22:05.897648+0000","last_unstale":"2024-09-16T10:22:40.877207+0000","last_undegraded":"2024-09-16T10:22:40.877207+0000","last_fullsized":"2024-09-16T10:22:40.877207+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:25.837614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.837614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:54.323539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":545,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.882801+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:22:40.882801+0000","last_peered":"2024-09-16T10:22:40.882801+0000","last_clean":"2024-09-16T10:22:40.882801+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:22:40.882801+0000","last_undegraded":"2024-09-16T10:22:40.882801+0000","last_fullsized":"2024-09-16T10:22:40.882801+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:08.375110+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":489,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887136+0000","last_change":"2024-09-16T10:21:59.329899+0000","last_active":"2024-09-16T10:22:40.887136+0000","last_peered":"2024-09-16T10:22:40.887136+0000","last_clean":"2024-09-16T10:22:40.887136+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:22:40.887136+0000","last_undegraded":"2024-09-16T10:22:40.887136+0000","last_fullsized":"2024-09-16T10:22:40.887136+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.329856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.329856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:59.820276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":263,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887029+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:22:40.887029+0000","last_peered":"2024-09-16T10:22:40.887029+0000","last_clean":"2024-09-16T10:22:40.887029+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:22:40.887029+0000","last_undegraded":"2024-09-16T10:22:40.887029+0000","last_fullsized":"2024-09-16T10:22:40.887029+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":535,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876427+0000","last_change":"2024-09-16T10:22:02.899911+0000","last_active":"2024-09-16T10:22:40.876427+0000","last_peered":"2024-09-16T10:22:40.876427+0000","last_clean":"2024-09-16T10:22:40.876427+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:22:40.876427+0000","last_undegraded":"2024-09-16T10:22:40.876427+0000","last_fullsized":"2024-09-16T10:22:40.876427+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.772413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"302'4103","reported_seq":4166,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.667669+0000","last_change":"2024-09-16T10:21:56.237148+0000","last_active":"2024-09-16T10:22:43.667669+0000","last_peered":"2024-09-16T10:22:43.667669+0000","last_clean":"2024-09-16T10:22:43.667669+0000","last_became_active":"2024-09-16T10:21:56.236822+0000","last_became_peered":"2024-09-16T10:21:56.236822+0000","last_unstale":"2024-09-16T10:22:43.667669+0000","last_undegraded":"2024-09-16T10:22:43.667669+0000","last_fullsized":"2024-09-16T10:22:43.667669+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4103,"log_dups_size":0,"ondisk_log_size":4103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.079994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268894208,"num_objects":4103,"num_object_clones":0,"num_object_copies":12309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4103,"num_write_kb":262592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":393,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887001+0000","last_change":"2024-09-16T10:21:50.864307+0000","last_active":"2024-09-16T10:22:40.887001+0000","last_peered":"2024-09-16T10:22:40.887001+0000","last_clean":"2024-09-16T10:22:40.887001+0000","last_became_active":"2024-09-16T10:21:50.864098+0000","last_became_peered":"2024-09-16T10:21:50.864098+0000","last_unstale":"2024-09-16T10:22:40.887001+0000","last_undegraded":"2024-09-16T10:22:40.887001+0000","last_fullsized":"2024-09-16T10:22:40.887001+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:44.027332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":550,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876078+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:22:40.876078+0000","last_peered":"2024-09-16T10:22:40.876078+0000","last_clean":"2024-09-16T10:22:40.876078+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:22:40.876078+0000","last_undegraded":"2024-09-16T10:22:40.876078+0000","last_fullsized":"2024-09-16T10:22:40.876078+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:49.638461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":526,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.891205+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:22:40.891205+0000","last_peered":"2024-09-16T10:22:40.891205+0000","last_clean":"2024-09-16T10:22:40.891205+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:22:40.891205+0000","last_undegraded":"2024-09-16T10:22:40.891205+0000","last_fullsized":"2024-09-16T10:22:40.891205+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:42.132239+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":638,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.882622+0000","last_change":"2024-09-16T10:22:02.357682+0000","last_active":"2024-09-16T10:22:40.882622+0000","last_peered":"2024-09-16T10:22:40.882622+0000","last_clean":"2024-09-16T10:22:40.882622+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:22:40.882622+0000","last_undegraded":"2024-09-16T10:22:40.882622+0000","last_fullsized":"2024-09-16T10:22:40.882622+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:22:02.357616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.357616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.613433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":520,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884434+0000","last_change":"2024-09-16T10:22:05.051429+0000","last_active":"2024-09-16T10:22:40.884434+0000","last_peered":"2024-09-16T10:22:40.884434+0000","last_clean":"2024-09-16T10:22:40.884434+0000","last_became_active":"2024-09-16T10:22:02.890865+0000","last_became_peered":"2024-09-16T10:22:02.890865+0000","last_unstale":"2024-09-16T10:22:40.884434+0000","last_undegraded":"2024-09-16T10:22:40.884434+0000","last_fullsized":"2024-09-16T10:22:40.884434+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:29.018103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"302'4203","reported_seq":4266,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.405150+0000","last_change":"2024-09-16T10:21:56.239042+0000","last_active":"2024-09-16T10:22:43.405150+0000","last_peered":"2024-09-16T10:22:43.405150+0000","last_clean":"2024-09-16T10:22:43.405150+0000","last_became_active":"2024-09-16T10:21:56.238825+0000","last_became_peered":"2024-09-16T10:21:56.238825+0000","last_unstale":"2024-09-16T10:22:43.405150+0000","last_undegraded":"2024-09-16T10:22:43.405150+0000","last_fullsized":"2024-09-16T10:22:43.405150+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275447808,"num_objects":4203,"num_object_clones":0,"num_object_copies":12609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4203,"num_write_kb":268992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":445,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887567+0000","last_change":"2024-09-16T10:21:39.142049+0000","last_active":"2024-09-16T10:22:40.887567+0000","last_peered":"2024-09-16T10:22:40.887567+0000","last_clean":"2024-09-16T10:22:40.887567+0000","last_became_active":"2024-09-16T10:21:39.141585+0000","last_became_peered":"2024-09-16T10:21:39.141585+0000","last_unstale":"2024-09-16T10:22:40.887567+0000","last_undegraded":"2024-09-16T10:22:40.887567+0000","last_fullsized":"2024-09-16T10:22:40.887567+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.164536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":553,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879260+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:22:40.879260+0000","last_peered":"2024-09-16T10:22:40.879260+0000","last_clean":"2024-09-16T10:22:40.879260+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:22:40.879260+0000","last_undegraded":"2024-09-16T10:22:40.879260+0000","last_fullsized":"2024-09-16T10:22:40.879260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.712019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":540,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879907+0000","last_change":"2024-09-16T10:22:02.898141+0000","last_active":"2024-09-16T10:22:40.879907+0000","last_peered":"2024-09-16T10:22:40.879907+0000","last_clean":"2024-09-16T10:22:40.879907+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:22:40.879907+0000","last_undegraded":"2024-09-16T10:22:40.879907+0000","last_fullsized":"2024-09-16T10:22:40.879907+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.918613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.918613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.461798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":627,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.893727+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:22:40.893727+0000","last_peered":"2024-09-16T10:22:40.893727+0000","last_clean":"2024-09-16T10:22:40.893727+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:22:40.893727+0000","last_undegraded":"2024-09-16T10:22:40.893727+0000","last_fullsized":"2024-09-16T10:22:40.893727+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.301996+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":535,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890539+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:22:40.890539+0000","last_peered":"2024-09-16T10:22:40.890539+0000","last_clean":"2024-09-16T10:22:40.890539+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:22:40.890539+0000","last_undegraded":"2024-09-16T10:22:40.890539+0000","last_fullsized":"2024-09-16T10:22:40.890539+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.592047+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"302'4345","reported_seq":4408,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.126829+0000","last_change":"2024-09-16T10:21:56.236599+0000","last_active":"2024-09-16T10:22:43.126829+0000","last_peered":"2024-09-16T10:22:43.126829+0000","last_clean":"2024-09-16T10:22:43.126829+0000","last_became_active":"2024-09-16T10:21:56.236217+0000","last_became_peered":"2024-09-16T10:21:56.236217+0000","last_unstale":"2024-09-16T10:22:43.126829+0000","last_undegraded":"2024-09-16T10:22:43.126829+0000","last_fullsized":"2024-09-16T10:22:43.126829+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:08.065697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284753920,"num_objects":4345,"num_object_clones":0,"num_object_copies":13035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":278080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":447,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887630+0000","last_change":"2024-09-16T10:21:42.939180+0000","last_active":"2024-09-16T10:22:40.887630+0000","last_peered":"2024-09-16T10:22:40.887630+0000","last_clean":"2024-09-16T10:22:40.887630+0000","last_became_active":"2024-09-16T10:21:42.938973+0000","last_became_peered":"2024-09-16T10:21:42.938973+0000","last_unstale":"2024-09-16T10:22:40.887630+0000","last_undegraded":"2024-09-16T10:22:40.887630+0000","last_fullsized":"2024-09-16T10:22:40.887630+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.732986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.732986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":565,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.893948+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:22:40.893948+0000","last_peered":"2024-09-16T10:22:40.893948+0000","last_clean":"2024-09-16T10:22:40.893948+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:22:40.893948+0000","last_undegraded":"2024-09-16T10:22:40.893948+0000","last_fullsized":"2024-09-16T10:22:40.893948+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":548,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.880465+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:22:40.880465+0000","last_peered":"2024-09-16T10:22:40.880465+0000","last_clean":"2024-09-16T10:22:40.880465+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:22:40.880465+0000","last_undegraded":"2024-09-16T10:22:40.880465+0000","last_fullsized":"2024-09-16T10:22:40.880465+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:27.439600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":533,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887187+0000","last_change":"2024-09-16T10:22:28.126110+0000","last_active":"2024-09-16T10:22:40.887187+0000","last_peered":"2024-09-16T10:22:40.887187+0000","last_clean":"2024-09-16T10:22:40.887187+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:22:40.887187+0000","last_undegraded":"2024-09-16T10:22:40.887187+0000","last_fullsized":"2024-09-16T10:22:40.887187+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:28.126042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:28.126042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:33.509938+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":528,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.878650+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:22:40.878650+0000","last_peered":"2024-09-16T10:22:40.878650+0000","last_clean":"2024-09-16T10:22:40.878650+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:22:40.878650+0000","last_undegraded":"2024-09-16T10:22:40.878650+0000","last_fullsized":"2024-09-16T10:22:40.878650+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"302'4290","reported_seq":4353,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.705618+0000","last_change":"2024-09-16T10:21:56.233704+0000","last_active":"2024-09-16T10:22:43.705618+0000","last_peered":"2024-09-16T10:22:43.705618+0000","last_clean":"2024-09-16T10:22:43.705618+0000","last_became_active":"2024-09-16T10:21:56.233167+0000","last_became_peered":"2024-09-16T10:21:56.233167+0000","last_unstale":"2024-09-16T10:22:43.705618+0000","last_undegraded":"2024-09-16T10:22:43.705618+0000","last_fullsized":"2024-09-16T10:22:43.705618+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.678798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281149440,"num_objects":4290,"num_object_clones":0,"num_object_copies":12870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4290,"num_write_kb":274560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":393,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879135+0000","last_change":"2024-09-16T10:22:05.898575+0000","last_active":"2024-09-16T10:22:40.879135+0000","last_peered":"2024-09-16T10:22:40.879135+0000","last_clean":"2024-09-16T10:22:40.879135+0000","last_became_active":"2024-09-16T10:22:05.898258+0000","last_became_peered":"2024-09-16T10:22:05.898258+0000","last_unstale":"2024-09-16T10:22:40.879135+0000","last_undegraded":"2024-09-16T10:22:40.879135+0000","last_fullsized":"2024-09-16T10:22:40.879135+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.941141+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"282'30","reported_seq":754,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.880434+0000","last_change":"2024-09-16T10:22:04.554377+0000","last_active":"2024-09-16T10:22:40.880434+0000","last_peered":"2024-09-16T10:22:40.880434+0000","last_clean":"2024-09-16T10:22:40.880434+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:22:40.880434+0000","last_undegraded":"2024-09-16T10:22:40.880434+0000","last_fullsized":"2024-09-16T10:22:40.880434+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'30","last_scrub_stamp":"2024-09-16T10:22:04.554260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.554260+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.708122+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":533,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.877134+0000","last_change":"2024-09-16T10:22:32.891207+0000","last_active":"2024-09-16T10:22:40.877134+0000","last_peered":"2024-09-16T10:22:40.877134+0000","last_clean":"2024-09-16T10:22:40.877134+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:22:40.877134+0000","last_undegraded":"2024-09-16T10:22:40.877134+0000","last_fullsized":"2024-09-16T10:22:40.877134+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:32.891126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.891126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:53.007811+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":528,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879507+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:22:40.879507+0000","last_peered":"2024-09-16T10:22:40.879507+0000","last_clean":"2024-09-16T10:22:40.879507+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:22:40.879507+0000","last_undegraded":"2024-09-16T10:22:40.879507+0000","last_fullsized":"2024-09-16T10:22:40.879507+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:50.811766+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":509,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.891163+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:22:40.891163+0000","last_peered":"2024-09-16T10:22:40.891163+0000","last_clean":"2024-09-16T10:22:40.891163+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:22:40.891163+0000","last_undegraded":"2024-09-16T10:22:40.891163+0000","last_fullsized":"2024-09-16T10:22:40.891163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.265494+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":529,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883547+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:22:40.883547+0000","last_peered":"2024-09-16T10:22:40.883547+0000","last_clean":"2024-09-16T10:22:40.883547+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:22:40.883547+0000","last_undegraded":"2024-09-16T10:22:40.883547+0000","last_fullsized":"2024-09-16T10:22:40.883547+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.835165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"302'4097","reported_seq":4160,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.405145+0000","last_change":"2024-09-16T10:21:56.236854+0000","last_active":"2024-09-16T10:22:43.405145+0000","last_peered":"2024-09-16T10:22:43.405145+0000","last_clean":"2024-09-16T10:22:43.405145+0000","last_became_active":"2024-09-16T10:21:56.236183+0000","last_became_peered":"2024-09-16T10:21:56.236183+0000","last_unstale":"2024-09-16T10:22:43.405145+0000","last_undegraded":"2024-09-16T10:22:43.405145+0000","last_fullsized":"2024-09-16T10:22:43.405145+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.259343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268500992,"num_objects":4097,"num_object_clones":0,"num_object_copies":12291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4097,"num_write_kb":262208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":652,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.882759+0000","last_change":"2024-09-16T10:22:32.033741+0000","last_active":"2024-09-16T10:22:40.882759+0000","last_peered":"2024-09-16T10:22:40.882759+0000","last_clean":"2024-09-16T10:22:40.882759+0000","last_became_active":"2024-09-16T10:21:27.006016+0000","last_became_peered":"2024-09-16T10:21:27.006016+0000","last_unstale":"2024-09-16T10:22:40.882759+0000","last_undegraded":"2024-09-16T10:22:40.882759+0000","last_fullsized":"2024-09-16T10:22:40.882759+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:32.033672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.033672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:42.295285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":533,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879024+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:22:40.879024+0000","last_peered":"2024-09-16T10:22:40.879024+0000","last_clean":"2024-09-16T10:22:40.879024+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:22:40.879024+0000","last_undegraded":"2024-09-16T10:22:40.879024+0000","last_fullsized":"2024-09-16T10:22:40.879024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:44.556505+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":532,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879666+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:22:40.879666+0000","last_peered":"2024-09-16T10:22:40.879666+0000","last_clean":"2024-09-16T10:22:40.879666+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:22:40.879666+0000","last_undegraded":"2024-09-16T10:22:40.879666+0000","last_fullsized":"2024-09-16T10:22:40.879666+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.551901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":56,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883322+0000","last_change":"2024-09-16T10:22:02.917475+0000","last_active":"2024-09-16T10:22:40.883322+0000","last_peered":"2024-09-16T10:22:40.883322+0000","last_clean":"2024-09-16T10:22:40.883322+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:22:40.883322+0000","last_undegraded":"2024-09-16T10:22:40.883322+0000","last_fullsized":"2024-09-16T10:22:40.883322+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.736419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":60,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884129+0000","last_change":"2024-09-16T10:22:29.905776+0000","last_active":"2024-09-16T10:22:40.884129+0000","last_peered":"2024-09-16T10:22:40.884129+0000","last_clean":"2024-09-16T10:22:40.884129+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:22:40.884129+0000","last_undegraded":"2024-09-16T10:22:40.884129+0000","last_fullsized":"2024-09-16T10:22:40.884129+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:57.728247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":530,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883129+0000","last_change":"2024-09-16T10:22:29.105512+0000","last_active":"2024-09-16T10:22:40.883129+0000","last_peered":"2024-09-16T10:22:40.883129+0000","last_clean":"2024-09-16T10:22:40.883129+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:22:40.883129+0000","last_undegraded":"2024-09-16T10:22:40.883129+0000","last_fullsized":"2024-09-16T10:22:40.883129+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.105178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.105178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:42.853519+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"302'4212","reported_seq":4592,"reported_epoch":302,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:22:43.698135+0000","last_change":"2024-09-16T10:22:37.049031+0000","last_active":"2024-09-16T10:22:43.698135+0000","last_peered":"2024-09-16T10:22:43.698135+0000","last_clean":"2024-09-16T10:22:35.458157+0000","last_became_active":"2024-09-16T10:22:37.003170+0000","last_became_peered":"2024-09-16T10:22:37.003170+0000","last_unstale":"2024-09-16T10:22:43.698135+0000","last_undegraded":"2024-09-16T10:22:36.998730+0000","last_fullsized":"2024-09-16T10:22:36.878645+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276037632,"num_objects":4212,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":7340,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4212,"num_write_kb":269568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[0,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,2,5,6","objects":1},{"shards":"0,5,6","objects":3669},{"shards":"0,6","objects":541}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":381,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886524+0000","last_change":"2024-09-16T10:21:55.226523+0000","last_active":"2024-09-16T10:22:40.886524+0000","last_peered":"2024-09-16T10:22:40.886524+0000","last_clean":"2024-09-16T10:22:40.886524+0000","last_became_active":"2024-09-16T10:21:55.226200+0000","last_became_peered":"2024-09-16T10:21:55.226200+0000","last_unstale":"2024-09-16T10:22:40.886524+0000","last_undegraded":"2024-09-16T10:22:40.886524+0000","last_fullsized":"2024-09-16T10:22:40.886524+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:09.610254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":605,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.891029+0000","last_change":"2024-09-16T10:22:24.213089+0000","last_active":"2024-09-16T10:22:40.891029+0000","last_peered":"2024-09-16T10:22:40.891029+0000","last_clean":"2024-09-16T10:22:40.891029+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:22:40.891029+0000","last_undegraded":"2024-09-16T10:22:40.891029+0000","last_fullsized":"2024-09-16T10:22:40.891029+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:22:24.213008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.213008+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.043515+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":549,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890684+0000","last_change":"2024-09-16T10:22:26.226703+0000","last_active":"2024-09-16T10:22:40.890684+0000","last_peered":"2024-09-16T10:22:40.890684+0000","last_clean":"2024-09-16T10:22:40.890684+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:22:40.890684+0000","last_undegraded":"2024-09-16T10:22:40.890684+0000","last_fullsized":"2024-09-16T10:22:40.890684+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.226632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.226632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:28.255751+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1085,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:41.830062+0000","last_change":"2024-09-16T10:21:58.009040+0000","last_active":"2024-09-16T10:22:41.830062+0000","last_peered":"2024-09-16T10:22:41.830062+0000","last_clean":"2024-09-16T10:22:41.830062+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:22:41.830062+0000","last_undegraded":"2024-09-16T10:22:41.830062+0000","last_fullsized":"2024-09-16T10:22:41.830062+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:21:58.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.008984+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.997371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":540,"num_read_kb":535,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":518,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879994+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:22:40.879994+0000","last_peered":"2024-09-16T10:22:40.879994+0000","last_clean":"2024-09-16T10:22:40.879994+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:22:40.879994+0000","last_undegraded":"2024-09-16T10:22:40.879994+0000","last_fullsized":"2024-09-16T10:22:40.879994+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.025944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":530,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.880215+0000","last_change":"2024-09-16T10:21:58.547682+0000","last_active":"2024-09-16T10:22:40.880215+0000","last_peered":"2024-09-16T10:22:40.880215+0000","last_clean":"2024-09-16T10:22:40.880215+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:22:40.880215+0000","last_undegraded":"2024-09-16T10:22:40.880215+0000","last_fullsized":"2024-09-16T10:22:40.880215+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.547634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.547634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.050150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"302'4222","reported_seq":4285,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.647446+0000","last_change":"2024-09-16T10:21:56.236351+0000","last_active":"2024-09-16T10:22:43.647446+0000","last_peered":"2024-09-16T10:22:43.647446+0000","last_clean":"2024-09-16T10:22:43.647446+0000","last_became_active":"2024-09-16T10:21:56.236070+0000","last_became_peered":"2024-09-16T10:21:56.236070+0000","last_unstale":"2024-09-16T10:22:43.647446+0000","last_undegraded":"2024-09-16T10:22:43.647446+0000","last_fullsized":"2024-09-16T10:22:43.647446+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.402455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276692992,"num_objects":4222,"num_object_clones":0,"num_object_copies":12666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4222,"num_write_kb":270208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":645,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887401+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:22:40.887401+0000","last_peered":"2024-09-16T10:22:40.887401+0000","last_clean":"2024-09-16T10:22:40.887401+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:22:40.887401+0000","last_undegraded":"2024-09-16T10:22:40.887401+0000","last_fullsized":"2024-09-16T10:22:40.887401+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:45.859035+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":498,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887076+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:22:40.887076+0000","last_peered":"2024-09-16T10:22:40.887076+0000","last_clean":"2024-09-16T10:22:40.887076+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:22:40.887076+0000","last_undegraded":"2024-09-16T10:22:40.887076+0000","last_fullsized":"2024-09-16T10:22:40.887076+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:57.819113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":632,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884031+0000","last_change":"2024-09-16T10:21:57.046814+0000","last_active":"2024-09-16T10:22:40.884031+0000","last_peered":"2024-09-16T10:22:40.884031+0000","last_clean":"2024-09-16T10:22:40.884031+0000","last_became_active":"2024-09-16T10:21:35.111405+0000","last_became_peered":"2024-09-16T10:21:35.111405+0000","last_unstale":"2024-09-16T10:22:40.884031+0000","last_undegraded":"2024-09-16T10:22:40.884031+0000","last_fullsized":"2024-09-16T10:22:40.884031+0000","mapping_epoch":55,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:57.046749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.046749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:03.768702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":524,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884087+0000","last_change":"2024-09-16T10:22:02.906870+0000","last_active":"2024-09-16T10:22:40.884087+0000","last_peered":"2024-09-16T10:22:40.884087+0000","last_clean":"2024-09-16T10:22:40.884087+0000","last_became_active":"2024-09-16T10:22:02.906315+0000","last_became_peered":"2024-09-16T10:22:02.906315+0000","last_unstale":"2024-09-16T10:22:40.884087+0000","last_undegraded":"2024-09-16T10:22:40.884087+0000","last_fullsized":"2024-09-16T10:22:40.884087+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.151113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":514,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879915+0000","last_change":"2024-09-16T10:21:56.923534+0000","last_active":"2024-09-16T10:22:40.879915+0000","last_peered":"2024-09-16T10:22:40.879915+0000","last_clean":"2024-09-16T10:22:40.879915+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:22:40.879915+0000","last_undegraded":"2024-09-16T10:22:40.879915+0000","last_fullsized":"2024-09-16T10:22:40.879915+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:56.923485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.923485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:59.840767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"302'4222","reported_seq":4284,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.222962+0000","last_change":"2024-09-16T10:21:56.234554+0000","last_active":"2024-09-16T10:22:43.222962+0000","last_peered":"2024-09-16T10:22:43.222962+0000","last_clean":"2024-09-16T10:22:43.222962+0000","last_became_active":"2024-09-16T10:21:56.234289+0000","last_became_peered":"2024-09-16T10:21:56.234289+0000","last_unstale":"2024-09-16T10:22:43.222962+0000","last_undegraded":"2024-09-16T10:22:43.222962+0000","last_fullsized":"2024-09-16T10:22:43.222962+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.348123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276692992,"num_objects":4222,"num_object_clones":0,"num_object_copies":12666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4222,"num_write_kb":270208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":559,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876891+0000","last_change":"2024-09-16T10:20:31.013510+0000","last_active":"2024-09-16T10:22:40.876891+0000","last_peered":"2024-09-16T10:22:40.876891+0000","last_clean":"2024-09-16T10:22:40.876891+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:22:40.876891+0000","last_undegraded":"2024-09-16T10:22:40.876891+0000","last_fullsized":"2024-09-16T10:22:40.876891+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:20:31.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.013316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.790395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":539,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876655+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:22:40.876655+0000","last_peered":"2024-09-16T10:22:40.876655+0000","last_clean":"2024-09-16T10:22:40.876655+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:22:40.876655+0000","last_undegraded":"2024-09-16T10:22:40.876655+0000","last_fullsized":"2024-09-16T10:22:40.876655+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:31.365218+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":526,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879911+0000","last_change":"2024-09-16T10:22:22.987395+0000","last_active":"2024-09-16T10:22:40.879911+0000","last_peered":"2024-09-16T10:22:40.879911+0000","last_clean":"2024-09-16T10:22:40.879911+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:22:40.879911+0000","last_undegraded":"2024-09-16T10:22:40.879911+0000","last_fullsized":"2024-09-16T10:22:40.879911+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.987351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.987351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.275819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":469,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886816+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:22:40.886816+0000","last_peered":"2024-09-16T10:22:40.886816+0000","last_clean":"2024-09-16T10:22:40.886816+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:22:40.886816+0000","last_undegraded":"2024-09-16T10:22:40.886816+0000","last_fullsized":"2024-09-16T10:22:40.886816+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"302'4222","reported_seq":4484,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.562082+0000","last_change":"2024-09-16T10:22:03.098658+0000","last_active":"2024-09-16T10:22:43.562082+0000","last_peered":"2024-09-16T10:22:43.562082+0000","last_clean":"2024-09-16T10:22:43.562082+0000","last_became_active":"2024-09-16T10:22:02.896249+0000","last_became_peered":"2024-09-16T10:22:02.896249+0000","last_unstale":"2024-09-16T10:22:43.562082+0000","last_undegraded":"2024-09-16T10:22:43.562082+0000","last_fullsized":"2024-09-16T10:22:43.562082+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.823024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276692992,"num_objects":4222,"num_object_clones":0,"num_object_copies":12666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4222,"num_write_kb":270208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":374,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879079+0000","last_change":"2024-09-16T10:22:03.029868+0000","last_active":"2024-09-16T10:22:40.879079+0000","last_peered":"2024-09-16T10:22:40.879079+0000","last_clean":"2024-09-16T10:22:40.879079+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:22:40.879079+0000","last_undegraded":"2024-09-16T10:22:40.879079+0000","last_fullsized":"2024-09-16T10:22:40.879079+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:27.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:27.405599+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:08.103053+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":599,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.877128+0000","last_change":"2024-09-16T10:22:03.026909+0000","last_active":"2024-09-16T10:22:40.877128+0000","last_peered":"2024-09-16T10:22:40.877128+0000","last_clean":"2024-09-16T10:22:40.877128+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:22:40.877128+0000","last_undegraded":"2024-09-16T10:22:40.877128+0000","last_fullsized":"2024-09-16T10:22:40.877128+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:21:53.705644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.705644+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:05.296962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":525,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887401+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:22:40.887401+0000","last_peered":"2024-09-16T10:22:40.887401+0000","last_clean":"2024-09-16T10:22:40.887401+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:22:40.887401+0000","last_undegraded":"2024-09-16T10:22:40.887401+0000","last_fullsized":"2024-09-16T10:22:40.887401+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:00.342008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":518,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.880203+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:22:40.880203+0000","last_peered":"2024-09-16T10:22:40.880203+0000","last_clean":"2024-09-16T10:22:40.880203+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:22:40.880203+0000","last_undegraded":"2024-09-16T10:22:40.880203+0000","last_fullsized":"2024-09-16T10:22:40.880203+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:01.268083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":60,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884295+0000","last_change":"2024-09-16T10:22:05.694026+0000","last_active":"2024-09-16T10:22:40.884295+0000","last_peered":"2024-09-16T10:22:40.884295+0000","last_clean":"2024-09-16T10:22:40.884295+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:22:40.884295+0000","last_undegraded":"2024-09-16T10:22:40.884295+0000","last_fullsized":"2024-09-16T10:22:40.884295+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.693972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.693972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.767050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"302'4197","reported_seq":4260,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.660376+0000","last_change":"2024-09-16T10:21:56.913566+0000","last_active":"2024-09-16T10:22:43.660376+0000","last_peered":"2024-09-16T10:22:43.660376+0000","last_clean":"2024-09-16T10:22:43.660376+0000","last_became_active":"2024-09-16T10:21:56.913186+0000","last_became_peered":"2024-09-16T10:21:56.913186+0000","last_unstale":"2024-09-16T10:22:43.660376+0000","last_undegraded":"2024-09-16T10:22:43.660376+0000","last_fullsized":"2024-09-16T10:22:43.660376+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:58.492599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275054592,"num_objects":4197,"num_object_clones":0,"num_object_copies":12591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4197,"num_write_kb":268608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":634,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883582+0000","last_change":"2024-09-16T10:22:02.905855+0000","last_active":"2024-09-16T10:22:40.883582+0000","last_peered":"2024-09-16T10:22:40.883582+0000","last_clean":"2024-09-16T10:22:40.883582+0000","last_became_active":"2024-09-16T10:22:02.905393+0000","last_became_peered":"2024-09-16T10:22:02.905393+0000","last_unstale":"2024-09-16T10:22:40.883582+0000","last_undegraded":"2024-09-16T10:22:40.883582+0000","last_fullsized":"2024-09-16T10:22:40.883582+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:21.240387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:21.240387+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":528,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876661+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:22:40.876661+0000","last_peered":"2024-09-16T10:22:40.876661+0000","last_clean":"2024-09-16T10:22:40.876661+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:22:40.876661+0000","last_undegraded":"2024-09-16T10:22:40.876661+0000","last_fullsized":"2024-09-16T10:22:40.876661+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":509,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887651+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:22:40.887651+0000","last_peered":"2024-09-16T10:22:40.887651+0000","last_clean":"2024-09-16T10:22:40.887651+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:22:40.887651+0000","last_undegraded":"2024-09-16T10:22:40.887651+0000","last_fullsized":"2024-09-16T10:22:40.887651+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.375612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":73,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884101+0000","last_change":"2024-09-16T10:22:24.799901+0000","last_active":"2024-09-16T10:22:40.884101+0000","last_peered":"2024-09-16T10:22:40.884101+0000","last_clean":"2024-09-16T10:22:40.884101+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:22:40.884101+0000","last_undegraded":"2024-09-16T10:22:40.884101+0000","last_fullsized":"2024-09-16T10:22:40.884101+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:32.615360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876616+0000","last_change":"2024-09-16T10:22:31.886830+0000","last_active":"2024-09-16T10:22:40.876616+0000","last_peered":"2024-09-16T10:22:40.876616+0000","last_clean":"2024-09-16T10:22:40.876616+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:22:40.876616+0000","last_undegraded":"2024-09-16T10:22:40.876616+0000","last_fullsized":"2024-09-16T10:22:40.876616+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:31.886666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:31.886666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:51.243131+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"302'4141","reported_seq":4392,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.405322+0000","last_change":"2024-09-16T10:22:03.102043+0000","last_active":"2024-09-16T10:22:43.405322+0000","last_peered":"2024-09-16T10:22:43.405322+0000","last_clean":"2024-09-16T10:22:43.405322+0000","last_became_active":"2024-09-16T10:22:02.898446+0000","last_became_peered":"2024-09-16T10:22:02.898446+0000","last_unstale":"2024-09-16T10:22:43.405322+0000","last_undegraded":"2024-09-16T10:22:43.405322+0000","last_fullsized":"2024-09-16T10:22:43.405322+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4141,"log_dups_size":0,"ondisk_log_size":4141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.816221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271384576,"num_objects":4141,"num_object_clones":0,"num_object_copies":12423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4141,"num_write_kb":265024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":529,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.893858+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:22:40.893858+0000","last_peered":"2024-09-16T10:22:40.893858+0000","last_clean":"2024-09-16T10:22:40.893858+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:22:40.893858+0000","last_undegraded":"2024-09-16T10:22:40.893858+0000","last_fullsized":"2024-09-16T10:22:40.893858+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879654+0000","last_change":"2024-09-16T10:21:57.915828+0000","last_active":"2024-09-16T10:22:40.879654+0000","last_peered":"2024-09-16T10:22:40.879654+0000","last_clean":"2024-09-16T10:22:40.879654+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:22:40.879654+0000","last_undegraded":"2024-09-16T10:22:40.879654+0000","last_fullsized":"2024-09-16T10:22:40.879654+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.915773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.915773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.518442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":52,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883697+0000","last_change":"2024-09-16T10:22:02.896967+0000","last_active":"2024-09-16T10:22:40.883697+0000","last_peered":"2024-09-16T10:22:40.883697+0000","last_clean":"2024-09-16T10:22:40.883697+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:22:40.883697+0000","last_undegraded":"2024-09-16T10:22:40.883697+0000","last_fullsized":"2024-09-16T10:22:40.883697+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.735258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":550,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890574+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:22:40.890574+0000","last_peered":"2024-09-16T10:22:40.890574+0000","last_clean":"2024-09-16T10:22:40.890574+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:22:40.890574+0000","last_undegraded":"2024-09-16T10:22:40.890574+0000","last_fullsized":"2024-09-16T10:22:40.890574+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.144396+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"302'4252","reported_seq":4315,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.383147+0000","last_change":"2024-09-16T10:21:56.234684+0000","last_active":"2024-09-16T10:22:43.383147+0000","last_peered":"2024-09-16T10:22:43.383147+0000","last_clean":"2024-09-16T10:22:43.383147+0000","last_became_active":"2024-09-16T10:21:56.234401+0000","last_became_peered":"2024-09-16T10:21:56.234401+0000","last_unstale":"2024-09-16T10:22:43.383147+0000","last_undegraded":"2024-09-16T10:22:43.383147+0000","last_fullsized":"2024-09-16T10:22:43.383147+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.262243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278659072,"num_objects":4252,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4252,"num_write_kb":272128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":568,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887197+0000","last_change":"2024-09-16T10:22:26.050158+0000","last_active":"2024-09-16T10:22:40.887197+0000","last_peered":"2024-09-16T10:22:40.887197+0000","last_clean":"2024-09-16T10:22:40.887197+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:22:40.887197+0000","last_undegraded":"2024-09-16T10:22:40.887197+0000","last_fullsized":"2024-09-16T10:22:40.887197+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:22:26.050070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.050070+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:43.078689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":526,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883403+0000","last_change":"2024-09-16T10:21:59.998410+0000","last_active":"2024-09-16T10:22:40.883403+0000","last_peered":"2024-09-16T10:22:40.883403+0000","last_clean":"2024-09-16T10:22:40.883403+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:22:40.883403+0000","last_undegraded":"2024-09-16T10:22:40.883403+0000","last_fullsized":"2024-09-16T10:22:40.883403+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.998357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.998357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.630364+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":60,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.904294+0000","last_change":"2024-09-16T10:22:26.818522+0000","last_active":"2024-09-16T10:22:40.904294+0000","last_peered":"2024-09-16T10:22:40.904294+0000","last_clean":"2024-09-16T10:22:40.904294+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:22:40.904294+0000","last_undegraded":"2024-09-16T10:22:40.904294+0000","last_fullsized":"2024-09-16T10:22:40.904294+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.818439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.818439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.045673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":528,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883910+0000","last_change":"2024-09-16T10:22:02.906954+0000","last_active":"2024-09-16T10:22:40.883910+0000","last_peered":"2024-09-16T10:22:40.883910+0000","last_clean":"2024-09-16T10:22:40.883910+0000","last_became_active":"2024-09-16T10:22:02.906524+0000","last_became_peered":"2024-09-16T10:22:02.906524+0000","last_unstale":"2024-09-16T10:22:40.883910+0000","last_undegraded":"2024-09-16T10:22:40.883910+0000","last_fullsized":"2024-09-16T10:22:40.883910+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.105240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.105240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"302'4221","reported_seq":4284,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.573893+0000","last_change":"2024-09-16T10:21:56.235273+0000","last_active":"2024-09-16T10:22:43.573893+0000","last_peered":"2024-09-16T10:22:43.573893+0000","last_clean":"2024-09-16T10:22:43.573893+0000","last_became_active":"2024-09-16T10:21:56.234987+0000","last_became_peered":"2024-09-16T10:21:56.234987+0000","last_unstale":"2024-09-16T10:22:43.573893+0000","last_undegraded":"2024-09-16T10:22:43.573893+0000","last_fullsized":"2024-09-16T10:22:43.573893+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:06.384245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276627456,"num_objects":4221,"num_object_clones":0,"num_object_copies":12663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":270144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":536,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876212+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:22:40.876212+0000","last_peered":"2024-09-16T10:22:40.876212+0000","last_clean":"2024-09-16T10:22:40.876212+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:22:40.876212+0000","last_undegraded":"2024-09-16T10:22:40.876212+0000","last_fullsized":"2024-09-16T10:22:40.876212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.885016+0000","last_change":"2024-09-16T10:22:34.020368+0000","last_active":"2024-09-16T10:22:40.885016+0000","last_peered":"2024-09-16T10:22:40.885016+0000","last_clean":"2024-09-16T10:22:40.885016+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:22:40.885016+0000","last_undegraded":"2024-09-16T10:22:40.885016+0000","last_fullsized":"2024-09-16T10:22:40.885016+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:34.020246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:34.020246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.286859+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":60,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884440+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:22:40.884440+0000","last_peered":"2024-09-16T10:22:40.884440+0000","last_clean":"2024-09-16T10:22:40.884440+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:22:40.884440+0000","last_undegraded":"2024-09-16T10:22:40.884440+0000","last_fullsized":"2024-09-16T10:22:40.884440+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.537291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":513,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887363+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:22:40.887363+0000","last_peered":"2024-09-16T10:22:40.887363+0000","last_clean":"2024-09-16T10:22:40.887363+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:22:40.887363+0000","last_undegraded":"2024-09-16T10:22:40.887363+0000","last_fullsized":"2024-09-16T10:22:40.887363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.386329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"302'4262","reported_seq":4325,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:43.696724+0000","last_change":"2024-09-16T10:21:56.236925+0000","last_active":"2024-09-16T10:22:43.696724+0000","last_peered":"2024-09-16T10:22:43.696724+0000","last_clean":"2024-09-16T10:22:43.696724+0000","last_became_active":"2024-09-16T10:21:56.236619+0000","last_became_peered":"2024-09-16T10:21:56.236619+0000","last_unstale":"2024-09-16T10:22:43.696724+0000","last_undegraded":"2024-09-16T10:22:43.696724+0000","last_fullsized":"2024-09-16T10:22:43.696724+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:56.551656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279314432,"num_objects":4262,"num_object_clones":0,"num_object_copies":12786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4262,"num_write_kb":272768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":531,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879136+0000","last_change":"2024-09-16T10:21:35.618881+0000","last_active":"2024-09-16T10:22:40.879136+0000","last_peered":"2024-09-16T10:22:40.879136+0000","last_clean":"2024-09-16T10:22:40.879136+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:22:40.879136+0000","last_undegraded":"2024-09-16T10:22:40.879136+0000","last_fullsized":"2024-09-16T10:22:40.879136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:21:35.618806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.618806+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:02.397177+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":530,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.885327+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:22:40.885327+0000","last_peered":"2024-09-16T10:22:40.885327+0000","last_clean":"2024-09-16T10:22:40.885327+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:22:40.885327+0000","last_undegraded":"2024-09-16T10:22:40.885327+0000","last_fullsized":"2024-09-16T10:22:40.885327+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.060231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":628,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883487+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:22:40.883487+0000","last_peered":"2024-09-16T10:22:40.883487+0000","last_clean":"2024-09-16T10:22:40.883487+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:22:40.883487+0000","last_undegraded":"2024-09-16T10:22:40.883487+0000","last_fullsized":"2024-09-16T10:22:40.883487+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:54.008607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":522,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879330+0000","last_change":"2024-09-16T10:22:00.982299+0000","last_active":"2024-09-16T10:22:40.879330+0000","last_peered":"2024-09-16T10:22:40.879330+0000","last_clean":"2024-09-16T10:22:40.879330+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:22:40.879330+0000","last_undegraded":"2024-09-16T10:22:40.879330+0000","last_fullsized":"2024-09-16T10:22:40.879330+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:00.982240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:00.982240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:01.375176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":516,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887159+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:22:40.887159+0000","last_peered":"2024-09-16T10:22:40.887159+0000","last_clean":"2024-09-16T10:22:40.887159+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:22:40.887159+0000","last_undegraded":"2024-09-16T10:22:40.887159+0000","last_fullsized":"2024-09-16T10:22:40.887159+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.931989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":529,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886982+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:22:40.886982+0000","last_peered":"2024-09-16T10:22:40.886982+0000","last_clean":"2024-09-16T10:22:40.886982+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:22:40.886982+0000","last_undegraded":"2024-09-16T10:22:40.886982+0000","last_fullsized":"2024-09-16T10:22:40.886982+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:44.298147+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":517,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879845+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:22:40.879845+0000","last_peered":"2024-09-16T10:22:40.879845+0000","last_clean":"2024-09-16T10:22:40.879845+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:22:40.879845+0000","last_undegraded":"2024-09-16T10:22:40.879845+0000","last_fullsized":"2024-09-16T10:22:40.879845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:04.628323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":497,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883052+0000","last_change":"2024-09-16T10:16:39.634491+0000","last_active":"2024-09-16T10:22:40.883052+0000","last_peered":"2024-09-16T10:22:40.883052+0000","last_clean":"2024-09-16T10:22:40.883052+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:22:40.883052+0000","last_undegraded":"2024-09-16T10:22:40.883052+0000","last_fullsized":"2024-09-16T10:22:40.883052+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:39.634446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:39.634446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.049994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":536,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.891118+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:22:40.891118+0000","last_peered":"2024-09-16T10:22:40.891118+0000","last_clean":"2024-09-16T10:22:40.891118+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:22:40.891118+0000","last_undegraded":"2024-09-16T10:22:40.891118+0000","last_fullsized":"2024-09-16T10:22:40.891118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.198247+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":525,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883759+0000","last_change":"2024-09-16T10:22:02.040477+0000","last_active":"2024-09-16T10:22:40.883759+0000","last_peered":"2024-09-16T10:22:40.883759+0000","last_clean":"2024-09-16T10:22:40.883759+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:22:40.883759+0000","last_undegraded":"2024-09-16T10:22:40.883759+0000","last_fullsized":"2024-09-16T10:22:40.883759+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.040428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.040428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.314331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":516,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886700+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:22:40.886700+0000","last_peered":"2024-09-16T10:22:40.886700+0000","last_clean":"2024-09-16T10:22:40.886700+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:22:40.886700+0000","last_undegraded":"2024-09-16T10:22:40.886700+0000","last_fullsized":"2024-09-16T10:22:40.886700+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.194439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":52,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884889+0000","last_change":"2024-09-16T10:22:02.903139+0000","last_active":"2024-09-16T10:22:40.884889+0000","last_peered":"2024-09-16T10:22:40.884889+0000","last_clean":"2024-09-16T10:22:40.884889+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:22:40.884889+0000","last_undegraded":"2024-09-16T10:22:40.884889+0000","last_fullsized":"2024-09-16T10:22:40.884889+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.701346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":529,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879814+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:22:40.879814+0000","last_peered":"2024-09-16T10:22:40.879814+0000","last_clean":"2024-09-16T10:22:40.879814+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:22:40.879814+0000","last_undegraded":"2024-09-16T10:22:40.879814+0000","last_fullsized":"2024-09-16T10:22:40.879814+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:48.765027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":532,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890501+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:22:40.890501+0000","last_peered":"2024-09-16T10:22:40.890501+0000","last_clean":"2024-09-16T10:22:40.890501+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:22:40.890501+0000","last_undegraded":"2024-09-16T10:22:40.890501+0000","last_fullsized":"2024-09-16T10:22:40.890501+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:58.559783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":672,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886759+0000","last_change":"2024-09-16T10:22:24.619056+0000","last_active":"2024-09-16T10:22:40.886759+0000","last_peered":"2024-09-16T10:22:40.886759+0000","last_clean":"2024-09-16T10:22:40.886759+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:22:40.886759+0000","last_undegraded":"2024-09-16T10:22:40.886759+0000","last_fullsized":"2024-09-16T10:22:40.886759+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.618991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.618991+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:31.047734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876698+0000","last_change":"2024-09-16T10:22:34.860226+0000","last_active":"2024-09-16T10:22:40.876698+0000","last_peered":"2024-09-16T10:22:40.876698+0000","last_clean":"2024-09-16T10:22:40.876698+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:22:40.876698+0000","last_undegraded":"2024-09-16T10:22:40.876698+0000","last_fullsized":"2024-09-16T10:22:40.876698+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:34.860161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:34.860161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:38.752001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":543,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.886415+0000","last_change":"2024-09-16T10:21:56.344549+0000","last_active":"2024-09-16T10:22:40.886415+0000","last_peered":"2024-09-16T10:22:40.886415+0000","last_clean":"2024-09-16T10:22:40.886415+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:22:40.886415+0000","last_undegraded":"2024-09-16T10:22:40.886415+0000","last_fullsized":"2024-09-16T10:22:40.886415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:56.344472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.344472+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.689637+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":531,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.884686+0000","last_change":"2024-09-16T10:22:02.891069+0000","last_active":"2024-09-16T10:22:40.884686+0000","last_peered":"2024-09-16T10:22:40.884686+0000","last_clean":"2024-09-16T10:22:40.884686+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:22:40.884686+0000","last_undegraded":"2024-09-16T10:22:40.884686+0000","last_fullsized":"2024-09-16T10:22:40.884686+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.101314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":635,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879676+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:22:40.879676+0000","last_peered":"2024-09-16T10:22:40.879676+0000","last_clean":"2024-09-16T10:22:40.879676+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:22:40.879676+0000","last_undegraded":"2024-09-16T10:22:40.879676+0000","last_fullsized":"2024-09-16T10:22:40.879676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.951237+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":512,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890694+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:22:40.890694+0000","last_peered":"2024-09-16T10:22:40.890694+0000","last_clean":"2024-09-16T10:22:40.890694+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:22:40.890694+0000","last_undegraded":"2024-09-16T10:22:40.890694+0000","last_fullsized":"2024-09-16T10:22:40.890694+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:41.183832+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":542,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876391+0000","last_change":"2024-09-16T10:22:10.878557+0000","last_active":"2024-09-16T10:22:40.876391+0000","last_peered":"2024-09-16T10:22:40.876391+0000","last_clean":"2024-09-16T10:22:40.876391+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:22:40.876391+0000","last_undegraded":"2024-09-16T10:22:40.876391+0000","last_fullsized":"2024-09-16T10:22:40.876391+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.735967+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":516,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.876867+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:22:40.876867+0000","last_peered":"2024-09-16T10:22:40.876867+0000","last_clean":"2024-09-16T10:22:40.876867+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:22:40.876867+0000","last_undegraded":"2024-09-16T10:22:40.876867+0000","last_fullsized":"2024-09-16T10:22:40.876867+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:40.688843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":517,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879108+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:22:40.879108+0000","last_peered":"2024-09-16T10:22:40.879108+0000","last_clean":"2024-09-16T10:22:40.879108+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:22:40.879108+0000","last_undegraded":"2024-09-16T10:22:40.879108+0000","last_fullsized":"2024-09-16T10:22:40.879108+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.522521+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":504,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883214+0000","last_change":"2024-09-16T10:20:54.392034+0000","last_active":"2024-09-16T10:22:40.883214+0000","last_peered":"2024-09-16T10:22:40.883214+0000","last_clean":"2024-09-16T10:22:40.883214+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:22:40.883214+0000","last_undegraded":"2024-09-16T10:22:40.883214+0000","last_fullsized":"2024-09-16T10:22:40.883214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.391970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.391970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":571,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879369+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:22:40.879369+0000","last_peered":"2024-09-16T10:22:40.879369+0000","last_clean":"2024-09-16T10:22:40.879369+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:22:40.879369+0000","last_undegraded":"2024-09-16T10:22:40.879369+0000","last_fullsized":"2024-09-16T10:22:40.879369+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.584296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":533,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883272+0000","last_change":"2024-09-16T10:22:25.105281+0000","last_active":"2024-09-16T10:22:40.883272+0000","last_peered":"2024-09-16T10:22:40.883272+0000","last_clean":"2024-09-16T10:22:40.883272+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:22:40.883272+0000","last_undegraded":"2024-09-16T10:22:40.883272+0000","last_fullsized":"2024-09-16T10:22:40.883272+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:25.105215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.105215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:33.272207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":473,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887414+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:22:40.887414+0000","last_peered":"2024-09-16T10:22:40.887414+0000","last_clean":"2024-09-16T10:22:40.887414+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:22:40.887414+0000","last_undegraded":"2024-09-16T10:22:40.887414+0000","last_fullsized":"2024-09-16T10:22:40.887414+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:43.285375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":496,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890150+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:22:40.890150+0000","last_peered":"2024-09-16T10:22:40.890150+0000","last_clean":"2024-09-16T10:22:40.890150+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:22:40.890150+0000","last_undegraded":"2024-09-16T10:22:40.890150+0000","last_fullsized":"2024-09-16T10:22:40.890150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.132782+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":523,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883490+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:22:40.883490+0000","last_peered":"2024-09-16T10:22:40.883490+0000","last_clean":"2024-09-16T10:22:40.883490+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:22:40.883490+0000","last_undegraded":"2024-09-16T10:22:40.883490+0000","last_fullsized":"2024-09-16T10:22:40.883490+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:53.285825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":525,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.879854+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:22:40.879854+0000","last_peered":"2024-09-16T10:22:40.879854+0000","last_clean":"2024-09-16T10:22:40.879854+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:22:40.879854+0000","last_undegraded":"2024-09-16T10:22:40.879854+0000","last_fullsized":"2024-09-16T10:22:40.879854+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.996569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":525,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883500+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:22:40.883500+0000","last_peered":"2024-09-16T10:22:40.883500+0000","last_clean":"2024-09-16T10:22:40.883500+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:22:40.883500+0000","last_undegraded":"2024-09-16T10:22:40.883500+0000","last_fullsized":"2024-09-16T10:22:40.883500+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":520,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887561+0000","last_change":"2024-09-16T10:21:31.395288+0000","last_active":"2024-09-16T10:22:40.887561+0000","last_peered":"2024-09-16T10:22:40.887561+0000","last_clean":"2024-09-16T10:22:40.887561+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:22:40.887561+0000","last_undegraded":"2024-09-16T10:22:40.887561+0000","last_fullsized":"2024-09-16T10:22:40.887561+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.395222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.395222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:52.580946+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":552,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.882712+0000","last_change":"2024-09-16T10:21:58.334088+0000","last_active":"2024-09-16T10:22:40.882712+0000","last_peered":"2024-09-16T10:22:40.882712+0000","last_clean":"2024-09-16T10:22:40.882712+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:22:40.882712+0000","last_undegraded":"2024-09-16T10:22:40.882712+0000","last_fullsized":"2024-09-16T10:22:40.882712+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:21:58.334025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.334025+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.713313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":522,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.891556+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:22:40.891556+0000","last_peered":"2024-09-16T10:22:40.891556+0000","last_clean":"2024-09-16T10:22:40.891556+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:22:40.891556+0000","last_undegraded":"2024-09-16T10:22:40.891556+0000","last_fullsized":"2024-09-16T10:22:40.891556+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:51.302753+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":516,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887178+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:22:40.887178+0000","last_peered":"2024-09-16T10:22:40.887178+0000","last_clean":"2024-09-16T10:22:40.887178+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:22:40.887178+0000","last_undegraded":"2024-09-16T10:22:40.887178+0000","last_fullsized":"2024-09-16T10:22:40.887178+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:22:41.864975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":282,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.882655+0000","last_change":"2024-09-16T10:22:02.898982+0000","last_active":"2024-09-16T10:22:40.882655+0000","last_peered":"2024-09-16T10:22:40.882655+0000","last_clean":"2024-09-16T10:22:40.882655+0000","last_became_active":"2024-09-16T10:22:02.898630+0000","last_became_peered":"2024-09-16T10:22:02.898630+0000","last_unstale":"2024-09-16T10:22:40.882655+0000","last_undegraded":"2024-09-16T10:22:40.882655+0000","last_fullsized":"2024-09-16T10:22:40.882655+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:20.058989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":513,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.880602+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:22:40.880602+0000","last_peered":"2024-09-16T10:22:40.880602+0000","last_clean":"2024-09-16T10:22:40.880602+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:22:40.880602+0000","last_undegraded":"2024-09-16T10:22:40.880602+0000","last_fullsized":"2024-09-16T10:22:40.880602+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.593004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":524,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.883039+0000","last_change":"2024-09-16T10:20:58.426297+0000","last_active":"2024-09-16T10:22:40.883039+0000","last_peered":"2024-09-16T10:22:40.883039+0000","last_clean":"2024-09-16T10:22:40.883039+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:22:40.883039+0000","last_undegraded":"2024-09-16T10:22:40.883039+0000","last_fullsized":"2024-09-16T10:22:40.883039+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.426235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.426235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":548,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.887673+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:22:40.887673+0000","last_peered":"2024-09-16T10:22:40.887673+0000","last_clean":"2024-09-16T10:22:40.887673+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:22:40.887673+0000","last_undegraded":"2024-09-16T10:22:40.887673+0000","last_fullsized":"2024-09-16T10:22:40.887673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.159732+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":531,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-16T10:22:40.890849+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:22:40.890849+0000","last_peered":"2024-09-16T10:22:40.890849+0000","last_clean":"2024-09-16T10:22:40.890849+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:22:40.890849+0000","last_undegraded":"2024-09-16T10:22:40.890849+0000","last_fullsized":"2024-09-16T10:22:40.890849+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.158658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4409786368,"num_objects":67288,"num_object_clones":0,"num_object_copies":201864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":7340,"num_objects_unfound":0,"num_objects_dirty":67288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67288,"num_write_kb":4306432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":395,"num_bytes_recovered":25886720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13258719232,"data_stored":13258719232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67288,"ondisk_log_size":67288,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":540,"num_read_kb":535,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234097,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2762660,"kb_used_data":2416336,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90982492,"statfs":{"total":95995035648,"available":93166071808,"internally_reserved":0,"allocated":2474328064,"data_stored":2473403100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528437,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124168,"kb_used_data":804216,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92620984,"statfs":{"total":95995035648,"available":94843887616,"internally_reserved":0,"allocated":823517184,"data_stored":822615698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790075,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2702240,"kb_used_data":2386448,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91042912,"statfs":{"total":95995035648,"available":93227941888,"internally_reserved":0,"allocated":2443722752,"data_stored":2442798916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986304,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1111656,"kb_used_data":805592,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92633496,"statfs":{"total":95995035648,"available":94856699904,"internally_reserved":0,"allocated":824926208,"data_stored":823992995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215237,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2506236,"kb_used_data":2127976,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91238916,"statfs":{"total":95995035648,"available":93428649984,"internally_reserved":0,"allocated":2179047424,"data_stored":2178099650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":387316168},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":985360,"kb_used_data":823680,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92759792,"statfs":{"total":95995035648,"available":94986027008,"internally_reserved":0,"allocated":843448320,"data_stored":842539457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574990,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2797436,"kb_used_data":2423340,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":90947716,"statfs":{"total":95995035648,"available":93130461184,"internally_reserved":0,"allocated":2481500160,"data_stored":2480547207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738516,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1399464,"kb_used_data":1083672,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92345688,"statfs":{"total":95995035648,"available":94561984512,"internally_reserved":0,"allocated":1109680128,"data_stored":1108747674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1113653248,"data_stored":1113653248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2514812928,"data_stored":2514812928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":846921728,"data_stored":846921728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2200502272,"data_stored":2200502272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":827195392,"data_stored":827195392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2452160512,"data_stored":2452160512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":822673408,"data_stored":822673408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2480799744,"data_stored":2480799744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:48.761 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T10:22:48.762 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-16T10:22:48.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:48.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:48.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:48.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:48.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:49.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:22:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:22:49.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:49.108 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:49.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:49.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:49.254 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:49.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:49.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:49.381 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:49.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:49.412 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 48 16 73198 73182 95.2678 64 0.0487582 0.0104802 2024-09-16T10:22:49.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:49.523 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:49.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:49.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:49.670 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:49.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:49.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[29993]: from='client.25729 v1:172.21.15.63:0/2700365690' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[29993]: from='client.25726 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[29993]: osdmap e303: 8 total, 8 up, 8 in 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:49.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[39470]: from='client.25729 v1:172.21.15.63:0/2700365690' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:22:49.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:22:49.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:22:49.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:49.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[39470]: from='client.25726 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:49.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[39470]: osdmap e303: 8 total, 8 up, 8 in 2024-09-16T10:22:49.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:49.772 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:49 smithi165 ceph-mon[32696]: from='client.25729 v1:172.21.15.63:0/2700365690' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:22:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:49 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:22:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:49 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:22:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:49 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:22:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:49 smithi165 ceph-mon[32696]: from='client.25726 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:22:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:49 smithi165 ceph-mon[32696]: osdmap e303: 8 total, 8 up, 8 in 2024-09-16T10:22:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:49.813 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:49.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:49.914 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:49.967 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:49.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:50.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:50.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:50.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:50.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:50.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:50.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:50.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:50.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:50.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:50.413 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 49 15 74329 74314 94.767 70.75 0.0569917 0.0105469 2024-09-16T10:22:50.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:50.532 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:50.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:50.633 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:50.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:50.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:50 smithi063 ceph-mon[29993]: pgmap v494: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 1.2 MiB/s, 18 objects/s recovering 2024-09-16T10:22:50.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:50 smithi063 ceph-mon[39470]: pgmap v494: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 1.2 MiB/s, 18 objects/s recovering 2024-09-16T10:22:50.781 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:50.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:50 smithi165 ceph-mon[32696]: pgmap v494: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 1.2 MiB/s, 18 objects/s recovering 2024-09-16T10:22:50.825 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:50.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:50.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:50.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:51.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:51.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:51.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:51.143 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:51.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:51.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:51.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:51.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:51.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:51.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:51.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:51.413 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 50 16 75426 75410 94.2417 68.5 0.00541853 0.0106049 2024-09-16T10:22:51.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:51.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:51.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:51.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:51 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:51.558 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:51 smithi063 ceph-mon[29993]: osdmap e304: 8 total, 8 up, 8 in 2024-09-16T10:22:51.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:51 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:51.558 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:51 smithi063 ceph-mon[39470]: osdmap e304: 8 total, 8 up, 8 in 2024-09-16T10:22:51.579 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:51.624 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:51.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:51.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:51.729 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:51.771 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:51.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:51.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:51 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:51.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:51 smithi165 ceph-mon[32696]: osdmap e304: 8 total, 8 up, 8 in 2024-09-16T10:22:51.868 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:22:51.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:51.969 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:51.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:51.989 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50949 8.a 8:59c98f0f:::benchmark_data_smithi063_101290_object50948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.71989559400001, 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0382288049999999, 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:51.990 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50949 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:51.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264644+0000", 2024-09-16T10:22:51.992 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.978e-06 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264667+0000", 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3649999999999999e-05 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264681+0000", 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3423e-05 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264688+0000", 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1010000000000003e-06 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265626+0000", 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00093838999999999999 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265747+0000", 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120319 2024-09-16T10:22:51.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.267052+0000", 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013057990000000001 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.281048+0000", 2024-09-16T10:22:51.995 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013995452 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283470+0000", 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024224540000000001 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283482+0000", 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1581e-05 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.996 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302818+0000", 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0193361350000001 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302871+0000", 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2522999999999998e-05 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50954 8.a 8:57f268c8:::benchmark_data_smithi063_101290_object50953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.71653516699999, 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0425692, 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50954 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.998 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:51.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268004+0000", 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1050000000000002e-06 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268031+0000", 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6468000000000001e-05 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268041+0000", 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0532e-05 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268048+0000", 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7259999999999996e-06 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.270471+0000", 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024233789999999998 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275200+0000", 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047289130000000004 2024-09-16T10:22:52.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275360+0000", 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000159531 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284217+0000", 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088568970000000007 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.002 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284226+0000", 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3559999999999999e-06 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.303128+0000", 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0189021700000001 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304057+0000", 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00092923599999999995 2024-09-16T10:22:52.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310454+0000", 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063961260000000002 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310571+0000", 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117761 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50965 8.a 8:5e5d9df4:::benchmark_data_smithi063_101290_object50964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.70607543200001, 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0323928659999999, 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.005 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50965 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278464+0000", 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.968e-06 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278491+0000", 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7061999999999999e-05 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278500+0000", 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4970000000000007e-06 2024-09-16T10:22:52.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278507+0000", 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5139999999999998e-06 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284249+0000", 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0057418360000000002 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.008 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284255+0000", 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4529999999999999e-06 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304143+0000", 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.019888039 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304299+0000", 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155651 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.009 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304463+0000", 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016370800000000001 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308275+0000", 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038120049999999998 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310595+0000", 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023207620000000001 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310754+0000", 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015848500000000001 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310795+0000", 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1655e-05 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310855+0000", 2024-09-16T10:22:52.011 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9231000000000003e-05 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50996 8.a 8:5adb42d1:::benchmark_data_smithi063_101290_object50995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.67944559099999, 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005949921, 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.012 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50996 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:22:52.013 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305093+0000", 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.779e-06 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305117+0000", 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3370000000000002e-05 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.014 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305127+0000", 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0284e-05 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305134+0000", 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9659999999999999e-06 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305192+0000", 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7587000000000003e-05 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305201+0000", 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2590000000000006e-06 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306679+0000", 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0014777290000001 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306790+0000", 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111675 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306942+0000", 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015151300000000001 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310399+0000", 2024-09-16T10:22:52.017 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034570529999999999 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310904+0000", 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00050482900000000004 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310943+0000", 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9048000000000003e-05 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.018 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310974+0000", 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1164999999999997e-05 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311042+0000", 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7663999999999995e-05 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.020 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51001 8.a 8:59406931:::benchmark_data_smithi063_101290_object51000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.020 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:22:52.020 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.67532847199999, 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0021313270000001, 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51001 2024-09-16T10:22:52.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.022 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309210+0000", 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2589999999999999e-06 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309227+0000", 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6765999999999998e-05 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309235+0000", 2024-09-16T10:22:52.023 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1969999999999996e-06 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309240+0000", 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6290000000000001e-06 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309279+0000", 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9199999999999997e-05 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309285+0000", 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8810000000000001e-06 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308323+0000", 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99903792999999996 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308464+0000", 2024-09-16T10:22:52.025 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014168000000000001 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308635+0000", 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000170383 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310405+0000", 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001770134 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.026 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311104+0000", 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069940900000000001 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311178+0000", 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3911000000000004e-05 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311237+0000", 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9113000000000001e-05 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.027 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311340+0000", 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102835 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51015 8.a 8:5f39a03a:::benchmark_data_smithi063_101290_object51014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.65902250299999, 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99112391399999999, 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.028 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51015 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325516+0000", 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-07 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.030 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325537+0000", 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1900999999999999e-05 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325554+0000", 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6937999999999999e-05 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.031 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325562+0000", 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3830000000000003e-06 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325628+0000", 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6418e-05 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.032 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325640+0000", 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2157000000000001e-05 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311387+0000", 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98574625999999999 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.033 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311498+0000", 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111768 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311624+0000", 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012560199999999999 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316394+0000", 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047700279999999999 2024-09-16T10:22:52.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316457+0000", 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2819999999999998e-05 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316471+0000", 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4236000000000001e-05 2024-09-16T10:22:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316522+0000", 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1001000000000002e-05 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316639+0000", 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116641 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51028 8.a 8:504c9504:::benchmark_data_smithi063_101290_object51027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.650640506, 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98302855600000005, 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51028 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333898+0000", 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.375e-06 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333916+0000", 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8236e-05 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333925+0000", 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.585e-06 2024-09-16T10:22:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333931+0000", 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1070000000000001e-06 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334057+0000", 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012610900000000001 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334083+0000", 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5327e-05 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.312988+0000", 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97890501699999999 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313110+0000", 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012285099999999999 2024-09-16T10:22:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313254+0000", 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014338900000000001 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316650+0000", 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033961529999999998 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316706+0000", 2024-09-16T10:22:52.042 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5878000000000003e-05 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316760+0000", 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4502000000000001e-05 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316833+0000", 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2632000000000005e-05 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.043 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316925+0000", 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2394999999999999e-05 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51060 8.a 8:500fc7dc:::benchmark_data_smithi063_101290_object51059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.627640372, 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96027958700000005, 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.044 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51060 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:22:52.045 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356899+0000", 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.821e-06 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356917+0000", 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7972999999999999e-05 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.046 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356926+0000", 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9679999999999995e-06 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356931+0000", 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356997+0000", 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6079000000000002e-05 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.047 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.357005+0000", 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7569999999999993e-06 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314700+0000", 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95769431000000005 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314827+0000", 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012752799999999999 2024-09-16T10:22:52.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314995+0000", 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016789100000000001 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316982+0000", 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986603 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.049 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316999+0000", 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7319000000000001e-05 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317035+0000", 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5655000000000003e-05 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317088+0000", 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3278e-05 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317177+0000", 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8486999999999995e-05 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51073 8.a 8:53b2a9fa:::benchmark_data_smithi063_101290_object51072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.616510809, 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.957136019, 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.051 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51073 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.052 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368028+0000", 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5939999999999999e-06 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368053+0000", 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5130000000000002e-05 2024-09-16T10:22:52.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368066+0000", 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2397e-05 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368074+0000", 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2789999999999998e-06 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368143+0000", 2024-09-16T10:22:52.054 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9558e-05 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368154+0000", 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0869999999999999e-05 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317228+0000", 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.949073273 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.055 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317346+0000", 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011878400000000001 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317477+0000", 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013062600000000001 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322458+0000", 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049812980000000003 2024-09-16T10:22:52.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322513+0000", 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4561999999999997e-05 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325080+0000", 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025674819999999998 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.057 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325125+0000", 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5065000000000003e-05 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325163+0000", 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7101000000000003e-05 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51087 8.a 8:51205303:::benchmark_data_smithi063_101290_object51086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.058 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.60555327599999, 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94635271099999996, 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51087 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.059 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378986+0000", 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9850000000000001e-06 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379013+0000", 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6911e-05 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379029+0000", 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5928999999999999e-05 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379037+0000", 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7100000000000007e-06 2024-09-16T10:22:52.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379115+0000", 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8634999999999995e-05 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379128+0000", 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2903e-05 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.062 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319060+0000", 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93993237200000002 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319187+0000", 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012621 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319358+0000", 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017167699999999999 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.063 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322553+0000", 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003194625 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325067+0000", 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025139170000000001 2024-09-16T10:22:52.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325170+0000", 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102948 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.065 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325265+0000", 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5570999999999997e-05 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325337+0000", 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1317999999999997e-05 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51105 8.a 8:558fcaa8:::benchmark_data_smithi063_101290_object51104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:22:52.066 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.59166102099999, 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93274633699999998, 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51105 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.067 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.068 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392878+0000", 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.516e-06 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392900+0000", 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2575e-05 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392913+0000", 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2196e-05 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392920+0000", 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7170000000000008e-06 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393546+0000", 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00062592699999999997 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.070 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393555+0000", 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7809999999999994e-06 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320780+0000", 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92722495100000002 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320895+0000", 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115375 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.071 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.321051+0000", 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155482 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325345+0000", 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042937840000000001 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.072 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325384+0000", 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9261999999999998e-05 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325522+0000", 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0001383 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325584+0000", 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-05 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325623+0000", 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8940999999999999e-05 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51170 8.a 8:55be5a3b:::benchmark_data_smithi063_101290_object51169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.53571314000001, 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87700012299999996, 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.074 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51170 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.075 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448826+0000", 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7749999999999999e-06 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448849+0000", 2024-09-16T10:22:52.076 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3298999999999998e-05 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448867+0000", 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7467999999999999e-05 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448874+0000", 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-06 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.077 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448936+0000", 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1383999999999994e-05 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448947+0000", 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1756000000000001e-05 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322695+0000", 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87374773699999997 2024-09-16T10:22:52.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322825+0000", 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000129579 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322994+0000", 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016874899999999999 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.079 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325349+0000", 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002355514 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325668+0000", 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000319034 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325723+0000", 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5376e-05 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325775+0000", 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1924e-05 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325824+0000", 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8918000000000002e-05 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51179 8.a 8:5e35a76f:::benchmark_data_smithi063_101290_object51178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.471417809, 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82446452000000003, 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51179 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.082 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513122+0000", 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5579999999999999e-06 2024-09-16T10:22:52.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513149+0000", 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7101e-05 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513166+0000", 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7017e-05 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.084 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513175+0000", 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6209999999999993e-06 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513268+0000", 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3287000000000003e-05 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513279+0000", 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0889e-05 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325865+0000", 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.812586527 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325989+0000", 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123347 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.326103+0000", 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011419199999999999 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331852+0000", 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005749466 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331901+0000", 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8854999999999999e-05 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337506+0000", 2024-09-16T10:22:52.087 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005604876 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337547+0000", 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0961999999999998e-05 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337584+0000", 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6822000000000001e-05 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51188 8.a 8:5734c940:::benchmark_data_smithi063_101290_object51187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 219.46140166699999, 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.814521685, 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51188 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.089 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.090 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523137+0000", 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.804e-06 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523159+0000", 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1447e-05 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523171+0000", 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2347000000000001e-05 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523179+0000", 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6310000000000003e-06 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523240+0000", 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1150999999999998e-05 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523248+0000", 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5190000000000003e-06 2024-09-16T10:22:52.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.327963+0000", 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.80471533100000003 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.093 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328066+0000", 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103287 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328230+0000", 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163521 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331938+0000", 2024-09-16T10:22:52.094 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037081639999999999 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331973+0000", 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5355999999999998e-05 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337597+0000", 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056237809999999996 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.095 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337637+0000", 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.019e-05 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337657+0000", 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0156000000000001e-05 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.096 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51286 8.1 8:897e7f6e:::benchmark_data_smithi063_101290_object51285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.66751070699999, 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.983161858, 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51286 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.097 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317028+0000", 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.141e-06 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317054+0000", 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6186e-05 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.099 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317065+0000", 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.08e-05 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317072+0000", 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3520000000000001e-06 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317688+0000", 2024-09-16T10:22:52.100 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061638600000000002 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317698+0000", 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.346e-06 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292346+0000", 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97464779999999995 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.101 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292464+0000", 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118582 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292563+0000", 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9047000000000002e-05 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296570+0000", 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040065379999999996 2024-09-16T10:22:52.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296637+0000", 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7547999999999998e-05 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300071+0000", 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003433298 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300139+0000", 2024-09-16T10:22:52.103 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8702999999999996e-05 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300188+0000", 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9131000000000001e-05 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51312 8.1 8:8f097c26:::benchmark_data_smithi063_101290_object51311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.64847705599999, 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96891611600000005, 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.105 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51312 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.106 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336063+0000", 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3949999999999999e-06 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336103+0000", 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0754000000000003e-05 2024-09-16T10:22:52.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336138+0000", 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4662000000000001e-05 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336160+0000", 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1752000000000001e-05 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.108 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336277+0000", 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117586 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336288+0000", 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0852999999999999e-05 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294143+0000", 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95785426600000001 2024-09-16T10:22:52.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.110 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.110 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.110 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294236+0000", 2024-09-16T10:22:52.110 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3010000000000006e-05 2024-09-16T10:22:52.110 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:52.111 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.111 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.111 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:52.111 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294323+0000", 2024-09-16T10:22:52.111 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7619000000000003e-05 2024-09-16T10:22:52.112 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.112 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.112 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.112 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298376+0000", 2024-09-16T10:22:52.112 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004053175 2024-09-16T10:22:52.112 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.112 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300258+0000", 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018817300000000001 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304863+0000", 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046049539999999996 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:22:52.113 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3490000000000004e-05 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304976+0000", 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9870000000000002e-05 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.114 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51316 8.1 8:858e6b14:::benchmark_data_smithi063_101290_object51315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.64715707900001, 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96772945600000004, 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51316 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.115 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337381+0000", 2024-09-16T10:22:52.116 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.192e-06 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337441+0000", 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9988000000000002e-05 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337452+0000", 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0857e-05 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.117 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337460+0000", 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2649999999999999e-06 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337520+0000", 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0377999999999997e-05 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337530+0000", 2024-09-16T10:22:52.118 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0416e-05 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295200+0000", 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95766991499999998 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295307+0000", 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010660199999999999 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:52.119 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295446+0000", 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013906699999999999 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300014+0000", 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045680249999999999 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300329+0000", 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031521199999999999 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305018+0000", 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046893569999999999 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305073+0000", 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4243000000000002e-05 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305110+0000", 2024-09-16T10:22:52.121 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6939e-05 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51363 8.1 8:83ceb830:::benchmark_data_smithi063_101290_object51362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.54563105400001, 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.86632090799999995, 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.122 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51363 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438908+0000", 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.454e-06 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.124 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438931+0000", 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3197000000000002e-05 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438945+0000", 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3703000000000001e-05 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438952+0000", 2024-09-16T10:22:52.125 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5259999999999996e-06 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439023+0000", 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0457000000000007e-05 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439034+0000", 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1141000000000001e-05 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.126 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296689+0000", 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.85765570300000005 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296811+0000", 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121229 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296957+0000", 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014621999999999999 2024-09-16T10:22:52.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300288+0000", 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033314 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300372+0000", 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3277999999999998e-05 2024-09-16T10:22:52.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305147+0000", 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047752990000000002 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305194+0000", 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6789999999999998e-05 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305227+0000", 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3510999999999998e-05 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51388 8.1 8:8d4128ae:::benchmark_data_smithi063_101290_object51387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:52.130 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.52091622, 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.84172315399999997, 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51388 2024-09-16T10:22:52.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.132 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463623+0000", 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463644+0000", 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0937999999999998e-05 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463654+0000", 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0390999999999999e-05 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463661+0000", 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2400000000000001e-06 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463714+0000", 2024-09-16T10:22:52.134 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3059999999999997e-05 2024-09-16T10:22:52.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.135 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.135 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463722+0000", 2024-09-16T10:22:52.135 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4769999999999997e-06 2024-09-16T10:22:52.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298409+0000", 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.83468682000000005 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298629+0000", 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220258 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:52.136 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298774+0000", 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145012 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300340+0000", 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001566352 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302114+0000", 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001773518 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305261+0000", 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003147721 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305310+0000", 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9051000000000002e-05 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.138 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305344+0000", 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3751000000000003e-05 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51396 8.1 8:8a8d0cb2:::benchmark_data_smithi063_101290_object51395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 217.502124119, 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82304446200000003, 2024-09-16T10:22:52.139 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:22:52.140 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:22:52.140 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:22:52.140 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:22:52.140 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:22:52.140 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:22:52.140 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51396 2024-09-16T10:22:52.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.141 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482414+0000", 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2300000000000001e-06 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482433+0000", 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8876e-05 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482445+0000", 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2170999999999999e-05 2024-09-16T10:22:52.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482453+0000", 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8939999999999993e-06 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482504+0000", 2024-09-16T10:22:52.143 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0698000000000001e-05 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482512+0000", 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1319999999999994e-06 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300411+0000", 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.81789900699999996 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300560+0000", 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014893799999999999 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300718+0000", 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015829700000000001 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:22:52.145 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302254+0000", 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015359429999999999 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302509+0000", 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025458499999999999 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305379+0000", 2024-09-16T10:22:52.146 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028699329999999999 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305426+0000", 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7330000000000003e-05 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305458+0000", 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1427999999999998e-05 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.147 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.148 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.148 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:22:52.148 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:22:52.148 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:22:52.197 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:52.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:52.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:52.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:52.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:52.413 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 51 16 76320 76304 93.4894 55.875 0.0152192 0.0106901 2024-09-16T10:22:52.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:52.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:52.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:52.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:52.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:52.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:52.682 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:52.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:52 smithi063 ceph-mon[29993]: pgmap v496: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:52.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:52 smithi063 ceph-mon[39470]: pgmap v496: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:52.741 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:52.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:52.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:52 smithi165 ceph-mon[32696]: pgmap v496: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:22:52.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:52.883 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:52.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:52.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:52.985 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:53.023 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:53.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:22:53.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:53.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:53.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:53.236 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:53.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:22:53.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:53.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:53.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:53.400 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 1 pg_num to 21 2024-09-16T10:22:53.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 52 16 77299 77283 92.8674 61.1875 0.0246379 0.0107584 2024-09-16T10:22:53.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:53.467 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:53.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:53.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:53.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/438759450' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[29993]: from='client.45683 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/438759450' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[39470]: from='client.45683 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:53.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:53.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:53.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/438759450' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:22:53.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:53 smithi165 ceph-mon[32696]: from='client.45683 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-16T10:22:53.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:53 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:22:53.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:22:53.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:53.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:53.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:53.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:53.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:54.100 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:54.142 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:54.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:54.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:54.277 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:54.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:54.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:54.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:54.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:54.414 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 53 16 78373 78357 92.3816 67.125 0.0155514 0.0108189 2024-09-16T10:22:54.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:54.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:54.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:54.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:54.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:54.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:54 smithi063 ceph-mon[39470]: pgmap v497: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 627 op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 616 KiB/s, 9 objects/s recovering 2024-09-16T10:22:54.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:54 smithi063 ceph-mon[39470]: from='client.45683 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T10:22:54.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:54 smithi063 ceph-mon[39470]: osdmap e305: 8 total, 8 up, 8 in 2024-09-16T10:22:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:54 smithi063 ceph-mon[29993]: pgmap v497: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 627 op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 616 KiB/s, 9 objects/s recovering 2024-09-16T10:22:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:54 smithi063 ceph-mon[29993]: from='client.45683 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T10:22:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:54 smithi063 ceph-mon[29993]: osdmap e305: 8 total, 8 up, 8 in 2024-09-16T10:22:54.786 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:54.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:54 smithi165 ceph-mon[32696]: pgmap v497: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 627 op/s; 1/217500 objects degraded (0.000%); 7187/217500 objects misplaced (3.304%); 616 KiB/s, 9 objects/s recovering 2024-09-16T10:22:54.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:54 smithi165 ceph-mon[32696]: from='client.45683 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T10:22:54.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:54 smithi165 ceph-mon[32696]: osdmap e305: 8 total, 8 up, 8 in 2024-09-16T10:22:54.821 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:54.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:54.923 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:54.967 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:54.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:55.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:55.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:55.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:55.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:55.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:55.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:55.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:55.375 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:55.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:55.414 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 54 16 79484 79468 91.9558 69.4375 0.0133804 0.0108678 2024-09-16T10:22:55.477 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:55.522 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:55.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:55.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:55.637 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:22:55.637 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:22:55.637 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:55 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:55.638 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:55.638 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:55 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:55.638 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:55.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:55.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:55.680 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:22:55.680 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:22:55.720 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:22:55.721 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:22:55.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:55.762 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:22:55.763 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:22:55.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:55.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:55.797 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:22:55.798 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:22:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:55 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:22:55.830 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:22:55.868 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:22:55.868 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:22:55.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:55.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:55.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:55.914 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:22:55.914 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:22:56.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:56.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:56.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:56.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:56.183 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:56.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:56.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:56.414 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 55 16 80566 80550 91.5133 67.625 0.0125443 0.0109219 2024-09-16T10:22:56.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:56.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:56.558 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:56.600 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:56.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:56.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[29993]: pgmap v499: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 747 B/s rd, 84 MiB/s wr, 1.34k op/s; 1/232845 objects degraded (0.000%); 7033/232845 objects misplaced (3.020%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:22:56.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T10:22:56.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[29993]: osdmap e306: 8 total, 8 up, 8 in 2024-09-16T10:22:56.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1191514188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:56.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/232845 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:56.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[39470]: pgmap v499: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 747 B/s rd, 84 MiB/s wr, 1.34k op/s; 1/232845 objects degraded (0.000%); 7033/232845 objects misplaced (3.020%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:22:56.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T10:22:56.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[39470]: osdmap e306: 8 total, 8 up, 8 in 2024-09-16T10:22:56.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1191514188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:56.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:56 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/232845 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:56.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:56 smithi165 ceph-mon[32696]: pgmap v499: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 747 B/s rd, 84 MiB/s wr, 1.34k op/s; 1/232845 objects degraded (0.000%); 7033/232845 objects misplaced (3.020%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:22:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:56 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T10:22:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:56 smithi165 ceph-mon[32696]: osdmap e306: 8 total, 8 up, 8 in 2024-09-16T10:22:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1191514188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:22:56.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:56 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/232845 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:22:56.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:56.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:56.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:56.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:56.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:57.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:57.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:57.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:57.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:57.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:57.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:57.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:57.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:57.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:57.414 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 56 16 81598 81582 91.0309 64.5 0.0653392 0.0109794 2024-09-16T10:22:57.415 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:22:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:22:57.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:57.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:57.630 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:57.680 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:57.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[29993]: osdmap e307: 8 total, 8 up, 8 in 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:57.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[39470]: osdmap e307: 8 total, 8 up, 8 in 2024-09-16T10:22:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:22:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:22:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:22:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:22:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:57 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:57.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:57 smithi165 ceph-mon[32696]: osdmap e307: 8 total, 8 up, 8 in 2024-09-16T10:22:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:57 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:22:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:57 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:22:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:57 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:22:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:57 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:22:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:57 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:57.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:57 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-16T10:22:57.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:57.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:57.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:22:57.944 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:57.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:22:58.046 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:58.057 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:58.119 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:58.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:58.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:58.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:58.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:58.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:58.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:58.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:58.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 57 16 82714 82698 90.6575 69.75 0.0567188 0.0110246 2024-09-16T10:22:58.484 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:58.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:22:58.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:22:58.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: pgmap v502: 168 pgs: 10 unknown, 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 852 op/s; 1/232845 objects degraded (0.000%); 7033/232845 objects misplaced (3.020%); 820 KiB/s, 12 objects/s recovering 2024-09-16T10:22:58.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T10:22:58.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: osdmap e308: 8 total, 8 up, 8 in 2024-09-16T10:22:58.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:22:58.525 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: pgmap v502: 168 pgs: 10 unknown, 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 852 op/s; 1/232845 objects degraded (0.000%); 7033/232845 objects misplaced (3.020%); 820 KiB/s, 12 objects/s recovering 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: osdmap e308: 8 total, 8 up, 8 in 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:22:58.526 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:58 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:22:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:58.527 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:58.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:58.628 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:58.674 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:58.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:58.685 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:22:58.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:22:58.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:22:58.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:22:58.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: pgmap v502: 168 pgs: 10 unknown, 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 852 op/s; 1/232845 objects degraded (0.000%); 7033/232845 objects misplaced (3.020%); 820 KiB/s, 12 objects/s recovering 2024-09-16T10:22:58.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T10:22:58.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: osdmap e308: 8 total, 8 up, 8 in 2024-09-16T10:22:58.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:22:58.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:22:58.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:22:58.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:58 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:22:58.704 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:22:58.704 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:22:58.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:58.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:58.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:22:58.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:22:58.944 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:58.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:22:59.046 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:59.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:59.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:59.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:22:59.249 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:59.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:22:59.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:22:59.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:59.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:59.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:22:59.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 58 15 83732 83717 90.1925 63.6875 0.0111698 0.0110799 2024-09-16T10:22:59.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:22:59.539 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:22:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:59.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:22:59.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:22:59.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:59.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[29993]: osdmap e309: 8 total, 8 up, 8 in 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:22:59.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[39470]: osdmap e309: 8 total, 8 up, 8 in 2024-09-16T10:22:59.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:59.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:22:59 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:22:59.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:22:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:59 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:22:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:59 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:22:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:59 smithi165 ceph-mon[32696]: osdmap e309: 8 total, 8 up, 8 in 2024-09-16T10:22:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:22:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:22:59 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:22:59.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:59.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:22:59.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:22:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:22:59.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:22:59.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:00.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:00.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:23:00.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:00.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:00.101 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:23:00.168 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:00.236 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:00.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:00.337 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:00.391 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:00.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:00.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 59 16 84712 84696 89.7009 61.1875 0.010769 0.0111407 2024-09-16T10:23:00.473 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:23:00.492 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:00.556 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:00.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:00.657 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:00.701 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:00.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:23:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: pgmap v505: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 8.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:23:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-16T10:23:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.12 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.c starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: osdmap e310: 8 total, 8 up, 8 in 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.f starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.b starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:23:00.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:23:00.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:23:00.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1792407944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:00.804 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:00.850 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:00.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:00.852 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:23:00.852 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: pgmap v505: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 8.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:23:00.852 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-16T10:23:00.852 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.852 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.12 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.c starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: osdmap e310: 8 total, 8 up, 8 in 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.f starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.b starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1792407944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:00.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: pgmap v505: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 8.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.14 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.14 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.d starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.12 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.11 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.12 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.12 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.855 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.10 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.e starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.c starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.c starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: osdmap e310: 8 total, 8 up, 8 in 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.f starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.f starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.13 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.b starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.b starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:23:00.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1792407944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:00.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:00.994 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:00.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:01.095 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:01.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:01.235 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:01.282 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:01.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:01.384 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:01.407 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:01.414 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:23:01.414077+0000 min lat: 0.00102889 max lat: 2.20645 avg lat: 0.0111873 2024-09-16T10:23:01.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:23:01.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 60 16 85803 85787 89.3423 68.1875 0.0197418 0.0111873 2024-09-16T10:23:01.430 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:01.439 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:01.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:01.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:01.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:23:01.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:01.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1792407944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:23:01.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[29993]: osdmap e311: 8 total, 8 up, 8 in 2024-09-16T10:23:01.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/249027 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:01.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[29993]: osdmap e312: 8 total, 8 up, 8 in 2024-09-16T10:23:01.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:23:01.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:01.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1792407944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:23:01.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[39470]: osdmap e311: 8 total, 8 up, 8 in 2024-09-16T10:23:01.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/249027 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:01.570 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:01 smithi063 ceph-mon[39470]: osdmap e312: 8 total, 8 up, 8 in 2024-09-16T10:23:01.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:01.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:01.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:01.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:01.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:01 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:23:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:01 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1792407944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:23:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:01 smithi165 ceph-mon[32696]: osdmap e311: 8 total, 8 up, 8 in 2024-09-16T10:23:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:01 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/249027 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:01 smithi165 ceph-mon[32696]: osdmap e312: 8 total, 8 up, 8 in 2024-09-16T10:23:01.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:01.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:01.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:01.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:01.977 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:01.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:02.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:02.126 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:02.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:02.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:02.239 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:23:02.239 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:23:02.239 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:23:02.240 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:23:02.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:02.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:02.282 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:23:02.282 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:23:02.334 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:23:02.334 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:23:02.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:02.381 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:23:02.381 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:23:02.394 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:02.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:02.397 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:02.397 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:02.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 61 16 86769 86753 88.8673 60.375 0.0123886 0.0112466 2024-09-16T10:23:02.430 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:23:02.430 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:23:02.467 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:23:02.467 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:23:02.496 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:02.504 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:23:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:23:02.542 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:23:02.543 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:23:02.560 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:02.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:02.576 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:23:02.576 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:23:02.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:02.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:02.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:02.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:23:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:23:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: pgmap v509: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 9.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:23:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:23:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:23:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 1.b scrub ok 2024-09-16T10:23:02.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:23:02.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:23:02.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:23:02.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: 1.10 scrub ok 2024-09-16T10:23:02.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1926211451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:02.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:02.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:02.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: pgmap v509: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 9.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 1.b scrub ok 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:23:02.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: 1.10 scrub ok 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1926211451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: pgmap v509: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 9.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 1.b scrub ok 2024-09-16T10:23:02.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:23:02.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:23:02.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:23:02.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: 1.10 scrub ok 2024-09-16T10:23:02.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1926211451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:02.965 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:02.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:03.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:03.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:03.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:03.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:03.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:03.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:03.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:03.371 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:23:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:03.376 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:03.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:03.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 62 16 87892 87876 88.5659 70.1875 0.0105831 0.0112851 2024-09-16T10:23:03.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:03.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:03.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:03.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:03.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:03.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:03.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:03 smithi063 ceph-mon[29993]: osdmap e313: 8 total, 8 up, 8 in 2024-09-16T10:23:03.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:03 smithi063 ceph-mon[29993]: 1.a scrub starts 2024-09-16T10:23:03.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:03 smithi063 ceph-mon[29993]: 1.a scrub ok 2024-09-16T10:23:03.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:03 smithi063 ceph-mon[39470]: osdmap e313: 8 total, 8 up, 8 in 2024-09-16T10:23:03.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:03 smithi063 ceph-mon[39470]: 1.a scrub starts 2024-09-16T10:23:03.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:03 smithi063 ceph-mon[39470]: 1.a scrub ok 2024-09-16T10:23:03.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:03.776 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:03.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:03 smithi165 ceph-mon[32696]: osdmap e313: 8 total, 8 up, 8 in 2024-09-16T10:23:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:03 smithi165 ceph-mon[32696]: 1.a scrub starts 2024-09-16T10:23:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:03 smithi165 ceph-mon[32696]: 1.a scrub ok 2024-09-16T10:23:03.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:03.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:03.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:03.982 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:04.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:23:04.009 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:04.096 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:04.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:04.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:04.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:04.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:04.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:04.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:04.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:04.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 63 15 88863 88848 88.1238 60.75 0.010771 0.0113429 2024-09-16T10:23:04.453 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:04.500 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:04.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:04.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:04.626 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:04.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:04.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:04.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:04.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[29993]: pgmap v511: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%) 2024-09-16T10:23:04.764 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[29993]: 6.15 scrub starts 2024-09-16T10:23:04.764 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[29993]: 6.15 scrub ok 2024-09-16T10:23:04.764 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[29993]: 3.8 scrub starts 2024-09-16T10:23:04.764 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[29993]: 3.8 scrub ok 2024-09-16T10:23:04.764 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[39470]: pgmap v511: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%) 2024-09-16T10:23:04.764 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[39470]: 6.15 scrub starts 2024-09-16T10:23:04.765 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[39470]: 6.15 scrub ok 2024-09-16T10:23:04.765 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[39470]: 3.8 scrub starts 2024-09-16T10:23:04.765 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:04 smithi063 ceph-mon[39470]: 3.8 scrub ok 2024-09-16T10:23:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:04.786 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:04.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:04 smithi165 ceph-mon[32696]: pgmap v511: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 8 remapped+peering, 2 unknown, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1/249027 objects degraded (0.000%); 6851/249027 objects misplaced (2.751%) 2024-09-16T10:23:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:04 smithi165 ceph-mon[32696]: 6.15 scrub starts 2024-09-16T10:23:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:04 smithi165 ceph-mon[32696]: 6.15 scrub ok 2024-09-16T10:23:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:04 smithi165 ceph-mon[32696]: 3.8 scrub starts 2024-09-16T10:23:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:04 smithi165 ceph-mon[32696]: 3.8 scrub ok 2024-09-16T10:23:04.887 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:04.933 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:04.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:05.035 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:05.048 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:23:05.048 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:23:05.048 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 313, 2024-09-16T10:23:05.048 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:23:05.048 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:23:05.048 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:23:02.498306+0000", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:22:01.866838+0000", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:23:05.049 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 9, 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:23:05.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:05.051 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 21, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 21, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 21, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 21, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 21, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.b", 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 275, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 274, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 274, 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:23:05.052 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "308", 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "275", 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "306", 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:05.053 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:05.054 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:05.055 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5199999809265137, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.8999999761581421, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5199999809265137, 2024-09-16T10:23:05.056 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.8999999761581421, 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:05.057 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:23:05.058 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:05.059 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:05.060 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:05.061 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:23:05.062 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:05.063 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:05.064 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:05.065 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:05.065 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:05.065 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:05.065 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:05.065 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:05.065 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:05.065 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:05.066 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:05.067 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:05.068 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.069 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.070 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:05.072 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:05.073 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:05.074 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.25, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.25, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.25, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.25, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.25, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.25, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.25, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.25, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:23:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 9, 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_2", 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:21:54.923489+0000", 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:23:05.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "279", 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:05.092 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:05.093 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:05.094 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.095 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2, 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2, 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.096 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 310, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.097 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.098 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:23:05.100 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.101 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 311, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.103 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:23:05.105 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.106 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 309, 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.107 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.108 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.109 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.110 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:23:05.111 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 309, 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.112 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:23:05.113 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:23:05.114 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.115 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 308, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:05.116 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.118 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:23:05.119 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 311, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.120 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.122 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.122 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.122 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:05.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:05.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:05.123 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:23:05.125 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 309, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:05.127 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.129 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 308, 2024-09-16T10:23:05.130 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.132 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.134 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:59.788186+0000", 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:23:01.692724+0000", 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:06.859608+0000", 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:37.120902+0000", 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:02.701780+0000", 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:26.467679+0000", 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:48.316098+0000", 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:23.134000+0000", 2024-09-16T10:23:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [ 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "9.b", 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: 2, 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: 6 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.10", 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 6, 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.13", 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.145 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [ 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.c", 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: 5, 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.d", 2024-09-16T10:23:05.146 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: 2, 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: 6 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.e", 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: 6 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.f", 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.11", 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: 5, 2024-09-16T10:23:05.148 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.12", 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.13", 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.149 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: 6 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.14", 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: 7, 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: 5 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:05.150 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "9.b", 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: 6 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:05.151 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:23:05.152 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:23:05.153 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:23:05.154 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:23:05.154 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:23:05.154 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:05.154 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:23:05.224 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:05.260 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:05.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:05.361 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:05.403 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:05.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:05.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 64 16 89778 89762 87.6394 57.125 0.0124427 0.0114049 2024-09-16T10:23:05.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:23:05.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:05.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:05.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:05.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:05.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:05.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:05.764 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:05.964 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:05.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:05.965 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:05 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/217821283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:23:05.965 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:05 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/217821283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:23:05.966 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:23:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:05 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/217821283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:23:06.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:06.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:06.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:06.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:06.245 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:06.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:06.347 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:06.393 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:06.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:06.415 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 65 16 90778 90762 87.2526 62.5 0.00603792 0.0114557 2024-09-16T10:23:06.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:06.522 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:06.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:06.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:06.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:06.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: pgmap v512: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 57 MiB/s wr, 905 op/s; 1/264708 objects degraded (0.000%); 6688/264708 objects misplaced (2.527%); 930 KiB/s, 14 objects/s recovering 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2809595520' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/264708 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2809595520' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: osdmap e314: 8 total, 8 up, 8 in 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: pgmap v512: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 57 MiB/s wr, 905 op/s; 1/264708 objects degraded (0.000%); 6688/264708 objects misplaced (2.527%); 930 KiB/s, 14 objects/s recovering 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-09-16T10:23:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2809595520' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:23:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/264708 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:23:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2809595520' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: osdmap e314: 8 total, 8 up, 8 in 2024-09-16T10:23:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:06 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:23:06.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:06.797 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:06.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:06.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:06.942 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:06.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:07.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: pgmap v512: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 57 MiB/s wr, 905 op/s; 1/264708 objects degraded (0.000%); 6688/264708 objects misplaced (2.527%); 930 KiB/s, 14 objects/s recovering 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2809595520' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/264708 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:23:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2809595520' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: osdmap e314: 8 total, 8 up, 8 in 2024-09-16T10:23:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:06 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:23:07.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:07.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:07.126 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:07.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:07.176 INFO:tasks.thrashosds.thrasher:j is {'epoch': 313, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:23:02.498306+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:22:01.866838+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 275, 'last_epoch_started': 274, 'last_epoch_clean': 274, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '308', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '275', 'last_force_op_resend_preluminous': '306', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5199999809265137, 'score_stable': 1.899999976158142, 'optimal_score': 1, 'raw_score_acting': 1.5199999809265137, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-16T10:21:54.923489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 310, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 311, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 308, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 311, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 308, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:59.788186+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:23:01.692724+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:06.859608+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:37.120902+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:02.701780+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:26.467679+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:48.316098+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:23.134000+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '9.b', 'osds': [2, 1, 6]}], 'pg_upmap_items': [{'pgid': '3.10', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.c', 'osds': [3, 5, 7]}, {'pgid': '1.d', 'osds': [2, 4, 6]}, {'pgid': '1.e', 'osds': [0, 3, 6]}, {'pgid': '1.f', 'osds': [6, 3, 7]}, {'pgid': '1.11', 'osds': [0, 5, 2]}, {'pgid': '1.12', 'osds': [0, 4, 7]}, {'pgid': '1.13', 'osds': [1, 4, 6]}, {'pgid': '1.14', 'osds': [3, 7, 5]}, {'pgid': '9.b', 'osds': [0, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:23:07.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:23:07.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:07.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:07.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:07.358 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:07.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:07.411 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:23:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:23:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:23:07.417 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 66 16 91898 91882 86.991 70 0.0336377 0.0114899 2024-09-16T10:23:07.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:07.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:07.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:07.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:07.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:07.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:07.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:07.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:07.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:07.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:07.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:07.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:07.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:07.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:08.032 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:08.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:08.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:08.134 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:08.200 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:08.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:08.206 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:23:08.206 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:23:08.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:23:08.206 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:23:08.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:65096 7.e 7:7bb27b3e:::benchmark_data_smithi063_73086_object65095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 460.63601709900001, 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.125924595, 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 65096 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563904+0000", 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563905+0000", 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.606e-06 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563926+0000", 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1009000000000002e-05 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563936+0000", 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6460000000000005e-06 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563942+0000", 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1589999999999998e-06 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.563991+0000", 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9279e-05 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564082+0000", 2024-09-16T10:23:08.210 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0890000000000006e-05 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.564170+0000", 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7435000000000006e-05 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.565824+0000", 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001654345 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.211 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.566103+0000", 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027876700000000002 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689664+0000", 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123561491 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689778+0000", 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113677 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:27.689828+0000", 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0290999999999997e-05 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70300 7.8 7:10588a9d:::benchmark_data_smithi063_73086_object70299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 456.29682766299999, 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12214414, 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.213 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.214 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.214 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.214 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.214 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.214 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70300 2024-09-16T10:23:08.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903093+0000", 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903095+0000", 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9099999999999999e-06 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903119+0000", 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3856999999999999e-05 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903132+0000", 2024-09-16T10:23:08.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2873e-05 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903139+0000", 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9739999999999996e-06 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903195+0000", 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6121e-05 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903309+0000", 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113962 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.903448+0000", 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013888799999999999 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905996+0000", 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025485479999999999 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909244+0000", 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032479420000000002 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.024965+0000", 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.115720986 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025185+0000", 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220177 2024-09-16T10:23:08.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025237+0000", 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1901999999999999e-05 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:70303 7.8 7:176924a2:::benchmark_data_smithi063_73086_object70302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 456.29452720799998, 2024-09-16T10:23:08.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120210123, 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 70303 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905394+0000", 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905396+0000", 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5900000000000002e-06 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.222 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905422+0000", 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6364999999999999e-05 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905438+0000", 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5795e-05 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.905448+0000", 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8020000000000003e-06 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906033+0000", 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00058453800000000001 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906161+0000", 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012883799999999999 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.906300+0000", 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013873499999999999 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.909282+0000", 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029820070000000001 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:31.910327+0000", 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001044332 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025288+0000", 2024-09-16T10:23:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11496116100000001 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025559+0000", 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027147 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:32.025604+0000", 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4490000000000003e-05 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72689 7.e 7:7f5699d3:::benchmark_data_smithi063_73086_object72688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 454.33043343700001, 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12073505599999999, 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72689 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869487+0000", 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869489+0000", 2024-09-16T10:23:08.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.237e-06 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869521+0000", 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2728999999999998e-05 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869531+0000", 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8989999999999996e-06 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.869539+0000", 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2209999999999997e-06 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872936+0000", 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003396296 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873090+0000", 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015414000000000001 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.873225+0000", 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013536000000000001 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878402+0000", 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0051764419999999998 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878413+0000", 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1810999999999999e-05 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.989923+0000", 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.111509897 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990168+0000", 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00024499400000000001 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.990222+0000", 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4030000000000003e-05 2024-09-16T10:23:08.232 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72696 7.e 7:7eb0d49d:::benchmark_data_smithi063_73086_object72695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 454.32713405200002, 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120396989, 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.233 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72696 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872787+0000", 2024-09-16T10:23:08.235 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.235 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.235 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872789+0000", 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.979e-06 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872820+0000", 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1522000000000001e-05 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872830+0000", 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7550000000000006e-06 2024-09-16T10:23:08.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.872837+0000", 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7399999999999998e-06 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875523+0000", 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026860759999999999 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875632+0000", 2024-09-16T10:23:08.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000109343 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.875757+0000", 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012532799999999999 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878452+0000", 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026941249999999999 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.880841+0000", 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023892340000000001 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993090+0000", 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.112248941 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993145+0000", 2024-09-16T10:23:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5043999999999997e-05 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.993184+0000", 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8902000000000001e-05 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72702 7.e 7:720829b1:::benchmark_data_smithi063_73086_object72701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:23:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 454.32187026600002, 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.119080255, 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72702 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:23:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878050+0000", 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878052+0000", 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3689999999999999e-06 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878078+0000", 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6140999999999999e-05 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878088+0000", 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0147e-05 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878096+0000", 2024-09-16T10:23:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5089999999999996e-06 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878485+0000", 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00038902000000000001 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878652+0000", 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016732800000000001 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.878733+0000", 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0873999999999999e-05 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.883480+0000", 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047467120000000002 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884355+0000", 2024-09-16T10:23:08.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087525899999999998 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997027+0000", 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11267184399999999 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997084+0000", 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.766e-05 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.997131+0000", 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6391999999999999e-05 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:72712 7.e 7:7d21703e:::benchmark_data_smithi063_73086_object72711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 454.31535015499998, 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.118747798, 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.247 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 72712 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884571+0000", 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884572+0000", 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4270000000000001e-06 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884607+0000", 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4535000000000001e-05 2024-09-16T10:23:08.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884617+0000", 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0774000000000001e-05 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.884625+0000", 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.644e-06 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.886978+0000", 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00235257 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887080+0000", 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010283 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.887167+0000", 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6706999999999995e-05 2024-09-16T10:23:08.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984059+0000", 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.096891692000000001 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:33.984099+0000", 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0516000000000003e-05 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003069+0000", 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.018970035999999999 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003200+0000", 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000130961 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:34.003318+0000", 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118106 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.44750.0:90069 7.e 7:7df0bbc7:::benchmark_data_smithi063_73086_object90068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 438.67621175300002, 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123224003, 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.44750", 2024-09-16T10:23:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 90069 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523709+0000", 2024-09-16T10:23:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523711+0000", 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6309999999999999e-06 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523734+0000", 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3002000000000001e-05 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523745+0000", 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1173e-05 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523752+0000", 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2579999999999998e-06 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523836+0000", 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3758999999999995e-05 2024-09-16T10:23:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.523945+0000", 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010883999999999999 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.524079+0000", 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013397600000000001 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:23:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640421+0000", 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116342404 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.640488+0000", 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6561000000000002e-05 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646786+0000", 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0062988599999999999 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646881+0000", 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4291999999999998e-05 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:15:49.646933+0000", 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2247000000000003e-05 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.260 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9.36b0083b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.736807016999997, 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41720589800000002, 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463115+0000", 2024-09-16T10:23:08.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7430000000000001e-06 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463140+0000", 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4477e-05 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463151+0000", 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0891e-05 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463161+0000", 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0020000000000001e-05 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464130+0000", 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00096871499999999998 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464137+0000", 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1749999999999999e-06 2024-09-16T10:23:08.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.265 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.265 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880306+0000", 2024-09-16T10:23:08.265 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41616966999999999 2024-09-16T10:23:08.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880320+0000", 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3207000000000001e-05 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9.669f9f4b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.695357403000003, 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37579170000000001, 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.266 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504565+0000", 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9400000000000001e-06 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504589+0000", 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3971000000000001e-05 2024-09-16T10:23:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504602+0000", 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2539e-05 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504610+0000", 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4460000000000001e-06 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504674+0000", 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3895999999999999e-05 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504682+0000", 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0260000000000007e-06 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880348+0000", 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37566548300000002 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880355+0000", 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3989999999999998e-06 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9.deba987b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.689196475000003, 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36966423799999998, 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.271 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510727+0000", 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8370000000000001e-06 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510777+0000", 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9957999999999998e-05 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510794+0000", 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7356000000000001e-05 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510820+0000", 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6071000000000002e-05 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.274 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510892+0000", 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1520999999999995e-05 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510901+0000", 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5920000000000001e-06 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880380+0000", 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36947966100000001 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880388+0000", 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.242e-06 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9.255df14b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.642385638999997, 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32288046799999998, 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.276 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557537+0000", 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9080000000000002e-06 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557562+0000", 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5131e-05 2024-09-16T10:23:08.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557575+0000", 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3039e-05 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557582+0000", 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0110000000000003e-06 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557855+0000", 2024-09-16T10:23:08.279 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027304500000000003 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557867+0000", 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1507e-05 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880409+0000", 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32254223399999998 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880416+0000", 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5930000000000002e-06 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9.3f093dfb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.598211059999997, 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27873467899999999, 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.281 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.282 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601712+0000", 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4609999999999998e-06 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601740+0000", 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8008999999999999e-05 2024-09-16T10:23:08.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601759+0000", 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8427000000000002e-05 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601767+0000", 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5539999999999998e-06 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601927+0000", 2024-09-16T10:23:08.284 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015967299999999999 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601935+0000", 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8040000000000002e-06 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.285 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880436+0000", 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27850148000000002 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880444+0000", 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2709999999999992e-06 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.289 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9.c91b333b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.572896114000002, 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25344918100000002, 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.290 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627028+0000", 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.948e-06 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627081+0000", 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3631999999999997e-05 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627100+0000", 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9191999999999999e-05 2024-09-16T10:23:08.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627110+0000", 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6409999999999997e-06 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627207+0000", 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6762999999999995e-05 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627215+0000", 2024-09-16T10:23:08.293 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2260000000000005e-06 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880467+0000", 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25325206 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880474+0000", 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7190000000000004e-06 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.294 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9.7e4a5ddb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.564622825999997, 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.245202226, 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:23:08.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635300+0000", 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2620000000000001e-06 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635352+0000", 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2079000000000001e-05 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635365+0000", 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3081000000000001e-05 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.298 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.298 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635373+0000", 2024-09-16T10:23:08.298 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6909999999999997e-06 2024-09-16T10:23:08.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.298 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.298 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.298 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635435+0000", 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2271000000000001e-05 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635445+0000", 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6099999999999995e-06 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880493+0000", 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.24504827800000001 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880500+0000", 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9539999999999998e-06 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.300 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9.61c3adbb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.557860267999999, 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23846684600000001, 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642062+0000", 2024-09-16T10:23:08.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0269999999999998e-06 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642104+0000", 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1473000000000001e-05 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642120+0000", 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6235e-05 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642127+0000", 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2470000000000002e-06 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642184+0000", 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6725999999999999e-05 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642192+0000", 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1200000000000002e-06 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880520+0000", 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23832803799999999 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880527+0000", 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9800000000000001e-06 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60273 9.b 9.bb30e96b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.509321812000003, 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.189954294, 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60273 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690601+0000", 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2479999999999999e-06 2024-09-16T10:23:08.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690656+0000", 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4951000000000003e-05 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690673+0000", 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6543999999999999e-05 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.308 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690680+0000", 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8329999999999994e-06 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690741+0000", 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0491000000000001e-05 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690754+0000", 2024-09-16T10:23:08.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2806000000000001e-05 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880547+0000", 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18979368599999999 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880553+0000", 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7350000000000001e-06 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60317 9.b 9.71cef2b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.476446232000001, 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.15710459700000001, 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60317 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.311 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723477+0000", 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4119999999999999e-06 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723606+0000", 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012882700000000001 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723622+0000", 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6436e-05 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723630+0000", 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.089e-06 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723701+0000", 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0729000000000007e-05 2024-09-16T10:23:08.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723710+0000", 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8140000000000007e-06 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880572+0000", 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.156862641 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880579+0000", 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6490000000000002e-06 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60326 9.b 9.d49aff0b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.466505071, 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14719354100000001, 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60326 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733418+0000", 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.385e-06 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733480+0000", 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2331999999999999e-05 2024-09-16T10:23:08.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733498+0000", 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8111e-05 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733506+0000", 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0129999999999993e-06 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733571+0000", 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4437000000000007e-05 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733582+0000", 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1073000000000001e-05 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880601+0000", 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.147019074 2024-09-16T10:23:08.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880609+0000", 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1159999999999999e-06 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60333 9.b 9.f3eeac1b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 32.462101740000001, 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14281871500000001, 2024-09-16T10:23:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60333 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:23:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737822+0000", 2024-09-16T10:23:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.649e-06 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737888+0000", 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6540999999999999e-05 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737908+0000", 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9419000000000002e-05 2024-09-16T10:23:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737917+0000", 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0790000000000007e-06 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.738071+0000", 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015447900000000001 2024-09-16T10:23:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.738079+0000", 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8590000000000005e-06 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880631+0000", 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.142552229 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:23:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880638+0000", 2024-09-16T10:23:08.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.46e-06 2024-09-16T10:23:08.327 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.327 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.327 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.327 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:08.327 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:08.327 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:23:08.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:08.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:08.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:08.416 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 67 16 92938 92922 86.6628 65 0.0135578 0.0115339 2024-09-16T10:23:08.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:08.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:08.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:08.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:08.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:08.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:08.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:08.753 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:08.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:08.855 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:08.896 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:08.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[29993]: pgmap v514: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 52 MiB/s wr, 824 op/s; 1/264708 objects degraded (0.000%); 6688/264708 objects misplaced (2.527%); 846 KiB/s, 13 objects/s recovering 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[29993]: 6.d scrub starts 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[29993]: 6.d scrub ok 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:08.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[39470]: pgmap v514: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 52 MiB/s wr, 824 op/s; 1/264708 objects degraded (0.000%); 6688/264708 objects misplaced (2.527%); 846 KiB/s, 13 objects/s recovering 2024-09-16T10:23:08.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:23:08.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:23:08.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[39470]: 6.d scrub starts 2024-09-16T10:23:08.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[39470]: 6.d scrub ok 2024-09-16T10:23:08.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:08.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:08.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:09.040 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:09.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:09.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:23:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-mon[32696]: pgmap v514: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 52 MiB/s wr, 824 op/s; 1/264708 objects degraded (0.000%); 6688/264708 objects misplaced (2.527%); 846 KiB/s, 13 objects/s recovering 2024-09-16T10:23:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:23:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:23:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-mon[32696]: 6.d scrub starts 2024-09-16T10:23:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-mon[32696]: 6.d scrub ok 2024-09-16T10:23:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:09.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:09.179 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:09.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:09.281 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:09.318 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:09.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:09.416 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 68 15 94053 94038 86.414 69.75 0.0125892 0.0115672 2024-09-16T10:23:09.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:09.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:09.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:09.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:09.597 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:09.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:09.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:23:09.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:09.772 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:09.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:09.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:09.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[39470]: osdmap e315: 8 total, 8 up, 8 in 2024-09-16T10:23:09.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:09.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[39470]: 3.1a scrub starts 2024-09-16T10:23:09.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[39470]: 3.1a scrub ok 2024-09-16T10:23:09.891 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[29993]: osdmap e315: 8 total, 8 up, 8 in 2024-09-16T10:23:09.891 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:09.891 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[29993]: 3.1a scrub starts 2024-09-16T10:23:09.891 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:09 smithi063 ceph-mon[29993]: 3.1a scrub ok 2024-09-16T10:23:09.900 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:09.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:23:09.939 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:09.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:10.028 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:23:10.028 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:23:10.041 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:09 smithi165 ceph-mon[32696]: osdmap e315: 8 total, 8 up, 8 in 2024-09-16T10:23:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:09 smithi165 ceph-mon[32696]: 3.1a scrub starts 2024-09-16T10:23:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:09 smithi165 ceph-mon[32696]: 3.1a scrub ok 2024-09-16T10:23:10.073 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:23:10.076 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:23:10.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:23:10.115 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:23:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:10.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:10.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:10.155 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:23:10.155 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:23:10.186 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:23:10.186 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:23:10.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:10.226 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:23:10.227 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:23:10.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:10.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:10.269 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:23:10.269 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:23:10.302 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:23:10.302 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:23:10.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:10.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:10.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:10.421 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 69 16 95004 94988 86.0156 59.375 0.0590086 0.0116207 2024-09-16T10:23:10.484 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:10.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:10.539 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:10.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:10.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:10.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:10.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:10.773 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:10.814 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:10.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:10.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:23:10.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[29993]: pgmap v516: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 763 B/s rd, 96 MiB/s wr, 1.53k op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:10.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:23:10.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[29993]: 6.1c scrub starts 2024-09-16T10:23:10.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:23:10.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[29993]: 6.1c scrub ok 2024-09-16T10:23:10.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2144814209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[39470]: pgmap v516: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 763 B/s rd, 96 MiB/s wr, 1.53k op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:23:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[39470]: 6.1c scrub starts 2024-09-16T10:23:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:23:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[39470]: 6.1c scrub ok 2024-09-16T10:23:10.932 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2144814209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:10.939 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:23:10.962 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:10.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:10 smithi165 ceph-mon[32696]: pgmap v516: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 763 B/s rd, 96 MiB/s wr, 1.53k op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:10 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:23:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:10 smithi165 ceph-mon[32696]: 6.1c scrub starts 2024-09-16T10:23:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:10 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:23:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:10 smithi165 ceph-mon[32696]: 6.1c scrub ok 2024-09-16T10:23:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2144814209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:11.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:11.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:11.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:11.194 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:11.233 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:11.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:11.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:11.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:11.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:11.421 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 70 16 96026 96010 85.6993 63.875 0.00814857 0.0116578 2024-09-16T10:23:11.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:11.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:11.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:11.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:11.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:11.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:11.733 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:11.774 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:11.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:11.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:11.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:11.919 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:11.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:11.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[29993]: 5.9 scrub starts 2024-09-16T10:23:11.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[29993]: 5.9 scrub ok 2024-09-16T10:23:11.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/279816 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:11.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[29993]: osdmap e316: 8 total, 8 up, 8 in 2024-09-16T10:23:11.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[39470]: 5.9 scrub starts 2024-09-16T10:23:11.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[39470]: 5.9 scrub ok 2024-09-16T10:23:11.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/279816 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:11.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:11 smithi063 ceph-mon[39470]: osdmap e316: 8 total, 8 up, 8 in 2024-09-16T10:23:12.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:12.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:11 smithi165 ceph-mon[32696]: 5.9 scrub starts 2024-09-16T10:23:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:11 smithi165 ceph-mon[32696]: 5.9 scrub ok 2024-09-16T10:23:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:11 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/279816 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:12.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:11 smithi165 ceph-mon[32696]: osdmap e316: 8 total, 8 up, 8 in 2024-09-16T10:23:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:12.065 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:12.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:12.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:12.213 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:12.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:12.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:12.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:12.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:12.421 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 71 16 97043 97027 85.3875 63.5625 0.0203639 0.0117022 2024-09-16T10:23:12.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:12.501 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:12.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:12.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:12.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:12.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[29993]: from='client.45719 v1:172.21.15.63:0/488546767' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:23:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[29993]: pgmap v518: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 839 op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 863 KiB/s, 13 objects/s recovering 2024-09-16T10:23:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[29993]: 4.13 scrub starts 2024-09-16T10:23:12.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[29993]: 4.13 scrub ok 2024-09-16T10:23:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[39470]: from='client.45719 v1:172.21.15.63:0/488546767' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:23:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[39470]: pgmap v518: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 839 op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 863 KiB/s, 13 objects/s recovering 2024-09-16T10:23:12.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[39470]: 4.13 scrub starts 2024-09-16T10:23:12.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:12 smithi063 ceph-mon[39470]: 4.13 scrub ok 2024-09-16T10:23:12.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:12.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:12.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:12.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:12.884 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:12.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:12.986 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:13.024 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:12 smithi165 ceph-mon[32696]: from='client.45719 v1:172.21.15.63:0/488546767' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:23:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:12 smithi165 ceph-mon[32696]: pgmap v518: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 839 op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 863 KiB/s, 13 objects/s recovering 2024-09-16T10:23:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:12 smithi165 ceph-mon[32696]: 4.13 scrub starts 2024-09-16T10:23:13.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:12 smithi165 ceph-mon[32696]: 4.13 scrub ok 2024-09-16T10:23:13.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:13.163 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:13.265 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:13.301 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:13.403 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:13.422 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 72 16 97995 97979 85.0279 59.5 0.00990598 0.0117561 2024-09-16T10:23:13.442 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:13.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:13.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:13.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:13.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:13.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:13.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:13.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:13.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:13.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:14.055 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:14.099 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:14.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:14.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:14.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:14.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:14.266 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-16T10:23:10.892147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6100230187,"num_objects":93272,"num_object_clones":0,"num_object_copies":279816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":6526,"num_objects_unfound":0,"num_objects_dirty":93272,"num_whiteouts":0,"num_read":1300,"num_read_kb":1063,"num_write":93522,"num_write_kb":5956838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":53679628,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71578,"ondisk_log_size":71578,"up":504,"acting":503,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":521,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20591404,"kb_used_data":17933148,"kb_used_omap":12,"kb_used_meta":2658035,"kb_avail":729369812,"statfs":{"total":767960285184,"available":746874687488,"internally_reserved":0,"allocated":18363543552,"data_stored":18355639161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2721828401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,4,10,10,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672582551,"num_objects":10263,"num_object_clones":0,"num_object_copies":30789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-325,"num_objects_unfound":0,"num_objects_dirty":10263,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":10260,"num_write_kb":656795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":10800023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.665791"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":573,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600882+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:23:08.600882+0000","last_peered":"2024-09-16T10:23:08.600882+0000","last_clean":"2024-09-16T10:23:08.600882+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:23:08.600882+0000","last_undegraded":"2024-09-16T10:23:08.600882+0000","last_fullsized":"2024-09-16T10:23:08.600882+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":602,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.903959+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:23:05.903959+0000","last_peered":"2024-09-16T10:23:05.903959+0000","last_clean":"2024-09-16T10:23:05.903959+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:23:05.903959+0000","last_undegraded":"2024-09-16T10:23:05.903959+0000","last_fullsized":"2024-09-16T10:23:05.903959+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:58.750050+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":551,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.572667+0000","last_change":"2024-09-16T10:22:02.564509+0000","last_active":"2024-09-16T10:23:08.572667+0000","last_peered":"2024-09-16T10:23:08.572667+0000","last_clean":"2024-09-16T10:23:08.572667+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:23:08.572667+0000","last_undegraded":"2024-09-16T10:23:08.572667+0000","last_fullsized":"2024-09-16T10:23:08.572667+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.564458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.564458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.322235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904310+0000","last_change":"2024-09-16T10:22:35.847447+0000","last_active":"2024-09-16T10:23:05.904310+0000","last_peered":"2024-09-16T10:23:05.904310+0000","last_clean":"2024-09-16T10:23:05.904310+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:23:05.904310+0000","last_undegraded":"2024-09-16T10:23:05.904310+0000","last_fullsized":"2024-09-16T10:23:05.904310+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:35.847382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:35.847382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.635415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":538,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591775+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:23:08.591775+0000","last_peered":"2024-09-16T10:23:08.591775+0000","last_clean":"2024-09-16T10:23:08.591775+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:23:08.591775+0000","last_undegraded":"2024-09-16T10:23:08.591775+0000","last_fullsized":"2024-09-16T10:23:08.591775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.317265+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":573,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.592347+0000","last_change":"2024-09-16T10:22:30.653658+0000","last_active":"2024-09-16T10:23:08.592347+0000","last_peered":"2024-09-16T10:23:08.592347+0000","last_clean":"2024-09-16T10:23:08.592347+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:23:08.592347+0000","last_undegraded":"2024-09-16T10:23:08.592347+0000","last_fullsized":"2024-09-16T10:23:08.592347+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:30.653580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:30.653580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:40.872677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":617,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591951+0000","last_change":"2024-09-16T10:22:02.945083+0000","last_active":"2024-09-16T10:23:08.591951+0000","last_peered":"2024-09-16T10:23:08.591951+0000","last_clean":"2024-09-16T10:23:08.591951+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:23:08.591951+0000","last_undegraded":"2024-09-16T10:23:08.591951+0000","last_fullsized":"2024-09-16T10:23:08.591951+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:20:31.270548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.270548+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.423393+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.903862+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:23:05.903862+0000","last_peered":"2024-09-16T10:23:05.903862+0000","last_clean":"2024-09-16T10:23:05.903862+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:23:05.903862+0000","last_undegraded":"2024-09-16T10:23:05.903862+0000","last_fullsized":"2024-09-16T10:23:05.903862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.803939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":565,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.892746+0000","last_change":"2024-09-16T10:23:00.881755+0000","last_active":"2024-09-16T10:23:05.892746+0000","last_peered":"2024-09-16T10:23:05.892746+0000","last_clean":"2024-09-16T10:23:05.892746+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:23:05.892746+0000","last_undegraded":"2024-09-16T10:23:05.892746+0000","last_fullsized":"2024-09-16T10:23:05.892746+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:00.881704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:00.881704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:08.353844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":568,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904678+0000","last_change":"2024-09-16T10:21:58.648495+0000","last_active":"2024-09-16T10:23:05.904678+0000","last_peered":"2024-09-16T10:23:05.904678+0000","last_clean":"2024-09-16T10:23:05.904678+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:23:05.904678+0000","last_undegraded":"2024-09-16T10:23:05.904678+0000","last_fullsized":"2024-09-16T10:23:05.904678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:21:58.648414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.648414+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890949+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:23:05.890949+0000","last_peered":"2024-09-16T10:23:05.890949+0000","last_clean":"2024-09-16T10:23:05.890949+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:23:05.890949+0000","last_undegraded":"2024-09-16T10:23:05.890949+0000","last_fullsized":"2024-09-16T10:23:05.890949+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.944396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":546,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600686+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:23:08.600686+0000","last_peered":"2024-09-16T10:23:08.600686+0000","last_clean":"2024-09-16T10:23:08.600686+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:23:08.600686+0000","last_undegraded":"2024-09-16T10:23:08.600686+0000","last_fullsized":"2024-09-16T10:23:08.600686+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.729203+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":525,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891179+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:23:05.891179+0000","last_peered":"2024-09-16T10:23:05.891179+0000","last_clean":"2024-09-16T10:23:05.891179+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:23:05.891179+0000","last_undegraded":"2024-09-16T10:23:05.891179+0000","last_fullsized":"2024-09-16T10:23:05.891179+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.926610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":562,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904666+0000","last_change":"2024-09-16T10:21:57.684221+0000","last_active":"2024-09-16T10:23:05.904666+0000","last_peered":"2024-09-16T10:23:05.904666+0000","last_clean":"2024-09-16T10:23:05.904666+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:23:05.904666+0000","last_undegraded":"2024-09-16T10:23:05.904666+0000","last_fullsized":"2024-09-16T10:23:05.904666+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:21:57.684172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.684172+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.186220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889174+0000","last_change":"2024-09-16T10:22:22.118256+0000","last_active":"2024-09-16T10:23:05.889174+0000","last_peered":"2024-09-16T10:23:05.889174+0000","last_clean":"2024-09-16T10:23:05.889174+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:23:05.889174+0000","last_undegraded":"2024-09-16T10:23:05.889174+0000","last_fullsized":"2024-09-16T10:23:05.889174+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.118194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.118194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:26.327315+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890561+0000","last_change":"2024-09-16T10:21:20.402146+0000","last_active":"2024-09-16T10:23:05.890561+0000","last_peered":"2024-09-16T10:23:05.890561+0000","last_clean":"2024-09-16T10:23:05.890561+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:23:05.890561+0000","last_undegraded":"2024-09-16T10:23:05.890561+0000","last_fullsized":"2024-09-16T10:23:05.890561+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.402080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.402080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:40.966067+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":537,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905518+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:23:05.905518+0000","last_peered":"2024-09-16T10:23:05.905518+0000","last_clean":"2024-09-16T10:23:05.905518+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:23:05.905518+0000","last_undegraded":"2024-09-16T10:23:05.905518+0000","last_fullsized":"2024-09-16T10:23:05.905518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:52.665306+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":550,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.607083+0000","last_change":"2024-09-16T10:22:27.210541+0000","last_active":"2024-09-16T10:23:08.607083+0000","last_peered":"2024-09-16T10:23:08.607083+0000","last_clean":"2024-09-16T10:23:08.607083+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:23:08.607083+0000","last_undegraded":"2024-09-16T10:23:08.607083+0000","last_fullsized":"2024-09-16T10:23:08.607083+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:34.232050+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":92,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896767+0000","last_change":"2024-09-16T10:22:02.948234+0000","last_active":"2024-09-16T10:23:05.896767+0000","last_peered":"2024-09-16T10:23:05.896767+0000","last_clean":"2024-09-16T10:23:05.896767+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:23:05.896767+0000","last_undegraded":"2024-09-16T10:23:05.896767+0000","last_fullsized":"2024-09-16T10:23:05.896767+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.703324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":543,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.580933+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:23:08.580933+0000","last_peered":"2024-09-16T10:23:08.580933+0000","last_clean":"2024-09-16T10:23:08.580933+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:23:08.580933+0000","last_undegraded":"2024-09-16T10:23:08.580933+0000","last_fullsized":"2024-09-16T10:23:08.580933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.535156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":496,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.594934+0000","last_change":"2024-09-16T10:21:57.358479+0000","last_active":"2024-09-16T10:23:08.594934+0000","last_peered":"2024-09-16T10:23:08.594934+0000","last_clean":"2024-09-16T10:23:08.594934+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:23:08.594934+0000","last_undegraded":"2024-09-16T10:23:08.594934+0000","last_fullsized":"2024-09-16T10:23:08.594934+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.358418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.358418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.586679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":546,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904810+0000","last_change":"2024-09-16T10:22:02.899574+0000","last_active":"2024-09-16T10:23:05.904810+0000","last_peered":"2024-09-16T10:23:05.904810+0000","last_clean":"2024-09-16T10:23:05.904810+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:23:05.904810+0000","last_undegraded":"2024-09-16T10:23:05.904810+0000","last_fullsized":"2024-09-16T10:23:05.904810+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:22.803930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:22.803930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.764435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":604,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.593450+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:23:08.593450+0000","last_peered":"2024-09-16T10:23:08.593450+0000","last_clean":"2024-09-16T10:23:08.593450+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:23:08.593450+0000","last_undegraded":"2024-09-16T10:23:08.593450+0000","last_fullsized":"2024-09-16T10:23:08.593450+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:07.520262+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":562,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905311+0000","last_change":"2024-09-16T10:22:05.779269+0000","last_active":"2024-09-16T10:23:05.905311+0000","last_peered":"2024-09-16T10:23:05.905311+0000","last_clean":"2024-09-16T10:23:05.905311+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:23:05.905311+0000","last_undegraded":"2024-09-16T10:23:05.905311+0000","last_fullsized":"2024-09-16T10:23:05.905311+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.779225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.779225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:07.069689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890320+0000","last_change":"2024-09-16T10:22:03.031843+0000","last_active":"2024-09-16T10:23:05.890320+0000","last_peered":"2024-09-16T10:23:05.890320+0000","last_clean":"2024-09-16T10:23:05.890320+0000","last_became_active":"2024-09-16T10:22:02.905797+0000","last_became_peered":"2024-09-16T10:22:02.905797+0000","last_unstale":"2024-09-16T10:23:05.890320+0000","last_undegraded":"2024-09-16T10:23:05.890320+0000","last_fullsized":"2024-09-16T10:23:05.890320+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:16:37.672406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:16:37.672406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:06.022236+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":550,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.572826+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:23:08.572826+0000","last_peered":"2024-09-16T10:23:08.572826+0000","last_clean":"2024-09-16T10:23:08.572826+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:23:08.572826+0000","last_undegraded":"2024-09-16T10:23:08.572826+0000","last_fullsized":"2024-09-16T10:23:08.572826+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":570,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603509+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:23:08.603509+0000","last_peered":"2024-09-16T10:23:08.603509+0000","last_clean":"2024-09-16T10:23:08.603509+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:23:08.603509+0000","last_undegraded":"2024-09-16T10:23:08.603509+0000","last_fullsized":"2024-09-16T10:23:08.603509+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":667,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.897235+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:23:05.897235+0000","last_peered":"2024-09-16T10:23:05.897235+0000","last_clean":"2024-09-16T10:23:05.897235+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:23:05.897235+0000","last_undegraded":"2024-09-16T10:23:05.897235+0000","last_fullsized":"2024-09-16T10:23:05.897235+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"315'5674","reported_seq":5985,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.695299+0000","last_change":"2024-09-16T10:22:05.902394+0000","last_active":"2024-09-16T10:23:08.695299+0000","last_peered":"2024-09-16T10:23:08.695299+0000","last_clean":"2024-09-16T10:23:08.695299+0000","last_became_active":"2024-09-16T10:22:05.900726+0000","last_became_peered":"2024-09-16T10:22:05.900726+0000","last_unstale":"2024-09-16T10:23:08.695299+0000","last_undegraded":"2024-09-16T10:23:08.695299+0000","last_fullsized":"2024-09-16T10:23:08.695299+0000","mapping_epoch":284,"log_start":"289'1538","ondisk_log_start":"289'1538","created":276,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4136,"log_dups_size":0,"ondisk_log_size":4136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371851264,"num_objects":5674,"num_object_clones":0,"num_object_copies":17022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":363136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":402,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890583+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:23:05.890583+0000","last_peered":"2024-09-16T10:23:05.890583+0000","last_clean":"2024-09-16T10:23:05.890583+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:23:05.890583+0000","last_undegraded":"2024-09-16T10:23:05.890583+0000","last_fullsized":"2024-09-16T10:23:05.890583+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:29.222578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":565,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.573347+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:23:08.573347+0000","last_peered":"2024-09-16T10:23:08.573347+0000","last_clean":"2024-09-16T10:23:08.573347+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:23:08.573347+0000","last_undegraded":"2024-09-16T10:23:08.573347+0000","last_fullsized":"2024-09-16T10:23:08.573347+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":559,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.592439+0000","last_change":"2024-09-16T10:23:03.779125+0000","last_active":"2024-09-16T10:23:08.592439+0000","last_peered":"2024-09-16T10:23:08.592439+0000","last_clean":"2024-09-16T10:23:08.592439+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:23:08.592439+0000","last_undegraded":"2024-09-16T10:23:08.592439+0000","last_fullsized":"2024-09-16T10:23:08.592439+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:03.779044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:03.779044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:21.866780+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":656,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905669+0000","last_change":"2024-09-16T10:21:22.928090+0000","last_active":"2024-09-16T10:23:05.905669+0000","last_peered":"2024-09-16T10:23:05.905669+0000","last_clean":"2024-09-16T10:23:05.905669+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:23:05.905669+0000","last_undegraded":"2024-09-16T10:23:05.905669+0000","last_fullsized":"2024-09-16T10:23:05.905669+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:21:22.928016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:22.928016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:44.692279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":538,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:07.772284+0000","last_change":"2024-09-16T10:23:07.772284+0000","last_active":"2024-09-16T10:23:07.772284+0000","last_peered":"2024-09-16T10:23:07.772284+0000","last_clean":"2024-09-16T10:23:07.772284+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:23:07.772284+0000","last_undegraded":"2024-09-16T10:23:07.772284+0000","last_fullsized":"2024-09-16T10:23:07.772284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:07.772163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:07.772163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:09.861820+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"314'5817","reported_seq":5904,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.157095+0000","last_change":"2024-09-16T10:21:56.237699+0000","last_active":"2024-09-16T10:23:08.157095+0000","last_peered":"2024-09-16T10:23:08.157095+0000","last_clean":"2024-09-16T10:23:08.157095+0000","last_became_active":"2024-09-16T10:21:56.237309+0000","last_became_peered":"2024-09-16T10:21:56.237309+0000","last_unstale":"2024-09-16T10:23:08.157095+0000","last_undegraded":"2024-09-16T10:23:08.157095+0000","last_fullsized":"2024-09-16T10:23:08.157095+0000","mapping_epoch":276,"log_start":"289'1704","ondisk_log_start":"289'1704","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4113,"log_dups_size":0,"ondisk_log_size":4113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:22.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381222912,"num_objects":5817,"num_object_clones":0,"num_object_copies":17451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5817,"num_write_kb":372288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":440,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905347+0000","last_change":"2024-09-16T10:22:58.441031+0000","last_active":"2024-09-16T10:23:05.905347+0000","last_peered":"2024-09-16T10:23:05.905347+0000","last_clean":"2024-09-16T10:23:05.905347+0000","last_became_active":"2024-09-16T10:22:56.409926+0000","last_became_peered":"2024-09-16T10:22:56.409926+0000","last_unstale":"2024-09-16T10:23:05.905347+0000","last_undegraded":"2024-09-16T10:23:05.905347+0000","last_fullsized":"2024-09-16T10:23:05.905347+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:25.837614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.837614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:27.745811+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":571,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591299+0000","last_change":"2024-09-16T10:18:07.989349+0000","last_active":"2024-09-16T10:23:08.591299+0000","last_peered":"2024-09-16T10:23:08.591299+0000","last_clean":"2024-09-16T10:23:08.591299+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:23:08.591299+0000","last_undegraded":"2024-09-16T10:23:08.591299+0000","last_fullsized":"2024-09-16T10:23:08.591299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:18:07.989263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:07.989263+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":516,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.595738+0000","last_change":"2024-09-16T10:21:59.329899+0000","last_active":"2024-09-16T10:23:08.595738+0000","last_peered":"2024-09-16T10:23:08.595738+0000","last_clean":"2024-09-16T10:23:08.595738+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:23:08.595738+0000","last_undegraded":"2024-09-16T10:23:08.595738+0000","last_fullsized":"2024-09-16T10:23:08.595738+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.329856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.329856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":287,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896393+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:23:05.896393+0000","last_peered":"2024-09-16T10:23:05.896393+0000","last_clean":"2024-09-16T10:23:05.896393+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:23:05.896393+0000","last_undegraded":"2024-09-16T10:23:05.896393+0000","last_fullsized":"2024-09-16T10:23:05.896393+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":559,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904597+0000","last_change":"2024-09-16T10:22:02.899911+0000","last_active":"2024-09-16T10:23:05.904597+0000","last_peered":"2024-09-16T10:23:05.904597+0000","last_clean":"2024-09-16T10:23:05.904597+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:23:05.904597+0000","last_undegraded":"2024-09-16T10:23:05.904597+0000","last_fullsized":"2024-09-16T10:23:05.904597+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.772413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"315'5685","reported_seq":5774,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.682521+0000","last_change":"2024-09-16T10:21:56.237148+0000","last_active":"2024-09-16T10:23:08.682521+0000","last_peered":"2024-09-16T10:23:08.682521+0000","last_clean":"2024-09-16T10:23:08.682521+0000","last_became_active":"2024-09-16T10:21:56.236822+0000","last_became_peered":"2024-09-16T10:21:56.236822+0000","last_unstale":"2024-09-16T10:23:08.682521+0000","last_undegraded":"2024-09-16T10:23:08.682521+0000","last_fullsized":"2024-09-16T10:23:08.682521+0000","mapping_epoch":276,"log_start":"289'1561","ondisk_log_start":"289'1561","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.079994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372572160,"num_objects":5685,"num_object_clones":0,"num_object_copies":17055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":363840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":434,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.601194+0000","last_change":"2024-09-16T10:22:58.454372+0000","last_active":"2024-09-16T10:23:08.601194+0000","last_peered":"2024-09-16T10:23:08.601194+0000","last_clean":"2024-09-16T10:23:08.601194+0000","last_became_active":"2024-09-16T10:22:56.446950+0000","last_became_peered":"2024-09-16T10:22:56.446950+0000","last_unstale":"2024-09-16T10:23:08.601194+0000","last_undegraded":"2024-09-16T10:23:08.601194+0000","last_fullsized":"2024-09-16T10:23:08.601194+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:27.829990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":574,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904189+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:23:05.904189+0000","last_peered":"2024-09-16T10:23:05.904189+0000","last_clean":"2024-09-16T10:23:05.904189+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:23:05.904189+0000","last_undegraded":"2024-09-16T10:23:05.904189+0000","last_fullsized":"2024-09-16T10:23:05.904189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.726490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":552,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603949+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:23:08.603949+0000","last_peered":"2024-09-16T10:23:08.603949+0000","last_clean":"2024-09-16T10:23:08.603949+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:23:08.603949+0000","last_undegraded":"2024-09-16T10:23:08.603949+0000","last_fullsized":"2024-09-16T10:23:08.603949+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":669,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591104+0000","last_change":"2024-09-16T10:22:02.357682+0000","last_active":"2024-09-16T10:23:08.591104+0000","last_peered":"2024-09-16T10:23:08.591104+0000","last_clean":"2024-09-16T10:23:08.591104+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:23:08.591104+0000","last_undegraded":"2024-09-16T10:23:08.591104+0000","last_fullsized":"2024-09-16T10:23:08.591104+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:22:02.357616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.357616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.613433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.888004+0000","last_change":"2024-09-16T10:22:05.051429+0000","last_active":"2024-09-16T10:23:05.888004+0000","last_peered":"2024-09-16T10:23:05.888004+0000","last_clean":"2024-09-16T10:23:05.888004+0000","last_became_active":"2024-09-16T10:22:02.890865+0000","last_became_peered":"2024-09-16T10:22:02.890865+0000","last_unstale":"2024-09-16T10:23:05.888004+0000","last_undegraded":"2024-09-16T10:23:05.888004+0000","last_fullsized":"2024-09-16T10:23:05.888004+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:29.018103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"314'5872","reported_seq":5959,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.456518+0000","last_change":"2024-09-16T10:21:56.239042+0000","last_active":"2024-09-16T10:23:08.456518+0000","last_peered":"2024-09-16T10:23:08.456518+0000","last_clean":"2024-09-16T10:23:08.456518+0000","last_became_active":"2024-09-16T10:21:56.238825+0000","last_became_peered":"2024-09-16T10:21:56.238825+0000","last_unstale":"2024-09-16T10:23:08.456518+0000","last_undegraded":"2024-09-16T10:23:08.456518+0000","last_fullsized":"2024-09-16T10:23:08.456518+0000","mapping_epoch":276,"log_start":"289'1443","ondisk_log_start":"289'1443","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.168677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384827392,"num_objects":5872,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":375808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":481,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.897090+0000","last_change":"2024-09-16T10:22:59.464649+0000","last_active":"2024-09-16T10:23:05.897090+0000","last_peered":"2024-09-16T10:23:05.897090+0000","last_clean":"2024-09-16T10:23:05.897090+0000","last_became_active":"2024-09-16T10:22:56.438108+0000","last_became_peered":"2024-09-16T10:22:56.438108+0000","last_unstale":"2024-09-16T10:23:05.897090+0000","last_undegraded":"2024-09-16T10:23:05.897090+0000","last_fullsized":"2024-09-16T10:23:05.897090+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:29.076225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":579,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.581515+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:23:08.581515+0000","last_peered":"2024-09-16T10:23:08.581515+0000","last_clean":"2024-09-16T10:23:08.581515+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:23:08.581515+0000","last_undegraded":"2024-09-16T10:23:08.581515+0000","last_fullsized":"2024-09-16T10:23:08.581515+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.712019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":564,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.892843+0000","last_change":"2024-09-16T10:22:02.898141+0000","last_active":"2024-09-16T10:23:05.892843+0000","last_peered":"2024-09-16T10:23:05.892843+0000","last_clean":"2024-09-16T10:23:05.892843+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:23:05.892843+0000","last_undegraded":"2024-09-16T10:23:05.892843+0000","last_fullsized":"2024-09-16T10:23:05.892843+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.918613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.918613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.461798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":658,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.601571+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:23:08.601571+0000","last_peered":"2024-09-16T10:23:08.601571+0000","last_clean":"2024-09-16T10:23:08.601571+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:23:08.601571+0000","last_undegraded":"2024-09-16T10:23:08.601571+0000","last_fullsized":"2024-09-16T10:23:08.601571+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":561,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600914+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:23:08.600914+0000","last_peered":"2024-09-16T10:23:08.600914+0000","last_clean":"2024-09-16T10:23:08.600914+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:23:08.600914+0000","last_undegraded":"2024-09-16T10:23:08.600914+0000","last_fullsized":"2024-09-16T10:23:08.600914+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"314'5958","reported_seq":6045,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.201683+0000","last_change":"2024-09-16T10:21:56.236599+0000","last_active":"2024-09-16T10:23:08.201683+0000","last_peered":"2024-09-16T10:23:08.201683+0000","last_clean":"2024-09-16T10:23:08.201683+0000","last_became_active":"2024-09-16T10:21:56.236217+0000","last_became_peered":"2024-09-16T10:21:56.236217+0000","last_unstale":"2024-09-16T10:23:08.201683+0000","last_undegraded":"2024-09-16T10:23:08.201683+0000","last_fullsized":"2024-09-16T10:23:08.201683+0000","mapping_epoch":276,"log_start":"289'1784","ondisk_log_start":"289'1784","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390463488,"num_objects":5958,"num_object_clones":0,"num_object_copies":17874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":381312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":486,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.897051+0000","last_change":"2024-09-16T10:22:58.444304+0000","last_active":"2024-09-16T10:23:05.897051+0000","last_peered":"2024-09-16T10:23:05.897051+0000","last_clean":"2024-09-16T10:23:05.897051+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:23:05.897051+0000","last_undegraded":"2024-09-16T10:23:05.897051+0000","last_fullsized":"2024-09-16T10:23:05.897051+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.732986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.732986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:28.085230+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":591,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603110+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:23:08.603110+0000","last_peered":"2024-09-16T10:23:08.603110+0000","last_clean":"2024-09-16T10:23:08.603110+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:23:08.603110+0000","last_undegraded":"2024-09-16T10:23:08.603110+0000","last_fullsized":"2024-09-16T10:23:08.603110+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":574,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.575136+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:23:08.575136+0000","last_peered":"2024-09-16T10:23:08.575136+0000","last_clean":"2024-09-16T10:23:08.575136+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:23:08.575136+0000","last_undegraded":"2024-09-16T10:23:08.575136+0000","last_fullsized":"2024-09-16T10:23:08.575136+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:27.439600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":557,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896566+0000","last_change":"2024-09-16T10:22:28.126110+0000","last_active":"2024-09-16T10:23:05.896566+0000","last_peered":"2024-09-16T10:23:05.896566+0000","last_clean":"2024-09-16T10:23:05.896566+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:23:05.896566+0000","last_undegraded":"2024-09-16T10:23:05.896566+0000","last_fullsized":"2024-09-16T10:23:05.896566+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:28.126042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:28.126042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:33.509938+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":554,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.572565+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:23:08.572565+0000","last_peered":"2024-09-16T10:23:08.572565+0000","last_clean":"2024-09-16T10:23:08.572565+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:23:08.572565+0000","last_undegraded":"2024-09-16T10:23:08.572565+0000","last_fullsized":"2024-09-16T10:23:08.572565+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"315'5894","reported_seq":5983,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.776762+0000","last_change":"2024-09-16T10:21:56.233704+0000","last_active":"2024-09-16T10:23:08.776762+0000","last_peered":"2024-09-16T10:23:08.776762+0000","last_clean":"2024-09-16T10:23:08.776762+0000","last_became_active":"2024-09-16T10:21:56.233167+0000","last_became_peered":"2024-09-16T10:21:56.233167+0000","last_unstale":"2024-09-16T10:23:08.776762+0000","last_undegraded":"2024-09-16T10:23:08.776762+0000","last_fullsized":"2024-09-16T10:23:08.776762+0000","mapping_epoch":276,"log_start":"286'795","ondisk_log_start":"286'795","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.678798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386269184,"num_objects":5894,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":377216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":432,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890435+0000","last_change":"2024-09-16T10:22:58.443914+0000","last_active":"2024-09-16T10:23:05.890435+0000","last_peered":"2024-09-16T10:23:05.890435+0000","last_clean":"2024-09-16T10:23:05.890435+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:23:05.890435+0000","last_undegraded":"2024-09-16T10:23:05.890435+0000","last_fullsized":"2024-09-16T10:23:05.890435+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:27.854542+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"282'30","reported_seq":785,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.575036+0000","last_change":"2024-09-16T10:22:04.554377+0000","last_active":"2024-09-16T10:23:08.575036+0000","last_peered":"2024-09-16T10:23:08.575036+0000","last_clean":"2024-09-16T10:23:08.575036+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:23:08.575036+0000","last_undegraded":"2024-09-16T10:23:08.575036+0000","last_fullsized":"2024-09-16T10:23:08.575036+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'30","last_scrub_stamp":"2024-09-16T10:22:04.554260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.554260+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.708122+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":557,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905057+0000","last_change":"2024-09-16T10:22:32.891207+0000","last_active":"2024-09-16T10:23:05.905057+0000","last_peered":"2024-09-16T10:23:05.905057+0000","last_clean":"2024-09-16T10:23:05.905057+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:23:05.905057+0000","last_undegraded":"2024-09-16T10:23:05.905057+0000","last_fullsized":"2024-09-16T10:23:05.905057+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:32.891126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.891126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:53.007811+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891056+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:23:05.891056+0000","last_peered":"2024-09-16T10:23:05.891056+0000","last_clean":"2024-09-16T10:23:05.891056+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:23:05.891056+0000","last_undegraded":"2024-09-16T10:23:05.891056+0000","last_fullsized":"2024-09-16T10:23:05.891056+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:51.930288+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":535,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.601878+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:23:08.601878+0000","last_peered":"2024-09-16T10:23:08.601878+0000","last_clean":"2024-09-16T10:23:08.601878+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:23:08.601878+0000","last_undegraded":"2024-09-16T10:23:08.601878+0000","last_fullsized":"2024-09-16T10:23:08.601878+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":555,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.592087+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:23:08.592087+0000","last_peered":"2024-09-16T10:23:08.592087+0000","last_clean":"2024-09-16T10:23:08.592087+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:23:08.592087+0000","last_undegraded":"2024-09-16T10:23:08.592087+0000","last_fullsized":"2024-09-16T10:23:08.592087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.835165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"314'5675","reported_seq":5762,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.452718+0000","last_change":"2024-09-16T10:21:56.236854+0000","last_active":"2024-09-16T10:23:08.452718+0000","last_peered":"2024-09-16T10:23:08.452718+0000","last_clean":"2024-09-16T10:23:08.452718+0000","last_became_active":"2024-09-16T10:21:56.236183+0000","last_became_peered":"2024-09-16T10:21:56.236183+0000","last_unstale":"2024-09-16T10:23:08.452718+0000","last_undegraded":"2024-09-16T10:23:08.452718+0000","last_fullsized":"2024-09-16T10:23:08.452718+0000","mapping_epoch":276,"log_start":"289'1262","ondisk_log_start":"289'1262","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371916800,"num_objects":5675,"num_object_clones":0,"num_object_copies":17025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":363200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":691,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889083+0000","last_change":"2024-09-16T10:22:59.461225+0000","last_active":"2024-09-16T10:23:05.889083+0000","last_peered":"2024-09-16T10:23:05.889083+0000","last_clean":"2024-09-16T10:23:05.889083+0000","last_became_active":"2024-09-16T10:22:56.424034+0000","last_became_peered":"2024-09-16T10:22:56.424034+0000","last_unstale":"2024-09-16T10:23:05.889083+0000","last_undegraded":"2024-09-16T10:23:05.889083+0000","last_fullsized":"2024-09-16T10:23:05.889083+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:32.033672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.033672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:28.792322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":559,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.573444+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:23:08.573444+0000","last_peered":"2024-09-16T10:23:08.573444+0000","last_clean":"2024-09-16T10:23:08.573444+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:23:08.573444+0000","last_undegraded":"2024-09-16T10:23:08.573444+0000","last_fullsized":"2024-09-16T10:23:08.573444+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":558,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.574217+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:23:08.574217+0000","last_peered":"2024-09-16T10:23:08.574217+0000","last_clean":"2024-09-16T10:23:08.574217+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:23:08.574217+0000","last_undegraded":"2024-09-16T10:23:08.574217+0000","last_fullsized":"2024-09-16T10:23:08.574217+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.551901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":80,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896331+0000","last_change":"2024-09-16T10:22:02.917475+0000","last_active":"2024-09-16T10:23:05.896331+0000","last_peered":"2024-09-16T10:23:05.896331+0000","last_clean":"2024-09-16T10:23:05.896331+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:23:05.896331+0000","last_undegraded":"2024-09-16T10:23:05.896331+0000","last_fullsized":"2024-09-16T10:23:05.896331+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.736419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":84,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896417+0000","last_change":"2024-09-16T10:22:29.905776+0000","last_active":"2024-09-16T10:23:05.896417+0000","last_peered":"2024-09-16T10:23:05.896417+0000","last_clean":"2024-09-16T10:23:05.896417+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:23:05.896417+0000","last_undegraded":"2024-09-16T10:23:05.896417+0000","last_fullsized":"2024-09-16T10:23:05.896417+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:57.728247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":554,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889497+0000","last_change":"2024-09-16T10:22:29.105512+0000","last_active":"2024-09-16T10:23:05.889497+0000","last_peered":"2024-09-16T10:23:05.889497+0000","last_clean":"2024-09-16T10:23:05.889497+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:23:05.889497+0000","last_undegraded":"2024-09-16T10:23:05.889497+0000","last_fullsized":"2024-09-16T10:23:05.889497+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.105178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.105178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:42.853519+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"315'5776","reported_seq":7413,"reported_epoch":315,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:08.810416+0000","last_change":"2024-09-16T10:22:37.049031+0000","last_active":"2024-09-16T10:23:08.810416+0000","last_peered":"2024-09-16T10:23:08.810416+0000","last_clean":"2024-09-16T10:22:35.458157+0000","last_became_active":"2024-09-16T10:22:37.003170+0000","last_became_peered":"2024-09-16T10:22:37.003170+0000","last_unstale":"2024-09-16T10:23:08.810416+0000","last_undegraded":"2024-09-16T10:22:36.998730+0000","last_fullsized":"2024-09-16T10:22:36.878645+0000","mapping_epoch":298,"log_start":"286'611","ondisk_log_start":"286'611","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378535936,"num_objects":5776,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":6521,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5776,"num_write_kb":369664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":33751040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[0,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":3260},{"shards":"0,6","objects":2515}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":422,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.590175+0000","last_change":"2024-09-16T10:22:58.474593+0000","last_active":"2024-09-16T10:23:08.590175+0000","last_peered":"2024-09-16T10:23:08.590175+0000","last_clean":"2024-09-16T10:23:08.590175+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:23:08.590175+0000","last_undegraded":"2024-09-16T10:23:08.590175+0000","last_fullsized":"2024-09-16T10:23:08.590175+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:27.619787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":631,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603727+0000","last_change":"2024-09-16T10:22:24.213089+0000","last_active":"2024-09-16T10:23:08.603727+0000","last_peered":"2024-09-16T10:23:08.603727+0000","last_clean":"2024-09-16T10:23:08.603727+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:23:08.603727+0000","last_undegraded":"2024-09-16T10:23:08.603727+0000","last_fullsized":"2024-09-16T10:23:08.603727+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:22:24.213008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.213008+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.043515+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":575,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.602235+0000","last_change":"2024-09-16T10:22:26.226703+0000","last_active":"2024-09-16T10:23:08.602235+0000","last_peered":"2024-09-16T10:23:08.602235+0000","last_clean":"2024-09-16T10:23:08.602235+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:23:08.602235+0000","last_undegraded":"2024-09-16T10:23:08.602235+0000","last_fullsized":"2024-09-16T10:23:08.602235+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.226632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.226632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:28.255751+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1134,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:06.862511+0000","last_change":"2024-09-16T10:21:58.009040+0000","last_active":"2024-09-16T10:23:06.862511+0000","last_peered":"2024-09-16T10:23:06.862511+0000","last_clean":"2024-09-16T10:23:06.862511+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:23:06.862511+0000","last_undegraded":"2024-09-16T10:23:06.862511+0000","last_fullsized":"2024-09-16T10:23:06.862511+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:21:58.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.008984+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:25.997371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":565,"num_read_kb":560,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":542,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891745+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:23:05.891745+0000","last_peered":"2024-09-16T10:23:05.891745+0000","last_clean":"2024-09-16T10:23:05.891745+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:23:05.891745+0000","last_undegraded":"2024-09-16T10:23:05.891745+0000","last_fullsized":"2024-09-16T10:23:05.891745+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.025944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":556,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.574719+0000","last_change":"2024-09-16T10:21:58.547682+0000","last_active":"2024-09-16T10:23:08.574719+0000","last_peered":"2024-09-16T10:23:08.574719+0000","last_clean":"2024-09-16T10:23:08.574719+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:23:08.574719+0000","last_undegraded":"2024-09-16T10:23:08.574719+0000","last_fullsized":"2024-09-16T10:23:08.574719+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.547634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.547634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.050150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"315'5866","reported_seq":5955,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.661228+0000","last_change":"2024-09-16T10:21:56.236351+0000","last_active":"2024-09-16T10:23:08.661228+0000","last_peered":"2024-09-16T10:23:08.661228+0000","last_clean":"2024-09-16T10:23:08.661228+0000","last_became_active":"2024-09-16T10:21:56.236070+0000","last_became_peered":"2024-09-16T10:21:56.236070+0000","last_unstale":"2024-09-16T10:23:08.661228+0000","last_undegraded":"2024-09-16T10:23:08.661228+0000","last_fullsized":"2024-09-16T10:23:08.661228+0000","mapping_epoch":276,"log_start":"289'1784","ondisk_log_start":"289'1784","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384434176,"num_objects":5866,"num_object_clones":0,"num_object_copies":17598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5866,"num_write_kb":375424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":674,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.897081+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:23:05.897081+0000","last_peered":"2024-09-16T10:23:05.897081+0000","last_clean":"2024-09-16T10:23:05.897081+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:23:05.897081+0000","last_undegraded":"2024-09-16T10:23:05.897081+0000","last_fullsized":"2024-09-16T10:23:05.897081+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":524,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.593021+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:23:08.593021+0000","last_peered":"2024-09-16T10:23:08.593021+0000","last_clean":"2024-09-16T10:23:08.593021+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:23:08.593021+0000","last_undegraded":"2024-09-16T10:23:08.593021+0000","last_fullsized":"2024-09-16T10:23:08.593021+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":668,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.888200+0000","last_change":"2024-09-16T10:22:58.443425+0000","last_active":"2024-09-16T10:23:05.888200+0000","last_peered":"2024-09-16T10:23:05.888200+0000","last_clean":"2024-09-16T10:23:05.888200+0000","last_became_active":"2024-09-16T10:22:56.424419+0000","last_became_peered":"2024-09-16T10:22:56.424419+0000","last_unstale":"2024-09-16T10:23:05.888200+0000","last_undegraded":"2024-09-16T10:23:05.888200+0000","last_fullsized":"2024-09-16T10:23:05.888200+0000","mapping_epoch":55,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:57.046749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.046749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:27.751482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":548,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890476+0000","last_change":"2024-09-16T10:22:02.906870+0000","last_active":"2024-09-16T10:23:05.890476+0000","last_peered":"2024-09-16T10:23:05.890476+0000","last_clean":"2024-09-16T10:23:05.890476+0000","last_became_active":"2024-09-16T10:22:02.906315+0000","last_became_peered":"2024-09-16T10:22:02.906315+0000","last_unstale":"2024-09-16T10:23:05.890476+0000","last_undegraded":"2024-09-16T10:23:05.890476+0000","last_fullsized":"2024-09-16T10:23:05.890476+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.151113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":538,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891649+0000","last_change":"2024-09-16T10:21:56.923534+0000","last_active":"2024-09-16T10:23:05.891649+0000","last_peered":"2024-09-16T10:23:05.891649+0000","last_clean":"2024-09-16T10:23:05.891649+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:23:05.891649+0000","last_undegraded":"2024-09-16T10:23:05.891649+0000","last_fullsized":"2024-09-16T10:23:05.891649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:56.923485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.923485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":444,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.574507+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:23:08.574507+0000","last_peered":"2024-09-16T10:23:08.574507+0000","last_clean":"2024-09-16T10:23:08.574507+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:23:08.574507+0000","last_undegraded":"2024-09-16T10:23:08.574507+0000","last_fullsized":"2024-09-16T10:23:08.574507+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:18.698762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"314'5868","reported_seq":5955,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.176087+0000","last_change":"2024-09-16T10:21:56.234554+0000","last_active":"2024-09-16T10:23:08.176087+0000","last_peered":"2024-09-16T10:23:08.176087+0000","last_clean":"2024-09-16T10:23:08.176087+0000","last_became_active":"2024-09-16T10:21:56.234289+0000","last_became_peered":"2024-09-16T10:21:56.234289+0000","last_unstale":"2024-09-16T10:23:08.176087+0000","last_undegraded":"2024-09-16T10:23:08.176087+0000","last_fullsized":"2024-09-16T10:23:08.176087+0000","mapping_epoch":276,"log_start":"289'1694","ondisk_log_start":"289'1694","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.348123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384565248,"num_objects":5868,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5868,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":583,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905650+0000","last_change":"2024-09-16T10:20:31.013510+0000","last_active":"2024-09-16T10:23:05.905650+0000","last_peered":"2024-09-16T10:23:05.905650+0000","last_clean":"2024-09-16T10:23:05.905650+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:23:05.905650+0000","last_undegraded":"2024-09-16T10:23:05.905650+0000","last_fullsized":"2024-09-16T10:23:05.905650+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:20:31.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.013316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.790395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":563,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904499+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:23:05.904499+0000","last_peered":"2024-09-16T10:23:05.904499+0000","last_clean":"2024-09-16T10:23:05.904499+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:23:05.904499+0000","last_undegraded":"2024-09-16T10:23:05.904499+0000","last_fullsized":"2024-09-16T10:23:05.904499+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:31.365218+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891508+0000","last_change":"2024-09-16T10:22:22.987395+0000","last_active":"2024-09-16T10:23:05.891508+0000","last_peered":"2024-09-16T10:23:05.891508+0000","last_clean":"2024-09-16T10:23:05.891508+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:23:05.891508+0000","last_undegraded":"2024-09-16T10:23:05.891508+0000","last_fullsized":"2024-09-16T10:23:05.891508+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.987351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.987351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.275819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":496,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.601005+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:23:08.601005+0000","last_peered":"2024-09-16T10:23:08.601005+0000","last_clean":"2024-09-16T10:23:08.601005+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:23:08.601005+0000","last_undegraded":"2024-09-16T10:23:08.601005+0000","last_fullsized":"2024-09-16T10:23:08.601005+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:49.885016+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"315'5796","reported_seq":6084,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.573657+0000","last_change":"2024-09-16T10:22:03.098658+0000","last_active":"2024-09-16T10:23:08.573657+0000","last_peered":"2024-09-16T10:23:08.573657+0000","last_clean":"2024-09-16T10:23:08.573657+0000","last_became_active":"2024-09-16T10:22:02.896249+0000","last_became_peered":"2024-09-16T10:22:02.896249+0000","last_unstale":"2024-09-16T10:23:08.573657+0000","last_undegraded":"2024-09-16T10:23:08.573657+0000","last_fullsized":"2024-09-16T10:23:08.573657+0000","mapping_epoch":281,"log_start":"289'1357","ondisk_log_start":"289'1357","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:15.823024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379846656,"num_objects":5796,"num_object_clones":0,"num_object_copies":17388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5796,"num_write_kb":370944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":401,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890338+0000","last_change":"2024-09-16T10:22:03.029868+0000","last_active":"2024-09-16T10:23:05.890338+0000","last_peered":"2024-09-16T10:23:05.890338+0000","last_clean":"2024-09-16T10:23:05.890338+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:23:05.890338+0000","last_undegraded":"2024-09-16T10:23:05.890338+0000","last_fullsized":"2024-09-16T10:23:05.890338+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:27.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:27.405599+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:58.883669+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":623,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905265+0000","last_change":"2024-09-16T10:22:03.026909+0000","last_active":"2024-09-16T10:23:05.905265+0000","last_peered":"2024-09-16T10:23:05.905265+0000","last_clean":"2024-09-16T10:23:05.905265+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:23:05.905265+0000","last_undegraded":"2024-09-16T10:23:05.905265+0000","last_fullsized":"2024-09-16T10:23:05.905265+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:21:53.705644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.705644+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":549,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896778+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:23:05.896778+0000","last_peered":"2024-09-16T10:23:05.896778+0000","last_clean":"2024-09-16T10:23:05.896778+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:23:05.896778+0000","last_undegraded":"2024-09-16T10:23:05.896778+0000","last_fullsized":"2024-09-16T10:23:05.896778+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":544,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.574828+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:23:08.574828+0000","last_peered":"2024-09-16T10:23:08.574828+0000","last_clean":"2024-09-16T10:23:08.574828+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:23:08.574828+0000","last_undegraded":"2024-09-16T10:23:08.574828+0000","last_fullsized":"2024-09-16T10:23:08.574828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":84,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.897312+0000","last_change":"2024-09-16T10:22:05.694026+0000","last_active":"2024-09-16T10:23:05.897312+0000","last_peered":"2024-09-16T10:23:05.897312+0000","last_clean":"2024-09-16T10:23:05.897312+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:23:05.897312+0000","last_undegraded":"2024-09-16T10:23:05.897312+0000","last_fullsized":"2024-09-16T10:23:05.897312+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.693972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.693972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.767050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"315'5746","reported_seq":5835,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.682966+0000","last_change":"2024-09-16T10:21:56.913566+0000","last_active":"2024-09-16T10:23:08.682966+0000","last_peered":"2024-09-16T10:23:08.682966+0000","last_clean":"2024-09-16T10:23:08.682966+0000","last_became_active":"2024-09-16T10:21:56.913186+0000","last_became_peered":"2024-09-16T10:21:56.913186+0000","last_unstale":"2024-09-16T10:23:08.682966+0000","last_undegraded":"2024-09-16T10:23:08.682966+0000","last_fullsized":"2024-09-16T10:23:08.682966+0000","mapping_epoch":276,"log_start":"289'1618","ondisk_log_start":"289'1618","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376569856,"num_objects":5746,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":367744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":661,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891869+0000","last_change":"2024-09-16T10:22:02.905855+0000","last_active":"2024-09-16T10:23:05.891869+0000","last_peered":"2024-09-16T10:23:05.891869+0000","last_clean":"2024-09-16T10:23:05.891869+0000","last_became_active":"2024-09-16T10:22:02.905393+0000","last_became_peered":"2024-09-16T10:22:02.905393+0000","last_unstale":"2024-09-16T10:23:05.891869+0000","last_undegraded":"2024-09-16T10:23:05.891869+0000","last_fullsized":"2024-09-16T10:23:05.891869+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:21.240387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:21.240387+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905455+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:23:05.905455+0000","last_peered":"2024-09-16T10:23:05.905455+0000","last_clean":"2024-09-16T10:23:05.905455+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:23:05.905455+0000","last_undegraded":"2024-09-16T10:23:05.905455+0000","last_fullsized":"2024-09-16T10:23:05.905455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:42.732608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":535,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.592273+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:23:08.592273+0000","last_peered":"2024-09-16T10:23:08.592273+0000","last_clean":"2024-09-16T10:23:08.592273+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:23:08.592273+0000","last_undegraded":"2024-09-16T10:23:08.592273+0000","last_fullsized":"2024-09-16T10:23:08.592273+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.375612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":102,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.897134+0000","last_change":"2024-09-16T10:22:24.799901+0000","last_active":"2024-09-16T10:23:05.897134+0000","last_peered":"2024-09-16T10:23:05.897134+0000","last_clean":"2024-09-16T10:23:05.897134+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:23:05.897134+0000","last_undegraded":"2024-09-16T10:23:05.897134+0000","last_fullsized":"2024-09-16T10:23:05.897134+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:32.615360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":559,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905514+0000","last_change":"2024-09-16T10:22:31.886830+0000","last_active":"2024-09-16T10:23:05.905514+0000","last_peered":"2024-09-16T10:23:05.905514+0000","last_clean":"2024-09-16T10:23:05.905514+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:23:05.905514+0000","last_undegraded":"2024-09-16T10:23:05.905514+0000","last_fullsized":"2024-09-16T10:23:05.905514+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:31.886666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:31.886666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:51.243131+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":100,"reported_epoch":315,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:08.603613+0000","last_change":"2024-09-16T10:22:59.486151+0000","last_active":"2024-09-16T10:23:08.603613+0000","last_peered":"2024-09-16T10:23:08.603613+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:22:59.484582+0000","last_became_peered":"2024-09-16T10:22:59.484582+0000","last_unstale":"2024-09-16T10:23:08.603613+0000","last_undegraded":"2024-09-16T10:23:08.603613+0000","last_fullsized":"2024-09-16T10:23:08.603613+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.732986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.732986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[0,3,6],"avail_no_missing":["0","3","4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"314'5751","reported_seq":6025,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.452896+0000","last_change":"2024-09-16T10:22:03.102043+0000","last_active":"2024-09-16T10:23:08.452896+0000","last_peered":"2024-09-16T10:23:08.452896+0000","last_clean":"2024-09-16T10:23:08.452896+0000","last_became_active":"2024-09-16T10:22:02.898446+0000","last_became_peered":"2024-09-16T10:22:02.898446+0000","last_unstale":"2024-09-16T10:23:08.452896+0000","last_undegraded":"2024-09-16T10:23:08.452896+0000","last_fullsized":"2024-09-16T10:23:08.452896+0000","mapping_epoch":281,"log_start":"289'1333","ondisk_log_start":"289'1333","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.816221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376897536,"num_objects":5751,"num_object_clones":0,"num_object_copies":17253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5751,"num_write_kb":368064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":555,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603020+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:23:08.603020+0000","last_peered":"2024-09-16T10:23:08.603020+0000","last_clean":"2024-09-16T10:23:08.603020+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:23:08.603020+0000","last_undegraded":"2024-09-16T10:23:08.603020+0000","last_fullsized":"2024-09-16T10:23:08.603020+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":558,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891231+0000","last_change":"2024-09-16T10:21:57.915828+0000","last_active":"2024-09-16T10:23:05.891231+0000","last_peered":"2024-09-16T10:23:05.891231+0000","last_clean":"2024-09-16T10:23:05.891231+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:23:05.891231+0000","last_undegraded":"2024-09-16T10:23:05.891231+0000","last_fullsized":"2024-09-16T10:23:05.891231+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.915773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.915773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:23.518442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":76,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.900337+0000","last_change":"2024-09-16T10:22:02.896967+0000","last_active":"2024-09-16T10:23:05.900337+0000","last_peered":"2024-09-16T10:23:05.900337+0000","last_clean":"2024-09-16T10:23:05.900337+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:23:05.900337+0000","last_undegraded":"2024-09-16T10:23:05.900337+0000","last_fullsized":"2024-09-16T10:23:05.900337+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.735258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":576,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.602148+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:23:08.602148+0000","last_peered":"2024-09-16T10:23:08.602148+0000","last_clean":"2024-09-16T10:23:08.602148+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:23:08.602148+0000","last_undegraded":"2024-09-16T10:23:08.602148+0000","last_fullsized":"2024-09-16T10:23:08.602148+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.144396+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":485,"reported_epoch":314,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:05.897215+0000","last_change":"2024-09-16T10:22:59.493493+0000","last_active":"2024-09-16T10:23:05.897215+0000","last_peered":"2024-09-16T10:23:05.897215+0000","last_clean":"2024-09-16T10:22:56.438866+0000","last_became_active":"2024-09-16T10:22:59.492636+0000","last_became_peered":"2024-09-16T10:22:59.492636+0000","last_unstale":"2024-09-16T10:23:05.897215+0000","last_undegraded":"2024-09-16T10:23:05.897215+0000","last_fullsized":"2024-09-16T10:23:05.897215+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,3,7],"avail_no_missing":["6","3","4","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"314'5892","reported_seq":5979,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.447171+0000","last_change":"2024-09-16T10:21:56.234684+0000","last_active":"2024-09-16T10:23:08.447171+0000","last_peered":"2024-09-16T10:23:08.447171+0000","last_clean":"2024-09-16T10:23:08.447171+0000","last_became_active":"2024-09-16T10:21:56.234401+0000","last_became_peered":"2024-09-16T10:21:56.234401+0000","last_unstale":"2024-09-16T10:23:08.447171+0000","last_undegraded":"2024-09-16T10:23:08.447171+0000","last_fullsized":"2024-09-16T10:23:08.447171+0000","mapping_epoch":276,"log_start":"289'1471","ondisk_log_start":"289'1471","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.262243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386138112,"num_objects":5892,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5892,"num_write_kb":377088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":592,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896672+0000","last_change":"2024-09-16T10:22:26.050158+0000","last_active":"2024-09-16T10:23:05.896672+0000","last_peered":"2024-09-16T10:23:05.896672+0000","last_clean":"2024-09-16T10:23:05.896672+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:23:05.896672+0000","last_undegraded":"2024-09-16T10:23:05.896672+0000","last_fullsized":"2024-09-16T10:23:05.896672+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:22:26.050070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.050070+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:43.078689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889760+0000","last_change":"2024-09-16T10:21:59.998410+0000","last_active":"2024-09-16T10:23:05.889760+0000","last_peered":"2024-09-16T10:23:05.889760+0000","last_clean":"2024-09-16T10:23:05.889760+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:23:05.889760+0000","last_undegraded":"2024-09-16T10:23:05.889760+0000","last_fullsized":"2024-09-16T10:23:05.889760+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.998357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.998357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":84,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896645+0000","last_change":"2024-09-16T10:22:26.818522+0000","last_active":"2024-09-16T10:23:05.896645+0000","last_peered":"2024-09-16T10:23:05.896645+0000","last_clean":"2024-09-16T10:23:05.896645+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:23:05.896645+0000","last_undegraded":"2024-09-16T10:23:05.896645+0000","last_fullsized":"2024-09-16T10:23:05.896645+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.818439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.818439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.045673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.888117+0000","last_change":"2024-09-16T10:22:02.906954+0000","last_active":"2024-09-16T10:23:05.888117+0000","last_peered":"2024-09-16T10:23:05.888117+0000","last_clean":"2024-09-16T10:23:05.888117+0000","last_became_active":"2024-09-16T10:22:02.906524+0000","last_became_peered":"2024-09-16T10:22:02.906524+0000","last_unstale":"2024-09-16T10:23:05.888117+0000","last_undegraded":"2024-09-16T10:23:05.888117+0000","last_fullsized":"2024-09-16T10:23:05.888117+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.105240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.105240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:41.931790+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":435,"reported_epoch":315,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:08.592694+0000","last_change":"2024-09-16T10:22:59.503845+0000","last_active":"2024-09-16T10:23:08.592694+0000","last_peered":"2024-09-16T10:23:08.592694+0000","last_clean":"2024-09-16T10:22:56.466317+0000","last_became_active":"2024-09-16T10:22:59.490973+0000","last_became_peered":"2024-09-16T10:22:59.490973+0000","last_unstale":"2024-09-16T10:23:08.592694+0000","last_undegraded":"2024-09-16T10:23:08.592694+0000","last_fullsized":"2024-09-16T10:23:08.592694+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":109227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"315'5895","reported_seq":5984,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.575045+0000","last_change":"2024-09-16T10:21:56.235273+0000","last_active":"2024-09-16T10:23:08.575045+0000","last_peered":"2024-09-16T10:23:08.575045+0000","last_clean":"2024-09-16T10:23:08.575045+0000","last_became_active":"2024-09-16T10:21:56.234987+0000","last_became_peered":"2024-09-16T10:21:56.234987+0000","last_unstale":"2024-09-16T10:23:08.575045+0000","last_undegraded":"2024-09-16T10:23:08.575045+0000","last_fullsized":"2024-09-16T10:23:08.575045+0000","mapping_epoch":276,"log_start":"289'1448","ondisk_log_start":"289'1448","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386334720,"num_objects":5895,"num_object_clones":0,"num_object_copies":17685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5895,"num_write_kb":377280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":560,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904363+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:23:05.904363+0000","last_peered":"2024-09-16T10:23:05.904363+0000","last_clean":"2024-09-16T10:23:05.904363+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:23:05.904363+0000","last_undegraded":"2024-09-16T10:23:05.904363+0000","last_fullsized":"2024-09-16T10:23:05.904363+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:41.732688+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":558,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889184+0000","last_change":"2024-09-16T10:22:34.020368+0000","last_active":"2024-09-16T10:23:05.889184+0000","last_peered":"2024-09-16T10:23:05.889184+0000","last_clean":"2024-09-16T10:23:05.889184+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:23:05.889184+0000","last_undegraded":"2024-09-16T10:23:05.889184+0000","last_fullsized":"2024-09-16T10:23:05.889184+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:34.020246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:34.020246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.286859+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":84,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.895903+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:23:05.895903+0000","last_peered":"2024-09-16T10:23:05.895903+0000","last_clean":"2024-09-16T10:23:05.895903+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:23:05.895903+0000","last_undegraded":"2024-09-16T10:23:05.895903+0000","last_fullsized":"2024-09-16T10:23:05.895903+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.537291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591877+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:23:08.591877+0000","last_peered":"2024-09-16T10:23:08.591877+0000","last_clean":"2024-09-16T10:23:08.591877+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:23:08.591877+0000","last_undegraded":"2024-09-16T10:23:08.591877+0000","last_fullsized":"2024-09-16T10:23:08.591877+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.386329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":365,"reported_epoch":314,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:05.896785+0000","last_change":"2024-09-16T10:22:59.469189+0000","last_active":"2024-09-16T10:23:05.896785+0000","last_peered":"2024-09-16T10:23:05.896785+0000","last_clean":"2024-09-16T10:22:01.242044+0000","last_became_active":"2024-09-16T10:22:59.465698+0000","last_became_peered":"2024-09-16T10:22:59.465698+0000","last_unstale":"2024-09-16T10:23:05.896785+0000","last_undegraded":"2024-09-16T10:23:05.896785+0000","last_fullsized":"2024-09-16T10:23:05.896785+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:25.837614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.837614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[2,4,6],"avail_no_missing":["2","1","4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.5","version":"315'5907","reported_seq":5996,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.804763+0000","last_change":"2024-09-16T10:21:56.236925+0000","last_active":"2024-09-16T10:23:08.804763+0000","last_peered":"2024-09-16T10:23:08.804763+0000","last_clean":"2024-09-16T10:23:08.804763+0000","last_became_active":"2024-09-16T10:21:56.236619+0000","last_became_peered":"2024-09-16T10:21:56.236619+0000","last_unstale":"2024-09-16T10:23:08.804763+0000","last_undegraded":"2024-09-16T10:23:08.804763+0000","last_fullsized":"2024-09-16T10:23:08.804763+0000","mapping_epoch":276,"log_start":"286'807","ondisk_log_start":"286'807","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387121152,"num_objects":5907,"num_object_clones":0,"num_object_copies":17721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5907,"num_write_kb":378048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":555,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890398+0000","last_change":"2024-09-16T10:21:35.618881+0000","last_active":"2024-09-16T10:23:05.890398+0000","last_peered":"2024-09-16T10:23:05.890398+0000","last_clean":"2024-09-16T10:23:05.890398+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:23:05.890398+0000","last_undegraded":"2024-09-16T10:23:05.890398+0000","last_fullsized":"2024-09-16T10:23:05.890398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:21:35.618806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.618806+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":554,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889593+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:23:05.889593+0000","last_peered":"2024-09-16T10:23:05.889593+0000","last_clean":"2024-09-16T10:23:05.889593+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:23:05.889593+0000","last_undegraded":"2024-09-16T10:23:05.889593+0000","last_fullsized":"2024-09-16T10:23:05.889593+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.060231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":657,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890478+0000","last_change":"2024-09-16T10:17:15.609963+0000","last_active":"2024-09-16T10:23:05.890478+0000","last_peered":"2024-09-16T10:23:05.890478+0000","last_clean":"2024-09-16T10:23:05.890478+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:23:05.890478+0000","last_undegraded":"2024-09-16T10:23:05.890478+0000","last_fullsized":"2024-09-16T10:23:05.890478+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:17:15.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:15.609910+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":546,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890728+0000","last_change":"2024-09-16T10:22:00.982299+0000","last_active":"2024-09-16T10:23:05.890728+0000","last_peered":"2024-09-16T10:23:05.890728+0000","last_clean":"2024-09-16T10:23:05.890728+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:23:05.890728+0000","last_undegraded":"2024-09-16T10:23:05.890728+0000","last_fullsized":"2024-09-16T10:23:05.890728+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:00.982240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:00.982240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":101,"reported_epoch":315,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:08.599558+0000","last_change":"2024-09-16T10:22:59.485096+0000","last_active":"2024-09-16T10:23:08.599558+0000","last_peered":"2024-09-16T10:23:08.599558+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:22:59.474762+0000","last_became_peered":"2024-09-16T10:22:59.474762+0000","last_unstale":"2024-09-16T10:23:08.599558+0000","last_undegraded":"2024-09-16T10:23:08.599558+0000","last_fullsized":"2024-09-16T10:23:08.599558+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:32.033672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.033672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896592+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:23:05.896592+0000","last_peered":"2024-09-16T10:23:05.896592+0000","last_clean":"2024-09-16T10:23:05.896592+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:23:05.896592+0000","last_undegraded":"2024-09-16T10:23:05.896592+0000","last_fullsized":"2024-09-16T10:23:05.896592+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":555,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.595601+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:23:08.595601+0000","last_peered":"2024-09-16T10:23:08.595601+0000","last_clean":"2024-09-16T10:23:08.595601+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:23:08.595601+0000","last_undegraded":"2024-09-16T10:23:08.595601+0000","last_fullsized":"2024-09-16T10:23:08.595601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.853715+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891460+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:23:05.891460+0000","last_peered":"2024-09-16T10:23:05.891460+0000","last_clean":"2024-09-16T10:23:05.891460+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:23:05.891460+0000","last_undegraded":"2024-09-16T10:23:05.891460+0000","last_fullsized":"2024-09-16T10:23:05.891460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":529,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.888767+0000","last_change":"2024-09-16T10:23:03.747587+0000","last_active":"2024-09-16T10:23:05.888767+0000","last_peered":"2024-09-16T10:23:05.888767+0000","last_clean":"2024-09-16T10:23:05.888767+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:23:05.888767+0000","last_undegraded":"2024-09-16T10:23:05.888767+0000","last_fullsized":"2024-09-16T10:23:05.888767+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:03.747523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:03.747523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:05.288516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":422,"reported_epoch":315,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:08.574681+0000","last_change":"2024-09-16T10:22:59.506200+0000","last_active":"2024-09-16T10:23:08.574681+0000","last_peered":"2024-09-16T10:23:08.574681+0000","last_clean":"2024-09-16T10:22:56.428443+0000","last_became_active":"2024-09-16T10:22:59.499206+0000","last_became_peered":"2024-09-16T10:22:59.499206+0000","last_unstale":"2024-09-16T10:23:08.574681+0000","last_undegraded":"2024-09-16T10:23:08.574681+0000","last_fullsized":"2024-09-16T10:23:08.574681+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,4,6],"avail_no_missing":["1","3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":562,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603844+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:23:08.603844+0000","last_peered":"2024-09-16T10:23:08.603844+0000","last_clean":"2024-09-16T10:23:08.603844+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:23:08.603844+0000","last_undegraded":"2024-09-16T10:23:08.603844+0000","last_fullsized":"2024-09-16T10:23:08.603844+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:14.198247+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890809+0000","last_change":"2024-09-16T10:22:02.040477+0000","last_active":"2024-09-16T10:23:05.890809+0000","last_peered":"2024-09-16T10:23:05.890809+0000","last_clean":"2024-09-16T10:23:05.890809+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:23:05.890809+0000","last_undegraded":"2024-09-16T10:23:05.890809+0000","last_fullsized":"2024-09-16T10:23:05.890809+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.040428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.040428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.314331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":542,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.595020+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:23:08.595020+0000","last_peered":"2024-09-16T10:23:08.595020+0000","last_clean":"2024-09-16T10:23:08.595020+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:23:08.595020+0000","last_undegraded":"2024-09-16T10:23:08.595020+0000","last_fullsized":"2024-09-16T10:23:08.595020+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.194439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":76,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896865+0000","last_change":"2024-09-16T10:22:02.903139+0000","last_active":"2024-09-16T10:23:05.896865+0000","last_peered":"2024-09-16T10:23:05.896865+0000","last_clean":"2024-09-16T10:23:05.896865+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:23:05.896865+0000","last_undegraded":"2024-09-16T10:23:05.896865+0000","last_fullsized":"2024-09-16T10:23:05.896865+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.701346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":112,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.890216+0000","last_change":"2024-09-16T10:23:01.854762+0000","last_active":"2024-09-16T10:23:05.890216+0000","last_peered":"2024-09-16T10:23:05.890216+0000","last_clean":"2024-09-16T10:23:05.890216+0000","last_became_active":"2024-09-16T10:23:00.902731+0000","last_became_peered":"2024-09-16T10:23:00.902731+0000","last_unstale":"2024-09-16T10:23:05.890216+0000","last_undegraded":"2024-09-16T10:23:05.890216+0000","last_fullsized":"2024-09-16T10:23:05.890216+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.854706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.854706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:06.610807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":553,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.892744+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:23:05.892744+0000","last_peered":"2024-09-16T10:23:05.892744+0000","last_clean":"2024-09-16T10:23:05.892744+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:23:05.892744+0000","last_undegraded":"2024-09-16T10:23:05.892744+0000","last_fullsized":"2024-09-16T10:23:05.892744+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:49.948413+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":559,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600072+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:23:08.600072+0000","last_peered":"2024-09-16T10:23:08.600072+0000","last_clean":"2024-09-16T10:23:08.600072+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:23:08.600072+0000","last_undegraded":"2024-09-16T10:23:08.600072+0000","last_fullsized":"2024-09-16T10:23:08.600072+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":704,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600912+0000","last_change":"2024-09-16T10:22:24.619056+0000","last_active":"2024-09-16T10:23:08.600912+0000","last_peered":"2024-09-16T10:23:08.600912+0000","last_clean":"2024-09-16T10:23:08.600912+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:23:08.600912+0000","last_undegraded":"2024-09-16T10:23:08.600912+0000","last_fullsized":"2024-09-16T10:23:08.600912+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.618991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.618991+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:31.047734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":558,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905569+0000","last_change":"2024-09-16T10:22:34.860226+0000","last_active":"2024-09-16T10:23:05.905569+0000","last_peered":"2024-09-16T10:23:05.905569+0000","last_clean":"2024-09-16T10:23:05.905569+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:23:05.905569+0000","last_undegraded":"2024-09-16T10:23:05.905569+0000","last_fullsized":"2024-09-16T10:23:05.905569+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:34.860161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:34.860161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:38.752001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":100,"reported_epoch":315,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:08.602315+0000","last_change":"2024-09-16T10:22:59.512532+0000","last_active":"2024-09-16T10:23:08.602315+0000","last_peered":"2024-09-16T10:23:08.602315+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:22:59.497296+0000","last_became_peered":"2024-09-16T10:22:59.497296+0000","last_unstale":"2024-09-16T10:23:08.602315+0000","last_undegraded":"2024-09-16T10:23:08.602315+0000","last_fullsized":"2024-09-16T10:23:08.602315+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,2],"avail_no_missing":["0","2","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":570,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600537+0000","last_change":"2024-09-16T10:21:56.344549+0000","last_active":"2024-09-16T10:23:08.600537+0000","last_peered":"2024-09-16T10:23:08.600537+0000","last_clean":"2024-09-16T10:23:08.600537+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:23:08.600537+0000","last_undegraded":"2024-09-16T10:23:08.600537+0000","last_fullsized":"2024-09-16T10:23:08.600537+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:56.344472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.344472+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.888279+0000","last_change":"2024-09-16T10:22:02.891069+0000","last_active":"2024-09-16T10:23:05.888279+0000","last_peered":"2024-09-16T10:23:05.888279+0000","last_clean":"2024-09-16T10:23:05.888279+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:23:05.888279+0000","last_undegraded":"2024-09-16T10:23:05.888279+0000","last_fullsized":"2024-09-16T10:23:05.888279+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:17.101314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":664,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.892531+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:23:05.892531+0000","last_peered":"2024-09-16T10:23:05.892531+0000","last_clean":"2024-09-16T10:23:05.892531+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:23:05.892531+0000","last_undegraded":"2024-09-16T10:23:05.892531+0000","last_fullsized":"2024-09-16T10:23:05.892531+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.951237+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":538,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603236+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:23:08.603236+0000","last_peered":"2024-09-16T10:23:08.603236+0000","last_clean":"2024-09-16T10:23:08.603236+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:23:08.603236+0000","last_undegraded":"2024-09-16T10:23:08.603236+0000","last_fullsized":"2024-09-16T10:23:08.603236+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":567,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.904752+0000","last_change":"2024-09-16T10:22:10.878557+0000","last_active":"2024-09-16T10:23:05.904752+0000","last_peered":"2024-09-16T10:23:05.904752+0000","last_clean":"2024-09-16T10:23:05.904752+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:23:05.904752+0000","last_undegraded":"2024-09-16T10:23:05.904752+0000","last_fullsized":"2024-09-16T10:23:05.904752+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:17:34.967567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:34.967567+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:11.735967+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.905634+0000","last_change":"2024-09-16T10:17:44.915579+0000","last_active":"2024-09-16T10:23:05.905634+0000","last_peered":"2024-09-16T10:23:05.905634+0000","last_clean":"2024-09-16T10:23:05.905634+0000","last_became_active":"2024-09-16T10:13:37.089849+0000","last_became_peered":"2024-09-16T10:13:37.089849+0000","last_unstale":"2024-09-16T10:23:05.905634+0000","last_undegraded":"2024-09-16T10:23:05.905634+0000","last_fullsized":"2024-09-16T10:23:05.905634+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:40.688843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":544,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.581319+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:23:08.581319+0000","last_peered":"2024-09-16T10:23:08.581319+0000","last_clean":"2024-09-16T10:23:08.581319+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:23:08.581319+0000","last_undegraded":"2024-09-16T10:23:08.581319+0000","last_fullsized":"2024-09-16T10:23:08.581319+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:13.522521+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":531,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591290+0000","last_change":"2024-09-16T10:20:54.392034+0000","last_active":"2024-09-16T10:23:08.591290+0000","last_peered":"2024-09-16T10:23:08.591290+0000","last_clean":"2024-09-16T10:23:08.591290+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:23:08.591290+0000","last_undegraded":"2024-09-16T10:23:08.591290+0000","last_fullsized":"2024-09-16T10:23:08.591290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.391970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.391970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:50.895339+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":598,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.583048+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:23:08.583048+0000","last_peered":"2024-09-16T10:23:08.583048+0000","last_clean":"2024-09-16T10:23:08.583048+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:23:08.583048+0000","last_undegraded":"2024-09-16T10:23:08.583048+0000","last_fullsized":"2024-09-16T10:23:08.583048+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:10.584296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":557,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889657+0000","last_change":"2024-09-16T10:22:25.105281+0000","last_active":"2024-09-16T10:23:05.889657+0000","last_peered":"2024-09-16T10:23:05.889657+0000","last_clean":"2024-09-16T10:23:05.889657+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:23:05.889657+0000","last_undegraded":"2024-09-16T10:23:05.889657+0000","last_fullsized":"2024-09-16T10:23:05.889657+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:25.105215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.105215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:33.272207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":499,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591965+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:23:08.591965+0000","last_peered":"2024-09-16T10:23:08.591965+0000","last_clean":"2024-09-16T10:23:08.591965+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:23:08.591965+0000","last_undegraded":"2024-09-16T10:23:08.591965+0000","last_fullsized":"2024-09-16T10:23:08.591965+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:44.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":523,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600541+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:23:08.600541+0000","last_peered":"2024-09-16T10:23:08.600541+0000","last_clean":"2024-09-16T10:23:08.600541+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:23:08.600541+0000","last_undegraded":"2024-09-16T10:23:08.600541+0000","last_fullsized":"2024-09-16T10:23:08.600541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":434,"reported_epoch":315,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:08.591462+0000","last_change":"2024-09-16T10:22:59.503573+0000","last_active":"2024-09-16T10:23:08.591462+0000","last_peered":"2024-09-16T10:23:08.591462+0000","last_clean":"2024-09-16T10:22:56.466247+0000","last_became_active":"2024-09-16T10:22:59.471947+0000","last_became_peered":"2024-09-16T10:22:59.471947+0000","last_unstale":"2024-09-16T10:23:08.591462+0000","last_undegraded":"2024-09-16T10:23:08.591462+0000","last_fullsized":"2024-09-16T10:23:08.591462+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[3,7,5],"avail_no_missing":["3","2","4","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":550,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591579+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:23:08.591579+0000","last_peered":"2024-09-16T10:23:08.591579+0000","last_clean":"2024-09-16T10:23:08.591579+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:23:08.591579+0000","last_undegraded":"2024-09-16T10:23:08.591579+0000","last_fullsized":"2024-09-16T10:23:08.591579+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:53.811978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":549,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.891531+0000","last_change":"2024-09-16T10:18:42.092839+0000","last_active":"2024-09-16T10:23:05.891531+0000","last_peered":"2024-09-16T10:23:05.891531+0000","last_clean":"2024-09-16T10:23:05.891531+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:23:05.891531+0000","last_undegraded":"2024-09-16T10:23:05.891531+0000","last_fullsized":"2024-09-16T10:23:05.891531+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:42.092762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.092762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:12.996569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.887545+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:23:05.887545+0000","last_peered":"2024-09-16T10:23:05.887545+0000","last_clean":"2024-09-16T10:23:05.887545+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:23:05.887545+0000","last_undegraded":"2024-09-16T10:23:05.887545+0000","last_fullsized":"2024-09-16T10:23:05.887545+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.747299+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":555,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.592228+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:23:08.592228+0000","last_peered":"2024-09-16T10:23:08.592228+0000","last_clean":"2024-09-16T10:23:08.592228+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:23:08.592228+0000","last_undegraded":"2024-09-16T10:23:08.592228+0000","last_fullsized":"2024-09-16T10:23:08.592228+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:16.625354+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":579,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591201+0000","last_change":"2024-09-16T10:21:58.334088+0000","last_active":"2024-09-16T10:23:08.591201+0000","last_peered":"2024-09-16T10:23:08.591201+0000","last_clean":"2024-09-16T10:23:08.591201+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:23:08.591201+0000","last_undegraded":"2024-09-16T10:23:08.591201+0000","last_fullsized":"2024-09-16T10:23:08.591201+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:21:58.334025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.334025+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:16.713313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":548,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.600966+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:23:08.600966+0000","last_peered":"2024-09-16T10:23:08.600966+0000","last_clean":"2024-09-16T10:23:08.600966+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:23:08.600966+0000","last_undegraded":"2024-09-16T10:23:08.600966+0000","last_fullsized":"2024-09-16T10:23:08.600966+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.896492+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:23:05.896492+0000","last_peered":"2024-09-16T10:23:05.896492+0000","last_clean":"2024-09-16T10:23:05.896492+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:23:05.896492+0000","last_undegraded":"2024-09-16T10:23:05.896492+0000","last_fullsized":"2024-09-16T10:23:05.896492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":306,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.889387+0000","last_change":"2024-09-16T10:22:02.898982+0000","last_active":"2024-09-16T10:23:05.889387+0000","last_peered":"2024-09-16T10:23:05.889387+0000","last_clean":"2024-09-16T10:23:05.889387+0000","last_became_active":"2024-09-16T10:22:02.898630+0000","last_became_peered":"2024-09-16T10:22:02.898630+0000","last_unstale":"2024-09-16T10:23:05.889387+0000","last_undegraded":"2024-09-16T10:23:05.889387+0000","last_fullsized":"2024-09-16T10:23:05.889387+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:20.058989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.580666+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:23:08.580666+0000","last_peered":"2024-09-16T10:23:08.580666+0000","last_clean":"2024-09-16T10:23:08.580666+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:23:08.580666+0000","last_undegraded":"2024-09-16T10:23:08.580666+0000","last_fullsized":"2024-09-16T10:23:08.580666+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.593004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":550,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.591560+0000","last_change":"2024-09-16T10:20:58.426297+0000","last_active":"2024-09-16T10:23:08.591560+0000","last_peered":"2024-09-16T10:23:08.591560+0000","last_clean":"2024-09-16T10:23:08.591560+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:23:08.591560+0000","last_undegraded":"2024-09-16T10:23:08.591560+0000","last_fullsized":"2024-09-16T10:23:08.591560+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.426235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.426235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:52.829722+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":572,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T10:23:05.897306+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:23:05.897306+0000","last_peered":"2024-09-16T10:23:05.897306+0000","last_clean":"2024-09-16T10:23:05.897306+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:23:05.897306+0000","last_undegraded":"2024-09-16T10:23:05.897306+0000","last_fullsized":"2024-09-16T10:23:05.897306+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:18.159732+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":557,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T10:23:08.603420+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:23:08.603420+0000","last_peered":"2024-09-16T10:23:08.603420+0000","last_clean":"2024-09-16T10:23:08.603420+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:23:08.603420+0000","last_undegraded":"2024-09-16T10:23:08.603420+0000","last_fullsized":"2024-09-16T10:23:08.603420+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:19.158658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6099566592,"num_objects":93072,"num_object_clones":0,"num_object_copies":279216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":6521,"num_objects_unfound":0,"num_objects_dirty":93072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93072,"num_write_kb":5956608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":52756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18483183616,"data_stored":18483183616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70862,"ondisk_log_size":70862,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":565,"num_read_kb":560,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":657456,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":25,"num_write":14,"num_write_kb":185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":920144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234102,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3680432,"kb_used_data":3328544,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":90064720,"statfs":{"total":95995035648,"available":92226273280,"internally_reserved":0,"allocated":3408429056,"data_stored":3407444236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":360315338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528442,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1444956,"kb_used_data":1102792,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92300196,"statfs":{"total":95995035648,"available":94515400704,"internally_reserved":0,"allocated":1129259008,"data_stored":1128297666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790080,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3522608,"kb_used_data":3206816,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":90222544,"statfs":{"total":95995035648,"available":92387885056,"internally_reserved":0,"allocated":3283779584,"data_stored":3282795892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986309,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1424700,"kb_used_data":1114472,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92320452,"statfs":{"total":95995035648,"available":94536142848,"internally_reserved":0,"allocated":1141219328,"data_stored":1140226259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215242,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3402376,"kb_used_data":2956088,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90342776,"statfs":{"total":95995035648,"available":92511002624,"internally_reserved":0,"allocated":3027034112,"data_stored":3026026482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":456980936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411472,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407328,"kb_used_data":1253968,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92337824,"statfs":{"total":95995035648,"available":94553931776,"internally_reserved":0,"allocated":1284063232,"data_stored":1283094513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3871376,"kb_used_data":3475004,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":89873776,"statfs":{"total":95995035648,"available":92030746624,"internally_reserved":0,"allocated":3558404096,"data_stored":3557391287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738521,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1837628,"kb_used_data":1495464,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91907524,"statfs":{"total":95995035648,"available":94113304576,"internally_reserved":0,"allocated":1531355136,"data_stored":1530362826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,6,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1537474560,"data_stored":1537474560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3598516224,"data_stored":3598516224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1289617408,"data_stored":1289617408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3051749376,"data_stored":3051749376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1145569280,"data_stored":1145569280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3303604224,"data_stored":3303604224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1130233856,"data_stored":1130233856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3426418688,"data_stored":3426418688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:14.273 INFO:tasks.thrashosds.thrasher:Setting 3.14 to [0, 6, 2, 7, 4, 1] 2024-09-16T10:23:14.273 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.14', '0', '6', '2', '7', '4', '1'] 2024-09-16T10:23:14.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 3.14 0 6 2 7 4 1 2024-09-16T10:23:14.307 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:14.314 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:14.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:14.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:14.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:14.422 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 73 16 99008 98992 84.7298 63.3125 0.0131013 0.0117975 2024-09-16T10:23:14.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:14.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:14.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:14.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:14.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[29993]: pgmap v519: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 697 B/s rd, 43 MiB/s wr, 686 op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 706 KiB/s, 11 objects/s recovering 2024-09-16T10:23:14.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[29993]: 3.16 scrub starts 2024-09-16T10:23:14.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[29993]: 3.16 scrub ok 2024-09-16T10:23:14.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[29993]: 4.0 scrub starts 2024-09-16T10:23:14.609 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[29993]: 4.0 scrub ok 2024-09-16T10:23:14.629 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:14.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:14.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:14.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:14.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:14.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:14.894 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:14.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:14.943 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[39470]: pgmap v519: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 697 B/s rd, 43 MiB/s wr, 686 op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 706 KiB/s, 11 objects/s recovering 2024-09-16T10:23:14.943 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[39470]: 3.16 scrub starts 2024-09-16T10:23:14.943 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[39470]: 3.16 scrub ok 2024-09-16T10:23:14.943 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[39470]: 4.0 scrub starts 2024-09-16T10:23:14.943 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:14 smithi063 ceph-mon[39470]: 4.0 scrub ok 2024-09-16T10:23:14.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:15.033 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:15.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:14 smithi165 ceph-mon[32696]: pgmap v519: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 697 B/s rd, 43 MiB/s wr, 686 op/s; 1/279816 objects degraded (0.000%); 6526/279816 objects misplaced (2.332%); 706 KiB/s, 11 objects/s recovering 2024-09-16T10:23:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:14 smithi165 ceph-mon[32696]: 3.16 scrub starts 2024-09-16T10:23:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:14 smithi165 ceph-mon[32696]: 3.16 scrub ok 2024-09-16T10:23:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:14 smithi165 ceph-mon[32696]: 4.0 scrub starts 2024-09-16T10:23:15.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:14 smithi165 ceph-mon[32696]: 4.0 scrub ok 2024-09-16T10:23:15.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:15.181 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:15.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:15.282 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:15.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:15.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:15.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:15.422 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 74 16 99962 99946 84.3905 59.625 0.0068286 0.0118443 2024-09-16T10:23:15.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:15.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:15.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:15.460 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:15.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:23:15.518 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:15.518 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:15.556 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:15.604 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:15.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:15.628 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:23:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/304092152' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:15.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:15 smithi063 ceph-mon[29993]: from='client.25783 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:15.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/304092152' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:15.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:15 smithi063 ceph-mon[39470]: from='client.25783 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:15.706 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:15.763 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:15.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:15.865 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:15.906 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:15.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:16.007 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:16.049 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:16.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:16.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/304092152' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:16.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:15 smithi165 ceph-mon[32696]: from='client.25783 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:16.151 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:16.192 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:16.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:16.293 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:16.336 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:16.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:16.422 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 75 16 100996 100980 84.1269 64.625 0.0110907 0.0118811 2024-09-16T10:23:16.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:16.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:16.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:16.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:16.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:16.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:16.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:16.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:16.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:16.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:16.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:16.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[29993]: pgmap v520: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 1/295014 objects degraded (0.000%); 6354/295014 objects misplaced (2.154%); 1.3 MiB/s, 20 objects/s recovering 2024-09-16T10:23:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[29993]: from='client.25783 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:23:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[29993]: osdmap e317: 8 total, 8 up, 8 in 2024-09-16T10:23:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/295014 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[29993]: osdmap e318: 8 total, 8 up, 8 in 2024-09-16T10:23:16.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[39470]: pgmap v520: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 1/295014 objects degraded (0.000%); 6354/295014 objects misplaced (2.154%); 1.3 MiB/s, 20 objects/s recovering 2024-09-16T10:23:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[39470]: from='client.25783 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:23:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[39470]: osdmap e317: 8 total, 8 up, 8 in 2024-09-16T10:23:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/295014 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:16 smithi063 ceph-mon[39470]: osdmap e318: 8 total, 8 up, 8 in 2024-09-16T10:23:16.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:17.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:16 smithi165 ceph-mon[32696]: pgmap v520: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 1/295014 objects degraded (0.000%); 6354/295014 objects misplaced (2.154%); 1.3 MiB/s, 20 objects/s recovering 2024-09-16T10:23:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:16 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:16 smithi165 ceph-mon[32696]: from='client.25783 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:23:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:16 smithi165 ceph-mon[32696]: osdmap e317: 8 total, 8 up, 8 in 2024-09-16T10:23:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:16 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/295014 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:17.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:16 smithi165 ceph-mon[32696]: osdmap e318: 8 total, 8 up, 8 in 2024-09-16T10:23:17.104 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.144 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:17.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:17.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:17.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:17.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:17.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:17.422 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 76 16 102182 102166 83.9953 74.125 0.00665619 0.0119002 2024-09-16T10:23:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:23:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:23:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:23:17.503 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:17.542 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:17.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:17.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:17.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:17.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:17.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:17.908 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:17.957 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:17.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:18.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:18.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:18.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:18.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:18.229 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:18.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:18.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:18.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:18.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:18.422 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 77 16 103204 103188 83.7339 63.875 0.0194673 0.0119377 2024-09-16T10:23:18.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:18.496 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:18.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:18.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:18.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:18.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:18.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:18.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:18.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:18.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:18.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:18.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:18.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:18 smithi063 ceph-mon[29993]: pgmap v523: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 807 B/s rd, 50 MiB/s wr, 799 op/s; 1/295014 objects degraded (0.000%); 6354/295014 objects misplaced (2.154%); 868 KiB/s, 13 objects/s recovering 2024-09-16T10:23:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/709105102' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]} : dispatch 2024-09-16T10:23:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:18 smithi063 ceph-mon[29993]: from='client.45728 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]} : dispatch 2024-09-16T10:23:18.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:18 smithi063 ceph-mon[39470]: pgmap v523: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 807 B/s rd, 50 MiB/s wr, 799 op/s; 1/295014 objects degraded (0.000%); 6354/295014 objects misplaced (2.154%); 868 KiB/s, 13 objects/s recovering 2024-09-16T10:23:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/709105102' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]} : dispatch 2024-09-16T10:23:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:18 smithi063 ceph-mon[39470]: from='client.45728 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]} : dispatch 2024-09-16T10:23:18.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:19.020 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:19.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:18 smithi165 ceph-mon[32696]: pgmap v523: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 807 B/s rd, 50 MiB/s wr, 799 op/s; 1/295014 objects degraded (0.000%); 6354/295014 objects misplaced (2.154%); 868 KiB/s, 13 objects/s recovering 2024-09-16T10:23:19.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/709105102' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]} : dispatch 2024-09-16T10:23:19.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:18 smithi165 ceph-mon[32696]: from='client.45728 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]} : dispatch 2024-09-16T10:23:19.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:23:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:23:19.093 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:23:19.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.185 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:19.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:19.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:19.294 INFO:teuthology.orchestra.run.smithi063.stderr:set 3.14 pg_upmap_items mapping to [0->6,2->7,4->1] 2024-09-16T10:23:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:19.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:19.417 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:19.422 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 78 16 104186 104170 83.4473 61.375 0.00292849 0.0119738 2024-09-16T10:23:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.441 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:19.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:19.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:19.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:19.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:19.696 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:19.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:19.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:19.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:19.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:19.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:19.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:20.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:20.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:19 smithi165 ceph-mon[32696]: from='client.45728 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]}]': finished 2024-09-16T10:23:20.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:19 smithi165 ceph-mon[32696]: osdmap e319: 8 total, 8 up, 8 in 2024-09-16T10:23:20.053 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:20.092 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:20.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:20.194 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:20.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:20.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:19 smithi063 ceph-mon[39470]: from='client.45728 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]}]': finished 2024-09-16T10:23:20.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:19 smithi063 ceph-mon[39470]: osdmap e319: 8 total, 8 up, 8 in 2024-09-16T10:23:20.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:20.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:19 smithi063 ceph-mon[29993]: from='client.45728 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.14", "id": [0, 6, 2, 7, 4, 1]}]': finished 2024-09-16T10:23:20.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:19 smithi063 ceph-mon[29993]: osdmap e319: 8 total, 8 up, 8 in 2024-09-16T10:23:20.234 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:20.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:20.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:20.380 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:20.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:20.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 79 16 105163 105147 83.1639 61.0625 0.0149832 0.0120182 2024-09-16T10:23:20.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:20.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:20.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:20.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:20.652 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:20.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:20.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:20.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:20.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:20.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:20.912 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:20.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:21.016 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:21.038 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:21.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:20 smithi165 ceph-mon[32696]: pgmap v525: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 1.8 MiB/s, 29 objects/s recovering 2024-09-16T10:23:21.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:20 smithi165 ceph-mon[32696]: osdmap e320: 8 total, 8 up, 8 in 2024-09-16T10:23:21.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:21.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:21.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:20 smithi063 ceph-mon[29993]: pgmap v525: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 1.8 MiB/s, 29 objects/s recovering 2024-09-16T10:23:21.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:20 smithi063 ceph-mon[29993]: osdmap e320: 8 total, 8 up, 8 in 2024-09-16T10:23:21.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:20 smithi063 ceph-mon[39470]: pgmap v525: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 1.8 MiB/s, 29 objects/s recovering 2024-09-16T10:23:21.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:20 smithi063 ceph-mon[39470]: osdmap e320: 8 total, 8 up, 8 in 2024-09-16T10:23:21.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:21.310 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:21.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:21.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:23:21.422171+0000 min lat: 0.00102889 max lat: 2.20645 avg lat: 0.0120542 2024-09-16T10:23:21.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:23:21.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 80 16 106187 106171 82.9243 64 0.0278753 0.0120542 2024-09-16T10:23:21.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:21.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:21.579 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:21.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:21.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:21.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:21.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:21.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:21.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:21.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:21.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:21 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/310617 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:21.978 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:21 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/310617 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:22.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:21 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/310617 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:22.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:22.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:22.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:22.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:22.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:22.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:22.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:22.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:22.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:22.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 81 16 107101 107085 82.6058 57.125 0.00427475 0.0121 2024-09-16T10:23:22.464 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:22.505 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:22.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:22.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:22.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:22.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:22.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:22.781 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:22.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:22.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:22.931 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:22.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:23.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:23.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:22 smithi165 ceph-mon[32696]: pgmap v527: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 912 B/s rd, 58 MiB/s wr, 928 op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 1.0 MiB/s, 16 objects/s recovering 2024-09-16T10:23:23.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:23.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:23.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:23.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:22 smithi063 ceph-mon[29993]: pgmap v527: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 912 B/s rd, 58 MiB/s wr, 928 op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 1.0 MiB/s, 16 objects/s recovering 2024-09-16T10:23:23.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:22 smithi063 ceph-mon[39470]: pgmap v527: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 912 B/s rd, 58 MiB/s wr, 928 op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 1.0 MiB/s, 16 objects/s recovering 2024-09-16T10:23:23.205 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:23.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:23.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:23.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:23.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:23.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 82 16 108187 108171 82.4261 67.875 0.0121719 0.0121267 2024-09-16T10:23:23.437 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:23.482 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:23.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:23.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:23.622 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:23.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:23.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:23.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:23.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:23.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:23.900 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:23.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:24.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:24.048 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:24.115 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:23:24.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:24.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:24.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:24.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:24.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:24.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 83 16 109284 109268 82.259 68.5625 0.0340232 0.0121501 2024-09-16T10:23:24.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:24.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:24.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:24.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:24.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:24.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:24 smithi063 ceph-mon[39470]: pgmap v528: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 698 B/s rd, 44 MiB/s wr, 710 op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 786 KiB/s, 12 objects/s recovering 2024-09-16T10:23:24.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:24.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:24.705 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:24 smithi063 ceph-mon[29993]: pgmap v528: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 698 B/s rd, 44 MiB/s wr, 710 op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 786 KiB/s, 12 objects/s recovering 2024-09-16T10:23:24.705 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:24.706 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:24.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:24.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:24 smithi165 ceph-mon[32696]: pgmap v528: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 698 B/s rd, 44 MiB/s wr, 710 op/s; 1/310617 objects degraded (0.000%); 6174/310617 objects misplaced (1.988%); 786 KiB/s, 12 objects/s recovering 2024-09-16T10:23:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:24.803 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:24.804 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:24.813 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:24.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:24.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:24.961 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:24.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:25.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:25.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:25.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:25.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:25.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:25.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:25.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:25.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:25.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:25.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 84 16 110316 110300 82.0476 64.5 0.021268 0.0121817 2024-09-16T10:23:25.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:25.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:25.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:25.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:25.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:25.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:25.699 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:23:25.700 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:23:25.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:25.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:25.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:25.748 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:23:25.749 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:23:25.783 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:23:25.784 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:23:25.786 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:25.786 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:25.823 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:23:25.823 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:23:25.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:25.861 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:23:25.862 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:23:25.876 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:25.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:25.897 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:23:25.898 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:23:25.938 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:23:25.938 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:23:25.971 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:23:25.972 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:23:25.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:26.023 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:26.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:26.124 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:26.166 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:26.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:26.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:26.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:26.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:26.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 85 16 111247 111231 81.7669 58.1875 0.0124461 0.0122241 2024-09-16T10:23:26.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:26.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:26.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:26.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:26.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:26.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:26.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:26 smithi063 ceph-mon[29993]: pgmap v529: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 1/325743 objects degraded (0.000%); 5992/325743 objects misplaced (1.839%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:23:26.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2114507053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:26.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:26 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/325743 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:26.696 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:26 smithi063 ceph-mon[39470]: pgmap v529: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 1/325743 objects degraded (0.000%); 5992/325743 objects misplaced (1.839%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:23:26.696 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2114507053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:26.696 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:26 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/325743 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:26.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:26.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:26.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:26.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:26.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:26 smithi165 ceph-mon[32696]: pgmap v529: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 1/325743 objects degraded (0.000%); 5992/325743 objects misplaced (1.839%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:23:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2114507053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:26 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/325743 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:26.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:26.907 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:26.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:27.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:27.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:27.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:27.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:27.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:27.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:27.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:27.296 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:23:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:23:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:27.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:27.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:27.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 86 16 112181 112165 81.4948 58.375 0.00795367 0.0122644 2024-09-16T10:23:27.438 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:23:27.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:27.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:27.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:27.592 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:27.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:27.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:27.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:23:27.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:23:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/113401013' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[29993]: from='client.45737 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:23:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:23:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/113401013' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:27 smithi063 ceph-mon[39470]: from='client.45737 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:27.737 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:27.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:27 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:23:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:27 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:23:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/113401013' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:27 smithi165 ceph-mon[32696]: from='client.45737 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:27.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:27.873 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:27.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:27.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:28.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:28.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:28.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:28.140 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:28.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:28.242 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:28.285 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:28.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:28.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:28.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:28.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:28.423 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 87 16 113286 113270 81.3519 69.0625 0.00599689 0.0122875 2024-09-16T10:23:28.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:28.559 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:28.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:28.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:28.691 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:28.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:28.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[29993]: pgmap v530: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 599 B/s rd, 58 MiB/s wr, 923 op/s; 1/325743 objects degraded (0.000%); 5992/325743 objects misplaced (1.839%); 1.3 MiB/s, 21 objects/s recovering 2024-09-16T10:23:28.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[29993]: from='client.45737 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[29993]: osdmap e321: 8 total, 8 up, 8 in 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[39470]: pgmap v530: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 599 B/s rd, 58 MiB/s wr, 923 op/s; 1/325743 objects degraded (0.000%); 5992/325743 objects misplaced (1.839%); 1.3 MiB/s, 21 objects/s recovering 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[39470]: from='client.45737 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[39470]: osdmap e321: 8 total, 8 up, 8 in 2024-09-16T10:23:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:23:28.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:23:28.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:28 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:23:28.703 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:23:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-mon[32696]: pgmap v530: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+remapped+backfill_toofull, 159 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 599 B/s rd, 58 MiB/s wr, 923 op/s; 1/325743 objects degraded (0.000%); 5992/325743 objects misplaced (1.839%); 1.3 MiB/s, 21 objects/s recovering 2024-09-16T10:23:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-mon[32696]: from='client.45737 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-mon[32696]: osdmap e321: 8 total, 8 up, 8 in 2024-09-16T10:23:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:23:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:23:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:28 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:23:28.793 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:28.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:28.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:23:28.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:29.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:29.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:29.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:29.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:29.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:29.256 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:29.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:29.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:23:29.398 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:29.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 88 16 114245 114229 81.1085 59.9375 0.00558277 0.0123237 2024-09-16T10:23:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:29.451 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:29.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:29.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:29.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:29.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:29 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:23:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:29 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:23:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:29.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:29 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:23:29.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:29 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:23:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:29.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:29.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:29.755 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:23:29.755 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:23:29.755 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T10:23:29.755 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:23:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:29 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:23:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:29 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:23:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:29.806 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:29.858 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:29.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:29.959 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:30.005 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:30.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:30.106 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:30.151 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:30.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:30.253 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:30.301 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:30.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:30.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:30.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 89 16 115255 115239 80.9064 63.125 0.00744922 0.012353 2024-09-16T10:23:30.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:30.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:30.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:30.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:30.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:30.662 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:30.740 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:30.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:30.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:30.784 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:30 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:23:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:30 smithi165 ceph-mon[32696]: pgmap v532: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 155 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 1.1 MiB/s, 18 objects/s recovering 2024-09-16T10:23:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:30 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:23:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:30 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:23:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:30 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:23:30.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:30.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:23:30.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[39470]: pgmap v532: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 155 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 1.1 MiB/s, 18 objects/s recovering 2024-09-16T10:23:30.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:23:30.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:23:30.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:23:30.885 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:23:30.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[29993]: pgmap v532: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 155 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 1.1 MiB/s, 18 objects/s recovering 2024-09-16T10:23:30.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:23:30.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:23:30.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:30 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:23:30.901 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:30.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:31.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:31.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:31.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:31.161 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:31.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:31.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:31.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:31.405 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total time run: 90.0129 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total writes made: 116228 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Write size: 65536 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object size: 65536 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Bandwidth (MB/sec): 80.7023 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Bandwidth: 31.7671 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max bandwidth (MB/sec): 181.938 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min bandwidth (MB/sec): 9.0625 2024-09-16T10:23:31.424 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average IOPS: 1291 2024-09-16T10:23:31.425 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev IOPS: 508.273 2024-09-16T10:23:31.425 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max IOPS: 2911 2024-09-16T10:23:31.425 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min IOPS: 145 2024-09-16T10:23:31.425 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average Latency(s): 0.0123865 2024-09-16T10:23:31.425 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Latency(s): 0.0316217 2024-09-16T10:23:31.425 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max latency(s): 2.20645 2024-09-16T10:23:31.425 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min latency(s): 0.00102889 2024-09-16T10:23:31.428 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Cleaning up (deleting benchmark objects) 2024-09-16T10:23:31.431 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:31.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:23:31.462 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:23:31.516 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:23:31.517 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:23:31.533 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:31.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:23:31.558 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:23:31.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.585 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:31.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:31.586 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:23:31.586 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:23:31.625 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:23:31.625 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:23:31.659 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:23:31.659 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:23:31.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:31.691 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:23:31.691 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:23:31.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:31.725 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:23:31.726 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:23:31.726 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:23:31.726 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:23:31.726 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:23:31.726 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[29993]: osdmap e322: 8 total, 8 up, 8 in 2024-09-16T10:23:31.726 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/340836 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:31.726 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1827774556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:31.726 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[29993]: osdmap e323: 8 total, 8 up, 8 in 2024-09-16T10:23:31.727 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:23:31.727 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:23:31.727 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:23:31.727 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[39470]: osdmap e322: 8 total, 8 up, 8 in 2024-09-16T10:23:31.727 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/340836 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:31.727 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1827774556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:31.727 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:31 smithi063 ceph-mon[39470]: osdmap e323: 8 total, 8 up, 8 in 2024-09-16T10:23:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:31 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:23:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:31 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:23:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:31 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:23:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:31 smithi165 ceph-mon[32696]: osdmap e322: 8 total, 8 up, 8 in 2024-09-16T10:23:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:31 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/340836 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1827774556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:31 smithi165 ceph-mon[32696]: osdmap e323: 8 total, 8 up, 8 in 2024-09-16T10:23:31.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:31.861 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:31.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:31.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:31.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:32.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:32.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:32.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:32.223 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:32.262 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:32.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:32.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:32.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:32.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:32.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:32.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:32.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:32.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:32.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:23:32.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:32.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:32.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:32.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:32.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:32.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[29993]: pgmap v534: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 155 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 1.4 MiB/s, 21 objects/s recovering 2024-09-16T10:23:32.791 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:23:32.791 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:23:32.791 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:23:32.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[39470]: pgmap v534: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 155 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 1.4 MiB/s, 21 objects/s recovering 2024-09-16T10:23:32.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:23:32.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:23:32.791 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:32 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:23:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:32 smithi165 ceph-mon[32696]: pgmap v534: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 155 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 1.4 MiB/s, 21 objects/s recovering 2024-09-16T10:23:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:32 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:23:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:32 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:23:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:32 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:23:32.853 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:32.853 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:32.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:32.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:32.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:33.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:33.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:33.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:33.136 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:33.188 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:33.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:33.290 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:33.343 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:33.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:33.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 323, 2024-09-16T10:23:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:23:31.458442+0000", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:22:01.866838+0000", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5806080, 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "noscrub", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:23:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 9, 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:23:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 21, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 21, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 21, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 21, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 21, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.b", 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 275, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 274, 2024-09-16T10:23:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 274, 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "308", 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "275", 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "306", 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:33.454 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:33.455 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:33.456 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:33.457 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5199999809265137, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.8999999761581421, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5199999809265137, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.8999999761581421, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:23:33.458 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:23:33.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:33.460 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:33.461 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:33.462 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:33.463 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:23:33.464 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:33.465 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:33.466 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:33.467 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:33.468 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:33.469 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:23:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:33.472 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.473 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:33.474 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:33.475 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:33.476 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.25, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.25, 2024-09-16T10:23:33.477 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.25, 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.25, 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:23:33.478 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:33.479 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.480 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:33.481 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.482 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:33.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.25, 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.25, 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.25, 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.25, 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:33.484 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:33.485 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:23:33.486 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:33.487 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:23:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 9, 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_2", 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:21:54.923489+0000", 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:23:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:23:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "279", 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:23:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:23:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:23:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:23:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2, 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2, 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 1, 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 1, 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 1 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:23:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:23:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:23:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 321, 2024-09-16T10:23:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:23:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:23:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:23:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 311, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:33.525 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:33.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:23:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:23:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:23:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:23:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:59.788186+0000", 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:23:01.692724+0000", 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:23:07.725846+0000", 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:37.120902+0000", 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:23:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:23:26.529413+0000", 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:26.467679+0000", 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:48.316098+0000", 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:22:23.134000+0000", 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [ 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "9.b", 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: 2, 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: 6 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.10", 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 6, 2024-09-16T10:23:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.14", 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 1 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.13", 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:23:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [ 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.c", 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: 5, 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.12", 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "9.b", 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-09-16T10:23:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: 6 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:23:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:23:33.552 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:23:33.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:33.665 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:33.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:33.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:23:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:23:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:23:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/538142794' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:23:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:23:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:23:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:23:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/538142794' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:23:33.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:33 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:23:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:33 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:23:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:33 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:23:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/538142794' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:23:33.803 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:33.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:33.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:33.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:23:33.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:33.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:34.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:34.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:34.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:34.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:34.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:34.295 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:34.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:34.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:34.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:34.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:34.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:34.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:34.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:23:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:34.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:34.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:34.666 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:34 smithi063 ceph-mon[29993]: pgmap v536: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 838 op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 896 KiB/s, 13 objects/s recovering 2024-09-16T10:23:34.666 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:34 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:34.666 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:34 smithi063 ceph-mon[39470]: pgmap v536: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 838 op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 896 KiB/s, 13 objects/s recovering 2024-09-16T10:23:34.666 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:34 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:34.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:34.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:34 smithi165 ceph-mon[32696]: pgmap v536: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 838 op/s; 1/340836 objects degraded (0.000%); 5824/340836 objects misplaced (1.709%); 896 KiB/s, 13 objects/s recovering 2024-09-16T10:23:34.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:34 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:34.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:34.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:34.856 INFO:tasks.thrashosds.thrasher:j is {'epoch': 323, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:23:31.458442+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:22:01.866838+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 275, 'last_epoch_started': 274, 'last_epoch_clean': 274, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '308', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '275', 'last_force_op_resend_preluminous': '306', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5199999809265137, 'score_stable': 1.899999976158142, 'optimal_score': 1, 'raw_score_acting': 1.5199999809265137, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-16T10:21:54.923489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 321, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 311, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:59.788186+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:23:01.692724+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:23:07.725846+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:37.120902+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:23:26.529413+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:26.467679+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:48.316098+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:22:23.134000+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '9.b', 'osds': [2, 1, 6]}], 'pg_upmap_items': [{'pgid': '3.10', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.c', 'osds': [3, 5, 7]}, {'pgid': '1.12', 'osds': [0, 4, 7]}, {'pgid': '9.b', 'osds': [0, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:23:34.856 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:23:34.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:34.873 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:34.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:34.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:34.896 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:34.896 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:34.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:35.034 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:35.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:35.136 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:35.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:35.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:35.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:35.339 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:35.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:35.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:35.483 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:35.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:35.558 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:35.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:35.655 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:35.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:35.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:23:35.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:35.801 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:35.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:35.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:35.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:35.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:36.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:36.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:36.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:36.124 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:36.155 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:36.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:36.206 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:23:36.206 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:23:36.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:36.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:36.245 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:23:36.245 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:23:36.277 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:23:36.277 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:23:36.312 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:23:36.312 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:23:36.331 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:36.348 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:23:36.348 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:23:36.380 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:23:36.380 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:23:36.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:36.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:36.421 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:23:36.421 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:23:36.464 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:23:36.464 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:23:36.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:36.530 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:36.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:36 smithi165 ceph-mon[32696]: pgmap v537: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.32k op/s; 1/341964 objects degraded (0.000%); 5652/341964 objects misplaced (1.653%); 1.4 MiB/s, 21 objects/s recovering 2024-09-16T10:23:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:36 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/341964 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1832205489' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:36 smithi165 ceph-mon[32696]: from='client.25813 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:36 smithi165 ceph-mon[32696]: osdmap e324: 8 total, 8 up, 8 in 2024-09-16T10:23:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4000105084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:36.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:36.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:36.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:36.758 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:23:36.758 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:23:36.758 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:23:36.758 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:23:36.758 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:23:36.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:36.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:36.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:36.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[29993]: pgmap v537: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.32k op/s; 1/341964 objects degraded (0.000%); 5652/341964 objects misplaced (1.653%); 1.4 MiB/s, 21 objects/s recovering 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/341964 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1832205489' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[29993]: from='client.25813 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[29993]: osdmap e324: 8 total, 8 up, 8 in 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4000105084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[39470]: pgmap v537: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.32k op/s; 1/341964 objects degraded (0.000%); 5652/341964 objects misplaced (1.653%); 1.4 MiB/s, 21 objects/s recovering 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/341964 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1832205489' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[39470]: from='client.25813 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:23:36.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[39470]: osdmap e324: 8 total, 8 up, 8 in 2024-09-16T10:23:36.921 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4000105084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:36.929 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:23:36.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:36.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:37.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:37.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:37.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:37.116 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:23:37.120 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:23:37.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:37.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:37.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:37.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:23:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:23:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:23:37.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:37.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:37.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:37.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:37.470 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:37.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:37.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:23:37.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:37.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:37.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:37.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:37.746 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:37.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:37.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:37 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:37.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:37 smithi165 ceph-mon[32696]: from='client.25813 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:23:37.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:37 smithi165 ceph-mon[32696]: osdmap e325: 8 total, 8 up, 8 in 2024-09-16T10:23:37.849 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:37.874 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:37.891 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:37.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:37.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:37 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:37.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:37 smithi063 ceph-mon[39470]: from='client.25813 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:23:37.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:37 smithi063 ceph-mon[39470]: osdmap e325: 8 total, 8 up, 8 in 2024-09-16T10:23:37.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:37 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:37.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:37 smithi063 ceph-mon[29993]: from='client.25813 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:23:37.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:37 smithi063 ceph-mon[29993]: osdmap e325: 8 total, 8 up, 8 in 2024-09-16T10:23:37.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:38.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:38.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:38.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:38.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:38.327 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:38.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:38.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:38.463 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:23:38.463 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:23:38.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:39780 7.1 7:87ba97cd:::benchmark_data_smithi063_73086_object39779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:09.255004+0000", 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 509.20435931399999, 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.132848357, 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39780 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255004+0000", 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255004+0000", 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255007+0000", 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4930000000000001e-06 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255051+0000", 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4141999999999999e-05 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255072+0000", 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0594e-05 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255084+0000", 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2075999999999999e-05 2024-09-16T10:23:38.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255147+0000", 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2867000000000006e-05 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255233+0000", 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6434999999999995e-05 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.255375+0000", 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014167199999999999 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.256696+0000", 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001321572 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.257082+0000", 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000385521 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.387743+0000", 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13066166400000001 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.387809+0000", 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5554000000000003e-05 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:09.387853+0000", 2024-09-16T10:23:38.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3767000000000003e-05 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70291 7.1 7:8090c1a5:::benchmark_data_smithi063_73086_object70290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.56117727100002, 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12180724800000001, 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.471 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70291 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898188+0000", 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898240+0000", 2024-09-16T10:23:38.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1740999999999998e-05 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898249+0000", 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4490000000000005e-06 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898254+0000", 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2569999999999998e-06 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898300+0000", 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5454000000000002e-05 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898383+0000", 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3005999999999998e-05 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898476+0000", 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3029000000000007e-05 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.905816+0000", 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073403699999999997 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906673+0000", 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085627000000000001 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019900+0000", 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11322701 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019950+0000", 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0862999999999997e-05 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019994+0000", 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3169e-05 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.478 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70294 7.1 7:8c6bcbeb:::benchmark_data_smithi063_73086_object70293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:23:38.478 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:23:38.478 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55911405799998, 2024-09-16T10:23:38.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119864807, 2024-09-16T10:23:38.478 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.478 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.478 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.479 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.479 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70294 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:23:38.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900252+0000", 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1119999999999999e-06 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900304+0000", 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1965999999999997e-05 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900313+0000", 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4690000000000003e-06 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900320+0000", 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.404e-06 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900758+0000", 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000438023 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900864+0000", 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106434 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900994+0000", 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129567 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906642+0000", 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056482299999999997 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906681+0000", 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9443000000000002e-05 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020041+0000", 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113360006 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020084+0000", 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2623999999999997e-05 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020114+0000", 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0528999999999999e-05 2024-09-16T10:23:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70297 7.1 7:8aa306c3:::benchmark_data_smithi063_73086_object70296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55884563000001, 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119738053, 2024-09-16T10:23:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70297 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900519+0000", 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1799999999999999e-06 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900602+0000", 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2973000000000007e-05 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900612+0000", 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3230000000000004e-06 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900618+0000", 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9489999999999999e-06 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903224+0000", 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026055459999999998 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903382+0000", 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015748800000000001 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903501+0000", 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119727 2024-09-16T10:23:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906766+0000", 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032648949999999999 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019591+0000", 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112824517 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020144+0000", 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055325600000000002 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020224+0000", 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9878999999999999e-05 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020256+0000", 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2320000000000002e-05 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70298 7.3 e106/81 7:c8a39fda:::benchmark_data_smithi063_73086_object70297:head v 106'4332, mlcod=106'4328)", 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55693075300002, 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.117018094, 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902434+0000", 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5570000000000001e-06 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902490+0000", 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5559000000000001e-05 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902501+0000", 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0879999999999999e-05 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902508+0000", 2024-09-16T10:23:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.204e-06 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902555+0000", 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6430000000000001e-05 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902566+0000", 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1922999999999999e-05 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906782+0000", 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042155300000000003 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019451+0000", 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112669011 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70300 7.8 e106/81 7:10588a9d:::benchmark_data_smithi063_73086_object70299:head v 106'4365, mlcod=106'4363)", 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55575508800001, 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118272198, 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903610+0000", 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0640000000000001e-06 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903703+0000", 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3548000000000006e-05 2024-09-16T10:23:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903721+0000", 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8036000000000001e-05 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903727+0000", 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.156e-06 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903787+0000", 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9658999999999999e-05 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903816+0000", 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8988000000000001e-05 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.021844+0000", 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11802747700000001 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.021881+0000", 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7270000000000001e-05 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70292 7.9 e106/81 7:99cfcedf:::benchmark_data_smithi063_73086_object70291:head v 106'4342, mlcod=106'4340)", 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.904195+0000", 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55516859400001, 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11532339799999999, 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904195+0000", 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904195+0000", 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904196+0000", 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1230000000000001e-06 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904230+0000", 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4264000000000003e-05 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904239+0000", 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6260000000000001e-06 2024-09-16T10:23:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904245+0000", 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5470000000000004e-06 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.905845+0000", 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016002760000000001 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.905862+0000", 2024-09-16T10:23:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6835000000000001e-05 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019483+0000", 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11362138200000001 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019518+0000", 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5345000000000001e-05 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70296 7.4 e106/81 7:2c63ac7c:::benchmark_data_smithi063_73086_object70295:head v 106'4337, mlcod=106'4335)", 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.904254+0000", 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55510961300001, 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11525693200000001, 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904254+0000", 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904254+0000", 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904255+0000", 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904283+0000", 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8755000000000002e-05 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904289+0000", 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2889999999999997e-06 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904293+0000", 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1559999999999997e-06 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904355+0000", 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1920000000000003e-05 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.904372+0000", 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6883000000000001e-05 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906808+0000", 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024367820000000002 2024-09-16T10:23:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019511+0000", 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112702711 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70303 7.8 e106/81 7:176924a2:::benchmark_data_smithi063_73086_object70302:head v 106'4366, mlcod=106'4363)", 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55272514500001, 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115509458, 2024-09-16T10:23:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906640+0000", 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.215e-06 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906678+0000", 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8124000000000003e-05 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906691+0000", 2024-09-16T10:23:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3244999999999999e-05 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906698+0000", 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2949999999999996e-06 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906760+0000", 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1163999999999999e-05 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906778+0000", 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8695e-05 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.022123+0000", 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115344601 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.022148+0000", 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5119000000000001e-05 2024-09-16T10:23:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.44750.0:70297 7.1 e106/81 ondisk, result = 0)", 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.906846+0000", 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.55251793999997, 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112753247, 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906846+0000", 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906846+0000", 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906847+0000", 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9999999999999996e-07 2024-09-16T10:23:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906851+0000", 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2570000000000001e-06 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019522+0000", 2024-09-16T10:23:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112670991 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019529+0000", 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019576+0000", 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7012000000000003e-05 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019584+0000", 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2840000000000006e-06 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019599+0000", 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4664000000000001e-05 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70304 7.3 e106/81 7:c3ecd6bf:::benchmark_data_smithi063_73086_object70303:head v 106'4333, mlcod=106'4331)", 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.909677+0000", 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 486.54968625599997, 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.114141138, 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909677+0000", 2024-09-16T10:23:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909677+0000", 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909679+0000", 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6270000000000001e-06 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.909762+0000", 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3071999999999993e-05 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019488+0000", 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10972603 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019500+0000", 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1464999999999999e-05 2024-09-16T10:23:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019619+0000", 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118899 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019639+0000", 2024-09-16T10:23:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200999999999999e-05 2024-09-16T10:23:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:38.525 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:38.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.023760+0000", 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041216530000000003 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.023819+0000", 2024-09-16T10:23:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8190999999999999e-05 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174929 7.1 7:83c2bca2:::benchmark_data_smithi063_73086_object55650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 384.59088617800001, 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12126697, 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174929 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868479+0000", 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2869999999999999e-06 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868482+0000", 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6799999999999999e-06 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868491+0000", 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4889999999999995e-06 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868498+0000", 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9450000000000004e-06 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868544+0000", 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5828000000000002e-05 2024-09-16T10:23:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868647+0000", 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102902 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.869028+0000", 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038142800000000001 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.871769+0000", 2024-09-16T10:23:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027412370000000001 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.871892+0000", 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012271299999999999 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989639+0000", 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.117746694 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989694+0000", 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5634000000000003e-05 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989744+0000", 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0133000000000003e-05 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174942 7.1 7:89bad1de:::benchmark_data_smithi063_73086_object55663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:23:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.878911+0000", 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 384.580452788, 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110946397, 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174942 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878911+0000", 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878911+0000", 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878913+0000", 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9069999999999999e-06 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878918+0000", 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4829999999999999e-06 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878928+0000", 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5070000000000006e-06 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.878935+0000", 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0520000000000004e-06 2024-09-16T10:23:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.879041+0000", 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106211 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.879153+0000", 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011212 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.879254+0000", 2024-09-16T10:23:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010049500000000001 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.882256+0000", 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030022820000000002 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.882267+0000", 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0906e-05 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989780+0000", 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.107513172 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989822+0000", 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2045000000000001e-05 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989857+0000", 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5216999999999998e-05 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:201826 7.4 e146/81 7:2bb12e82:::benchmark_data_smithi063_73086_object82547:head v 146'12527, mlcod=146'12525)", 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 357.09584349699998, 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13052909300000001, 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363522+0000", 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.686e-06 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363529+0000", 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5209999999999997e-06 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363541+0000", 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.163e-05 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363549+0000", 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2870000000000004e-06 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363609+0000", 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0059000000000002e-05 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363629+0000", 2024-09-16T10:23:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9953999999999999e-05 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.494008+0000", 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13037901800000001 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.494049+0000", 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0938e-05 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:201829 7.1 7:86c5e7dd:::benchmark_data_smithi063_73086_object82550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 357.09549668400001, 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13543617699999999, 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 201829 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363869+0000", 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.748e-06 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363874+0000", 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4380000000000003e-06 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489286+0000", 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12541170800000001 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489301+0000", 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4732999999999999e-05 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489378+0000", 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7651000000000006e-05 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489473+0000", 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4846999999999995e-05 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489580+0000", 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106932 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.497200+0000", 2024-09-16T10:23:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076199780000000003 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.497235+0000", 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4955e-05 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499237+0000", 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020019410000000001 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499273+0000", 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6531e-05 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499303+0000", 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9714999999999999e-05 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:201820 7.9 e146/81 7:907729a7:::benchmark_data_smithi063_73086_object82541:head v 146'12370, mlcod=146'12367)", 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 357.090899036, 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13076132600000001, 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368466+0000", 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5910000000000001e-06 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368474+0000", 2024-09-16T10:23:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4989999999999997e-06 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489290+0000", 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120816008 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489300+0000", 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0530000000000001e-05 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489358+0000", 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7380999999999998e-05 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489379+0000", 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1596000000000001e-05 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499198+0000", 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098188400000000006 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499226+0000", 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7881e-05 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:217879 7.1 7:80692134:::benchmark_data_smithi063_73086_object98600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:57.648897+0000", 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 340.810466378, 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.108476776, 2024-09-16T10:23:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 217879 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.648897+0000", 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.648897+0000", 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.648900+0000", 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.272e-06 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.648905+0000", 2024-09-16T10:23:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6210000000000001e-06 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.648926+0000", 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0797000000000001e-05 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.648933+0000", 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.648989+0000", 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5619999999999999e-05 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.649075+0000", 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6310000000000005e-05 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.649165+0000", 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0020999999999998e-05 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.654207+0000", 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050414600000000002 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.656399+0000", 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002192856 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.757301+0000", 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100902043 2024-09-16T10:23:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.757335+0000", 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.349e-05 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:57.757374+0000", 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9125000000000002e-05 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:130408 8.2 8:42c64008:::benchmark_data_smithi063_101290_object3088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:13.779352+0000", 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 204.680011482, 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11462703, 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 130408 2024-09-16T10:23:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.779352+0000", 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.779352+0000", 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.779354+0000", 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.928e-06 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.779359+0000", 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.339e-06 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.779371+0000", 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1980999999999999e-05 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.779379+0000", 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.780179+0000", 2024-09-16T10:23:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000799667 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.780442+0000", 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026333800000000001 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.780736+0000", 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029381399999999999 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.785848+0000", 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005112821 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.787439+0000", 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015909609999999999 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.893904+0000", 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106464059 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.893943+0000", 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9013000000000001e-05 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.893979+0000", 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6699000000000002e-05 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:213900 8.2 8:4fe36798:::benchmark_data_smithi063_101290_object86580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:13.109732+0000", 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 145.34963183900001, 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106675095, 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213900 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109732+0000", 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109732+0000", 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109734+0000", 2024-09-16T10:23:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1160000000000002e-06 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109741+0000", 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7179999999999999e-06 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109754+0000", 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3772e-05 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109761+0000", 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5470000000000002e-06 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109834+0000", 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2780000000000005e-05 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.109958+0000", 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012455800000000001 2024-09-16T10:23:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.110093+0000", 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013471499999999999 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.111909+0000", 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018156680000000001 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.123359+0000", 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01145058 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.216301+0000", 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092941629999999997 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.216359+0000", 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8366999999999999e-05 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.216407+0000", 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7644e-05 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:231096 8.2 8:4cceeac5:::benchmark_data_smithi063_101290_object103776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 133.704642264, 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115767351, 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 231096 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754723+0000", 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754727+0000", 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2300000000000002e-06 2024-09-16T10:23:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754734+0000", 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7850000000000003e-06 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754739+0000", 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7979999999999996e-06 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754843+0000", 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010442000000000001 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754957+0000", 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113748 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:23:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.755066+0000", 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109279 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758427+0000", 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033604519999999999 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758995+0000", 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056835100000000003 2024-09-16T10:23:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870387+0000", 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11139234100000001 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870444+0000", 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6450999999999999e-05 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870489+0000", 2024-09-16T10:23:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4863000000000001e-05 2024-09-16T10:23:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:38.583 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:23:38.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:38.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:38 smithi165 ceph-mon[32696]: from='client.45767 v1:172.21.15.63:0/1459470952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:23:38.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:38 smithi165 ceph-mon[32696]: pgmap v540: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 876 op/s; 1/341964 objects degraded (0.000%); 5652/341964 objects misplaced (1.653%); 917 KiB/s, 14 objects/s recovering 2024-09-16T10:23:38.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:38.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:38.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:38.696 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:38 smithi063 ceph-mon[29993]: from='client.45767 v1:172.21.15.63:0/1459470952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:23:38.696 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:38 smithi063 ceph-mon[29993]: pgmap v540: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 876 op/s; 1/341964 objects degraded (0.000%); 5652/341964 objects misplaced (1.653%); 917 KiB/s, 14 objects/s recovering 2024-09-16T10:23:38.696 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:38.697 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:38 smithi063 ceph-mon[39470]: from='client.45767 v1:172.21.15.63:0/1459470952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:23:38.697 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:38 smithi063 ceph-mon[39470]: pgmap v540: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 876 op/s; 1/341964 objects degraded (0.000%); 5652/341964 objects misplaced (1.653%); 917 KiB/s, 14 objects/s recovering 2024-09-16T10:23:38.697 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:38.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:38.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:38.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:38.906 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:38.948 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:38.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:39.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:39.051 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:23:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:23:39.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:39.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:39.100 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2024-09-16T10:23:36.918076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7457808451,"num_objects":113988,"num_object_clones":0,"num_object_copies":341964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":5652,"num_objects_unfound":0,"num_objects_dirty":113988,"num_whiteouts":0,"num_read":1326,"num_read_kb":1089,"num_write":119120,"num_write_kb":7438823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":82318860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72217,"ondisk_log_size":72217,"up":504,"acting":503,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":517,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25222752,"kb_used_data":22310408,"kb_used_omap":12,"kb_used_meta":2912051,"kb_avail":724738464,"statfs":{"total":767960285184,"available":742132187136,"internally_reserved":0,"allocated":22845857792,"data_stored":22837580617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2981940785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24576024,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-172,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5258,"num_write_kb":180225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":5636096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.456908"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":589,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467179+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:23:31.467179+0000","last_peered":"2024-09-16T10:23:31.467179+0000","last_clean":"2024-09-16T10:23:31.467179+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:23:31.467179+0000","last_undegraded":"2024-09-16T10:23:31.467179+0000","last_fullsized":"2024-09-16T10:23:31.467179+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":621,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464962+0000","last_change":"2024-09-16T10:19:55.724523+0000","last_active":"2024-09-16T10:23:31.464962+0000","last_peered":"2024-09-16T10:23:31.464962+0000","last_clean":"2024-09-16T10:23:31.464962+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:23:31.464962+0000","last_undegraded":"2024-09-16T10:23:31.464962+0000","last_fullsized":"2024-09-16T10:23:31.464962+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:19:55.724447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:55.724447+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:58.750050+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":568,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.475912+0000","last_change":"2024-09-16T10:22:02.564509+0000","last_active":"2024-09-16T10:23:31.475912+0000","last_peered":"2024-09-16T10:23:31.475912+0000","last_clean":"2024-09-16T10:23:31.475912+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:23:31.475912+0000","last_undegraded":"2024-09-16T10:23:31.475912+0000","last_fullsized":"2024-09-16T10:23:31.475912+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.564458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.564458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":562,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465304+0000","last_change":"2024-09-16T10:22:35.847447+0000","last_active":"2024-09-16T10:23:31.465304+0000","last_peered":"2024-09-16T10:23:31.465304+0000","last_clean":"2024-09-16T10:23:31.465304+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:23:31.465304+0000","last_undegraded":"2024-09-16T10:23:31.465304+0000","last_fullsized":"2024-09-16T10:23:31.465304+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:35.847382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:35.847382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.635415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":554,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.483120+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:23:31.483120+0000","last_peered":"2024-09-16T10:23:31.483120+0000","last_clean":"2024-09-16T10:23:31.483120+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:23:31.483120+0000","last_undegraded":"2024-09-16T10:23:31.483120+0000","last_fullsized":"2024-09-16T10:23:31.483120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:24.904699+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":589,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.481092+0000","last_change":"2024-09-16T10:22:30.653658+0000","last_active":"2024-09-16T10:23:31.481092+0000","last_peered":"2024-09-16T10:23:31.481092+0000","last_clean":"2024-09-16T10:23:31.481092+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:23:31.481092+0000","last_undegraded":"2024-09-16T10:23:31.481092+0000","last_fullsized":"2024-09-16T10:23:31.481092+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:30.653580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:30.653580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:40.872677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":634,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.481177+0000","last_change":"2024-09-16T10:22:02.945083+0000","last_active":"2024-09-16T10:23:31.481177+0000","last_peered":"2024-09-16T10:23:31.481177+0000","last_clean":"2024-09-16T10:23:31.481177+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:23:31.481177+0000","last_undegraded":"2024-09-16T10:23:31.481177+0000","last_fullsized":"2024-09-16T10:23:31.481177+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:20:31.270548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.270548+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:19.923508+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":559,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464877+0000","last_change":"2024-09-16T10:18:37.614574+0000","last_active":"2024-09-16T10:23:31.464877+0000","last_peered":"2024-09-16T10:23:31.464877+0000","last_clean":"2024-09-16T10:23:31.464877+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:23:31.464877+0000","last_undegraded":"2024-09-16T10:23:31.464877+0000","last_fullsized":"2024-09-16T10:23:31.464877+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:37.614530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:37.614530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:19.787139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":583,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470625+0000","last_change":"2024-09-16T10:23:00.881755+0000","last_active":"2024-09-16T10:23:31.470625+0000","last_peered":"2024-09-16T10:23:31.470625+0000","last_clean":"2024-09-16T10:23:31.470625+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:23:31.470625+0000","last_undegraded":"2024-09-16T10:23:31.470625+0000","last_fullsized":"2024-09-16T10:23:31.470625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:00.881704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:00.881704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:08.353844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":586,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465611+0000","last_change":"2024-09-16T10:21:58.648495+0000","last_active":"2024-09-16T10:23:31.465611+0000","last_peered":"2024-09-16T10:23:31.465611+0000","last_clean":"2024-09-16T10:23:31.465611+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:23:31.465611+0000","last_undegraded":"2024-09-16T10:23:31.465611+0000","last_fullsized":"2024-09-16T10:23:31.465611+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:21:58.648414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.648414+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:16.721272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":559,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469844+0000","last_change":"2024-09-16T10:19:51.099949+0000","last_active":"2024-09-16T10:23:31.469844+0000","last_peered":"2024-09-16T10:23:31.469844+0000","last_clean":"2024-09-16T10:23:31.469844+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:23:31.469844+0000","last_undegraded":"2024-09-16T10:23:31.469844+0000","last_fullsized":"2024-09-16T10:23:31.469844+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:51.099878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:51.099878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:18.900721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":563,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466749+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:23:31.466749+0000","last_peered":"2024-09-16T10:23:31.466749+0000","last_clean":"2024-09-16T10:23:31.466749+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:23:31.466749+0000","last_undegraded":"2024-09-16T10:23:31.466749+0000","last_fullsized":"2024-09-16T10:23:31.466749+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.729203+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":543,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469964+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:23:31.469964+0000","last_peered":"2024-09-16T10:23:31.469964+0000","last_clean":"2024-09-16T10:23:31.469964+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:23:31.469964+0000","last_undegraded":"2024-09-16T10:23:31.469964+0000","last_fullsized":"2024-09-16T10:23:31.469964+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.926610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":580,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465714+0000","last_change":"2024-09-16T10:21:57.684221+0000","last_active":"2024-09-16T10:23:31.465714+0000","last_peered":"2024-09-16T10:23:31.465714+0000","last_clean":"2024-09-16T10:23:31.465714+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:23:31.465714+0000","last_undegraded":"2024-09-16T10:23:31.465714+0000","last_fullsized":"2024-09-16T10:23:31.465714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-16T10:21:57.684172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.684172+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:29.735932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":559,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462618+0000","last_change":"2024-09-16T10:22:22.118256+0000","last_active":"2024-09-16T10:23:31.462618+0000","last_peered":"2024-09-16T10:23:31.462618+0000","last_clean":"2024-09-16T10:23:31.462618+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:23:31.462618+0000","last_undegraded":"2024-09-16T10:23:31.462618+0000","last_fullsized":"2024-09-16T10:23:31.462618+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.118194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.118194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":571,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470331+0000","last_change":"2024-09-16T10:21:20.402146+0000","last_active":"2024-09-16T10:23:31.470331+0000","last_peered":"2024-09-16T10:23:31.470331+0000","last_clean":"2024-09-16T10:23:31.470331+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:23:31.470331+0000","last_undegraded":"2024-09-16T10:23:31.470331+0000","last_fullsized":"2024-09-16T10:23:31.470331+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.402080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.402080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:40.966067+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":555,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467123+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:23:31.467123+0000","last_peered":"2024-09-16T10:23:31.467123+0000","last_clean":"2024-09-16T10:23:31.467123+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:23:31.467123+0000","last_undegraded":"2024-09-16T10:23:31.467123+0000","last_fullsized":"2024-09-16T10:23:31.467123+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:52.665306+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":566,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477928+0000","last_change":"2024-09-16T10:22:27.210541+0000","last_active":"2024-09-16T10:23:31.477928+0000","last_peered":"2024-09-16T10:23:31.477928+0000","last_clean":"2024-09-16T10:23:31.477928+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:23:31.477928+0000","last_undegraded":"2024-09-16T10:23:31.477928+0000","last_fullsized":"2024-09-16T10:23:31.477928+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:34.232050+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":110,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469798+0000","last_change":"2024-09-16T10:22:02.948234+0000","last_active":"2024-09-16T10:23:31.469798+0000","last_peered":"2024-09-16T10:23:31.469798+0000","last_clean":"2024-09-16T10:23:31.469798+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:23:31.469798+0000","last_undegraded":"2024-09-16T10:23:31.469798+0000","last_fullsized":"2024-09-16T10:23:31.469798+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:19:14.579662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:14.579662+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":559,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477901+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:23:31.477901+0000","last_peered":"2024-09-16T10:23:31.477901+0000","last_clean":"2024-09-16T10:23:31.477901+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:23:31.477901+0000","last_undegraded":"2024-09-16T10:23:31.477901+0000","last_fullsized":"2024-09-16T10:23:31.477901+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":512,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.488670+0000","last_change":"2024-09-16T10:21:57.358479+0000","last_active":"2024-09-16T10:23:31.488670+0000","last_peered":"2024-09-16T10:23:31.488670+0000","last_clean":"2024-09-16T10:23:31.488670+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:23:31.488670+0000","last_undegraded":"2024-09-16T10:23:31.488670+0000","last_fullsized":"2024-09-16T10:23:31.488670+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.358418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.358418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":572,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465748+0000","last_change":"2024-09-16T10:23:08.721407+0000","last_active":"2024-09-16T10:23:31.465748+0000","last_peered":"2024-09-16T10:23:31.465748+0000","last_clean":"2024-09-16T10:23:31.465748+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:23:31.465748+0000","last_undegraded":"2024-09-16T10:23:31.465748+0000","last_fullsized":"2024-09-16T10:23:31.465748+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:08.721339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:08.721339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:12.259384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":620,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480470+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:23:31.480470+0000","last_peered":"2024-09-16T10:23:31.480470+0000","last_clean":"2024-09-16T10:23:31.480470+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:23:31.480470+0000","last_undegraded":"2024-09-16T10:23:31.480470+0000","last_fullsized":"2024-09-16T10:23:31.480470+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:07.520262+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":580,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466967+0000","last_change":"2024-09-16T10:22:05.779269+0000","last_active":"2024-09-16T10:23:31.466967+0000","last_peered":"2024-09-16T10:23:31.466967+0000","last_clean":"2024-09-16T10:23:31.466967+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:23:31.466967+0000","last_undegraded":"2024-09-16T10:23:31.466967+0000","last_fullsized":"2024-09-16T10:23:31.466967+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.779225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.779225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:28.761246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":570,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.463663+0000","last_change":"2024-09-16T10:23:09.749818+0000","last_active":"2024-09-16T10:23:31.463663+0000","last_peered":"2024-09-16T10:23:31.463663+0000","last_clean":"2024-09-16T10:23:31.463663+0000","last_became_active":"2024-09-16T10:22:02.905797+0000","last_became_peered":"2024-09-16T10:22:02.905797+0000","last_unstale":"2024-09-16T10:23:31.463663+0000","last_undegraded":"2024-09-16T10:23:31.463663+0000","last_fullsized":"2024-09-16T10:23:31.463663+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:23:09.749746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:09.749746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:18.299941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":567,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.476043+0000","last_change":"2024-09-16T10:17:13.137316+0000","last_active":"2024-09-16T10:23:31.476043+0000","last_peered":"2024-09-16T10:23:31.476043+0000","last_clean":"2024-09-16T10:23:31.476043+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:23:31.476043+0000","last_undegraded":"2024-09-16T10:23:31.476043+0000","last_fullsized":"2024-09-16T10:23:31.476043+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:17:13.137191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:13.137191+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":586,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.468275+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:23:31.468275+0000","last_peered":"2024-09-16T10:23:31.468275+0000","last_clean":"2024-09-16T10:23:31.468275+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:23:31.468275+0000","last_undegraded":"2024-09-16T10:23:31.468275+0000","last_fullsized":"2024-09-16T10:23:31.468275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":690,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466748+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:23:31.466748+0000","last_peered":"2024-09-16T10:23:31.466748+0000","last_clean":"2024-09-16T10:23:31.466748+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:23:31.466748+0000","last_undegraded":"2024-09-16T10:23:31.466748+0000","last_fullsized":"2024-09-16T10:23:31.466748+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"323'7248","reported_seq":7575,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.666372+0000","last_change":"2024-09-16T10:22:05.902394+0000","last_active":"2024-09-16T10:23:33.666372+0000","last_peered":"2024-09-16T10:23:33.666372+0000","last_clean":"2024-09-16T10:23:33.666372+0000","last_became_active":"2024-09-16T10:22:05.900726+0000","last_became_peered":"2024-09-16T10:22:05.900726+0000","last_unstale":"2024-09-16T10:23:33.666372+0000","last_undegraded":"2024-09-16T10:23:33.666372+0000","last_fullsized":"2024-09-16T10:23:33.666372+0000","mapping_epoch":284,"log_start":"295'3038","ondisk_log_start":"295'3038","created":276,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453771264,"num_objects":6924,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":453504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":420,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469606+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:23:31.469606+0000","last_peered":"2024-09-16T10:23:31.469606+0000","last_clean":"2024-09-16T10:23:31.469606+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:23:31.469606+0000","last_undegraded":"2024-09-16T10:23:31.469606+0000","last_fullsized":"2024-09-16T10:23:31.469606+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:29.222578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":581,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.476376+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:23:31.476376+0000","last_peered":"2024-09-16T10:23:31.476376+0000","last_clean":"2024-09-16T10:23:31.476376+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:23:31.476376+0000","last_undegraded":"2024-09-16T10:23:31.476376+0000","last_fullsized":"2024-09-16T10:23:31.476376+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":575,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.481625+0000","last_change":"2024-09-16T10:23:03.779125+0000","last_active":"2024-09-16T10:23:31.481625+0000","last_peered":"2024-09-16T10:23:31.481625+0000","last_clean":"2024-09-16T10:23:31.481625+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:23:31.481625+0000","last_undegraded":"2024-09-16T10:23:31.481625+0000","last_fullsized":"2024-09-16T10:23:31.481625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:03.779044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:03.779044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:21.866780+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":679,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467224+0000","last_change":"2024-09-16T10:21:22.928090+0000","last_active":"2024-09-16T10:23:31.467224+0000","last_peered":"2024-09-16T10:23:31.467224+0000","last_clean":"2024-09-16T10:23:31.467224+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:23:31.467224+0000","last_undegraded":"2024-09-16T10:23:31.467224+0000","last_fullsized":"2024-09-16T10:23:31.467224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:21:22.928016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:22.928016+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:44.692279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":556,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469832+0000","last_change":"2024-09-16T10:23:07.772284+0000","last_active":"2024-09-16T10:23:31.469832+0000","last_peered":"2024-09-16T10:23:31.469832+0000","last_clean":"2024-09-16T10:23:31.469832+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:23:31.469832+0000","last_undegraded":"2024-09-16T10:23:31.469832+0000","last_fullsized":"2024-09-16T10:23:31.469832+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:07.772163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:07.772163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:09.861820+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"323'7419","reported_seq":7525,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.230853+0000","last_change":"2024-09-16T10:21:56.237699+0000","last_active":"2024-09-16T10:23:33.230853+0000","last_peered":"2024-09-16T10:23:33.230853+0000","last_clean":"2024-09-16T10:23:33.230853+0000","last_became_active":"2024-09-16T10:21:56.237309+0000","last_became_peered":"2024-09-16T10:21:56.237309+0000","last_unstale":"2024-09-16T10:23:33.230853+0000","last_undegraded":"2024-09-16T10:23:33.230853+0000","last_fullsized":"2024-09-16T10:23:33.230853+0000","mapping_epoch":276,"log_start":"295'3204","ondisk_log_start":"295'3204","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4215,"log_dups_size":0,"ondisk_log_size":4215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468320280,"num_objects":7147,"num_object_clones":0,"num_object_copies":21441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7419,"num_write_kb":466049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":461,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466231+0000","last_change":"2024-09-16T10:23:30.466595+0000","last_active":"2024-09-16T10:23:31.466231+0000","last_peered":"2024-09-16T10:23:31.466231+0000","last_clean":"2024-09-16T10:23:31.466231+0000","last_became_active":"2024-09-16T10:22:56.409926+0000","last_became_peered":"2024-09-16T10:22:56.409926+0000","last_unstale":"2024-09-16T10:23:31.466231+0000","last_undegraded":"2024-09-16T10:23:31.466231+0000","last_fullsized":"2024-09-16T10:23:31.466231+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:25.837614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.837614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:57.753277+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":595,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480866+0000","last_change":"2024-09-16T10:23:13.871790+0000","last_active":"2024-09-16T10:23:31.480866+0000","last_peered":"2024-09-16T10:23:31.480866+0000","last_clean":"2024-09-16T10:23:31.480866+0000","last_became_active":"2024-09-16T10:13:39.105787+0000","last_became_peered":"2024-09-16T10:13:39.105787+0000","last_unstale":"2024-09-16T10:23:31.480866+0000","last_undegraded":"2024-09-16T10:23:31.480866+0000","last_fullsized":"2024-09-16T10:23:31.480866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:23:13.871560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.871560+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:41.472395+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":532,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.489177+0000","last_change":"2024-09-16T10:21:59.329899+0000","last_active":"2024-09-16T10:23:31.489177+0000","last_peered":"2024-09-16T10:23:31.489177+0000","last_clean":"2024-09-16T10:23:31.489177+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:23:31.489177+0000","last_undegraded":"2024-09-16T10:23:31.489177+0000","last_fullsized":"2024-09-16T10:23:31.489177+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.329856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.329856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":306,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465070+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:23:31.465070+0000","last_peered":"2024-09-16T10:23:31.465070+0000","last_clean":"2024-09-16T10:23:31.465070+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:23:31.465070+0000","last_undegraded":"2024-09-16T10:23:31.465070+0000","last_fullsized":"2024-09-16T10:23:31.465070+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":577,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465583+0000","last_change":"2024-09-16T10:22:02.899911+0000","last_active":"2024-09-16T10:23:31.465583+0000","last_peered":"2024-09-16T10:23:31.465583+0000","last_clean":"2024-09-16T10:23:31.465583+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:23:31.465583+0000","last_undegraded":"2024-09-16T10:23:31.465583+0000","last_fullsized":"2024-09-16T10:23:31.465583+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.711749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.711749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:15.750630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"323'7315","reported_seq":7420,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.718234+0000","last_change":"2024-09-16T10:21:56.237148+0000","last_active":"2024-09-16T10:23:33.718234+0000","last_peered":"2024-09-16T10:23:33.718234+0000","last_clean":"2024-09-16T10:23:33.718234+0000","last_became_active":"2024-09-16T10:21:56.236822+0000","last_became_peered":"2024-09-16T10:21:56.236822+0000","last_unstale":"2024-09-16T10:23:33.718234+0000","last_undegraded":"2024-09-16T10:23:33.718234+0000","last_fullsized":"2024-09-16T10:23:33.718234+0000","mapping_epoch":276,"log_start":"295'3061","ondisk_log_start":"295'3061","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457375744,"num_objects":6979,"num_object_clones":0,"num_object_copies":20937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":457408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":453,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.484548+0000","last_change":"2024-09-16T10:23:30.466646+0000","last_active":"2024-09-16T10:23:31.484548+0000","last_peered":"2024-09-16T10:23:31.484548+0000","last_clean":"2024-09-16T10:23:31.484548+0000","last_became_active":"2024-09-16T10:22:56.446950+0000","last_became_peered":"2024-09-16T10:22:56.446950+0000","last_unstale":"2024-09-16T10:23:31.484548+0000","last_undegraded":"2024-09-16T10:23:31.484548+0000","last_fullsized":"2024-09-16T10:23:31.484548+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:57.952234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":592,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465175+0000","last_change":"2024-09-16T10:19:46.683161+0000","last_active":"2024-09-16T10:23:31.465175+0000","last_peered":"2024-09-16T10:23:31.465175+0000","last_clean":"2024-09-16T10:23:31.465175+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:23:31.465175+0000","last_undegraded":"2024-09-16T10:23:31.465175+0000","last_fullsized":"2024-09-16T10:23:31.465175+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-16T10:19:46.683078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:46.683078+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.726490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":568,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469921+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:23:31.469921+0000","last_peered":"2024-09-16T10:23:31.469921+0000","last_clean":"2024-09-16T10:23:31.469921+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:23:31.469921+0000","last_undegraded":"2024-09-16T10:23:31.469921+0000","last_fullsized":"2024-09-16T10:23:31.469921+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":690,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480526+0000","last_change":"2024-09-16T10:22:02.357682+0000","last_active":"2024-09-16T10:23:31.480526+0000","last_peered":"2024-09-16T10:23:31.480526+0000","last_clean":"2024-09-16T10:23:31.480526+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:23:31.480526+0000","last_undegraded":"2024-09-16T10:23:31.480526+0000","last_fullsized":"2024-09-16T10:23:31.480526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:22:02.357616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.357616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":562,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462818+0000","last_change":"2024-09-16T10:22:05.051429+0000","last_active":"2024-09-16T10:23:31.462818+0000","last_peered":"2024-09-16T10:23:31.462818+0000","last_clean":"2024-09-16T10:23:31.462818+0000","last_became_active":"2024-09-16T10:22:02.890865+0000","last_became_peered":"2024-09-16T10:22:02.890865+0000","last_unstale":"2024-09-16T10:23:31.462818+0000","last_undegraded":"2024-09-16T10:23:31.462818+0000","last_fullsized":"2024-09-16T10:23:31.462818+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"323'7456","reported_seq":7561,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.355804+0000","last_change":"2024-09-16T10:21:56.239042+0000","last_active":"2024-09-16T10:23:33.355804+0000","last_peered":"2024-09-16T10:23:33.355804+0000","last_clean":"2024-09-16T10:23:33.355804+0000","last_became_active":"2024-09-16T10:21:56.238825+0000","last_became_peered":"2024-09-16T10:21:56.238825+0000","last_unstale":"2024-09-16T10:23:33.355804+0000","last_undegraded":"2024-09-16T10:23:33.355804+0000","last_fullsized":"2024-09-16T10:23:33.355804+0000","mapping_epoch":276,"log_start":"295'3043","ondisk_log_start":"295'3043","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471072768,"num_objects":7188,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":502,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466514+0000","last_change":"2024-09-16T10:23:30.461819+0000","last_active":"2024-09-16T10:23:31.466514+0000","last_peered":"2024-09-16T10:23:31.466514+0000","last_clean":"2024-09-16T10:23:31.466514+0000","last_became_active":"2024-09-16T10:22:56.438108+0000","last_became_peered":"2024-09-16T10:22:56.438108+0000","last_unstale":"2024-09-16T10:23:31.466514+0000","last_undegraded":"2024-09-16T10:23:31.466514+0000","last_fullsized":"2024-09-16T10:23:31.466514+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:59.898276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":595,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480990+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:23:31.480990+0000","last_peered":"2024-09-16T10:23:31.480990+0000","last_clean":"2024-09-16T10:23:31.480990+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:23:31.480990+0000","last_undegraded":"2024-09-16T10:23:31.480990+0000","last_fullsized":"2024-09-16T10:23:31.480990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":582,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470623+0000","last_change":"2024-09-16T10:22:02.898141+0000","last_active":"2024-09-16T10:23:31.470623+0000","last_peered":"2024-09-16T10:23:31.470623+0000","last_clean":"2024-09-16T10:23:31.470623+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:23:31.470623+0000","last_undegraded":"2024-09-16T10:23:31.470623+0000","last_fullsized":"2024-09-16T10:23:31.470623+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.918613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.918613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":679,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470377+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:23:31.470377+0000","last_peered":"2024-09-16T10:23:31.470377+0000","last_clean":"2024-09-16T10:23:31.470377+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:23:31.470377+0000","last_undegraded":"2024-09-16T10:23:31.470377+0000","last_fullsized":"2024-09-16T10:23:31.470377+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":577,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466965+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:23:31.466965+0000","last_peered":"2024-09-16T10:23:31.466965+0000","last_clean":"2024-09-16T10:23:31.466965+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:23:31.466965+0000","last_undegraded":"2024-09-16T10:23:31.466965+0000","last_fullsized":"2024-09-16T10:23:31.466965+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"323'7636","reported_seq":7741,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.239722+0000","last_change":"2024-09-16T10:21:56.236599+0000","last_active":"2024-09-16T10:23:33.239722+0000","last_peered":"2024-09-16T10:23:33.239722+0000","last_clean":"2024-09-16T10:23:33.239722+0000","last_became_active":"2024-09-16T10:21:56.236217+0000","last_became_peered":"2024-09-16T10:21:56.236217+0000","last_unstale":"2024-09-16T10:23:33.239722+0000","last_undegraded":"2024-09-16T10:23:33.239722+0000","last_fullsized":"2024-09-16T10:23:33.239722+0000","mapping_epoch":276,"log_start":"295'3384","ondisk_log_start":"295'3384","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:30.621052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483000320,"num_objects":7370,"num_object_clones":0,"num_object_copies":22110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7636,"num_write_kb":480192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":507,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466628+0000","last_change":"2024-09-16T10:23:30.461691+0000","last_active":"2024-09-16T10:23:31.466628+0000","last_peered":"2024-09-16T10:23:31.466628+0000","last_clean":"2024-09-16T10:23:31.466628+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:23:31.466628+0000","last_undegraded":"2024-09-16T10:23:31.466628+0000","last_fullsized":"2024-09-16T10:23:31.466628+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.732986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.732986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:58.850851+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":607,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477652+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:23:31.477652+0000","last_peered":"2024-09-16T10:23:31.477652+0000","last_clean":"2024-09-16T10:23:31.477652+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:23:31.477652+0000","last_undegraded":"2024-09-16T10:23:31.477652+0000","last_fullsized":"2024-09-16T10:23:31.477652+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":590,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477366+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:23:31.477366+0000","last_peered":"2024-09-16T10:23:31.477366+0000","last_clean":"2024-09-16T10:23:31.477366+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:23:31.477366+0000","last_undegraded":"2024-09-16T10:23:31.477366+0000","last_fullsized":"2024-09-16T10:23:31.477366+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":575,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465469+0000","last_change":"2024-09-16T10:22:28.126110+0000","last_active":"2024-09-16T10:23:31.465469+0000","last_peered":"2024-09-16T10:23:31.465469+0000","last_clean":"2024-09-16T10:23:31.465469+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:23:31.465469+0000","last_undegraded":"2024-09-16T10:23:31.465469+0000","last_fullsized":"2024-09-16T10:23:31.465469+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:28.126042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:28.126042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:33.509938+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":570,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.475772+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:23:31.475772+0000","last_peered":"2024-09-16T10:23:31.475772+0000","last_clean":"2024-09-16T10:23:31.475772+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:23:31.475772+0000","last_undegraded":"2024-09-16T10:23:31.475772+0000","last_fullsized":"2024-09-16T10:23:31.475772+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"323'7568","reported_seq":7672,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.884227+0000","last_change":"2024-09-16T10:21:56.233704+0000","last_active":"2024-09-16T10:23:33.884227+0000","last_peered":"2024-09-16T10:23:33.884227+0000","last_clean":"2024-09-16T10:23:33.884227+0000","last_became_active":"2024-09-16T10:21:56.233167+0000","last_became_peered":"2024-09-16T10:21:56.233167+0000","last_unstale":"2024-09-16T10:23:33.884227+0000","last_undegraded":"2024-09-16T10:23:33.884227+0000","last_fullsized":"2024-09-16T10:23:33.884227+0000","mapping_epoch":276,"log_start":"292'2395","ondisk_log_start":"292'2395","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471334912,"num_objects":7192,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7568,"num_write_kb":472320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":453,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469394+0000","last_change":"2024-09-16T10:23:30.466273+0000","last_active":"2024-09-16T10:23:31.469394+0000","last_peered":"2024-09-16T10:23:31.469394+0000","last_clean":"2024-09-16T10:23:31.469394+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:23:31.469394+0000","last_undegraded":"2024-09-16T10:23:31.469394+0000","last_fullsized":"2024-09-16T10:23:31.469394+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:57.933943+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"282'30","reported_seq":806,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477267+0000","last_change":"2024-09-16T10:22:04.554377+0000","last_active":"2024-09-16T10:23:31.477267+0000","last_peered":"2024-09-16T10:23:31.477267+0000","last_clean":"2024-09-16T10:23:31.477267+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:23:31.477267+0000","last_undegraded":"2024-09-16T10:23:31.477267+0000","last_fullsized":"2024-09-16T10:23:31.477267+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'30","last_scrub_stamp":"2024-09-16T10:22:04.554260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.554260+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":575,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465983+0000","last_change":"2024-09-16T10:22:32.891207+0000","last_active":"2024-09-16T10:23:31.465983+0000","last_peered":"2024-09-16T10:23:31.465983+0000","last_clean":"2024-09-16T10:23:31.465983+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:23:31.465983+0000","last_undegraded":"2024-09-16T10:23:31.465983+0000","last_fullsized":"2024-09-16T10:23:31.465983+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:32.891126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.891126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:53.007811+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":570,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470199+0000","last_change":"2024-09-16T10:17:40.466121+0000","last_active":"2024-09-16T10:23:31.470199+0000","last_peered":"2024-09-16T10:23:31.470199+0000","last_clean":"2024-09-16T10:23:31.470199+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:23:31.470199+0000","last_undegraded":"2024-09-16T10:23:31.470199+0000","last_fullsized":"2024-09-16T10:23:31.470199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:17:40.466041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:17:40.466041+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:51.930288+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":551,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470562+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:23:31.470562+0000","last_peered":"2024-09-16T10:23:31.470562+0000","last_clean":"2024-09-16T10:23:31.470562+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:23:31.470562+0000","last_undegraded":"2024-09-16T10:23:31.470562+0000","last_fullsized":"2024-09-16T10:23:31.470562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":571,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480866+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:23:31.480866+0000","last_peered":"2024-09-16T10:23:31.480866+0000","last_clean":"2024-09-16T10:23:31.480866+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:23:31.480866+0000","last_undegraded":"2024-09-16T10:23:31.480866+0000","last_fullsized":"2024-09-16T10:23:31.480866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.835165+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"323'7231","reported_seq":7336,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.389327+0000","last_change":"2024-09-16T10:21:56.236854+0000","last_active":"2024-09-16T10:23:33.389327+0000","last_peered":"2024-09-16T10:23:33.389327+0000","last_clean":"2024-09-16T10:23:33.389327+0000","last_became_active":"2024-09-16T10:21:56.236183+0000","last_became_peered":"2024-09-16T10:21:56.236183+0000","last_unstale":"2024-09-16T10:23:33.389327+0000","last_undegraded":"2024-09-16T10:23:33.389327+0000","last_fullsized":"2024-09-16T10:23:33.389327+0000","mapping_epoch":276,"log_start":"293'2762","ondisk_log_start":"293'2762","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458424320,"num_objects":6995,"num_object_clones":0,"num_object_copies":20985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":455232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":712,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462589+0000","last_change":"2024-09-16T10:23:30.467368+0000","last_active":"2024-09-16T10:23:31.462589+0000","last_peered":"2024-09-16T10:23:31.462589+0000","last_clean":"2024-09-16T10:23:31.462589+0000","last_became_active":"2024-09-16T10:22:56.424034+0000","last_became_peered":"2024-09-16T10:22:56.424034+0000","last_unstale":"2024-09-16T10:23:31.462589+0000","last_undegraded":"2024-09-16T10:23:31.462589+0000","last_fullsized":"2024-09-16T10:23:31.462589+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:32.033672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.033672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:59.620117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":575,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.476457+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:23:31.476457+0000","last_peered":"2024-09-16T10:23:31.476457+0000","last_clean":"2024-09-16T10:23:31.476457+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:23:31.476457+0000","last_undegraded":"2024-09-16T10:23:31.476457+0000","last_fullsized":"2024-09-16T10:23:31.476457+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":574,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.476677+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:23:31.476677+0000","last_peered":"2024-09-16T10:23:31.476677+0000","last_clean":"2024-09-16T10:23:31.476677+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:23:31.476677+0000","last_undegraded":"2024-09-16T10:23:31.476677+0000","last_fullsized":"2024-09-16T10:23:31.476677+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":98,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.468670+0000","last_change":"2024-09-16T10:22:02.917475+0000","last_active":"2024-09-16T10:23:31.468670+0000","last_peered":"2024-09-16T10:23:31.468670+0000","last_clean":"2024-09-16T10:23:31.468670+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:23:31.468670+0000","last_undegraded":"2024-09-16T10:23:31.468670+0000","last_fullsized":"2024-09-16T10:23:31.468670+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":102,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.472481+0000","last_change":"2024-09-16T10:22:29.905776+0000","last_active":"2024-09-16T10:23:31.472481+0000","last_peered":"2024-09-16T10:23:31.472481+0000","last_clean":"2024-09-16T10:23:31.472481+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:23:31.472481+0000","last_undegraded":"2024-09-16T10:23:31.472481+0000","last_fullsized":"2024-09-16T10:23:31.472481+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:57.728247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":572,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464254+0000","last_change":"2024-09-16T10:22:29.105512+0000","last_active":"2024-09-16T10:23:31.464254+0000","last_peered":"2024-09-16T10:23:31.464254+0000","last_clean":"2024-09-16T10:23:31.464254+0000","last_became_active":"2024-09-16T10:13:43.283210+0000","last_became_peered":"2024-09-16T10:13:43.283210+0000","last_unstale":"2024-09-16T10:23:31.464254+0000","last_undegraded":"2024-09-16T10:23:31.464254+0000","last_fullsized":"2024-09-16T10:23:31.464254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.105178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.105178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:42.853519+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"323'7352","reported_seq":10316,"reported_epoch":323,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:33.847277+0000","last_change":"2024-09-16T10:22:37.049031+0000","last_active":"2024-09-16T10:23:33.847277+0000","last_peered":"2024-09-16T10:23:33.847277+0000","last_clean":"2024-09-16T10:22:35.458157+0000","last_became_active":"2024-09-16T10:22:37.003170+0000","last_became_peered":"2024-09-16T10:22:37.003170+0000","last_unstale":"2024-09-16T10:23:33.847277+0000","last_undegraded":"2024-09-16T10:22:36.998730+0000","last_fullsized":"2024-09-16T10:22:36.878645+0000","mapping_epoch":298,"log_start":"290'2111","ondisk_log_start":"290'2111","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458883072,"num_objects":7002,"num_object_clones":0,"num_object_copies":21006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":5647,"num_objects_unfound":0,"num_objects_dirty":7002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7352,"num_write_kb":459328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":62390272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[0,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":2823},{"shards":"0,6","objects":4178}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":441,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.478651+0000","last_change":"2024-09-16T10:23:30.471180+0000","last_active":"2024-09-16T10:23:31.478651+0000","last_peered":"2024-09-16T10:23:31.478651+0000","last_clean":"2024-09-16T10:23:31.478651+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:23:31.478651+0000","last_undegraded":"2024-09-16T10:23:31.478651+0000","last_fullsized":"2024-09-16T10:23:31.478651+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:57.703306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":647,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.468484+0000","last_change":"2024-09-16T10:22:24.213089+0000","last_active":"2024-09-16T10:23:31.468484+0000","last_peered":"2024-09-16T10:23:31.468484+0000","last_clean":"2024-09-16T10:23:31.468484+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:23:31.468484+0000","last_undegraded":"2024-09-16T10:23:31.468484+0000","last_fullsized":"2024-09-16T10:23:31.468484+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:22:24.213008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.213008+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":591,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467801+0000","last_change":"2024-09-16T10:22:26.226703+0000","last_active":"2024-09-16T10:23:31.467801+0000","last_peered":"2024-09-16T10:23:31.467801+0000","last_clean":"2024-09-16T10:23:31.467801+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:23:31.467801+0000","last_undegraded":"2024-09-16T10:23:31.467801+0000","last_fullsized":"2024-09-16T10:23:31.467801+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.226632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.226632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1177,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.891148+0000","last_change":"2024-09-16T10:21:58.009040+0000","last_active":"2024-09-16T10:23:31.891148+0000","last_peered":"2024-09-16T10:23:31.891148+0000","last_clean":"2024-09-16T10:23:31.891148+0000","last_became_active":"2024-09-16T10:13:36.082762+0000","last_became_peered":"2024-09-16T10:13:36.082762+0000","last_unstale":"2024-09-16T10:23:31.891148+0000","last_undegraded":"2024-09-16T10:23:31.891148+0000","last_fullsized":"2024-09-16T10:23:31.891148+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:21:58.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.008984+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":560,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470884+0000","last_change":"2024-09-16T10:19:47.107063+0000","last_active":"2024-09-16T10:23:31.470884+0000","last_peered":"2024-09-16T10:23:31.470884+0000","last_clean":"2024-09-16T10:23:31.470884+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:23:31.470884+0000","last_undegraded":"2024-09-16T10:23:31.470884+0000","last_fullsized":"2024-09-16T10:23:31.470884+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.106993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.106993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:15.881113+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":572,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.476862+0000","last_change":"2024-09-16T10:21:58.547682+0000","last_active":"2024-09-16T10:23:31.476862+0000","last_peered":"2024-09-16T10:23:31.476862+0000","last_clean":"2024-09-16T10:23:31.476862+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:23:31.476862+0000","last_undegraded":"2024-09-16T10:23:31.476862+0000","last_fullsized":"2024-09-16T10:23:31.476862+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.547634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.547634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"323'7471","reported_seq":7576,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.704285+0000","last_change":"2024-09-16T10:21:56.236351+0000","last_active":"2024-09-16T10:23:33.704285+0000","last_peered":"2024-09-16T10:23:33.704285+0000","last_clean":"2024-09-16T10:23:33.704285+0000","last_became_active":"2024-09-16T10:21:56.236070+0000","last_became_peered":"2024-09-16T10:21:56.236070+0000","last_unstale":"2024-09-16T10:23:33.704285+0000","last_undegraded":"2024-09-16T10:23:33.704285+0000","last_fullsized":"2024-09-16T10:23:33.704285+0000","mapping_epoch":276,"log_start":"296'3284","ondisk_log_start":"296'3284","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468123648,"num_objects":7143,"num_object_clones":0,"num_object_copies":21429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7471,"num_write_kb":467648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":697,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466863+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:23:31.466863+0000","last_peered":"2024-09-16T10:23:31.466863+0000","last_clean":"2024-09-16T10:23:31.466863+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:23:31.466863+0000","last_undegraded":"2024-09-16T10:23:31.466863+0000","last_fullsized":"2024-09-16T10:23:31.466863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":540,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.481872+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:23:31.481872+0000","last_peered":"2024-09-16T10:23:31.481872+0000","last_clean":"2024-09-16T10:23:31.481872+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:23:31.481872+0000","last_undegraded":"2024-09-16T10:23:31.481872+0000","last_fullsized":"2024-09-16T10:23:31.481872+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":690,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462774+0000","last_change":"2024-09-16T10:23:30.463978+0000","last_active":"2024-09-16T10:23:31.462774+0000","last_peered":"2024-09-16T10:23:31.462774+0000","last_clean":"2024-09-16T10:23:31.462774+0000","last_became_active":"2024-09-16T10:22:56.424419+0000","last_became_peered":"2024-09-16T10:22:56.424419+0000","last_unstale":"2024-09-16T10:23:31.462774+0000","last_undegraded":"2024-09-16T10:23:31.462774+0000","last_fullsized":"2024-09-16T10:23:31.462774+0000","mapping_epoch":55,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:57.046749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.046749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:23:58.629959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":566,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464492+0000","last_change":"2024-09-16T10:22:02.906870+0000","last_active":"2024-09-16T10:23:31.464492+0000","last_peered":"2024-09-16T10:23:31.464492+0000","last_clean":"2024-09-16T10:23:31.464492+0000","last_became_active":"2024-09-16T10:22:02.906315+0000","last_became_peered":"2024-09-16T10:22:02.906315+0000","last_unstale":"2024-09-16T10:23:31.464492+0000","last_undegraded":"2024-09-16T10:23:31.464492+0000","last_fullsized":"2024-09-16T10:23:31.464492+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:16:43.622339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:16:43.622339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:15.739114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":556,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470687+0000","last_change":"2024-09-16T10:21:56.923534+0000","last_active":"2024-09-16T10:23:31.470687+0000","last_peered":"2024-09-16T10:23:31.470687+0000","last_clean":"2024-09-16T10:23:31.470687+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:23:31.470687+0000","last_undegraded":"2024-09-16T10:23:31.470687+0000","last_fullsized":"2024-09-16T10:23:31.470687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:56.923485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.923485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:17.854554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":460,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.478339+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:23:31.478339+0000","last_peered":"2024-09-16T10:23:31.478339+0000","last_clean":"2024-09-16T10:23:31.478339+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:23:31.478339+0000","last_undegraded":"2024-09-16T10:23:31.478339+0000","last_fullsized":"2024-09-16T10:23:31.478339+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:18.698762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"323'7498","reported_seq":7602,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.130969+0000","last_change":"2024-09-16T10:21:56.234554+0000","last_active":"2024-09-16T10:23:33.130969+0000","last_peered":"2024-09-16T10:23:33.130969+0000","last_clean":"2024-09-16T10:23:33.130969+0000","last_became_active":"2024-09-16T10:21:56.234289+0000","last_became_peered":"2024-09-16T10:21:56.234289+0000","last_unstale":"2024-09-16T10:23:33.130969+0000","last_undegraded":"2024-09-16T10:23:33.130969+0000","last_fullsized":"2024-09-16T10:23:33.130969+0000","mapping_epoch":276,"log_start":"295'3294","ondisk_log_start":"295'3294","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474742784,"num_objects":7244,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":471744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":601,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467213+0000","last_change":"2024-09-16T10:20:31.013510+0000","last_active":"2024-09-16T10:23:31.467213+0000","last_peered":"2024-09-16T10:23:31.467213+0000","last_clean":"2024-09-16T10:23:31.467213+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:23:31.467213+0000","last_undegraded":"2024-09-16T10:23:31.467213+0000","last_fullsized":"2024-09-16T10:23:31.467213+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:20:31.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.013316+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:26.752137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":581,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465458+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:23:31.465458+0000","last_peered":"2024-09-16T10:23:31.465458+0000","last_clean":"2024-09-16T10:23:31.465458+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:23:31.465458+0000","last_undegraded":"2024-09-16T10:23:31.465458+0000","last_fullsized":"2024-09-16T10:23:31.465458+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":568,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470493+0000","last_change":"2024-09-16T10:22:22.987395+0000","last_active":"2024-09-16T10:23:31.470493+0000","last_peered":"2024-09-16T10:23:31.470493+0000","last_clean":"2024-09-16T10:23:31.470493+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:23:31.470493+0000","last_undegraded":"2024-09-16T10:23:31.470493+0000","last_fullsized":"2024-09-16T10:23:31.470493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:22.987351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:22.987351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":512,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.484455+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:23:31.484455+0000","last_peered":"2024-09-16T10:23:31.484455+0000","last_clean":"2024-09-16T10:23:31.484455+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:23:31.484455+0000","last_undegraded":"2024-09-16T10:23:31.484455+0000","last_fullsized":"2024-09-16T10:23:31.484455+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:49.885016+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"323'7322","reported_seq":7626,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.691050+0000","last_change":"2024-09-16T10:22:03.098658+0000","last_active":"2024-09-16T10:23:33.691050+0000","last_peered":"2024-09-16T10:23:33.691050+0000","last_clean":"2024-09-16T10:23:33.691050+0000","last_became_active":"2024-09-16T10:22:02.896249+0000","last_became_peered":"2024-09-16T10:22:02.896249+0000","last_unstale":"2024-09-16T10:23:33.691050+0000","last_undegraded":"2024-09-16T10:23:33.691050+0000","last_fullsized":"2024-09-16T10:23:33.691050+0000","mapping_epoch":281,"log_start":"295'3016","ondisk_log_start":"295'3016","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462159872,"num_objects":7052,"num_object_clones":0,"num_object_copies":21156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":459968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":419,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469302+0000","last_change":"2024-09-16T10:22:03.029868+0000","last_active":"2024-09-16T10:23:31.469302+0000","last_peered":"2024-09-16T10:23:31.469302+0000","last_clean":"2024-09-16T10:23:31.469302+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:23:31.469302+0000","last_undegraded":"2024-09-16T10:23:31.469302+0000","last_fullsized":"2024-09-16T10:23:31.469302+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:27.405599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:27.405599+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:58.883669+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":649,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466102+0000","last_change":"2024-09-16T10:23:09.748114+0000","last_active":"2024-09-16T10:23:31.466102+0000","last_peered":"2024-09-16T10:23:31.466102+0000","last_clean":"2024-09-16T10:23:31.466102+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:23:31.466102+0000","last_undegraded":"2024-09-16T10:23:31.466102+0000","last_fullsized":"2024-09-16T10:23:31.466102+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:23:09.748031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:09.748031+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:19.242452+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":567,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465880+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:23:31.465880+0000","last_peered":"2024-09-16T10:23:31.465880+0000","last_clean":"2024-09-16T10:23:31.465880+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:23:31.465880+0000","last_undegraded":"2024-09-16T10:23:31.465880+0000","last_fullsized":"2024-09-16T10:23:31.465880+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":560,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.476940+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:23:31.476940+0000","last_peered":"2024-09-16T10:23:31.476940+0000","last_clean":"2024-09-16T10:23:31.476940+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:23:31.476940+0000","last_undegraded":"2024-09-16T10:23:31.476940+0000","last_fullsized":"2024-09-16T10:23:31.476940+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":102,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469336+0000","last_change":"2024-09-16T10:22:05.694026+0000","last_active":"2024-09-16T10:23:31.469336+0000","last_peered":"2024-09-16T10:23:31.469336+0000","last_clean":"2024-09-16T10:23:31.469336+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:23:31.469336+0000","last_undegraded":"2024-09-16T10:23:31.469336+0000","last_fullsized":"2024-09-16T10:23:31.469336+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.693972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.693972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"323'7287","reported_seq":7392,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.676960+0000","last_change":"2024-09-16T10:21:56.913566+0000","last_active":"2024-09-16T10:23:33.676960+0000","last_peered":"2024-09-16T10:23:33.676960+0000","last_clean":"2024-09-16T10:23:33.676960+0000","last_became_active":"2024-09-16T10:21:56.913186+0000","last_became_peered":"2024-09-16T10:21:56.913186+0000","last_unstale":"2024-09-16T10:23:33.676960+0000","last_undegraded":"2024-09-16T10:23:33.676960+0000","last_fullsized":"2024-09-16T10:23:33.676960+0000","mapping_epoch":276,"log_start":"295'3118","ondisk_log_start":"295'3118","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455147520,"num_objects":6945,"num_object_clones":0,"num_object_copies":20835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":455424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":679,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462954+0000","last_change":"2024-09-16T10:22:02.905855+0000","last_active":"2024-09-16T10:23:31.462954+0000","last_peered":"2024-09-16T10:23:31.462954+0000","last_clean":"2024-09-16T10:23:31.462954+0000","last_became_active":"2024-09-16T10:22:02.905393+0000","last_became_peered":"2024-09-16T10:22:02.905393+0000","last_unstale":"2024-09-16T10:23:31.462954+0000","last_undegraded":"2024-09-16T10:23:31.462954+0000","last_fullsized":"2024-09-16T10:23:31.462954+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:21.240387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:21.240387+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:16.718102+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":570,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467119+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:23:31.467119+0000","last_peered":"2024-09-16T10:23:31.467119+0000","last_clean":"2024-09-16T10:23:31.467119+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:23:31.467119+0000","last_undegraded":"2024-09-16T10:23:31.467119+0000","last_fullsized":"2024-09-16T10:23:31.467119+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:42.732608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":551,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.483499+0000","last_change":"2024-09-16T10:19:16.839163+0000","last_active":"2024-09-16T10:23:31.483499+0000","last_peered":"2024-09-16T10:23:31.483499+0000","last_clean":"2024-09-16T10:23:31.483499+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:23:31.483499+0000","last_undegraded":"2024-09-16T10:23:31.483499+0000","last_fullsized":"2024-09-16T10:23:31.483499+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.839071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.839071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:18.874860+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":125,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469148+0000","last_change":"2024-09-16T10:22:24.799901+0000","last_active":"2024-09-16T10:23:31.469148+0000","last_peered":"2024-09-16T10:23:31.469148+0000","last_clean":"2024-09-16T10:23:31.469148+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:23:31.469148+0000","last_undegraded":"2024-09-16T10:23:31.469148+0000","last_fullsized":"2024-09-16T10:23:31.469148+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:32.615360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":577,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467077+0000","last_change":"2024-09-16T10:22:31.886830+0000","last_active":"2024-09-16T10:23:31.467077+0000","last_peered":"2024-09-16T10:23:31.467077+0000","last_clean":"2024-09-16T10:23:31.467077+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:23:31.467077+0000","last_undegraded":"2024-09-16T10:23:31.467077+0000","last_fullsized":"2024-09-16T10:23:31.467077+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:31.886666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:31.886666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:51.243131+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":129,"reported_epoch":323,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:31.724473+0000","last_change":"2024-09-16T10:23:31.724324+0000","last_active":"2024-09-16T10:23:31.724473+0000","last_peered":"2024-09-16T10:23:31.724473+0000","last_clean":"2024-09-16T10:23:31.724473+0000","last_became_active":"2024-09-16T10:23:31.480326+0000","last_became_peered":"2024-09-16T10:23:31.480326+0000","last_unstale":"2024-09-16T10:23:31.724473+0000","last_undegraded":"2024-09-16T10:23:31.724473+0000","last_fullsized":"2024-09-16T10:23:31.724473+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:30.732986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:30.732986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"323'7371","reported_seq":7664,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.444592+0000","last_change":"2024-09-16T10:22:03.102043+0000","last_active":"2024-09-16T10:23:33.444592+0000","last_peered":"2024-09-16T10:23:33.444592+0000","last_clean":"2024-09-16T10:23:33.444592+0000","last_became_active":"2024-09-16T10:22:02.898446+0000","last_became_peered":"2024-09-16T10:22:02.898446+0000","last_unstale":"2024-09-16T10:23:33.444592+0000","last_undegraded":"2024-09-16T10:23:33.444592+0000","last_fullsized":"2024-09-16T10:23:33.444592+0000","mapping_epoch":281,"log_start":"294'2933","ondisk_log_start":"294'2933","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465895424,"num_objects":7109,"num_object_clones":0,"num_object_copies":21327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7371,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":571,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477545+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:23:31.477545+0000","last_peered":"2024-09-16T10:23:31.477545+0000","last_clean":"2024-09-16T10:23:31.477545+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:23:31.477545+0000","last_undegraded":"2024-09-16T10:23:31.477545+0000","last_fullsized":"2024-09-16T10:23:31.477545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":576,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470139+0000","last_change":"2024-09-16T10:21:57.915828+0000","last_active":"2024-09-16T10:23:31.470139+0000","last_peered":"2024-09-16T10:23:31.470139+0000","last_clean":"2024-09-16T10:23:31.470139+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:23:31.470139+0000","last_undegraded":"2024-09-16T10:23:31.470139+0000","last_fullsized":"2024-09-16T10:23:31.470139+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.915773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.915773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":94,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.468522+0000","last_change":"2024-09-16T10:22:02.896967+0000","last_active":"2024-09-16T10:23:31.468522+0000","last_peered":"2024-09-16T10:23:31.468522+0000","last_clean":"2024-09-16T10:23:31.468522+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:23:31.468522+0000","last_undegraded":"2024-09-16T10:23:31.468522+0000","last_fullsized":"2024-09-16T10:23:31.468522+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":592,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467666+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:23:31.467666+0000","last_peered":"2024-09-16T10:23:31.467666+0000","last_clean":"2024-09-16T10:23:31.467666+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:23:31.467666+0000","last_undegraded":"2024-09-16T10:23:31.467666+0000","last_fullsized":"2024-09-16T10:23:31.467666+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":520,"reported_epoch":323,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:31.908919+0000","last_change":"2024-09-16T10:23:31.908756+0000","last_active":"2024-09-16T10:23:31.908919+0000","last_peered":"2024-09-16T10:23:31.908919+0000","last_clean":"2024-09-16T10:23:31.908919+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:23:31.908919+0000","last_undegraded":"2024-09-16T10:23:31.908919+0000","last_fullsized":"2024-09-16T10:23:31.908919+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.578529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.578529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"323'7564","reported_seq":7669,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.408670+0000","last_change":"2024-09-16T10:21:56.234684+0000","last_active":"2024-09-16T10:23:33.408670+0000","last_peered":"2024-09-16T10:23:33.408670+0000","last_clean":"2024-09-16T10:23:33.408670+0000","last_became_active":"2024-09-16T10:21:56.234401+0000","last_became_peered":"2024-09-16T10:21:56.234401+0000","last_unstale":"2024-09-16T10:23:33.408670+0000","last_undegraded":"2024-09-16T10:23:33.408670+0000","last_fullsized":"2024-09-16T10:23:33.408670+0000","mapping_epoch":276,"log_start":"295'3071","ondisk_log_start":"295'3071","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476184576,"num_objects":7266,"num_object_clones":0,"num_object_copies":21798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7564,"num_write_kb":474560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":610,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465563+0000","last_change":"2024-09-16T10:22:26.050158+0000","last_active":"2024-09-16T10:23:31.465563+0000","last_peered":"2024-09-16T10:23:31.465563+0000","last_clean":"2024-09-16T10:23:31.465563+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:23:31.465563+0000","last_undegraded":"2024-09-16T10:23:31.465563+0000","last_fullsized":"2024-09-16T10:23:31.465563+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:22:26.050070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.050070+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:43.078689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":568,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464617+0000","last_change":"2024-09-16T10:21:59.998410+0000","last_active":"2024-09-16T10:23:31.464617+0000","last_peered":"2024-09-16T10:23:31.464617+0000","last_clean":"2024-09-16T10:23:31.464617+0000","last_became_active":"2024-09-16T10:13:37.093403+0000","last_became_peered":"2024-09-16T10:13:37.093403+0000","last_unstale":"2024-09-16T10:23:31.464617+0000","last_undegraded":"2024-09-16T10:23:31.464617+0000","last_fullsized":"2024-09-16T10:23:31.464617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:59.998357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:59.998357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:24.616954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":102,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477496+0000","last_change":"2024-09-16T10:22:26.818522+0000","last_active":"2024-09-16T10:23:31.477496+0000","last_peered":"2024-09-16T10:23:31.477496+0000","last_clean":"2024-09-16T10:23:31.477496+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:23:31.477496+0000","last_undegraded":"2024-09-16T10:23:31.477496+0000","last_fullsized":"2024-09-16T10:23:31.477496+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.818439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.818439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.045673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":570,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462587+0000","last_change":"2024-09-16T10:22:02.906954+0000","last_active":"2024-09-16T10:23:31.462587+0000","last_peered":"2024-09-16T10:23:31.462587+0000","last_clean":"2024-09-16T10:23:31.462587+0000","last_became_active":"2024-09-16T10:22:02.906524+0000","last_became_peered":"2024-09-16T10:22:02.906524+0000","last_unstale":"2024-09-16T10:23:31.462587+0000","last_undegraded":"2024-09-16T10:23:31.462587+0000","last_fullsized":"2024-09-16T10:23:31.462587+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:51.105240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:51.105240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:41.931790+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":455,"reported_epoch":323,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:31.481388+0000","last_change":"2024-09-16T10:23:29.505937+0000","last_active":"2024-09-16T10:23:31.481388+0000","last_peered":"2024-09-16T10:23:31.481388+0000","last_clean":"2024-09-16T10:22:56.466317+0000","last_became_active":"2024-09-16T10:22:59.490973+0000","last_became_peered":"2024-09-16T10:22:59.490973+0000","last_unstale":"2024-09-16T10:23:31.481388+0000","last_undegraded":"2024-09-16T10:23:31.481388+0000","last_fullsized":"2024-09-16T10:23:31.481388+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":109227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"323'7444","reported_seq":7549,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.710905+0000","last_change":"2024-09-16T10:21:56.235273+0000","last_active":"2024-09-16T10:23:33.710905+0000","last_peered":"2024-09-16T10:23:33.710905+0000","last_clean":"2024-09-16T10:23:33.710905+0000","last_became_active":"2024-09-16T10:21:56.234987+0000","last_became_peered":"2024-09-16T10:21:56.234987+0000","last_unstale":"2024-09-16T10:23:33.710905+0000","last_undegraded":"2024-09-16T10:23:33.710905+0000","last_fullsized":"2024-09-16T10:23:33.710905+0000","mapping_epoch":276,"log_start":"295'3148","ondisk_log_start":"295'3148","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466354176,"num_objects":7116,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7444,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":578,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465362+0000","last_change":"2024-09-16T10:17:42.921934+0000","last_active":"2024-09-16T10:23:31.465362+0000","last_peered":"2024-09-16T10:23:31.465362+0000","last_clean":"2024-09-16T10:23:31.465362+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:23:31.465362+0000","last_undegraded":"2024-09-16T10:23:31.465362+0000","last_fullsized":"2024-09-16T10:23:31.465362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:17:42.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:42.921838+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:41.732688+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":576,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464123+0000","last_change":"2024-09-16T10:22:34.020368+0000","last_active":"2024-09-16T10:23:31.464123+0000","last_peered":"2024-09-16T10:23:31.464123+0000","last_clean":"2024-09-16T10:23:31.464123+0000","last_became_active":"2024-09-16T10:13:37.092120+0000","last_became_peered":"2024-09-16T10:13:37.092120+0000","last_unstale":"2024-09-16T10:23:31.464123+0000","last_undegraded":"2024-09-16T10:23:31.464123+0000","last_fullsized":"2024-09-16T10:23:31.464123+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:34.020246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:34.020246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:55.286859+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":102,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469260+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:23:31.469260+0000","last_peered":"2024-09-16T10:23:31.469260+0000","last_clean":"2024-09-16T10:23:31.469260+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:23:31.469260+0000","last_undegraded":"2024-09-16T10:23:31.469260+0000","last_fullsized":"2024-09-16T10:23:31.469260+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":555,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.483204+0000","last_change":"2024-09-16T10:18:33.893934+0000","last_active":"2024-09-16T10:23:31.483204+0000","last_peered":"2024-09-16T10:23:31.483204+0000","last_clean":"2024-09-16T10:23:31.483204+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:23:31.483204+0000","last_undegraded":"2024-09-16T10:23:31.483204+0000","last_fullsized":"2024-09-16T10:23:31.483204+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:33.893877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:33.893877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:16.874923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":396,"reported_epoch":323,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:31.639265+0000","last_change":"2024-09-16T10:23:31.639131+0000","last_active":"2024-09-16T10:23:31.639265+0000","last_peered":"2024-09-16T10:23:31.639265+0000","last_clean":"2024-09-16T10:23:31.639265+0000","last_became_active":"2024-09-16T10:23:31.478391+0000","last_became_peered":"2024-09-16T10:23:31.478391+0000","last_unstale":"2024-09-16T10:23:31.639265+0000","last_undegraded":"2024-09-16T10:23:31.639265+0000","last_fullsized":"2024-09-16T10:23:31.639265+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:25.837614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.837614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"323'7488","reported_seq":7593,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:33.898036+0000","last_change":"2024-09-16T10:21:56.236925+0000","last_active":"2024-09-16T10:23:33.898036+0000","last_peered":"2024-09-16T10:23:33.898036+0000","last_clean":"2024-09-16T10:23:33.898036+0000","last_became_active":"2024-09-16T10:21:56.236619+0000","last_became_peered":"2024-09-16T10:21:56.236619+0000","last_unstale":"2024-09-16T10:23:33.898036+0000","last_undegraded":"2024-09-16T10:23:33.898036+0000","last_fullsized":"2024-09-16T10:23:33.898036+0000","mapping_epoch":276,"log_start":"291'2307","ondisk_log_start":"291'2307","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466354176,"num_objects":7116,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":573,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469395+0000","last_change":"2024-09-16T10:21:35.618881+0000","last_active":"2024-09-16T10:23:31.469395+0000","last_peered":"2024-09-16T10:23:31.469395+0000","last_clean":"2024-09-16T10:23:31.469395+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:23:31.469395+0000","last_undegraded":"2024-09-16T10:23:31.469395+0000","last_fullsized":"2024-09-16T10:23:31.469395+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:21:35.618806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.618806+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:29.886741+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":572,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464447+0000","last_change":"2024-09-16T10:18:16.603631+0000","last_active":"2024-09-16T10:23:31.464447+0000","last_peered":"2024-09-16T10:23:31.464447+0000","last_clean":"2024-09-16T10:23:31.464447+0000","last_became_active":"2024-09-16T10:13:37.092472+0000","last_became_peered":"2024-09-16T10:13:37.092472+0000","last_unstale":"2024-09-16T10:23:31.464447+0000","last_undegraded":"2024-09-16T10:23:31.464447+0000","last_fullsized":"2024-09-16T10:23:31.464447+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:18:16.603554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:16.603554+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:22.620982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":688,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462655+0000","last_change":"2024-09-16T10:23:10.768397+0000","last_active":"2024-09-16T10:23:31.462655+0000","last_peered":"2024-09-16T10:23:31.462655+0000","last_clean":"2024-09-16T10:23:31.462655+0000","last_became_active":"2024-09-16T10:13:41.112699+0000","last_became_peered":"2024-09-16T10:13:41.112699+0000","last_unstale":"2024-09-16T10:23:31.462655+0000","last_undegraded":"2024-09-16T10:23:31.462655+0000","last_fullsized":"2024-09-16T10:23:31.462655+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:23:10.768339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:23:10.768339+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:15.945009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":564,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469719+0000","last_change":"2024-09-16T10:22:00.982299+0000","last_active":"2024-09-16T10:23:31.469719+0000","last_peered":"2024-09-16T10:23:31.469719+0000","last_clean":"2024-09-16T10:23:31.469719+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:23:31.469719+0000","last_undegraded":"2024-09-16T10:23:31.469719+0000","last_fullsized":"2024-09-16T10:23:31.469719+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:00.982240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:00.982240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:25.915741+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":121,"reported_epoch":323,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:23:31.467050+0000","last_change":"2024-09-16T10:23:29.491115+0000","last_active":"2024-09-16T10:23:31.467050+0000","last_peered":"2024-09-16T10:23:31.467050+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:22:59.474762+0000","last_became_peered":"2024-09-16T10:22:59.474762+0000","last_unstale":"2024-09-16T10:23:31.467050+0000","last_undegraded":"2024-09-16T10:23:31.467050+0000","last_fullsized":"2024-09-16T10:23:31.467050+0000","mapping_epoch":309,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:22:32.033672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:22:32.033672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":559,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.468840+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:23:31.468840+0000","last_peered":"2024-09-16T10:23:31.468840+0000","last_clean":"2024-09-16T10:23:31.468840+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:23:31.468840+0000","last_undegraded":"2024-09-16T10:23:31.468840+0000","last_fullsized":"2024-09-16T10:23:31.468840+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":572,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.489008+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:23:31.489008+0000","last_peered":"2024-09-16T10:23:31.489008+0000","last_clean":"2024-09-16T10:23:31.489008+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:23:31.489008+0000","last_undegraded":"2024-09-16T10:23:31.489008+0000","last_fullsized":"2024-09-16T10:23:31.489008+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.853715+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":560,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470101+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:23:31.470101+0000","last_peered":"2024-09-16T10:23:31.470101+0000","last_clean":"2024-09-16T10:23:31.470101+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:23:31.470101+0000","last_undegraded":"2024-09-16T10:23:31.470101+0000","last_fullsized":"2024-09-16T10:23:31.470101+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":548,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464228+0000","last_change":"2024-09-16T10:23:03.747587+0000","last_active":"2024-09-16T10:23:31.464228+0000","last_peered":"2024-09-16T10:23:31.464228+0000","last_clean":"2024-09-16T10:23:31.464228+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:23:31.464228+0000","last_undegraded":"2024-09-16T10:23:31.464228+0000","last_fullsized":"2024-09-16T10:23:31.464228+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:03.747523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:03.747523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:05.288516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":458,"reported_epoch":323,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:31.782984+0000","last_change":"2024-09-16T10:23:31.762746+0000","last_active":"2024-09-16T10:23:31.782984+0000","last_peered":"2024-09-16T10:23:31.782984+0000","last_clean":"2024-09-16T10:23:31.782984+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:23:31.782984+0000","last_undegraded":"2024-09-16T10:23:31.782984+0000","last_fullsized":"2024-09-16T10:23:31.782984+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:49.328864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:49.328864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":579,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.468571+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:23:31.468571+0000","last_peered":"2024-09-16T10:23:31.468571+0000","last_clean":"2024-09-16T10:23:31.468571+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:23:31.468571+0000","last_undegraded":"2024-09-16T10:23:31.468571+0000","last_fullsized":"2024-09-16T10:23:31.468571+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":568,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.463156+0000","last_change":"2024-09-16T10:22:02.040477+0000","last_active":"2024-09-16T10:23:31.463156+0000","last_peered":"2024-09-16T10:23:31.463156+0000","last_clean":"2024-09-16T10:23:31.463156+0000","last_became_active":"2024-09-16T10:13:37.094947+0000","last_became_peered":"2024-09-16T10:13:37.094947+0000","last_unstale":"2024-09-16T10:23:31.463156+0000","last_undegraded":"2024-09-16T10:23:31.463156+0000","last_fullsized":"2024-09-16T10:23:31.463156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.040428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.040428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":559,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.488752+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:23:31.488752+0000","last_peered":"2024-09-16T10:23:31.488752+0000","last_clean":"2024-09-16T10:23:31.488752+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:23:31.488752+0000","last_undegraded":"2024-09-16T10:23:31.488752+0000","last_fullsized":"2024-09-16T10:23:31.488752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:47.194439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":94,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469938+0000","last_change":"2024-09-16T10:22:02.903139+0000","last_active":"2024-09-16T10:23:31.469938+0000","last_peered":"2024-09-16T10:23:31.469938+0000","last_clean":"2024-09-16T10:23:31.469938+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:23:31.469938+0000","last_undegraded":"2024-09-16T10:23:31.469938+0000","last_fullsized":"2024-09-16T10:23:31.469938+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":130,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469233+0000","last_change":"2024-09-16T10:23:01.854762+0000","last_active":"2024-09-16T10:23:31.469233+0000","last_peered":"2024-09-16T10:23:31.469233+0000","last_clean":"2024-09-16T10:23:31.469233+0000","last_became_active":"2024-09-16T10:23:00.902731+0000","last_became_peered":"2024-09-16T10:23:00.902731+0000","last_unstale":"2024-09-16T10:23:31.469233+0000","last_undegraded":"2024-09-16T10:23:31.469233+0000","last_fullsized":"2024-09-16T10:23:31.469233+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.854706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.854706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:06.610807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":572,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470547+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:23:31.470547+0000","last_peered":"2024-09-16T10:23:31.470547+0000","last_clean":"2024-09-16T10:23:31.470547+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:23:31.470547+0000","last_undegraded":"2024-09-16T10:23:31.470547+0000","last_fullsized":"2024-09-16T10:23:31.470547+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:49.948413+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":575,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467766+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:23:31.467766+0000","last_peered":"2024-09-16T10:23:31.467766+0000","last_clean":"2024-09-16T10:23:31.467766+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:23:31.467766+0000","last_undegraded":"2024-09-16T10:23:31.467766+0000","last_fullsized":"2024-09-16T10:23:31.467766+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":725,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.484464+0000","last_change":"2024-09-16T10:22:24.619056+0000","last_active":"2024-09-16T10:23:31.484464+0000","last_peered":"2024-09-16T10:23:31.484464+0000","last_clean":"2024-09-16T10:23:31.484464+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:23:31.484464+0000","last_undegraded":"2024-09-16T10:23:31.484464+0000","last_fullsized":"2024-09-16T10:23:31.484464+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.618991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.618991+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":577,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467447+0000","last_change":"2024-09-16T10:22:34.860226+0000","last_active":"2024-09-16T10:23:31.467447+0000","last_peered":"2024-09-16T10:23:31.467447+0000","last_clean":"2024-09-16T10:23:31.467447+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:23:31.467447+0000","last_undegraded":"2024-09-16T10:23:31.467447+0000","last_fullsized":"2024-09-16T10:23:31.467447+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:34.860161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:34.860161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:38.752001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":133,"reported_epoch":323,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:32.232201+0000","last_change":"2024-09-16T10:23:32.228409+0000","last_active":"2024-09-16T10:23:32.232201+0000","last_peered":"2024-09-16T10:23:32.232201+0000","last_clean":"2024-09-16T10:23:32.232201+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:23:32.232201+0000","last_undegraded":"2024-09-16T10:23:32.232201+0000","last_fullsized":"2024-09-16T10:23:32.232201+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:20:48.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:20:48.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":586,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.484115+0000","last_change":"2024-09-16T10:21:56.344549+0000","last_active":"2024-09-16T10:23:31.484115+0000","last_peered":"2024-09-16T10:23:31.484115+0000","last_clean":"2024-09-16T10:23:31.484115+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:23:31.484115+0000","last_undegraded":"2024-09-16T10:23:31.484115+0000","last_fullsized":"2024-09-16T10:23:31.484115+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:56.344472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:56.344472+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":574,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.463382+0000","last_change":"2024-09-16T10:22:02.891069+0000","last_active":"2024-09-16T10:23:31.463382+0000","last_peered":"2024-09-16T10:23:31.463382+0000","last_clean":"2024-09-16T10:23:31.463382+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:23:31.463382+0000","last_undegraded":"2024-09-16T10:23:31.463382+0000","last_fullsized":"2024-09-16T10:23:31.463382+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.569244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.569244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:18.720772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":688,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.469964+0000","last_change":"2024-09-16T10:19:54.181080+0000","last_active":"2024-09-16T10:23:31.469964+0000","last_peered":"2024-09-16T10:23:31.469964+0000","last_clean":"2024-09-16T10:23:31.469964+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:23:31.469964+0000","last_undegraded":"2024-09-16T10:23:31.469964+0000","last_fullsized":"2024-09-16T10:23:31.469964+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:19:54.181012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:54.181012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:24.954934+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":555,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467981+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:23:31.467981+0000","last_peered":"2024-09-16T10:23:31.467981+0000","last_clean":"2024-09-16T10:23:31.467981+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:23:31.467981+0000","last_undegraded":"2024-09-16T10:23:31.467981+0000","last_fullsized":"2024-09-16T10:23:31.467981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":593,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465745+0000","last_change":"2024-09-16T10:23:11.758636+0000","last_active":"2024-09-16T10:23:31.465745+0000","last_peered":"2024-09-16T10:23:31.465745+0000","last_clean":"2024-09-16T10:23:31.465745+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:23:31.465745+0000","last_undegraded":"2024-09-16T10:23:31.465745+0000","last_fullsized":"2024-09-16T10:23:31.465745+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:23:11.758543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:11.758543+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:19.195891+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.476773+0000","last_change":"2024-09-16T10:23:19.734511+0000","last_active":"2024-09-16T10:23:31.476773+0000","last_peered":"2024-09-16T10:23:31.476773+0000","last_clean":"2024-09-16T10:23:31.476773+0000","last_became_active":"2024-09-16T10:23:19.734145+0000","last_became_peered":"2024-09-16T10:23:19.734145+0000","last_unstale":"2024-09-16T10:23:31.476773+0000","last_undegraded":"2024-09-16T10:23:31.476773+0000","last_fullsized":"2024-09-16T10:23:31.476773+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:44.915515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:17:44.915515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":560,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480164+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:23:31.480164+0000","last_peered":"2024-09-16T10:23:31.480164+0000","last_clean":"2024-09-16T10:23:31.480164+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:23:31.480164+0000","last_undegraded":"2024-09-16T10:23:31.480164+0000","last_fullsized":"2024-09-16T10:23:31.480164+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":547,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480378+0000","last_change":"2024-09-16T10:20:54.392034+0000","last_active":"2024-09-16T10:23:31.480378+0000","last_peered":"2024-09-16T10:23:31.480378+0000","last_clean":"2024-09-16T10:23:31.480378+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:23:31.480378+0000","last_undegraded":"2024-09-16T10:23:31.480378+0000","last_fullsized":"2024-09-16T10:23:31.480378+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:54.391970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:54.391970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:50.895339+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":614,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.481093+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:23:31.481093+0000","last_peered":"2024-09-16T10:23:31.481093+0000","last_clean":"2024-09-16T10:23:31.481093+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:23:31.481093+0000","last_undegraded":"2024-09-16T10:23:31.481093+0000","last_fullsized":"2024-09-16T10:23:31.481093+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":576,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.464529+0000","last_change":"2024-09-16T10:22:25.105281+0000","last_active":"2024-09-16T10:23:31.464529+0000","last_peered":"2024-09-16T10:23:31.464529+0000","last_clean":"2024-09-16T10:23:31.464529+0000","last_became_active":"2024-09-16T10:13:37.089606+0000","last_became_peered":"2024-09-16T10:13:37.089606+0000","last_unstale":"2024-09-16T10:23:31.464529+0000","last_undegraded":"2024-09-16T10:23:31.464529+0000","last_fullsized":"2024-09-16T10:23:31.464529+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:25.105215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.105215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:33.272207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":516,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.483280+0000","last_change":"2024-09-16T10:18:33.825634+0000","last_active":"2024-09-16T10:23:31.483280+0000","last_peered":"2024-09-16T10:23:31.483280+0000","last_clean":"2024-09-16T10:23:31.483280+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:23:31.483280+0000","last_undegraded":"2024-09-16T10:23:31.483280+0000","last_fullsized":"2024-09-16T10:23:31.483280+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:35.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:17:35.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:44.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":539,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466612+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:23:31.466612+0000","last_peered":"2024-09-16T10:23:31.466612+0000","last_clean":"2024-09-16T10:23:31.466612+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:23:31.466612+0000","last_undegraded":"2024-09-16T10:23:31.466612+0000","last_fullsized":"2024-09-16T10:23:31.466612+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":463,"reported_epoch":323,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:31.960166+0000","last_change":"2024-09-16T10:23:31.960007+0000","last_active":"2024-09-16T10:23:31.960166+0000","last_peered":"2024-09-16T10:23:31.960166+0000","last_clean":"2024-09-16T10:23:31.960166+0000","last_became_active":"2024-09-16T10:23:31.473535+0000","last_became_peered":"2024-09-16T10:23:31.473535+0000","last_unstale":"2024-09-16T10:23:31.960166+0000","last_undegraded":"2024-09-16T10:23:31.960166+0000","last_fullsized":"2024-09-16T10:23:31.960166+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:21:19.429310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:21:19.429310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":566,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480897+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:23:31.480897+0000","last_peered":"2024-09-16T10:23:31.480897+0000","last_clean":"2024-09-16T10:23:31.480897+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:23:31.480897+0000","last_undegraded":"2024-09-16T10:23:31.480897+0000","last_fullsized":"2024-09-16T10:23:31.480897+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:53.811978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":576,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.470213+0000","last_change":"2024-09-16T10:23:13.803633+0000","last_active":"2024-09-16T10:23:31.470213+0000","last_peered":"2024-09-16T10:23:31.470213+0000","last_clean":"2024-09-16T10:23:31.470213+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:23:31.470213+0000","last_undegraded":"2024-09-16T10:23:31.470213+0000","last_fullsized":"2024-09-16T10:23:31.470213+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:13.803568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.803568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:41.657748+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":568,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462282+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:23:31.462282+0000","last_peered":"2024-09-16T10:23:31.462282+0000","last_clean":"2024-09-16T10:23:31.462282+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:23:31.462282+0000","last_undegraded":"2024-09-16T10:23:31.462282+0000","last_fullsized":"2024-09-16T10:23:31.462282+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:48.747299+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":571,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.483423+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:23:31.483423+0000","last_peered":"2024-09-16T10:23:31.483423+0000","last_clean":"2024-09-16T10:23:31.483423+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:23:31.483423+0000","last_undegraded":"2024-09-16T10:23:31.483423+0000","last_fullsized":"2024-09-16T10:23:31.483423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:16.625354+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":595,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.480747+0000","last_change":"2024-09-16T10:21:58.334088+0000","last_active":"2024-09-16T10:23:31.480747+0000","last_peered":"2024-09-16T10:23:31.480747+0000","last_clean":"2024-09-16T10:23:31.480747+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:23:31.480747+0000","last_undegraded":"2024-09-16T10:23:31.480747+0000","last_fullsized":"2024-09-16T10:23:31.480747+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-16T10:21:58.334025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.334025+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":565,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.467276+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:23:31.467276+0000","last_peered":"2024-09-16T10:23:31.467276+0000","last_clean":"2024-09-16T10:23:31.467276+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:23:31.467276+0000","last_undegraded":"2024-09-16T10:23:31.467276+0000","last_fullsized":"2024-09-16T10:23:31.467276+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":559,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.465184+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:23:31.465184+0000","last_peered":"2024-09-16T10:23:31.465184+0000","last_clean":"2024-09-16T10:23:31.465184+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:23:31.465184+0000","last_undegraded":"2024-09-16T10:23:31.465184+0000","last_fullsized":"2024-09-16T10:23:31.465184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":325,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.462526+0000","last_change":"2024-09-16T10:22:02.898982+0000","last_active":"2024-09-16T10:23:31.462526+0000","last_peered":"2024-09-16T10:23:31.462526+0000","last_clean":"2024-09-16T10:23:31.462526+0000","last_became_active":"2024-09-16T10:22:02.898630+0000","last_became_peered":"2024-09-16T10:22:02.898630+0000","last_unstale":"2024-09-16T10:23:31.462526+0000","last_undegraded":"2024-09-16T10:23:31.462526+0000","last_fullsized":"2024-09-16T10:23:31.462526+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:44.487416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:44.487416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:24:23.624940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":555,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.477633+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:23:31.477633+0000","last_peered":"2024-09-16T10:23:31.477633+0000","last_clean":"2024-09-16T10:23:31.477633+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:23:31.477633+0000","last_undegraded":"2024-09-16T10:23:31.477633+0000","last_fullsized":"2024-09-16T10:23:31.477633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":567,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.485429+0000","last_change":"2024-09-16T10:20:58.426297+0000","last_active":"2024-09-16T10:23:31.485429+0000","last_peered":"2024-09-16T10:23:31.485429+0000","last_clean":"2024-09-16T10:23:31.485429+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:23:31.485429+0000","last_undegraded":"2024-09-16T10:23:31.485429+0000","last_fullsized":"2024-09-16T10:23:31.485429+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.426235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.426235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:52.829722+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":590,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.466770+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:23:31.466770+0000","last_peered":"2024-09-16T10:23:31.466770+0000","last_clean":"2024-09-16T10:23:31.466770+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:23:31.466770+0000","last_undegraded":"2024-09-16T10:23:31.466770+0000","last_fullsized":"2024-09-16T10:23:31.466770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":573,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-16T10:23:31.468163+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:23:31.468163+0000","last_peered":"2024-09-16T10:23:31.468163+0000","last_clean":"2024-09-16T10:23:31.468163+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:23:31.468163+0000","last_undegraded":"2024-09-16T10:23:31.468163+0000","last_fullsized":"2024-09-16T10:23:31.468163+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7457144856,"num_objects":113788,"num_object_clones":0,"num_object_copies":341364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":5647,"num_objects_unfound":0,"num_objects_dirty":113788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118670,"num_write_kb":7438593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":81395712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22723899392,"data_stored":22723887176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71501,"ondisk_log_size":71501,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":657456,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":25,"num_write":14,"num_write_kb":185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":920144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234107,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4526180,"kb_used_data":4147920,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":89218972,"statfs":{"total":95995035648,"available":91360227328,"internally_reserved":0,"allocated":4247470080,"data_stored":4246436117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528447,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1717444,"kb_used_data":1357236,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92027708,"statfs":{"total":95995035648,"available":94236372992,"internally_reserved":0,"allocated":1389809664,"data_stored":1388803251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790085,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4278564,"kb_used_data":3926672,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":89466588,"statfs":{"total":95995035648,"available":91613786112,"internally_reserved":0,"allocated":4020912128,"data_stored":4019879293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986314,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1701228,"kb_used_data":1386836,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92043924,"statfs":{"total":95995035648,"available":94252978176,"internally_reserved":0,"allocated":1420120064,"data_stored":1419081924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215247,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4132148,"kb_used_data":3659428,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":89613004,"statfs":{"total":95995035648,"available":91763716096,"internally_reserved":0,"allocated":3747254272,"data_stored":3746201571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":483981768},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411477,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1864204,"kb_used_data":1630332,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91880948,"statfs":{"total":95995035648,"available":94086090752,"internally_reserved":0,"allocated":1669459968,"data_stored":1668446178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575000,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4803004,"kb_used_data":4366444,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":88942148,"statfs":{"total":95995035648,"available":91076759552,"internally_reserved":0,"allocated":4471238656,"data_stored":4470176704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738526,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2199980,"kb_used_data":1835540,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91545172,"statfs":{"total":95995035648,"available":93742256128,"internally_reserved":0,"allocated":1879592960,"data_stored":1878555579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1863385088,"data_stored":1863385088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4468314112,"data_stored":4468310040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1658978304,"data_stored":1658978304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3717464064,"data_stored":3717464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1411645440,"data_stored":1411645440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3998420992,"data_stored":3998416920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1382219776,"data_stored":1382219776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4223471616,"data_stored":4223467544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:39.106 INFO:tasks.thrashosds.thrasher:Setting 4.0 to [4, 6, 2] 2024-09-16T10:23:39.106 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.0', '4', '6', '2'] 2024-09-16T10:23:39.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 4.0 4 6 2 2024-09-16T10:23:39.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:39.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:39.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:39.312 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:39.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:39.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:39.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:39.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:39.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:39.562 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:39.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:39.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:39.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:39.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:23:39.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:39.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:39.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:39 smithi165 ceph-mon[32696]: 9.d scrub starts 2024-09-16T10:23:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:39 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:23:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:39 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:23:39.863 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[29993]: 9.d scrub starts 2024-09-16T10:23:39.863 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:39.864 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:23:39.864 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:23:39.864 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[39470]: 9.d scrub starts 2024-09-16T10:23:39.864 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:39.864 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:23:39.864 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:39 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:23:39.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:39.909 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:39.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:40.011 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:40.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:23:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:40.051 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:40.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:40.080 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:40.152 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:40.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:40.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:40.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:40.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:40.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:40.492 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:40.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:40.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:23:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:40.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:40.653 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:23:40.653 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:23:40.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.653 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:126313 9.f 9:f6b5c9ec:::benchmark_data_smithi063_124717_object10082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-16T10:23:40.653 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:23:40.633829+0000", 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.016917102999999999, 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016946512, 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126313 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633829+0000", 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633829+0000", 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633830+0000", 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7799999999999999e-06 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633836+0000", 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.728e-06 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633847+0000", 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0507e-05 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:40.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633855+0000", 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7620000000000001e-06 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633944+0000", 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8876999999999996e-05 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.634104+0000", 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159574 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.634270+0000", 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016630200000000001 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.638483+0000", 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042123530000000003 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.643610+0000", 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005127492 2024-09-16T10:23:40.658 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:126314 9.f 9:fe80e53d:::benchmark_data_smithi063_124717_object10083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:23:40.633875+0000", 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.016870532000000001, 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016991421, 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:40.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126314 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633875+0000", 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:40.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633875+0000", 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633876+0000", 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1799999999999995e-07 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633879+0000", 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6800000000000002e-06 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633891+0000", 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2031000000000001e-05 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.633895+0000", 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8940000000000001e-06 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.636688+0000", 2024-09-16T10:23:40.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027929090000000001 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.636785+0000", 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6658999999999997e-05 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.636865+0000", 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9933999999999997e-05 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:40.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.641554+0000", 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046885520000000003 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.643645+0000", 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020914110000000001 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:126318 9.f 9:fd9886c6:::benchmark_data_smithi063_124717_object10087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:23:40.637858+0000", 2024-09-16T10:23:40.664 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.012888258, 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013077790000000001, 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126318 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.637858+0000", 2024-09-16T10:23:40.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.637858+0000", 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.637859+0000", 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8530000000000001e-06 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:40.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.637864+0000", 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8099999999999997e-06 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.637877+0000", 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2347999999999999e-05 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.637884+0000", 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0639999999999996e-06 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.638517+0000", 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063361299999999999 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.638647+0000", 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012967299999999999 2024-09-16T10:23:40.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.638783+0000", 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013645899999999999 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.641669+0000", 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002885898 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.643968+0000", 2024-09-16T10:23:40.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022989080000000001 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:126322 9.a 9:585f317f:::benchmark_data_smithi063_124717_object10091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:23:40.647962+0000", 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0027833979999999999, 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030391300000000001, 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:23:40.670 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126322 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.647962+0000", 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.647962+0000", 2024-09-16T10:23:40.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.647964+0000", 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3549999999999999e-06 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.647968+0000", 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1180000000000002e-06 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.647978+0000", 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0202e-05 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.647983+0000", 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4550000000000003e-06 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.648036+0000", 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2185000000000003e-05 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.648150+0000", 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114572 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.648265+0000", 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115041 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.650784+0000", 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025187999999999999 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:126323 9.8 e325/276 9:1f5992d0:::benchmark_data_smithi063_124717_object10092:head v 325'7944, mlcod=325'7942)", 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:23:40.649839+0000", 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00090679400000000005, 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001221946, 2024-09-16T10:23:40.675 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649839+0000", 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649839+0000", 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:40.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649840+0000", 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.172e-06 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649846+0000", 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0039999999999996e-06 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649864+0000", 2024-09-16T10:23:40.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7884999999999999e-05 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649871+0000", 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0600000000000002e-06 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649936+0000", 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4960000000000001e-05 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:40.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:23:40.649955+0000", 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8918000000000001e-05 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 5 2024-09-16T10:23:40.679 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:23:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:40.697 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:40.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:40.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:40 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:23:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:40 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:23:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:40 smithi165 ceph-mon[32696]: pgmap v541: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.39k op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:40.852 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:40.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:40.885 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:40 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:23:40.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:40 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:23:40.886 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:40 smithi063 ceph-mon[29993]: pgmap v541: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.39k op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:40.886 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:40 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:23:40.886 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:40 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:23:40.886 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:40 smithi063 ceph-mon[39470]: pgmap v541: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.39k op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.5 MiB/s, 24 objects/s recovering 2024-09-16T10:23:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:set 4.0 pg_upmap mapping to [4,6,2] 2024-09-16T10:23:40.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:40.991 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:40.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:41.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:41.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:41.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:41.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:41.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:41.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:41.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:41.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:41.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:41.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:41.528 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:41.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:41.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:23:41.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:41.665 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:41.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:41.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:41.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:41.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:41.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:41 smithi165 ceph-mon[32696]: 9.d scrub ok 2024-09-16T10:23:41.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1934646442' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]} : dispatch 2024-09-16T10:23:41.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:41 smithi165 ceph-mon[32696]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]} : dispatch 2024-09-16T10:23:41.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:41 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:41.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:41 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/325365 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:41.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:41 smithi165 ceph-mon[32696]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]}]': finished 2024-09-16T10:23:41.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:41 smithi165 ceph-mon[32696]: osdmap e326: 8 total, 8 up, 8 in 2024-09-16T10:23:41.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:41.907 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:41.944 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:23:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:41.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:41.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:41.945 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[29993]: 9.d scrub ok 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1934646442' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]} : dispatch 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[29993]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]} : dispatch 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/325365 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[29993]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]}]': finished 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[29993]: osdmap e326: 8 total, 8 up, 8 in 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[39470]: 9.d scrub ok 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1934646442' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]} : dispatch 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[39470]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]} : dispatch 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:23:41.946 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/325365 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:41.947 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[39470]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.0", "id": [4, 6, 2]}]': finished 2024-09-16T10:23:41.947 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:41 smithi063 ceph-mon[39470]: osdmap e326: 8 total, 8 up, 8 in 2024-09-16T10:23:42.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:42.085 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:42.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:42.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:42.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:42.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:42.342 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:42.382 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:42.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:42.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:42.525 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:42.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:42.562 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:23:42.562 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:23:42.562 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:23:42.563 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:23:42.563 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:23:42.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:42.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:42.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:42.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: pgmap v543: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 922 op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:23:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3991350494' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: from='client.45779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: from='client.45779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[29993]: osdmap e327: 8 total, 8 up, 8 in 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: pgmap v543: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 922 op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3991350494' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: from='client.45779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:23:42.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: from='client.45779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:42.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:42 smithi063 ceph-mon[39470]: osdmap e327: 8 total, 8 up, 8 in 2024-09-16T10:23:42.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:42.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:42.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:42.801 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:42.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: pgmap v543: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 922 op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:23:42.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3991350494' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:42.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: from='client.45779 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:23:42.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:23:42.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:23:42.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:23:42.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: from='client.45779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:23:42.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:42 smithi165 ceph-mon[32696]: osdmap e327: 8 total, 8 up, 8 in 2024-09-16T10:23:42.886 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:42.926 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:42.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:43.028 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:43.076 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:43.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:43.178 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:43.231 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:43.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:43.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:23:43.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:43.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:43.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:43.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:43.480 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:23:43.480 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:23:43.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:43.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:43.520 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:23:43.521 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:23:43.551 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:23:43.551 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:23:43.589 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:23:43.590 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:23:43.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:43.624 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:23:43.624 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:23:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:43.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:43.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:43.627 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:43.660 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:23:43.661 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:23:43.662 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: 4.9 scrub starts 2024-09-16T10:23:43.662 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: 4.9 scrub ok 2024-09-16T10:23:43.662 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: 6.8 scrub starts 2024-09-16T10:23:43.662 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: 6.8 scrub ok 2024-09-16T10:23:43.662 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:23:43.662 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:23:43.663 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: 9.8 scrub starts 2024-09-16T10:23:43.663 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3606171112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:43.663 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: 4.9 scrub starts 2024-09-16T10:23:43.663 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: 4.9 scrub ok 2024-09-16T10:23:43.663 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: 6.8 scrub starts 2024-09-16T10:23:43.664 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: 6.8 scrub ok 2024-09-16T10:23:43.664 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:23:43.664 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:23:43.664 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: 9.8 scrub starts 2024-09-16T10:23:43.664 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3606171112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:43.690 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:23:43.690 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:23:43.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:43.731 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:23:43.732 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:23:43.770 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:43.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:43.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:43.910 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:43.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:43.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: 4.9 scrub starts 2024-09-16T10:23:43.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: 4.9 scrub ok 2024-09-16T10:23:43.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: 6.8 scrub starts 2024-09-16T10:23:43.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: 6.8 scrub ok 2024-09-16T10:23:43.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:23:43.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:23:43.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: 9.8 scrub starts 2024-09-16T10:23:43.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3606171112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:44.011 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.036 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:44.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:44.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:44.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:44.209 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:44.209 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:44.277 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:44.318 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:44.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:44.462 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:44.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:44.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:44.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:44.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:44.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:23:44.832 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:44 smithi063 ceph-mon[29993]: pgmap v545: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 646 B/s rd, 0 B/s wr, 874 op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.0 MiB/s, 18 objects/s recovering 2024-09-16T10:23:44.832 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:44 smithi063 ceph-mon[29993]: 9.6 scrub starts 2024-09-16T10:23:44.833 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:44 smithi063 ceph-mon[39470]: pgmap v545: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 646 B/s rd, 0 B/s wr, 874 op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.0 MiB/s, 18 objects/s recovering 2024-09-16T10:23:44.833 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:44 smithi063 ceph-mon[39470]: 9.6 scrub starts 2024-09-16T10:23:44.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:44.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:44.947 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:44 smithi165 ceph-mon[32696]: pgmap v545: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 646 B/s rd, 0 B/s wr, 874 op/s; 1/325365 objects degraded (0.000%); 5440/325365 objects misplaced (1.672%); 1.0 MiB/s, 18 objects/s recovering 2024-09-16T10:23:44.947 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:44 smithi165 ceph-mon[32696]: 9.6 scrub starts 2024-09-16T10:23:44.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:44.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:23:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.027 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:45.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:45.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:45.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:45.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:45.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:45.397 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:45.410 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:45.446 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:45.456 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:45.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:45.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:45.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:45.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:45.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:45 smithi165 ceph-mon[32696]: 9.7 scrub starts 2024-09-16T10:23:45.833 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:45.870 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:45.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:45 smithi063 ceph-mon[29993]: 9.7 scrub starts 2024-09-16T10:23:45.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:45 smithi063 ceph-mon[39470]: 9.7 scrub starts 2024-09-16T10:23:45.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:46.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:46.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:46.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:46.147 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:46.148 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:46.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:46.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:46.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:46.405 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:46.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.547 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:46.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:46.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:46.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:46.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:46.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:46.843 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:46 smithi165 ceph-mon[32696]: 9.8 scrub ok 2024-09-16T10:23:46.843 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:46 smithi165 ceph-mon[32696]: pgmap v546: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 1/309816 objects degraded (0.000%); 5262/309816 objects misplaced (1.698%); 1.5 MiB/s, 25 objects/s recovering 2024-09-16T10:23:46.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:46 smithi165 ceph-mon[32696]: 9.6 scrub ok 2024-09-16T10:23:46.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:46 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/309816 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:46.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:46 smithi165 ceph-mon[32696]: osdmap e328: 8 total, 8 up, 8 in 2024-09-16T10:23:46.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:23:46.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:46.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[29993]: 9.8 scrub ok 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[29993]: pgmap v546: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 1/309816 objects degraded (0.000%); 5262/309816 objects misplaced (1.698%); 1.5 MiB/s, 25 objects/s recovering 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[29993]: 9.6 scrub ok 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/309816 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[29993]: osdmap e328: 8 total, 8 up, 8 in 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[39470]: 9.8 scrub ok 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[39470]: pgmap v546: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 1/309816 objects degraded (0.000%); 5262/309816 objects misplaced (1.698%); 1.5 MiB/s, 25 objects/s recovering 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[39470]: 9.6 scrub ok 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/309816 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:46 smithi063 ceph-mon[39470]: osdmap e328: 8 total, 8 up, 8 in 2024-09-16T10:23:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:47.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:47.030 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:23:47.030 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:23:47.030 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-16T10:23:47.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-16T10:23:47.114 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:47.187 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:47.290 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:47.334 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:47.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:23:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.350 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:47.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:47.384 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:23:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:23:47.451 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:47.526 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:47.536 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:47.620 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:23:47.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:47.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[29993]: 9.7 scrub ok 2024-09-16T10:23:47.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:23:47.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:23:47.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4087936958' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:47.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:23:47.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:23:47.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:23:47.652 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[39470]: 9.7 scrub ok 2024-09-16T10:23:47.652 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:23:47.652 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:23:47.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4087936958' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:47.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:23:47.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:23:47.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:47 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:23:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:47.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:47.755 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:47 smithi165 ceph-mon[32696]: 9.7 scrub ok 2024-09-16T10:23:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:47 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:23:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:47 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:23:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4087936958' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:23:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:47 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:23:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:47 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:23:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:47 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:23:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.810 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:47.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:47.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:47.961 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:48.063 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:48.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:48.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:48.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:48.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:48.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:48.370 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:48.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:48.472 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:48.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:48.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:48.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:48.639 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:48.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:48.741 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:48.786 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:48.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:48.870 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: 5.13 scrub starts 2024-09-16T10:23:48.870 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: 5.13 scrub ok 2024-09-16T10:23:48.870 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: pgmap v548: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 817 op/s; 1/309816 objects degraded (0.000%); 5262/309816 objects misplaced (1.698%); 898 KiB/s, 15 objects/s recovering 2024-09-16T10:23:48.870 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4087936958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: osdmap e329: 8 total, 8 up, 8 in 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: 5.13 scrub starts 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: 5.13 scrub ok 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: pgmap v548: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 817 op/s; 1/309816 objects degraded (0.000%); 5262/309816 objects misplaced (1.698%); 898 KiB/s, 15 objects/s recovering 2024-09-16T10:23:48.871 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:48.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4087936958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:23:48.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: osdmap e329: 8 total, 8 up, 8 in 2024-09-16T10:23:48.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:23:48.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:23:48.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:48 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:48.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:48.943 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:48.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:49.039 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:49.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:49.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: 5.13 scrub starts 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: 5.13 scrub ok 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: pgmap v548: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 817 op/s; 1/309816 objects degraded (0.000%); 5262/309816 objects misplaced (1.698%); 898 KiB/s, 15 objects/s recovering 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4087936958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: osdmap e329: 8 total, 8 up, 8 in 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:23:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:48 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:23:49.083 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:23:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:23:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:49.104 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:49.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:49.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:49.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:49.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:49.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:49.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:49.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:49.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:49.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:49.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:49.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:49.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:49.753 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:49.793 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:49.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:49.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:49.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:49.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:49.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:49 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:23:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:49 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:23:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:49 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:23:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:49 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:23:50.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:49 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:23:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:49 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:23:50.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:50.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:50.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:50.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:50.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:50.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:50.330 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:50.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:50.364 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:50.365 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:50.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:50.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:50.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:50.564 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:50.610 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:50.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:50.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:50.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:23:50.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:50.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:50.845 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:50.899 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:50.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:50 smithi063 ceph-mon[29993]: pgmap v550: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 1.48k op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 1.6 MiB/s, 26 objects/s recovering 2024-09-16T10:23:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:50 smithi063 ceph-mon[29993]: osdmap e330: 8 total, 8 up, 8 in 2024-09-16T10:23:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:50 smithi063 ceph-mon[39470]: pgmap v550: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 1.48k op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 1.6 MiB/s, 26 objects/s recovering 2024-09-16T10:23:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:50 smithi063 ceph-mon[39470]: osdmap e330: 8 total, 8 up, 8 in 2024-09-16T10:23:51.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:51.039 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:51.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:50 smithi165 ceph-mon[32696]: pgmap v550: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 1.48k op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 1.6 MiB/s, 26 objects/s recovering 2024-09-16T10:23:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:50 smithi165 ceph-mon[32696]: osdmap e330: 8 total, 8 up, 8 in 2024-09-16T10:23:51.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:51.187 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:51.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:51.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:51.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:51.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:51.419 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:51.460 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:51.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:51.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:51.608 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:51.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:51.709 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:51.733 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:51.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:51.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:51.871 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:51.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:51.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:51 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1/295020 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:51 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1/295020 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:51.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:52.012 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:51 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1/295020 objects degraded (0.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:52.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:52.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:52.238 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:52.278 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:52.381 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:52.403 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:52.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:52.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:52.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:52.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:23:52.637 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:52.677 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:52.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:52.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:52 smithi165 ceph-mon[32696]: pgmap v552: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 941 op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 1.0 MiB/s, 16 objects/s recovering 2024-09-16T10:23:52.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:52.917 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:52 smithi063 ceph-mon[29993]: pgmap v552: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 941 op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 1.0 MiB/s, 16 objects/s recovering 2024-09-16T10:23:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:52 smithi063 ceph-mon[39470]: pgmap v552: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 941 op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 1.0 MiB/s, 16 objects/s recovering 2024-09-16T10:23:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:52.963 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:52.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:52.966 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:53.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:53.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:53.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:53.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:53.239 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:53.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:53.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:53.377 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:53.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:53.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:53.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:53.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:53.514 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:53.514 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:53.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:53.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:53.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:53.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:53.759 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:53.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:53.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:53.897 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:53.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:53.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:53.998 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.037 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:54.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:54.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:23:54.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:54.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:54.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:54.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:54.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:54.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:54.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:23:54.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:54.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:54.668 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:54.792 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:54.803 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.828 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:54.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:54.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:54.935 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:54 smithi165 ceph-mon[32696]: pgmap v553: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 771 op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 864 KiB/s, 13 objects/s recovering 2024-09-16T10:23:54.951 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:54.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:54.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:54 smithi063 ceph-mon[29993]: pgmap v553: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 771 op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 864 KiB/s, 13 objects/s recovering 2024-09-16T10:23:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:54 smithi063 ceph-mon[39470]: pgmap v553: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 771 op/s; 1/295020 objects degraded (0.000%); 5064/295020 objects misplaced (1.716%); 864 KiB/s, 13 objects/s recovering 2024-09-16T10:23:55.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:55.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:55.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:55.184 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:55.222 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:55.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:55.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:55.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:55.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:55.464 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:55.502 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:55.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:55.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:55.614 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:55.614 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:55.645 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:55.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:55.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:55.781 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:55.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:55.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:55.920 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:55.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:56.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:56.068 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:56.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:56.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:56.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:56.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:56.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:56.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:56.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:56.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:23:56.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:56.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:56.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:56.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:56.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:56.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:56.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:56.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:56.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:56.749 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:56.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:56.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:56.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:56.943 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:56 smithi165 ceph-mon[32696]: pgmap v554: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.37k op/s; 164/279138 objects degraded (0.059%); 4699/279138 objects misplaced (1.683%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:23:56.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:56 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:23:56.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:56 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:23:56.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:56 smithi165 ceph-mon[32696]: osdmap e331: 8 total, 8 up, 8 in 2024-09-16T10:23:56.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:56 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 164/279138 objects degraded (0.059%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:56.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:56 smithi165 ceph-mon[32696]: osdmap e332: 8 total, 8 up, 8 in 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[29993]: pgmap v554: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.37k op/s; 164/279138 objects degraded (0.059%); 4699/279138 objects misplaced (1.683%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[29993]: osdmap e331: 8 total, 8 up, 8 in 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 164/279138 objects degraded (0.059%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[29993]: osdmap e332: 8 total, 8 up, 8 in 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[39470]: pgmap v554: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.37k op/s; 164/279138 objects degraded (0.059%); 4699/279138 objects misplaced (1.683%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:23:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:23:56.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:23:56.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[39470]: osdmap e331: 8 total, 8 up, 8 in 2024-09-16T10:23:56.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 164/279138 objects degraded (0.059%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:23:56.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:56 smithi063 ceph-mon[39470]: osdmap e332: 8 total, 8 up, 8 in 2024-09-16T10:23:56.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:57.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:57.115 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:23:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:23:57.240 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:57.369 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:23:57.369 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:23:57.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:57.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:57.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:23:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:23:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:23:57.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.563 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:57.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:57.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:57.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:23:57.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:23:57.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:57.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:57.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:58.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:58.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:58.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:58.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:23:58.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:58.228 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:58.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:58.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:58.378 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:58.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:23:58.432 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:23:58.432 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-16T10:23:58.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:58.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:58.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:58.525 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:23:58.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:58.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:58.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:58.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:58.785 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:58.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:58.872 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:23:58.872 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:23:58.873 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:58.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:58.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:23:58.908 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:23:58.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:58.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:58.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:58 smithi063 ceph-mon[39470]: pgmap v557: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 694 op/s; 164/279138 objects degraded (0.059%); 4699/279138 objects misplaced (1.683%); 848 KiB/s, 13 objects/s recovering 2024-09-16T10:23:58.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:58 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:23:58.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:58 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:23:58.955 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:58 smithi063 ceph-mon[29993]: pgmap v557: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 694 op/s; 164/279138 objects degraded (0.059%); 4699/279138 objects misplaced (1.683%); 848 KiB/s, 13 objects/s recovering 2024-09-16T10:23:58.955 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:58 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:23:58.956 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:58 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:23:58.956 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:23:58.956 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:23:58.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:58 smithi165 ceph-mon[32696]: pgmap v557: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 165 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 671 B/s rd, 0 B/s wr, 694 op/s; 164/279138 objects degraded (0.059%); 4699/279138 objects misplaced (1.683%); 848 KiB/s, 13 objects/s recovering 2024-09-16T10:23:58.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:58 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:23:58.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:58 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:23:58.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:23:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:23:58.986 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:23:58.987 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:23:59.012 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:23:59.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:59.022 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:23:59.022 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:23:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:59.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:59.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:23:59.065 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:23:59.065 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:23:59.097 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:23:59.098 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:23:59.134 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:23:59.134 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72703 7.5 7:ace6f8ac:::benchmark_data_smithi063_73086_object72702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.878319+0000", 2024-09-16T10:23:59.139 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 505.25455963899998, 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10653797499999999, 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72703 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878319+0000", 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878319+0000", 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878321+0000", 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0449999999999999e-06 2024-09-16T10:23:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878367+0000", 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5639999999999997e-05 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878379+0000", 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2323e-05 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878386+0000", 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7989999999999996e-06 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878441+0000", 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.469e-05 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878556+0000", 2024-09-16T10:23:59.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115494 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.878705+0000", 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014857800000000001 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.883091+0000", 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043867230000000004 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.886390+0000", 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032982129999999999 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984773+0000", 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098383089000000007 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984822+0000", 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8921000000000001e-05 2024-09-16T10:23:59.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984857+0000", 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5460000000000003e-05 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:72706 7.5 7:affb7f46:::benchmark_data_smithi063_73086_object72705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:33.879155+0000", 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 505.253723923, 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105803568, 2024-09-16T10:23:59.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 72706 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879155+0000", 2024-09-16T10:23:59.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879155+0000", 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879156+0000", 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879225+0000", 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9202000000000006e-05 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879240+0000", 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4399000000000001e-05 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.879246+0000", 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4729999999999997e-06 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.880777+0000", 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001531325 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.880894+0000", 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116352 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.881025+0000", 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013082400000000001 2024-09-16T10:23:59.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.883350+0000", 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023250409999999999 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.886489+0000", 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031394550000000002 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984891+0000", 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098401706000000005 2024-09-16T10:23:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984925+0000", 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4462000000000003e-05 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:33.984958+0000", 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3092000000000001e-05 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174952 7.5 7:a881b411:::benchmark_data_smithi063_73086_object55673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.888558+0000", 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 405.24432070699999, 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10417939399999999, 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174952 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888558+0000", 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888558+0000", 2024-09-16T10:23:59.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888561+0000", 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.571e-06 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888568+0000", 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888584+0000", 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6036e-05 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888593+0000", 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5999999999999996e-06 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888667+0000", 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3226000000000007e-05 2024-09-16T10:23:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888797+0000", 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013077999999999999 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888907+0000", 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109959 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.893981+0000", 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050738579999999997 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896037+0000", 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020559530000000001 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992624+0000", 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096586728999999996 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992695+0000", 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1174999999999995e-05 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992737+0000", 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2330999999999998e-05 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174953 7.5 7:a2672bfb:::benchmark_data_smithi063_73086_object55674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.888773+0000", 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 405.24410592700002, 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10411178, 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174953 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888773+0000", 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888773+0000", 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888774+0000", 2024-09-16T10:23:59.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2799999999999995e-07 2024-09-16T10:23:59.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888778+0000", 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0210000000000001e-06 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888783+0000", 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8449999999999999e-06 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888787+0000", 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9060000000000004e-06 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.891543+0000", 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027551820000000001 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.891659+0000", 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0001165 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.891779+0000", 2024-09-16T10:23:59.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011995 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895997+0000", 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042179310000000003 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896225+0000", 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022793800000000001 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992799+0000", 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096574229999999997 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992845+0000", 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6041999999999997e-05 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.992885+0000", 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9406999999999997e-05 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.166 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174954 7.5 7:a9783837:::benchmark_data_smithi063_73086_object55675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.888820+0000", 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 405.24405898999998, 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112695169, 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.167 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174954 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888820+0000", 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888820+0000", 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888821+0000", 2024-09-16T10:23:59.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:23:59.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8400000000000003e-07 2024-09-16T10:23:59.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888824+0000", 2024-09-16T10:23:59.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5030000000000002e-06 2024-09-16T10:23:59.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888831+0000", 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2389999999999996e-06 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.888837+0000", 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7250000000000002e-06 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.894010+0000", 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051734229999999999 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.894103+0000", 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2594000000000001e-05 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.894171+0000", 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7414000000000002e-05 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896108+0000", 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019373540000000001 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.896336+0000", 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000227777 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.001448+0000", 2024-09-16T10:23:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105112742 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.001482+0000", 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4014999999999999e-05 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.001515+0000", 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499000000000002e-05 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:200795 7.5 7:a5355407:::benchmark_data_smithi063_73086_object81516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.235559+0000", 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 378.89731931099999, 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.088954024000000007, 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.175 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200795 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235559+0000", 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235559+0000", 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235561+0000", 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.356e-06 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235565+0000", 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6439999999999999e-06 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235577+0000", 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2064e-05 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.235585+0000", 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2509999999999997e-06 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.237700+0000", 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021148600000000001 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.237830+0000", 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013032899999999999 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.237962+0000", 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131977 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.244826+0000", 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068638509999999998 2024-09-16T10:23:59.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.246316+0000", 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001490571 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324433+0000", 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078116624999999995 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324476+0000", 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.282e-05 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.324513+0000", 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7676000000000003e-05 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:200801 7.5 7:a8e7ac74:::benchmark_data_smithi063_73086_object81522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:23:59.181 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.240067+0000", 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 378.89281209400002, 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092800275000000002, 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200801 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240067+0000", 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240067+0000", 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240069+0000", 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0999999999999998e-06 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240074+0000", 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4090000000000002e-06 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240087+0000", 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2661e-05 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240095+0000", 2024-09-16T10:23:59.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8590000000000005e-06 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240719+0000", 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062448799999999997 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240840+0000", 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012078699999999999 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.240961+0000", 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121004 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.244830+0000", 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038687109999999999 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.246349+0000", 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015198130000000001 2024-09-16T10:23:59.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332752+0000", 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.086402822000000004 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332814+0000", 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1289000000000002e-05 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332867+0000", 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3331999999999997e-05 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:200804 7.5 7:a1363d7b:::benchmark_data_smithi063_73086_object81525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:40.241179+0000", 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 378.89169933300002, 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.091818583999999995, 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.188 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200804 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241179+0000", 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241179+0000", 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241181+0000", 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3629999999999999e-06 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241185+0000", 2024-09-16T10:23:59.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.34e-06 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241195+0000", 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0254999999999999e-05 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.241202+0000", 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4420000000000003e-06 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.242942+0000", 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017397510000000001 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.243058+0000", 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116129 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.243163+0000", 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104888 2024-09-16T10:23:59.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.244949+0000", 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001786953 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.246401+0000", 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014510980000000001 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332921+0000", 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.086520606 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332966+0000", 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4706000000000002e-05 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:40.332998+0000", 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2053e-05 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:28376 9.d 9:b64251d8:::benchmark_data_smithi063_124717_object28375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:15.605145+0000", 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 103.527733449, 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092685612000000001, 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 28376 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605145+0000", 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605145+0000", 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605146+0000", 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0800000000000004e-07 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605183+0000", 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7338000000000001e-05 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605193+0000", 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0032e-05 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.605200+0000", 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8580000000000002e-06 2024-09-16T10:23:59.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.607011+0000", 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001811177 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.607134+0000", 2024-09-16T10:23:59.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122776 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.607219+0000", 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.488e-05 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.608041+0000", 2024-09-16T10:23:59.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082233800000000004 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.608276+0000", 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000234837 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.697740+0000", 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089463903999999997 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.697775+0000", 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4635000000000003e-05 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:15.697831+0000", 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6029000000000002e-05 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.202 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34105 9.e 9:7031eac9:::benchmark_data_smithi063_124717_object34104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.010625+0000", 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.122253402, 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11631763000000001, 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34105 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.010625+0000", 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.010625+0000", 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.010627+0000", 2024-09-16T10:23:59.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2619999999999999e-06 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.011187+0000", 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056009600000000005 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.011197+0000", 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0321e-05 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.011204+0000", 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6370000000000002e-06 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.012054+0000", 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085072100000000003 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.012304+0000", 2024-09-16T10:23:59.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024928 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.012420+0000", 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116605 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.015009+0000", 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025885779999999998 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.015053+0000", 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3781e-05 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126825+0000", 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11177234599999999 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126888+0000", 2024-09-16T10:23:59.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3040999999999995e-05 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126943+0000", 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4962e-05 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34123 9.e 9:700d37e5:::benchmark_data_smithi063_124717_object34122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.209 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.019216+0000", 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.11366271599999, 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11142177, 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34123 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.210 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019216+0000", 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019216+0000", 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019218+0000", 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9039999999999999e-06 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019248+0000", 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9603000000000001e-05 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019260+0000", 2024-09-16T10:23:59.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2011000000000001e-05 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019265+0000", 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1529999999999996e-06 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019313+0000", 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8605e-05 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019556+0000", 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024223199999999999 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019687+0000", 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013166600000000001 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.021167+0000", 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014795399999999999 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.021868+0000", 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070079799999999998 2024-09-16T10:23:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130568+0000", 2024-09-16T10:23:59.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10870049900000001 2024-09-16T10:23:59.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130602+0000", 2024-09-16T10:23:59.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3524e-05 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130638+0000", 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6235000000000001e-05 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.218 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34126 9.3 9:c6eaf356:::benchmark_data_smithi063_124717_object34125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.019477+0000", 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.113401576, 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111048233, 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34126 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019477+0000", 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019477+0000", 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019478+0000", 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6600000000000005e-07 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019547+0000", 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8853e-05 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019555+0000", 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8769999999999993e-06 2024-09-16T10:23:59.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019559+0000", 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1930000000000003e-06 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019610+0000", 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1343e-05 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.019887+0000", 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027674900000000002 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020069+0000", 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000181509 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.022182+0000", 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002113541 2024-09-16T10:23:59.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.126572+0000", 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104389754 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130383+0000", 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003811156 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130476+0000", 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2854000000000002e-05 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130525+0000", 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9537999999999999e-05 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34128 9.d 9:b9451fa2:::benchmark_data_smithi063_124717_object34127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.225 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.020108+0000", 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.112770411, 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110568416, 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34128 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020108+0000", 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020108+0000", 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020110+0000", 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3200000000000001e-06 2024-09-16T10:23:59.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020282+0000", 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017282899999999999 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020299+0000", 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7045e-05 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020306+0000", 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8159999999999996e-06 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020348+0000", 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2012999999999999e-05 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020456+0000", 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107539 2024-09-16T10:23:59.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.020569+0000", 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011336900000000001 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.023688+0000", 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031182919999999999 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.023701+0000", 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3773e-05 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130591+0000", 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10688929899999999 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130641+0000", 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0253999999999997e-05 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130677+0000", 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5867000000000001e-05 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:34135 9.e 9:7f486d23:::benchmark_data_smithi063_124717_object34134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.024112+0000", 2024-09-16T10:23:59.232 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.10876720900001, 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106646003, 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 34135 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024112+0000", 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024112+0000", 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024113+0000", 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.885e-06 2024-09-16T10:23:59.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024216+0000", 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102953 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024227+0000", 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0966e-05 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024233+0000", 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5339999999999999e-06 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024283+0000", 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9676999999999999e-05 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024367+0000", 2024-09-16T10:23:59.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4297999999999998e-05 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.024472+0000", 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010469200000000001 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.025659+0000", 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00118731 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.026080+0000", 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042150899999999998 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130689+0000", 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104608439 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130725+0000", 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6517000000000003e-05 2024-09-16T10:23:59.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.130758+0000", 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2223e-05 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35062 9.3 9:c7e49ef2:::benchmark_data_smithi063_124717_object35061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.657982+0000", 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 99.474896775999994, 2024-09-16T10:23:59.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098016333999999997, 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35062 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.657982+0000", 2024-09-16T10:23:59.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.657982+0000", 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.657983+0000", 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.519e-06 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658015+0000", 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1820999999999999e-05 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658031+0000", 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877999999999999e-05 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658038+0000", 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7619999999999998e-06 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658103+0000", 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5547000000000001e-05 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658224+0000", 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120492 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.658343+0000", 2024-09-16T10:23:59.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118836 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.659790+0000", 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014468860000000001 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.661285+0000", 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001495106 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.755758+0000", 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.094472798999999996 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.755824+0000", 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6662000000000007e-05 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.755998+0000", 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174026 2024-09-16T10:23:59.245 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35071 9.e 9:7ced6e27:::benchmark_data_smithi063_124717_object35070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.663307+0000", 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 99.469571525999996, 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092814160000000007, 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.246 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35071 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663307+0000", 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663307+0000", 2024-09-16T10:23:59.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663309+0000", 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.381e-06 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663380+0000", 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1410999999999998e-05 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663389+0000", 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.397e-06 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663395+0000", 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5310000000000001e-06 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663442+0000", 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7074999999999999e-05 2024-09-16T10:23:59.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663550+0000", 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107712 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.663665+0000", 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115765 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666160+0000", 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024947010000000002 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666207+0000", 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6604999999999998e-05 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.756015+0000", 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089808288 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.756074+0000", 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9097e-05 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.756121+0000", 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7197000000000002e-05 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35072 9.e 9:7d9eb204:::benchmark_data_smithi063_124717_object35071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.664392+0000", 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 99.468486780999996, 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097734269999999998, 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35072 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.253 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664392+0000", 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664392+0000", 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664393+0000", 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.548e-06 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664514+0000", 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012095900000000001 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664528+0000", 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3368e-05 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664535+0000", 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.007e-06 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665039+0000", 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050449100000000003 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665152+0000", 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011252999999999999 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665321+0000", 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016960599999999999 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666269+0000", 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00094797700000000002 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666889+0000", 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062005199999999995 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.762030+0000", 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095141030000000001 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.762084+0000", 2024-09-16T10:23:59.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3996000000000003e-05 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.762126+0000", 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1705999999999997e-05 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:35073 9.d 9:b3b70288:::benchmark_data_smithi063_124717_object35072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:19.664788+0000", 2024-09-16T10:23:59.259 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 99.468090617000001, 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095865565999999999, 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 35073 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664788+0000", 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664788+0000", 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664789+0000", 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664859+0000", 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9968999999999994e-05 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664873+0000", 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3570999999999999e-05 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664880+0000", 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2810000000000003e-06 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.664925+0000", 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4685000000000003e-05 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665060+0000", 2024-09-16T10:23:59.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013474400000000001 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.665196+0000", 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013607199999999999 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666424+0000", 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012285359999999999 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.666735+0000", 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000310833 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.760561+0000", 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093826309999999996 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.760614+0000", 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2232999999999999e-05 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:19.760654+0000", 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0052000000000002e-05 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:37532 9.3 9:cf32c78d:::benchmark_data_smithi063_124717_object37531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:21.177962+0000", 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 97.954916448999995, 2024-09-16T10:23:59.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.091552389999999997, 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 37532 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177962+0000", 2024-09-16T10:23:59.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177962+0000", 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177963+0000", 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6600000000000005e-07 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177993+0000", 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9892000000000001e-05 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.177999+0000", 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-06 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178004+0000", 2024-09-16T10:23:59.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6609999999999999e-06 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178045+0000", 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1409999999999998e-05 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178151+0000", 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106105 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:23:59.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.178269+0000", 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118021 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.181252+0000", 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00298307 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.183294+0000", 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020413699999999998 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.269406+0000", 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.086112094 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.269475+0000", 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9246999999999996e-05 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.269515+0000", 2024-09-16T10:23:59.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9453999999999999e-05 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:37541 9.3 9:c2ca3a24:::benchmark_data_smithi063_124717_object37540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:22:21.186221+0000", 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 97.946657857000005, 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.092424884999999998, 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:23:59.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 37541 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186221+0000", 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186221+0000", 2024-09-16T10:23:59.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186222+0000", 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6249999999999999e-06 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186275+0000", 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2549000000000001e-05 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186285+0000", 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7650000000000005e-06 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186290+0000", 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9219999999999996e-06 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186329+0000", 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9509999999999999e-05 2024-09-16T10:23:59.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186417+0000", 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7609999999999996e-05 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.186501+0000", 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3737000000000001e-05 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.187593+0000", 2024-09-16T10:23:59.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010920649999999999 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.266582+0000", 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078989689000000002 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.278548+0000", 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011965748 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.278600+0000", 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1845000000000003e-05 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:22:21.278646+0000", 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5819999999999998e-05 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:23:59.280 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:23:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:59.281 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:59.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:23:59.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:23:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:59.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:59.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:23:59.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:23:59.545 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:59.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:23:59.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:23:59.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:59.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:23:59.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:23:59.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:59.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:23:59.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2745376729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:59.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:23:59.776 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:23:59.776 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:23:59.776 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:23:59.776 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2745376729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:23:59.776 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:23:59 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:23:59.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:23:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:59.828 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:23:59.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:23:59.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:23:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:59.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:23:59.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:59 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:24:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:59 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:24:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2745376729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:23:59 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:24:00.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:00.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:00.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:00.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:00.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:00.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:00.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:24:00.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:00.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:00.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:00.485 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:00.528 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:00.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:00.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:00.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:00.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:00.727 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:24:00.766 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:00.809 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:00.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:00.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:00.939 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:00.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:00.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[39470]: pgmap v558: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:24:00.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:24:00.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[39470]: osdmap e333: 8 total, 8 up, 8 in 2024-09-16T10:24:00.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:24:00.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:24:00.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3600934901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:00.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[29993]: pgmap v558: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:24:00.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:24:00.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[29993]: osdmap e333: 8 total, 8 up, 8 in 2024-09-16T10:24:00.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:24:00.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:24:00.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3600934901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:01.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:00 smithi165 ceph-mon[32696]: pgmap v558: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:24:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:00 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:24:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:00 smithi165 ceph-mon[32696]: osdmap e333: 8 total, 8 up, 8 in 2024-09-16T10:24:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:00 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:24:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:00 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:24:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3600934901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:01.078 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:01.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:01.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:01.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:01.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:01.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:01.346 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:01.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:01.448 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:01.483 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:01.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:01.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:01.628 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:01.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:01.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:01.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:01.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:01.772 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:24:01.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:01.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3600934901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:01.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[29993]: osdmap e334: 8 total, 8 up, 8 in 2024-09-16T10:24:01.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 419/263034 objects degraded (0.159%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:01.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[29993]: 1.c scrub starts 2024-09-16T10:24:01.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:24:01.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:01.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3600934901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:01.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[39470]: osdmap e334: 8 total, 8 up, 8 in 2024-09-16T10:24:01.778 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 419/263034 objects degraded (0.159%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:01.779 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:01 smithi063 ceph-mon[39470]: 1.c scrub starts 2024-09-16T10:24:01.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:01.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:01.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:01.895 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:01.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:01.999 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:24:01.999 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:24:01.999 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:24:01.999 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:24:01.999 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:24:02.000 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:24:02.000 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:24:02.000 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:24:02.000 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:24:02.000 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:24:02.000 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:24:02.000 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:24:02.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:02.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:01 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:24:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:01 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:02.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3600934901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:02.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:01 smithi165 ceph-mon[32696]: osdmap e334: 8 total, 8 up, 8 in 2024-09-16T10:24:02.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:01 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 419/263034 objects degraded (0.159%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:02.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:01 smithi165 ceph-mon[32696]: 1.c scrub starts 2024-09-16T10:24:02.116 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:02.183 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:02.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:02.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:02.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:02.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:02.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:02.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:02.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:02.564 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:02.603 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:02.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:02.689 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:24:02.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:02.780 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:24:02.780 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:149509 9.b 9:d75e5eb5:::benchmark_data_smithi063_124717_object33278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:02.775519+0000", 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0022113150000000002, 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022412259999999998, 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for rw locks", 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:02.781 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 149509 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775519+0000", 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775519+0000", 2024-09-16T10:24:02.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775521+0000", 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.536e-06 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775525+0000", 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6500000000000004e-06 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:02.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775533+0000", 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.875e-06 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775539+0000", 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6130000000000003e-06 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775590+0000", 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1038999999999997e-05 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for rw locks", 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775723+0000", 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013313800000000001 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDPGPushReply(9.b 334/298 [PushReplyOp(9:d75e5eb5:::benchmark_data_smithi063_124717_object33278:head)])", 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:02.775932+0000", 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0017983669999999999, 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018991800000000001, 2024-09-16T10:24:02.785 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775932+0000", 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775932+0000", 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775933+0000", 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.31e-06 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775939+0000", 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1619999999999996e-06 2024-09-16T10:24:02.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:02.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775950+0000", 2024-09-16T10:24:02.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0688e-05 2024-09-16T10:24:02.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:02.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.775958+0000", 2024-09-16T10:24:02.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2439999999999993e-06 2024-09-16T10:24:02.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:02.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:02.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:02.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:02.776028+0000", 2024-09-16T10:24:02.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9627000000000003e-05 2024-09-16T10:24:02.790 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:02.790 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:02.790 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:02.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:02.791 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:24:02.791 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 2 2024-09-16T10:24:02.791 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:24:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:02.807 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:02.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:02.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:02.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:02.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:03.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:03.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:02 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:24:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:02 smithi165 ceph-mon[32696]: pgmap v561: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 963 op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:24:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:02 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:24:03.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:03.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:03.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:03.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:03.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:03.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:02 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:24:03.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:02 smithi063 ceph-mon[39470]: pgmap v561: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 963 op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:24:03.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:02 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:24:03.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:02 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:24:03.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:02 smithi063 ceph-mon[29993]: pgmap v561: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 963 op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:24:03.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:02 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:24:03.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:03.347 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:03.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:24:03.356 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:24:03.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-16T10:24:03.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:03.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:03.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:03.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:03.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:03.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:03.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:24:03.706 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:03.762 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:03.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:03.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:03.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:03.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:03.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:04.035 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:03 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:24:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:03 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:24:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:03 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:24:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:03 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:24:04.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:04.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:24:04.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:03 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:24:04.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:04.311 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:04.413 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.452 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:04.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:04.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:04.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:04.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:04.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:04.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:04.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:04.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:04.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:04.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:04 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:24:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:04 smithi063 ceph-mon[29993]: pgmap v562: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 696 B/s rd, 0 B/s wr, 731 op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 854 KiB/s, 13 objects/s recovering 2024-09-16T10:24:04.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:04 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:24:04.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:04 smithi063 ceph-mon[39470]: pgmap v562: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 696 B/s rd, 0 B/s wr, 731 op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 854 KiB/s, 13 objects/s recovering 2024-09-16T10:24:04.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:04.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:04 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:24:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:04 smithi165 ceph-mon[32696]: pgmap v562: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 162 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 696 B/s rd, 0 B/s wr, 731 op/s; 419/263034 objects degraded (0.159%); 4248/263034 objects misplaced (1.615%); 854 KiB/s, 13 objects/s recovering 2024-09-16T10:24:05.076 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:05.117 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:05.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:05.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:05.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:05.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:05.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:05.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:05.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:05.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:05.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:24:05.558 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:05.570 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:05.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:05.628 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:05.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:05.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:05.800 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:05.854 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:05.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:05.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:05.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:05.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:06.048 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:05 smithi063 ceph-mon[29993]: pgmap v563: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 621/246390 objects degraded (0.252%); 3806/246390 objects misplaced (1.545%); 1.7 MiB/s, 27 objects/s recovering 2024-09-16T10:24:06.049 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:05 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:24:06.049 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:05 smithi063 ceph-mon[39470]: pgmap v563: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 621/246390 objects degraded (0.252%); 3806/246390 objects misplaced (1.545%); 1.7 MiB/s, 27 objects/s recovering 2024-09-16T10:24:06.049 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:05 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:24:06.086 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:06.132 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:06.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:06.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:06.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:06.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:05 smithi165 ceph-mon[32696]: pgmap v563: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 621/246390 objects degraded (0.252%); 3806/246390 objects misplaced (1.545%); 1.7 MiB/s, 27 objects/s recovering 2024-09-16T10:24:06.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:05 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:24:06.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:06.406 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:06.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:06.508 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:06.549 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:06.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:06.632 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:24:06.632 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:24:06.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:24:06.676 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:24:06.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:06.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:06.720 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:24:06.721 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:24:06.757 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:24:06.758 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:24:06.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:06.798 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:24:06.799 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:24:06.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:06.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:06.848 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:24:06.848 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:24:06.889 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:24:06.889 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:24:06.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:06 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 621/246390 objects degraded (0.252%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:06.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/456110322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:06.892 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:06 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 621/246390 objects degraded (0.252%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:06.893 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/456110322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:06.900 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:06.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:06.923 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:24:06.923 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:24:06.923 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:24:06.924 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:24:06.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:24:06.931 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:24:06.969 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:06.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:07.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:07.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:07.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:07.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:24:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:24:07.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:07.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:07.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:06 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 621/246390 objects degraded (0.252%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/456110322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:07.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:07.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:07.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:07.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:07.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:07.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:07.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:07.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:07.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:07.711 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:07.752 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:07.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:07.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:07.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:07.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:07.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:07.983 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:08.019 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:08.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:08.078 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:08.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:08.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:08.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:08.194 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:24:08.194 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:24:08.194 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[29993]: osdmap e335: 8 total, 8 up, 8 in 2024-09-16T10:24:08.194 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[29993]: pgmap v565: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 736 op/s; 621/246390 objects degraded (0.252%); 3806/246390 objects misplaced (1.545%); 1.0 MiB/s, 15 objects/s recovering 2024-09-16T10:24:08.194 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:24:08.194 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:24:08.194 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[39470]: osdmap e335: 8 total, 8 up, 8 in 2024-09-16T10:24:08.195 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:07 smithi063 ceph-mon[39470]: pgmap v565: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 736 op/s; 621/246390 objects degraded (0.252%); 3806/246390 objects misplaced (1.545%); 1.0 MiB/s, 15 objects/s recovering 2024-09-16T10:24:08.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:08.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:08.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:07 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:24:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:07 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:24:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:07 smithi165 ceph-mon[32696]: osdmap e335: 8 total, 8 up, 8 in 2024-09-16T10:24:08.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:07 smithi165 ceph-mon[32696]: pgmap v565: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 736 op/s; 621/246390 objects degraded (0.252%); 3806/246390 objects misplaced (1.545%); 1.0 MiB/s, 15 objects/s recovering 2024-09-16T10:24:08.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:08.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:08.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:08.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:08.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:08.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:08.662 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:08.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:08.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:08.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:08.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:08.872 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:08.872 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:08 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:08.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:08.873 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:08 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:08.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:08.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:09.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:09.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:08 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:09.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:24:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:24:09.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:09.147 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:09.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:09.249 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:09.291 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:09.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:09.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:09.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:09.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:09.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:09.570 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:09.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:09.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:24:09.705 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:24:09.706 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-16T10:24:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:09.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:09.764 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:24:09.764 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:24:09.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:09.877 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:09.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:09.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:10.021 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:10.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:10.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:10.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:10.195 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:09 smithi063 ceph-mon[29993]: pgmap v566: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.7 MiB/s, 27 objects/s recovering 2024-09-16T10:24:10.195 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:09 smithi063 ceph-mon[39470]: pgmap v566: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.7 MiB/s, 27 objects/s recovering 2024-09-16T10:24:10.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:10.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:09 smithi165 ceph-mon[32696]: pgmap v566: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.7 MiB/s, 27 objects/s recovering 2024-09-16T10:24:10.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:10.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:10.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:10.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:10.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:10.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:10.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:10.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:10.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:24:10.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:24:10.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:10.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:10.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:10.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:10.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:10.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:10.925 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:11.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:11.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:11.112 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:11.144 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:11.240 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:11.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:11.272 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:10 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 838/229842 objects degraded (0.365%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:11.272 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:10 smithi063 ceph-mon[39470]: osdmap e336: 8 total, 8 up, 8 in 2024-09-16T10:24:11.272 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:10 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 838/229842 objects degraded (0.365%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:11.272 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:10 smithi063 ceph-mon[29993]: osdmap e336: 8 total, 8 up, 8 in 2024-09-16T10:24:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:10 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 838/229842 objects degraded (0.365%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:11.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:10 smithi165 ceph-mon[32696]: osdmap e336: 8 total, 8 up, 8 in 2024-09-16T10:24:11.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:11.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:11.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:11.373 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:11.374 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:11.465 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:11.488 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:11.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:11.589 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:11.632 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:11.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:11.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:11.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:11.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:11.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:11.900 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:11.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:11.981 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:24:12.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:12.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:12.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:12.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:12.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:12.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:12.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2254045518' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:11 smithi165 ceph-mon[32696]: pgmap v568: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.9 MiB/s, 29 objects/s recovering 2024-09-16T10:24:12.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:12.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:12.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2254045518' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:12.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:11 smithi063 ceph-mon[39470]: pgmap v568: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.9 MiB/s, 29 objects/s recovering 2024-09-16T10:24:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2254045518' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:11 smithi063 ceph-mon[29993]: pgmap v568: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.9 MiB/s, 29 objects/s recovering 2024-09-16T10:24:12.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:12.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:12.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:12.674 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:12.776 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:12.816 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:12.918 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:12.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:12.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:13.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:13.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:13.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:13.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:13.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:12 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:13.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2254045518' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:24:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[39470]: osdmap e337: 8 total, 8 up, 8 in 2024-09-16T10:24:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:12 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2254045518' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:24:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[29993]: osdmap e337: 8 total, 8 up, 8 in 2024-09-16T10:24:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:13.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:12 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:12 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2254045518' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:24:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:12 smithi165 ceph-mon[32696]: osdmap e337: 8 total, 8 up, 8 in 2024-09-16T10:24:13.337 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:13.371 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:13.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:13.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:13.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:13.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:13.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:13.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:13.651 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:13.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:13.753 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:13.795 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:13.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:13.896 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:13.923 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:13.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:14.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:14.048 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:14.150 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:14.173 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:14.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:14.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:14.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:13 smithi165 ceph-mon[32696]: 3.3 scrub starts 2024-09-16T10:24:14.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:13 smithi165 ceph-mon[32696]: 3.3 scrub ok 2024-09-16T10:24:14.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:13 smithi165 ceph-mon[32696]: pgmap v570: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 858 op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:24:14.337 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[29993]: 3.3 scrub starts 2024-09-16T10:24:14.337 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[29993]: 3.3 scrub ok 2024-09-16T10:24:14.337 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[29993]: pgmap v570: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 858 op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:24:14.337 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[39470]: 3.3 scrub starts 2024-09-16T10:24:14.337 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[39470]: 3.3 scrub ok 2024-09-16T10:24:14.338 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:13 smithi063 ceph-mon[39470]: pgmap v570: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 858 op/s; 838/229842 objects degraded (0.365%); 3361/229842 objects misplaced (1.462%); 1.1 MiB/s, 17 objects/s recovering 2024-09-16T10:24:14.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:14.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:14.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:14.569 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:14.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:14.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:14.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:14.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:14.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:14.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:14.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:15.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:15.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:15.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:15.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:15.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:15.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:15.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:15.393 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:15.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:15.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:15.545 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:15.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:15.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:15.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:15.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:15.694 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:24:15.694 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:24:15.694 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:24:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:24:15.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:15.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:15.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:15.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:15.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:16.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:16.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:16.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:16.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:16.232 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:16.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:16.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:16.369 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:16.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:16.471 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:16.519 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:16.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:16.621 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:16.659 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:16.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[29993]: pgmap v571: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1062/214542 objects degraded (0.495%); 2899/214542 objects misplaced (1.351%); 1.8 MiB/s, 29 objects/s recovering 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1062/214542 objects degraded (0.495%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[29993]: osdmap e338: 8 total, 8 up, 8 in 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[39470]: pgmap v571: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1062/214542 objects degraded (0.495%); 2899/214542 objects misplaced (1.351%); 1.8 MiB/s, 29 objects/s recovering 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1062/214542 objects degraded (0.495%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[39470]: osdmap e338: 8 total, 8 up, 8 in 2024-09-16T10:24:16.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:24:16.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:16 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:24:16.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:16.796 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:16.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:16.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:16 smithi165 ceph-mon[32696]: pgmap v571: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1062/214542 objects degraded (0.495%); 2899/214542 objects misplaced (1.351%); 1.8 MiB/s, 29 objects/s recovering 2024-09-16T10:24:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:16 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1062/214542 objects degraded (0.495%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:16 smithi165 ceph-mon[32696]: osdmap e338: 8 total, 8 up, 8 in 2024-09-16T10:24:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:16 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:24:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:16 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:24:16.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:16.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:16.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:17.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:17.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:17.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:17.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:17.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:17.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:17.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:17.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:24:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:24:17.294 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:17.336 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:17.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:17.437 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:17.476 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:17.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:17.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:17.662 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:17.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:17.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[29993]: 4.10 scrub starts 2024-09-16T10:24:17.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[29993]: 4.10 scrub ok 2024-09-16T10:24:17.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[29993]: 6.d scrub starts 2024-09-16T10:24:17.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[29993]: 6.d scrub ok 2024-09-16T10:24:17.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[29993]: 5.4 scrub starts 2024-09-16T10:24:17.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[29993]: 5.4 scrub ok 2024-09-16T10:24:17.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[29993]: osdmap e339: 8 total, 8 up, 8 in 2024-09-16T10:24:17.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[39470]: 4.10 scrub starts 2024-09-16T10:24:17.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[39470]: 4.10 scrub ok 2024-09-16T10:24:17.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[39470]: 6.d scrub starts 2024-09-16T10:24:17.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[39470]: 6.d scrub ok 2024-09-16T10:24:17.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[39470]: 5.4 scrub starts 2024-09-16T10:24:17.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[39470]: 5.4 scrub ok 2024-09-16T10:24:17.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:17 smithi063 ceph-mon[39470]: osdmap e339: 8 total, 8 up, 8 in 2024-09-16T10:24:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:24:17.723 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:24:17.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-16T10:24:17.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:17.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:17.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:17.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:17 smithi165 ceph-mon[32696]: 4.10 scrub starts 2024-09-16T10:24:17.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:17 smithi165 ceph-mon[32696]: 4.10 scrub ok 2024-09-16T10:24:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:17 smithi165 ceph-mon[32696]: 6.d scrub starts 2024-09-16T10:24:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:17 smithi165 ceph-mon[32696]: 6.d scrub ok 2024-09-16T10:24:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:17 smithi165 ceph-mon[32696]: 5.4 scrub starts 2024-09-16T10:24:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:17 smithi165 ceph-mon[32696]: 5.4 scrub ok 2024-09-16T10:24:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:17 smithi165 ceph-mon[32696]: osdmap e339: 8 total, 8 up, 8 in 2024-09-16T10:24:17.896 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:17.943 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:17.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:17.993 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:24:17.993 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:24:18.042 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:24:18.043 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:24:18.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:18.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:18.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:18.084 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:24:18.084 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:24:18.129 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:24:18.130 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:24:18.159 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:24:18.160 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:24:18.171 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:18.197 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:24:18.197 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:24:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:18.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:18.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:18.235 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:24:18.235 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:24:18.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:24:18.273 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:24:18.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:18.367 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:18.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:18.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:18.513 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:18.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:18.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:18.654 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:18.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:18.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:18 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:24:18.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:18 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:24:18.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:18 smithi165 ceph-mon[32696]: pgmap v574: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 850 op/s; 1062/214542 objects degraded (0.495%); 2899/214542 objects misplaced (1.351%); 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:24:18.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1669366736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:18.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:24:18.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:24:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[29993]: pgmap v574: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 850 op/s; 1062/214542 objects degraded (0.495%); 2899/214542 objects misplaced (1.351%); 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:24:18.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1669366736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:18.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:24:18.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:24:18.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[39470]: pgmap v574: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 850 op/s; 1062/214542 objects degraded (0.495%); 2899/214542 objects misplaced (1.351%); 1.2 MiB/s, 19 objects/s recovering 2024-09-16T10:24:18.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1669366736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:18.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:18.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:18.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:18.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:18.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:18.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:19.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:19.051 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:24:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:24:19.057 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:19.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:19.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:19.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:19.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:19.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:19.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:19.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:19.425 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:19.459 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:19.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:19.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:19.601 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:19.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:19.700 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:24:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:24:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:24:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:24:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51238 8.e 8:7236be56:::benchmark_data_smithi063_101290_object51237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 304.92041563100003, 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96343729, 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:19.703 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:19.703 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51238 2024-09-16T10:24:19.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.703 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341637+0000", 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.621e-06 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341663+0000", 2024-09-16T10:24:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6026000000000001e-05 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341674+0000", 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0871e-05 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341682+0000", 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8390000000000007e-06 2024-09-16T10:24:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341780+0000", 2024-09-16T10:24:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8233999999999995e-05 2024-09-16T10:24:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341791+0000", 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1426999999999999e-05 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296465+0000", 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95467394000000005 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296604+0000", 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013824400000000001 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296727+0000", 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012366699999999999 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304672+0000", 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079445339999999996 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304896+0000", 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000224117 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304941+0000", 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5374999999999998e-05 2024-09-16T10:24:19.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305028+0000", 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6447999999999996e-05 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305073+0000", 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4947000000000002e-05 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51239 8.e 8:7166e593:::benchmark_data_smithi063_101290_object51238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 304.919997454, 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96520916099999998, 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51239 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.712 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342055+0000", 2024-09-16T10:24:19.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.051e-06 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342073+0000", 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8325000000000002e-05 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342083+0000", 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9850000000000001e-06 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342090+0000", 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2119999999999997e-06 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342196+0000", 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010593400000000001 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342244+0000", 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8272000000000001e-05 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.715 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299084+0000", 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95683958300000005 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299216+0000", 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131595 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299311+0000", 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5445000000000005e-05 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305117+0000", 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058058169999999996 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305157+0000", 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9836999999999998e-05 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307167+0000", 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020101390000000002 2024-09-16T10:24:19.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307221+0000", 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4317000000000002e-05 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307263+0000", 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1649000000000001e-05 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51244 8.e 8:7a51d24a:::benchmark_data_smithi063_101290_object51243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 304.91677632300002, 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96214958799999994, 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51244 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.719 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345277+0000", 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9810000000000002e-06 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345305+0000", 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7858999999999999e-05 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345320+0000", 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5268e-05 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345328+0000", 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2900000000000002e-06 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345404+0000", 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6241000000000004e-05 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345415+0000", 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0672e-05 2024-09-16T10:24:19.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300950+0000", 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95553457900000005 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301082+0000", 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013229299999999999 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301242+0000", 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015961099999999999 2024-09-16T10:24:19.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307271+0000", 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060297889999999998 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307310+0000", 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8288999999999999e-05 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307349+0000", 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9499999999999998e-05 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307391+0000", 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1779000000000001e-05 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307424+0000", 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3436999999999999e-05 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51260 8.e 8:7df47e01:::benchmark_data_smithi063_101290_object51259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 304.89981540600002, 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95708149499999995, 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.726 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51260 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362238+0000", 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4140000000000001e-06 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362265+0000", 2024-09-16T10:24:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7183999999999999e-05 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362279+0000", 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3832000000000001e-05 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362287+0000", 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6990000000000003e-06 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362347+0000", 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0560000000000003e-05 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362356+0000", 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1990000000000006e-06 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305199+0000", 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94284296700000003 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305311+0000", 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011281900000000001 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305449+0000", 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000137708 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309902+0000", 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044530100000000003 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312979+0000", 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030772859999999998 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317533+0000", 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045533479999999996 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319259+0000", 2024-09-16T10:24:19.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017263129999999999 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319317+0000", 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8155999999999997e-05 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9:dc100d6c:::benchmark_data_smithi063_124717_object60003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:24:19.733 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.380451737, 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1260944720000001, 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:24:19.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881601+0000", 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.841e-06 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881621+0000", 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9976e-05 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881629+0000", 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2120000000000003e-06 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881636+0000", 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.472e-06 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881700+0000", 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4152000000000005e-05 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881707+0000", 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882746+0000", 2024-09-16T10:24:19.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001038574 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882755+0000", 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0480000000000004e-06 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007615+0000", 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.124860408 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007694+0000", 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8479000000000005e-05 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9:d2f9f966:::benchmark_data_smithi063_124717_object60060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.380404613, 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1652373410000001, 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.739 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881647+0000", 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7599999999999997e-07 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881671+0000", 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4284e-05 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881677+0000", 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3619999999999997e-06 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881681+0000", 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1980000000000003e-06 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881840+0000", 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015907000000000001 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting_for_map not empty", 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881846+0000", 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6250000000000004e-06 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883410+0000", 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001564085 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883417+0000", 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4749999999999996e-06 2024-09-16T10:24:19.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008717+0000", 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.125299746 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008819+0000", 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102545 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008934+0000", 2024-09-16T10:24:19.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000114923 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044911+0000", 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035976248000000002 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044965+0000", 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4756000000000002e-05 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045041+0000", 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5828000000000007e-05 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045130+0000", 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9221000000000005e-05 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045181+0000", 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0691e-05 2024-09-16T10:24:19.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046884+0000", 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001702708 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.747 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9:de195d7b:::benchmark_data_smithi063_124717_object60066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.380355544, 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1671491920000001, 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5400000000000001e-07 2024-09-16T10:24:19.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881718+0000", 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1948999999999999e-05 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881724+0000", 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7080000000000002e-06 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881728+0000", 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.946e-06 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883442+0000", 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00171441 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883449+0000", 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4309999999999999e-06 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010618+0000", 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1271692849999999 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010714+0000", 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6600000000000003e-05 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010840+0000", 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012543600000000001 2024-09-16T10:24:19.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044920+0000", 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034079854 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045083+0000", 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163688 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047061+0000", 2024-09-16T10:24:19.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019780269999999998 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047155+0000", 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3529999999999994e-05 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047228+0000", 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2822000000000002e-05 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048845+0000", 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016169520000000001 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9:d28fbaa4:::benchmark_data_smithi063_124717_object60144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.38029885, 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.171973098, 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.755 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881753+0000", 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8699999999999995e-07 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881772+0000", 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9159000000000001e-05 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881779+0000", 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.362e-06 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881784+0000", 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2580000000000003e-06 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883478+0000", 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016937180000000001 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883485+0000", 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3000000000000004e-06 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012549+0000", 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.129063785 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012705+0000", 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015551399999999999 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012901+0000", 2024-09-16T10:24:19.760 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019668299999999999 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046970+0000", 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034068909000000001 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047098+0000", 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012766899999999999 2024-09-16T10:24:19.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048912+0000", 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001813797 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051278+0000", 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023664480000000002 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051344+0000", 2024-09-16T10:24:19.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5415999999999998e-05 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053725+0000", 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002381593 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9:dfbc90fc:::benchmark_data_smithi063_124717_object60211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.380255824, 2024-09-16T10:24:19.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1744191719999999, 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:24:19.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881796+0000", 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3499999999999996e-07 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881881+0000", 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5458999999999996e-05 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881889+0000", 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9060000000000002e-06 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881896+0000", 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7830000000000001e-06 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883507+0000", 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016115249999999999 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883514+0000", 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-06 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.014943+0000", 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1314289719999999 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015122+0000", 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000178982 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015285+0000", 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016376499999999999 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046978+0000", 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031692310000000001 2024-09-16T10:24:19.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051202+0000", 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042246860000000001 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053812+0000", 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026092720000000002 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053866+0000", 2024-09-16T10:24:19.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4876000000000001e-05 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053932+0000", 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5314000000000005e-05 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056214+0000", 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002282634 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.770 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9:dcccd893:::benchmark_data_smithi063_124717_object60240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.38014658100001, 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2232440419999999, 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:24:19.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881905+0000", 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2900000000000003e-07 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881928+0000", 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2747000000000001e-05 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881935+0000", 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3150000000000003e-06 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881941+0000", 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1639999999999998e-06 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883538+0000", 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015963939999999999 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883543+0000", 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6540000000000004e-06 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056263+0000", 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.172719834 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056408+0000", 2024-09-16T10:24:19.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014513800000000001 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056584+0000", 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017557299999999999 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064153+0000", 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0075692820000000001 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068713+0000", 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045601620000000004 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068767+0000", 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3881000000000001e-05 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101726+0000", 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032958901999999998 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101782+0000", 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5609999999999998e-05 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105149+0000", 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033666569999999999 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9:dbba527e:::benchmark_data_smithi063_124717_object60254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.38010226599999, 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2264516890000001, 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.779 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:24:19.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1399999999999997e-07 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881971+0000", 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1461999999999999e-05 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881979+0000", 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.259e-06 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881985+0000", 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7320000000000003e-06 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883562+0000", 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001577171 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883568+0000", 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7910000000000001e-06 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059019+0000", 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1754510929999999 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059188+0000", 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016928700000000001 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059360+0000", 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017208100000000001 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068663+0000", 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0093023089999999999 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068831+0000", 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016804999999999999 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082474+0000", 2024-09-16T10:24:19.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013643597 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105299+0000", 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.022824816000000001 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105363+0000", 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4088e-05 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108400+0000", 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003037339 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9:ddb5c386:::benchmark_data_smithi063_124717_object60260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.380059069, 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2293183249999999, 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.786 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881993+0000", 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9999999999999997e-07 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882015+0000", 2024-09-16T10:24:19.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1946e-05 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882021+0000", 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9430000000000003e-06 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882028+0000", 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7599999999999997e-06 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883589+0000", 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001560875 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883595+0000", 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0719999999999996e-06 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061597+0000", 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1780015660000001 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061781+0000", 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000183909 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.062032+0000", 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025166899999999999 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068729+0000", 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0066966940000000004 2024-09-16T10:24:19.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068881+0000", 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000151965 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101683+0000", 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032802338 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108512+0000", 2024-09-16T10:24:19.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0068289229999999998 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108574+0000", 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1630000000000005e-05 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.111310+0000", 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027364350000000002 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60273 9.b 9:d6970cdd:::benchmark_data_smithi063_124717_object60272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.380015281, 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2328018119999999, 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60273 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.794 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:24:19.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3499999999999996e-07 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882058+0000", 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1236000000000001e-05 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882064+0000", 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5230000000000001e-06 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882070+0000", 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4670000000000003e-06 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883625+0000", 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015556820000000001 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883632+0000", 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6610000000000003e-06 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064190+0000", 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1805575690000001 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064363+0000", 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000173124 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064567+0000", 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020472000000000001 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068929+0000", 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043613879999999999 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076738+0000", 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0078089300000000004 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105256+0000", 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.028517793 2024-09-16T10:24:19.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112019+0000", 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067630110000000002 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112083+0000", 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4938999999999996e-05 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114838+0000", 2024-09-16T10:24:19.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027541340000000001 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60317 9.b 9:d4f738e0:::benchmark_data_smithi063_124717_object60316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.379974543, 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.235504929, 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.801 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60317 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:24:19.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5799999999999999e-07 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882098+0000", 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0758e-05 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882104+0000", 2024-09-16T10:24:19.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3239999999999999e-06 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882110+0000", 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5879999999999997e-06 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883724+0000", 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016140670000000001 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883731+0000", 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068967+0000", 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1852353390000001 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069080+0000", 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113442 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069174+0000", 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3623999999999997e-05 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079562+0000", 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010388029 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082423+0000", 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028609740000000001 2024-09-16T10:24:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114907+0000", 2024-09-16T10:24:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032484065999999999 2024-09-16T10:24:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114978+0000", 2024-09-16T10:24:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0919000000000003e-05 2024-09-16T10:24:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.115036+0000", 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7878999999999999e-05 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117581+0000", 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025458019999999998 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60326 9.b 9:d0ff592b:::benchmark_data_smithi063_124717_object60325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.379928085, 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2387679700000001, 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60326 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882124+0000", 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.99e-07 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882145+0000", 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1650000000000001e-05 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882151+0000", 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2110000000000003e-06 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882158+0000", 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6669999999999999e-06 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883760+0000", 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001601781 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883766+0000", 2024-09-16T10:24:19.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4019999999999999e-06 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071354+0000", 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1875881050000001 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071501+0000", 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014622300000000001 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071673+0000", 2024-09-16T10:24:19.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000172518 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082372+0000", 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010698874000000001 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082515+0000", 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000143409 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117647+0000", 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035131837999999999 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117792+0000", 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014443299999999999 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117866+0000", 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4274000000000001e-05 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120891+0000", 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030249859999999999 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.817 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60333 9.b 9:d83577cf:::benchmark_data_smithi063_124717_object60332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.379885802, 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2414839360000001, 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.818 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60333 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:24:19.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882166+0000", 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0100000000000005e-07 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882188+0000", 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2252999999999999e-05 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882221+0000", 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2459999999999998e-05 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882227+0000", 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.015e-06 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883790+0000", 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015635759999999999 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883797+0000", 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.567e-06 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076778+0000", 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1929816879999999 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076959+0000", 2024-09-16T10:24:19.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000180607 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.077150+0000", 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019081600000000001 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082436+0000", 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0052863240000000002 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082555+0000", 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119305 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120947+0000", 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.038391740000000001 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120989+0000", 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2107999999999997e-05 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.121049+0000", 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9888e-05 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123649+0000", 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025999880000000001 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.826 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60362 9.b 9:d2c32f51:::benchmark_data_smithi063_124717_object60361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.826 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:24:19.826 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.379814681, 2024-09-16T10:24:19.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.244097462, 2024-09-16T10:24:19.826 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60362 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882237+0000", 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0399999999999995e-07 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882261+0000", 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4269000000000001e-05 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882268+0000", 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8619999999999997e-06 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882274+0000", 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1829999999999999e-06 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883834+0000", 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015597930000000001 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883840+0000", 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.251e-06 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079597+0000", 2024-09-16T10:24:19.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1957566589999999 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079755+0000", 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015786399999999999 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.080192+0000", 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00043717099999999998 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.831 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105209+0000", 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.025016842000000001 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114928+0000", 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0097189919999999992 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123709+0000", 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0087810240000000001 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123762+0000", 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2567e-05 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123828+0000", 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6799999999999997e-05 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126334+0000", 2024-09-16T10:24:19.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025054809999999999 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60366 9.b 9:d4ec99e6:::benchmark_data_smithi063_124717_object60365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.379769182, 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.246752562, 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.834 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60366 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:24:19.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7899999999999998e-07 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883494+0000", 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012115959999999999 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883505+0000", 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.111e-05 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883513+0000", 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4320000000000002e-06 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883866+0000", 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00035366599999999998 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883873+0000", 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2910000000000004e-06 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099038+0000", 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2151655379999999 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099212+0000", 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017424000000000001 2024-09-16T10:24:19.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099404+0000", 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000191366 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108466+0000", 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0090625159999999996 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117602+0000", 2024-09-16T10:24:19.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0091353030000000009 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126413+0000", 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088110150000000002 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126572+0000", 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015944599999999999 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126628+0000", 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5819999999999997e-05 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129034+0000", 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024066439999999999 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60423 9.b 9:db831613:::benchmark_data_smithi063_124717_object60422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.841 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.378504729, 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2882874870000001, 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60423 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.842 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883548+0000", 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.206e-06 2024-09-16T10:24:19.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883603+0000", 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5036e-05 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883614+0000", 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1352000000000001e-05 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883624+0000", 2024-09-16T10:24:19.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0519999999999999e-05 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883899+0000", 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027465699999999998 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883905+0000", 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3119999999999999e-06 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129084+0000", 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.245178672 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129268+0000", 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018391999999999999 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129423+0000", 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015478000000000001 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136532+0000", 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071096270000000003 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136582+0000", 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9962e-05 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.152752+0000", 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016169444000000002 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168726+0000", 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015974559999999999 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168790+0000", 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3474999999999998e-05 2024-09-16T10:24:19.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.171834+0000", 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030439640000000001 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60452 9.b 9:d848878f:::benchmark_data_smithi063_124717_object60451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:24:19.849 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 102.319536257, 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.233084683, 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60452 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942516+0000", 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5969999999999999e-06 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942536+0000", 2024-09-16T10:24:19.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9970999999999999e-05 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942548+0000", 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2057e-05 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942556+0000", 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0350000000000001e-06 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942605+0000", 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8770999999999998e-05 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942611+0000", 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1040000000000003e-06 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131546+0000", 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.188934811 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131704+0000", 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015802399999999999 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131894+0000", 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018940499999999999 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136598+0000", 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047045409999999996 2024-09-16T10:24:19.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136646+0000", 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8204999999999997e-05 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168552+0000", 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031905880999999997 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172759+0000", 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042071809999999999 2024-09-16T10:24:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172950+0000", 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019102799999999999 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.175599+0000", 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026490720000000001 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:19.856 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:19.857 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:24:19.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:19.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:19.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:19.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:19.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:20.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:20.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:20.247 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:20.288 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:20.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:20.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:20.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:20.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:20.590 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:20 smithi063 ceph-mon[29993]: pgmap v575: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.41k op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 2.4 MiB/s, 37 objects/s recovering 2024-09-16T10:24:20.590 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:20 smithi063 ceph-mon[39470]: pgmap v575: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.41k op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 2.4 MiB/s, 37 objects/s recovering 2024-09-16T10:24:20.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:24:20.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:20.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:20.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:20 smithi165 ceph-mon[32696]: pgmap v575: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.41k op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 2.4 MiB/s, 37 objects/s recovering 2024-09-16T10:24:20.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:20.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:20.982 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:20.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:21.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:21.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:21.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:21.214 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:21.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:21.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:21.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:21.373 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:21.396 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:21.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:21.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:21.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[29993]: 6.b scrub starts 2024-09-16T10:24:21.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[29993]: 6.b scrub ok 2024-09-16T10:24:21.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1238/199131 objects degraded (0.622%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:21.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[29993]: osdmap e340: 8 total, 8 up, 8 in 2024-09-16T10:24:21.548 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[39470]: 6.b scrub starts 2024-09-16T10:24:21.548 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[39470]: 6.b scrub ok 2024-09-16T10:24:21.548 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1238/199131 objects degraded (0.622%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:21.548 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:21 smithi063 ceph-mon[39470]: osdmap e340: 8 total, 8 up, 8 in 2024-09-16T10:24:21.549 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:21.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:21.569 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:24:21.651 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:21.716 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:21.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:21 smithi165 ceph-mon[32696]: 6.b scrub starts 2024-09-16T10:24:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:21 smithi165 ceph-mon[32696]: 6.b scrub ok 2024-09-16T10:24:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:21 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1238/199131 objects degraded (0.622%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:21 smithi165 ceph-mon[32696]: osdmap e340: 8 total, 8 up, 8 in 2024-09-16T10:24:21.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:21.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:21.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:21.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:21.979 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:21.997 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:21.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:22.089 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:24:22.089 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:24:22.089 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:24:22.089 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:24:22.090 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:24:22.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:22.139 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:22.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:22.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:22.286 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:22.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:22.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:22.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:22.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[29993]: pgmap v577: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 856 op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 1.6 MiB/s, 25 objects/s recovering 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[39470]: pgmap v577: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 856 op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 1.6 MiB/s, 25 objects/s recovering 2024-09-16T10:24:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:24:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:22 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:24:22.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:22.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:22.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:22.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:22.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:22.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:22.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:22 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:24:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:22 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:24:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:22 smithi165 ceph-mon[32696]: pgmap v577: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 856 op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 1.6 MiB/s, 25 objects/s recovering 2024-09-16T10:24:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:22 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:24:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:22 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:24:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:22.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:22.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:22.907 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:22.934 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:22.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:23.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:23.070 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:23.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:23.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:23.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:23.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:23.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:23.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:24:23.417 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:23.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:23.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:23.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:23.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:23.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:23 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:24:23.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:23 smithi063 ceph-mon[29993]: 6.2 scrub ok 2024-09-16T10:24:23.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:23 smithi063 ceph-mon[29993]: osdmap e341: 8 total, 8 up, 8 in 2024-09-16T10:24:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:23 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:24:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:23 smithi063 ceph-mon[39470]: 6.2 scrub ok 2024-09-16T10:24:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:23 smithi063 ceph-mon[39470]: osdmap e341: 8 total, 8 up, 8 in 2024-09-16T10:24:23.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:23.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:23 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:24:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:23 smithi165 ceph-mon[32696]: 6.2 scrub ok 2024-09-16T10:24:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:23 smithi165 ceph-mon[32696]: osdmap e341: 8 total, 8 up, 8 in 2024-09-16T10:24:23.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:23.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:23.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:23.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:23.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:24.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:24.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:24.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:24.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:24.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:24.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:24.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:24.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:24.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:24.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:24.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:24.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:24.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:24.624 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:24.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[29993]: pgmap v579: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 818 op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[29993]: 3.13 scrub starts 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[29993]: 3.13 scrub ok 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:24:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[39470]: pgmap v579: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 818 op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:24:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[39470]: 3.13 scrub starts 2024-09-16T10:24:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[39470]: 3.13 scrub ok 2024-09-16T10:24:24.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:24.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:24.769 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:24.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:24 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:24:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:24 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:24:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:24 smithi165 ceph-mon[32696]: pgmap v579: 168 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 165 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 818 op/s; 1238/199131 objects degraded (0.622%); 2411/199131 objects misplaced (1.211%); 1.6 MiB/s, 24 objects/s recovering 2024-09-16T10:24:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:24 smithi165 ceph-mon[32696]: 3.13 scrub starts 2024-09-16T10:24:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:24 smithi165 ceph-mon[32696]: 3.13 scrub ok 2024-09-16T10:24:24.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:24.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:24.913 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:24.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:25.015 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:25.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:25.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:25.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:25.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:25.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:25.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:25.346 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:25.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:25.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:25.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:25.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:25.509 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:25.571 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:24:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173423 9.a e341/276 9:55d1c994:::benchmark_data_smithi063_124717_object57192:head v 341'10601, mlcod=341'10599)", 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.559736+0000", 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0066742429999999998, 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067004860000000003, 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559736+0000", 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559736+0000", 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559738+0000", 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1629999999999999e-06 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559744+0000", 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.878e-06 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559752+0000", 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6059999999999997e-06 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559758+0000", 2024-09-16T10:24:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.212e-06 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559939+0000", 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000180979 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559959+0000", 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0139000000000001e-05 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173427 9.f e341/276 9:fa820d31:::benchmark_data_smithi063_124717_object57196:head v 341'10889, mlcod=341'10887)", 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.559777+0000", 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0066339249999999997, 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067153830000000001, 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559777+0000", 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559777+0000", 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559777+0000", 2024-09-16T10:24:25.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9200000000000001e-07 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559782+0000", 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3889999999999998e-06 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559787+0000", 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3079999999999998e-06 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559791+0000", 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3800000000000004e-06 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.561540+0000", 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017483589999999999 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.561558+0000", 2024-09-16T10:24:25.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8618999999999999e-05 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173425 9.b e341/298 9:d1d14686:::benchmark_data_smithi063_124717_object57194:head v 341'10765, mlcod=285'472)", 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.559821+0000", 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.006590116, 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067226259999999998, 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:24:25.580 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559821+0000", 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559821+0000", 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559821+0000", 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7000000000000004e-07 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559827+0000", 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3720000000000004e-06 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559833+0000", 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6069999999999996e-06 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.559838+0000", 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8949999999999997e-06 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.560371+0000", 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00053253299999999996 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.560392+0000", 2024-09-16T10:24:25.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1007000000000001e-05 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173428 9.b e341/298 9:d201b82a:::benchmark_data_smithi063_124717_object57197:head v 341'10766, mlcod=285'472)", 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.562540+0000", 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0038704989999999999, 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040556790000000004, 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:24:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562540+0000", 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562540+0000", 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562541+0000", 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0219999999999999e-06 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562547+0000", 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0939999999999996e-06 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562560+0000", 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2573e-05 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562566+0000", 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.032e-06 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565194+0000", 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002627866 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565218+0000", 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4122999999999999e-05 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173430 9.e e341/276 9:7dcf7088:::benchmark_data_smithi063_124717_object57199:head v 341'11235, mlcod=341'11232)", 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.562593+0000", 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0038178980000000001, 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040546799999999997, 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562593+0000", 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562593+0000", 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562593+0000", 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.44e-07 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562602+0000", 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.106e-06 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562608+0000", 2024-09-16T10:24:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1909999999999997e-06 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562612+0000", 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2980000000000002e-06 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562951+0000", 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00033918700000000002 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562972+0000", 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0625e-05 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:173433 9.0 9:02e893de:::benchmark_data_smithi063_124717_object57202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.563689+0000", 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.002721963, 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030120139999999999, 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 173433 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563689+0000", 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563689+0000", 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563691+0000", 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1739999999999999e-06 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563696+0000", 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5130000000000004e-06 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563708+0000", 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1691999999999999e-05 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563715+0000", 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1489999999999997e-06 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563771+0000", 2024-09-16T10:24:25.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5374999999999998e-05 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.563884+0000", 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113354 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.564128+0000", 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00024398700000000001 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173432 9.6 e341/281)", 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.564048+0000", 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.002362907, 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027131669999999998, 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.564048+0000", 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.564048+0000", 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.564049+0000", 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.221e-06 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.564055+0000", 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6760000000000004e-06 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.564063+0000", 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2640000000000008e-06 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.564069+0000", 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9839999999999998e-06 2024-09-16T10:24:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173429 9.2 e341/284 9:4da81f03:::benchmark_data_smithi063_124717_object57198:head v 341'10537, mlcod=341'10535)", 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.565154+0000", 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0012565899999999999, 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017478280000000001, 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565154+0000", 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565154+0000", 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565155+0000", 2024-09-16T10:24:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.305e-06 2024-09-16T10:24:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565162+0000", 2024-09-16T10:24:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8669999999999996e-06 2024-09-16T10:24:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565171+0000", 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.619e-06 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565177+0000", 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2929999999999997e-06 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565295+0000", 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011745599999999999 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565311+0000", 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6379e-05 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173437 9.3 e341/276)", 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.566202+0000", 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.000208858, 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00075738699999999999, 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.605 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566202+0000", 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566202+0000", 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566203+0000", 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.203e-06 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566211+0000", 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4360000000000002e-06 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566221+0000", 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.346e-06 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566227+0000", 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1539999999999999e-06 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.608 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.45596.0:173433 9.0 e341/281)", 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.566266+0000", 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.00014466500000000001, 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000728412, 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566266+0000", 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566266+0000", 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566267+0000", 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4099999999999998e-07 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566270+0000", 2024-09-16T10:24:25.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8139999999999998e-06 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566279+0000", 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5699999999999993e-06 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566284+0000", 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.101e-06 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173435 9.b e341/298)", 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.566819+0000", 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9995918, 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00021189099999999999, 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566819+0000", 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566819+0000", 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566820+0000", 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2890000000000001e-06 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566826+0000", 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1550000000000004e-06 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566836+0000", 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3789999999999995e-06 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.566842+0000", 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0739999999999998e-06 2024-09-16T10:24:25.614 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.45596.0:173421 9.3 e341/276 9:c7a01cd0:::benchmark_data_smithi063_124717_object57190:head v 341'10978, mlcod=341'10974)", 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.556719+0000", 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0096917470000000006, 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010351503999999999, 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.556719+0000", 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.556719+0000", 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.556719+0000", 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0100000000000005e-07 2024-09-16T10:24:25.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.556723+0000", 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.794e-06 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.556728+0000", 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8930000000000004e-06 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.556732+0000", 2024-09-16T10:24:25.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0690000000000003e-06 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557245+0000", 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00051266500000000004 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557267+0000", 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1795000000000001e-05 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:173424 9.4 9:2ad72949:::benchmark_data_smithi063_124717_object57193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:24:25.557287+0000", 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0091240709999999992, 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0098338639999999995, 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:24:25.619 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 173424 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557287+0000", 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557287+0000", 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557288+0000", 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.719e-06 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557303+0000", 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4887e-05 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557313+0000", 2024-09-16T10:24:25.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8819999999999996e-06 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557320+0000", 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557380+0000", 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9240000000000002e-05 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:25.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557491+0000", 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111816 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,4", 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.557600+0000", 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010854299999999999 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.562471+0000", 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0048705620000000002 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:25.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:24:25.565124+0000", 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026530910000000002 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 13 2024-09-16T10:24:25.624 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:24:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:25.669 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:25.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:25.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:25 smithi063 ceph-mon[29993]: 3.f scrub starts 2024-09-16T10:24:25.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:25 smithi063 ceph-mon[29993]: 3.f scrub ok 2024-09-16T10:24:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:25 smithi063 ceph-mon[39470]: 3.f scrub starts 2024-09-16T10:24:25.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:25 smithi063 ceph-mon[39470]: 3.f scrub ok 2024-09-16T10:24:25.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:25 smithi165 ceph-mon[32696]: 3.f scrub starts 2024-09-16T10:24:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:25 smithi165 ceph-mon[32696]: 3.f scrub ok 2024-09-16T10:24:25.813 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:25.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:25.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:25.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:25.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:26.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:26.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:26.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:26.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:26.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:26.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:26.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:26.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:26.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:24:26.442 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:24:26.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-16T10:24:26.453 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:26.536 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:26.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:26.638 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:26.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[39470]: pgmap v580: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 1434/182886 objects degraded (0.784%); 1935/182886 objects misplaced (1.058%); 2.3 MiB/s, 36 objects/s recovering 2024-09-16T10:24:26.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[39470]: 3.b scrub starts 2024-09-16T10:24:26.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[39470]: 3.b scrub ok 2024-09-16T10:24:26.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1434/182886 objects degraded (0.784%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:26.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[39470]: osdmap e342: 8 total, 8 up, 8 in 2024-09-16T10:24:26.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[29993]: pgmap v580: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 1434/182886 objects degraded (0.784%); 1935/182886 objects misplaced (1.058%); 2.3 MiB/s, 36 objects/s recovering 2024-09-16T10:24:26.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[29993]: 3.b scrub starts 2024-09-16T10:24:26.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[29993]: 3.b scrub ok 2024-09-16T10:24:26.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1434/182886 objects degraded (0.784%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:26.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:26 smithi063 ceph-mon[29993]: osdmap e342: 8 total, 8 up, 8 in 2024-09-16T10:24:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:26.678 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:26.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:26.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:26 smithi165 ceph-mon[32696]: pgmap v580: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 1434/182886 objects degraded (0.784%); 1935/182886 objects misplaced (1.058%); 2.3 MiB/s, 36 objects/s recovering 2024-09-16T10:24:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:26 smithi165 ceph-mon[32696]: 3.b scrub starts 2024-09-16T10:24:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:26 smithi165 ceph-mon[32696]: 3.b scrub ok 2024-09-16T10:24:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:26 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1434/182886 objects degraded (0.784%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:26 smithi165 ceph-mon[32696]: osdmap e342: 8 total, 8 up, 8 in 2024-09-16T10:24:26.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:26.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:26.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:26.976 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:26.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:27.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:27.118 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:27.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:27.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:24:27.221 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:27.275 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:27.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:27.333 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:24:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:24:27.342 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:27.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:27.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:27.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:27.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:27.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:27.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:27.633 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:27.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:27.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[29993]: 5.1b scrub starts 2024-09-16T10:24:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[29993]: 5.1b scrub ok 2024-09-16T10:24:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:24:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:24:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[39470]: 5.1b scrub starts 2024-09-16T10:24:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[39470]: 5.1b scrub ok 2024-09-16T10:24:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:24:27.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:27 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:24:27.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:27 smithi165 ceph-mon[32696]: 5.1b scrub starts 2024-09-16T10:24:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:27 smithi165 ceph-mon[32696]: 5.1b scrub ok 2024-09-16T10:24:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:27 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:24:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:27 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:24:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:27.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:27.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:27.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:27.944 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:27.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:28.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:28.086 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:28.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:28.129 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:28.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:28.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:28.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:28.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:28.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:28.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:28.426 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:24:28.426 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:24:28.432 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:24:28.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:28.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:28.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:28.480 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:24:28.480 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:24:28.497 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: 5.15 scrub starts 2024-09-16T10:24:28.497 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: 5.15 scrub ok 2024-09-16T10:24:28.497 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: pgmap v582: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 854 op/s; 1434/182886 objects degraded (0.784%); 1935/182886 objects misplaced (1.058%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:24:28.498 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:24:28.498 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:24:28.498 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3875157696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:28.498 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: from='client.45836 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:28.500 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:24:28.500 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:24:28.500 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:24:28.500 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2854542326' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:28.501 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: 5.15 scrub starts 2024-09-16T10:24:28.501 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: 5.15 scrub ok 2024-09-16T10:24:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: pgmap v582: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 854 op/s; 1434/182886 objects degraded (0.784%); 1935/182886 objects misplaced (1.058%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:24:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:24:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:24:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3875157696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: from='client.45836 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:28.502 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:24:28.503 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:24:28.503 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:24:28.503 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2854542326' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:28.521 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:24:28.522 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:24:28.556 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:24:28.556 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:24:28.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:28.594 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:24:28.594 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:24:28.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:28.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:28.634 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:24:28.635 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:24:28.671 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:24:28.671 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:24:28.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: 5.15 scrub starts 2024-09-16T10:24:28.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: 5.15 scrub ok 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: pgmap v582: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 807 B/s rd, 0 B/s wr, 854 op/s; 1434/182886 objects degraded (0.784%); 1935/182886 objects misplaced (1.058%); 1.4 MiB/s, 22 objects/s recovering 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3875157696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: from='client.45836 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:24:28.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2854542326' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:28.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:28.741 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:24:28.742 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:24:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:28.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:28.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:28.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:28.890 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:28.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:28.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:29.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:24:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:24:29.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:29.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:29.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:29.262 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:29.297 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:29.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:29.444 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:29.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:29.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:29.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:29 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:24:29.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:29 smithi165 ceph-mon[32696]: 1.5 scrub ok 2024-09-16T10:24:29.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:29 smithi165 ceph-mon[32696]: 1.4 scrub ok 2024-09-16T10:24:29.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:29 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:24:29.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:29 smithi165 ceph-mon[32696]: from='client.45836 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:29.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:29 smithi165 ceph-mon[32696]: osdmap e343: 8 total, 8 up, 8 in 2024-09-16T10:24:29.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:29.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[29993]: 1.5 scrub ok 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[29993]: 1.4 scrub ok 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[29993]: from='client.45836 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[29993]: osdmap e343: 8 total, 8 up, 8 in 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:29.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:24:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[39470]: 1.5 scrub ok 2024-09-16T10:24:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[39470]: 1.4 scrub ok 2024-09-16T10:24:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:24:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[39470]: from='client.45836 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[39470]: osdmap e343: 8 total, 8 up, 8 in 2024-09-16T10:24:29.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:29.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:29.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.857 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:29.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:29.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:29.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:29.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:30.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:30.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:30.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:30.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:30.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:30.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:30.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:30.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:30.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:30.475 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:30.518 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:30.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:30.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:30.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:30.659 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:30.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:30.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:30.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:30.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:30 smithi165 ceph-mon[32696]: pgmap v584: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 2.4 MiB/s, 43 objects/s recovering 2024-09-16T10:24:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:30 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:24:30.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:30 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:24:30.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:30.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:30.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:30.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:30.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:30 smithi063 ceph-mon[29993]: pgmap v584: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 2.4 MiB/s, 43 objects/s recovering 2024-09-16T10:24:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:30 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:24:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:30 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:24:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:30 smithi063 ceph-mon[39470]: pgmap v584: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 2.4 MiB/s, 43 objects/s recovering 2024-09-16T10:24:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:30 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:24:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:30 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:24:31.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:31.063 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:31.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:31.165 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:31.212 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:31.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:31.313 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:31.354 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:31.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:31.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:31.501 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:31.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:31.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:31.628 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:24:31.628 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:24:31.629 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:24:31.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:24:31.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: 1.1 scrub ok 2024-09-16T10:24:31.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:24:31.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:24:31.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:24:31.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:24:31.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1521/167229 objects degraded (0.910%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:31.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[29993]: osdmap e344: 8 total, 8 up, 8 in 2024-09-16T10:24:31.630 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:24:31.630 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:24:31.631 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: 1.1 scrub ok 2024-09-16T10:24:31.631 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:24:31.631 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:24:31.631 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:24:31.631 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:24:31.631 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1521/167229 objects degraded (0.910%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:31.631 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:31 smithi063 ceph-mon[39470]: osdmap e344: 8 total, 8 up, 8 in 2024-09-16T10:24:31.661 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:31.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:31.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: 1.1 scrub ok 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1521/167229 objects degraded (0.910%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:31 smithi165 ceph-mon[32696]: osdmap e344: 8 total, 8 up, 8 in 2024-09-16T10:24:31.805 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:31.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:31.907 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:31.946 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:31.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:32.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:32.087 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:32.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:32.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:32.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:32.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:32.268 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:32.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:32.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:32.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:32.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:32.495 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:32.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:32.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:32.642 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:32.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:32.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[29993]: pgmap v586: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 870 op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 1.3 MiB/s, 26 objects/s recovering 2024-09-16T10:24:32.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[29993]: 1.c scrub starts 2024-09-16T10:24:32.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[29993]: 1.c scrub ok 2024-09-16T10:24:32.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:24:32.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[29993]: 1.7 scrub ok 2024-09-16T10:24:32.689 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[39470]: pgmap v586: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 870 op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 1.3 MiB/s, 26 objects/s recovering 2024-09-16T10:24:32.689 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[39470]: 1.c scrub starts 2024-09-16T10:24:32.690 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[39470]: 1.c scrub ok 2024-09-16T10:24:32.690 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:24:32.690 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:32 smithi063 ceph-mon[39470]: 1.7 scrub ok 2024-09-16T10:24:32.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:24:32.746 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:24:32.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-16T10:24:32.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:32 smithi165 ceph-mon[32696]: pgmap v586: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 870 op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 1.3 MiB/s, 26 objects/s recovering 2024-09-16T10:24:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:32 smithi165 ceph-mon[32696]: 1.c scrub starts 2024-09-16T10:24:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:32 smithi165 ceph-mon[32696]: 1.c scrub ok 2024-09-16T10:24:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:32 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:24:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:32 smithi165 ceph-mon[32696]: 1.7 scrub ok 2024-09-16T10:24:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:32.828 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:32.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:32.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:32.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:32.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:33.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:33.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:33.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:33.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:33.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:33.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:24:33.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:33.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:33.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:33.573 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:33.595 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:33.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.e scrub ok 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.d scrub ok 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.11 scrub ok 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:24:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[29993]: 1.13 scrub ok 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.e scrub ok 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.d scrub ok 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.11 scrub ok 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:24:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:33 smithi063 ceph-mon[39470]: 1.13 scrub ok 2024-09-16T10:24:33.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:24:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.e scrub ok 2024-09-16T10:24:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:24:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.d scrub ok 2024-09-16T10:24:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:24:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.11 scrub ok 2024-09-16T10:24:33.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:24:33.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:33 smithi165 ceph-mon[32696]: 1.13 scrub ok 2024-09-16T10:24:33.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:33.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:33.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:33.995 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:33.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:34.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:24:34.083 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:24:34.083 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:24:34.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:34.152 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:34.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:34.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:34.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:34.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:34.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:34.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:34.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:34.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:34.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:34.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:34.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:34.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:34.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:34.773 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:34.799 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:34.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:34.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:34 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:24:34.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:34 smithi165 ceph-mon[32696]: 1.14 scrub ok 2024-09-16T10:24:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:34 smithi165 ceph-mon[32696]: pgmap v587: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 1.1 MiB/s, 22 objects/s recovering 2024-09-16T10:24:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:34 smithi165 ceph-mon[32696]: 3.d scrub starts 2024-09-16T10:24:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:34 smithi165 ceph-mon[32696]: 3.d scrub ok 2024-09-16T10:24:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:34 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:24:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:34 smithi165 ceph-mon[32696]: 1.f scrub ok 2024-09-16T10:24:34.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:34.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:34.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:24:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[29993]: 1.14 scrub ok 2024-09-16T10:24:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[29993]: pgmap v587: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 1.1 MiB/s, 22 objects/s recovering 2024-09-16T10:24:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[29993]: 3.d scrub starts 2024-09-16T10:24:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[29993]: 3.d scrub ok 2024-09-16T10:24:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:24:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[29993]: 1.f scrub ok 2024-09-16T10:24:34.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:24:34.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[39470]: 1.14 scrub ok 2024-09-16T10:24:34.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[39470]: pgmap v587: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 1521/167229 objects degraded (0.910%); 1524/167229 objects misplaced (0.911%); 1.1 MiB/s, 22 objects/s recovering 2024-09-16T10:24:34.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[39470]: 3.d scrub starts 2024-09-16T10:24:34.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[39470]: 3.d scrub ok 2024-09-16T10:24:34.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:24:34.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:34 smithi063 ceph-mon[39470]: 1.f scrub ok 2024-09-16T10:24:35.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:35.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:35.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:35.206 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:35.308 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:35.331 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:35.432 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.457 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:35.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:35.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:35.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:35.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:35.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:35 smithi165 ceph-mon[32696]: 6.7 scrub starts 2024-09-16T10:24:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:35 smithi165 ceph-mon[32696]: 6.7 scrub ok 2024-09-16T10:24:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:35 smithi165 ceph-mon[32696]: 4.e scrub starts 2024-09-16T10:24:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:35 smithi165 ceph-mon[32696]: 4.e scrub ok 2024-09-16T10:24:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:35 smithi165 ceph-mon[32696]: 9.e scrub starts 2024-09-16T10:24:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:35 smithi165 ceph-mon[32696]: 9.e scrub ok 2024-09-16T10:24:35.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:35.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:35.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:35.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:35.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[29993]: 6.7 scrub starts 2024-09-16T10:24:35.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[29993]: 6.7 scrub ok 2024-09-16T10:24:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[29993]: 4.e scrub starts 2024-09-16T10:24:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[29993]: 4.e scrub ok 2024-09-16T10:24:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[29993]: 9.e scrub starts 2024-09-16T10:24:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[29993]: 9.e scrub ok 2024-09-16T10:24:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[39470]: 6.7 scrub starts 2024-09-16T10:24:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[39470]: 6.7 scrub ok 2024-09-16T10:24:35.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[39470]: 4.e scrub starts 2024-09-16T10:24:35.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[39470]: 4.e scrub ok 2024-09-16T10:24:35.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[39470]: 9.e scrub starts 2024-09-16T10:24:35.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:35 smithi063 ceph-mon[39470]: 9.e scrub ok 2024-09-16T10:24:35.979 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:35.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:36.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:36.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:36.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:36.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:36.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:36.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:36.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:36.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:36.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:36.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:36.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:36.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:36.570 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:24:36.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:36.628 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:36.640 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:36.728 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:36.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:36.799 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:24:36.799 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:24:36.799 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[39470]: 5.1c scrub starts 2024-09-16T10:24:36.799 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[39470]: 5.1c scrub ok 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[39470]: pgmap v588: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.33k op/s; 1356/150882 objects degraded (0.899%); 1359/150882 objects misplaced (0.901%); 931 KiB/s, 40 objects/s recovering 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1356/150882 objects degraded (0.899%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[29993]: 5.1c scrub starts 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[29993]: 5.1c scrub ok 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[29993]: pgmap v588: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.33k op/s; 1356/150882 objects degraded (0.899%); 1359/150882 objects misplaced (0.901%); 931 KiB/s, 40 objects/s recovering 2024-09-16T10:24:36.800 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:36 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1356/150882 objects degraded (0.899%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:36 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:24:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:36 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:24:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:36 smithi165 ceph-mon[32696]: 5.1c scrub starts 2024-09-16T10:24:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:36 smithi165 ceph-mon[32696]: 5.1c scrub ok 2024-09-16T10:24:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:36 smithi165 ceph-mon[32696]: pgmap v588: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.33k op/s; 1356/150882 objects degraded (0.899%); 1359/150882 objects misplaced (0.901%); 931 KiB/s, 40 objects/s recovering 2024-09-16T10:24:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:36 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1356/150882 objects degraded (0.899%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:36.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:36.857 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:36.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:36.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:36.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:36.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:37.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:37.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:37.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:37.208 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:37.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:37.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:37.352 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:37.377 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:37.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:37.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:24:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:24:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:24:37.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:37.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:37.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:37.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:37.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:37.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:37.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:37.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:37.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:37.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:37.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:38.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:38.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:38.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:38.152 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:38.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:38.284 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:38.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:38.321 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:24:38.322 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:24:38.322 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:24:38.323 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:24:38.323 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:24:38.323 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:24:38.323 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:24:38.324 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:24:38.324 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:24:38.324 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:24:38.324 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:24:38.324 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:38.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:38.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:38.497 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:38.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:24:38.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 4.8 scrub starts 2024-09-16T10:24:38.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 4.8 scrub ok 2024-09-16T10:24:38.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 4.1b scrub starts 2024-09-16T10:24:38.584 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 4.1b scrub ok 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: pgmap v589: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 1356/150882 objects degraded (0.899%); 1359/150882 objects misplaced (0.901%); 841 KiB/s, 36 objects/s recovering 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 4.19 scrub starts 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: 4.19 scrub ok 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3564082628' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[29993]: from='client.45854 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 4.8 scrub starts 2024-09-16T10:24:38.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 4.8 scrub ok 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 4.1b scrub starts 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 4.1b scrub ok 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: pgmap v589: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 1356/150882 objects degraded (0.899%); 1359/150882 objects misplaced (0.901%); 841 KiB/s, 36 objects/s recovering 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 4.19 scrub starts 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: 4.19 scrub ok 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3564082628' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:24:38.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:38 smithi063 ceph-mon[39470]: from='client.45854 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:24:38.599 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:38.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:38.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:38.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:38.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:38.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:38.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 4.8 scrub starts 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 4.8 scrub ok 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 4.1b scrub starts 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 4.1b scrub ok 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: pgmap v589: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 1356/150882 objects degraded (0.899%); 1359/150882 objects misplaced (0.901%); 841 KiB/s, 36 objects/s recovering 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 4.19 scrub starts 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: 4.19 scrub ok 2024-09-16T10:24:38.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3564082628' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:24:38.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-mon[32696]: from='client.45854 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:24:38.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:24:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:24:38.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:38.908 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:38.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:39.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:39.050 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:39.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:39.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:39.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:39.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:39.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:39.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:39.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:39.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:39.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:39.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:39.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:39 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:39 smithi165 ceph-mon[32696]: from='client.45854 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:24:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:39 smithi165 ceph-mon[32696]: osdmap e345: 8 total, 8 up, 8 in 2024-09-16T10:24:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:39.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:39.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:39.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[29993]: from='client.45854 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[29993]: osdmap e345: 8 total, 8 up, 8 in 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[39470]: from='client.45854 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[39470]: osdmap e345: 8 total, 8 up, 8 in 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:39.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:39.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:40.063 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:40.102 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:40.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:40.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:40.231 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:40.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:40.332 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:40.373 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:40.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:40.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:40.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:40.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:40.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:40.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:40.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:40.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:40 smithi063 ceph-mon[29993]: pgmap v591: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.25k op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 40 objects/s recovering 2024-09-16T10:24:40.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:40 smithi063 ceph-mon[39470]: pgmap v591: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.25k op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 40 objects/s recovering 2024-09-16T10:24:40.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:40.754 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:24:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:40.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:40.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:40.793 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:24:40.793 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:24:40.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:40 smithi165 ceph-mon[32696]: pgmap v591: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.25k op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 40 objects/s recovering 2024-09-16T10:24:40.827 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:24:40.827 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:24:40.860 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:24:40.860 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:24:40.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:40.895 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:24:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:24:40.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:40.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:40.931 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:24:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:24:40.964 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:24:41.000 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:24:41.000 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:24:41.004 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:41.053 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:41.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:41.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:41.199 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:41.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:24:41.292 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:24:41.293 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-16T10:24:41.301 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:41.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:41.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:41.530 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:41.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:41.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/593768945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:41.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:41 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1181/135882 objects degraded (0.869%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:41.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:41 smithi063 ceph-mon[29993]: osdmap e346: 8 total, 8 up, 8 in 2024-09-16T10:24:41.630 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/593768945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:41.630 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:41 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1181/135882 objects degraded (0.869%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:41.630 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:41 smithi063 ceph-mon[39470]: osdmap e346: 8 total, 8 up, 8 in 2024-09-16T10:24:41.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:41.686 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:41.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:41.697 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:24:41.697 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:24:41.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:24:41.697 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50949 8.a 8:59c98f0f:::benchmark_data_smithi063_101290_object50948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.42865147499998, 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0382288049999999, 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50949 2024-09-16T10:24:41.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264644+0000", 2024-09-16T10:24:41.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.978e-06 2024-09-16T10:24:41.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264667+0000", 2024-09-16T10:24:41.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3649999999999999e-05 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264681+0000", 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3423e-05 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264688+0000", 2024-09-16T10:24:41.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1010000000000003e-06 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265626+0000", 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00093838999999999999 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265747+0000", 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120319 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.267052+0000", 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013057990000000001 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.281048+0000", 2024-09-16T10:24:41.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013995452 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283470+0000", 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024224540000000001 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283482+0000", 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1581e-05 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302818+0000", 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0193361350000001 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302871+0000", 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2522999999999998e-05 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50954 8.a 8:57f268c8:::benchmark_data_smithi063_101290_object50953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.42529104800002, 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0425692, 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.706 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50954 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268004+0000", 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1050000000000002e-06 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268031+0000", 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6468000000000001e-05 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268041+0000", 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0532e-05 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268048+0000", 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7259999999999996e-06 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.270471+0000", 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024233789999999998 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275200+0000", 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047289130000000004 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275360+0000", 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000159531 2024-09-16T10:24:41.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284217+0000", 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088568970000000007 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284226+0000", 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3559999999999999e-06 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.303128+0000", 2024-09-16T10:24:41.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0189021700000001 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304057+0000", 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00092923599999999995 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310454+0000", 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063961260000000002 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310571+0000", 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117761 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.713 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.714 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.714 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50965 8.a 8:5e5d9df4:::benchmark_data_smithi063_101290_object50964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.714 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:24:41.714 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.41483131299998, 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0323928659999999, 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50965 2024-09-16T10:24:41.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278464+0000", 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.968e-06 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278491+0000", 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7061999999999999e-05 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278500+0000", 2024-09-16T10:24:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4970000000000007e-06 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278507+0000", 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5139999999999998e-06 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284249+0000", 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0057418360000000002 2024-09-16T10:24:41.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284255+0000", 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4529999999999999e-06 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304143+0000", 2024-09-16T10:24:41.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.019888039 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304299+0000", 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155651 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304463+0000", 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016370800000000001 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308275+0000", 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038120049999999998 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310595+0000", 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023207620000000001 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310754+0000", 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015848500000000001 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310795+0000", 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1655e-05 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310855+0000", 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9231000000000003e-05 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.722 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50996 8.a 8:5adb42d1:::benchmark_data_smithi063_101290_object50995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.38820147199999, 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005949921, 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.723 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50996 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:24:41.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305093+0000", 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.779e-06 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305117+0000", 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3370000000000002e-05 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305127+0000", 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0284e-05 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305134+0000", 2024-09-16T10:24:41.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9659999999999999e-06 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305192+0000", 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7587000000000003e-05 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305201+0000", 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2590000000000006e-06 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306679+0000", 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0014777290000001 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306790+0000", 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111675 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306942+0000", 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015151300000000001 2024-09-16T10:24:41.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310399+0000", 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034570529999999999 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310904+0000", 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00050482900000000004 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310943+0000", 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9048000000000003e-05 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310974+0000", 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1164999999999997e-05 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311042+0000", 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7663999999999995e-05 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51001 8.a 8:59406931:::benchmark_data_smithi063_101290_object51000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.38408435299999, 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0021313270000001, 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.731 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51001 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309210+0000", 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2589999999999999e-06 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309227+0000", 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6765999999999998e-05 2024-09-16T10:24:41.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309235+0000", 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1969999999999996e-06 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309240+0000", 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6290000000000001e-06 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309279+0000", 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9199999999999997e-05 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309285+0000", 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8810000000000001e-06 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308323+0000", 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99903792999999996 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308464+0000", 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014168000000000001 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308635+0000", 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000170383 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310405+0000", 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001770134 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311104+0000", 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069940900000000001 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311178+0000", 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3911000000000004e-05 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311237+0000", 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9113000000000001e-05 2024-09-16T10:24:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311340+0000", 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102835 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51015 8.a 8:5f39a03a:::benchmark_data_smithi063_101290_object51014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.36777838400002, 2024-09-16T10:24:41.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99112391399999999, 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51015 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:24:41.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325516+0000", 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-07 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325537+0000", 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1900999999999999e-05 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325554+0000", 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6937999999999999e-05 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325562+0000", 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3830000000000003e-06 2024-09-16T10:24:41.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325628+0000", 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6418e-05 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325640+0000", 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2157000000000001e-05 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311387+0000", 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98574625999999999 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311498+0000", 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111768 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311624+0000", 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012560199999999999 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316394+0000", 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047700279999999999 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316457+0000", 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2819999999999998e-05 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316471+0000", 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4236000000000001e-05 2024-09-16T10:24:41.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316522+0000", 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1001000000000002e-05 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316639+0000", 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116641 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51028 8.a 8:504c9504:::benchmark_data_smithi063_101290_object51027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.359396387, 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98302855600000005, 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.746 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51028 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333898+0000", 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.375e-06 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333916+0000", 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8236e-05 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333925+0000", 2024-09-16T10:24:41.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.585e-06 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333931+0000", 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1070000000000001e-06 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334057+0000", 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012610900000000001 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334083+0000", 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5327e-05 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.312988+0000", 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97890501699999999 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313110+0000", 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012285099999999999 2024-09-16T10:24:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313254+0000", 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014338900000000001 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316650+0000", 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033961529999999998 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316706+0000", 2024-09-16T10:24:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5878000000000003e-05 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316760+0000", 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4502000000000001e-05 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316833+0000", 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2632000000000005e-05 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316925+0000", 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2394999999999999e-05 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51060 8.a 8:500fc7dc:::benchmark_data_smithi063_101290_object51059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:24:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.33639625299998, 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96027958700000005, 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51060 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:24:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356899+0000", 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.821e-06 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356917+0000", 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7972999999999999e-05 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356926+0000", 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9679999999999995e-06 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356931+0000", 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:24:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356997+0000", 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6079000000000002e-05 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.357005+0000", 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7569999999999993e-06 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314700+0000", 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95769431000000005 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314827+0000", 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012752799999999999 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314995+0000", 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016789100000000001 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316982+0000", 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986603 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316999+0000", 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7319000000000001e-05 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317035+0000", 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5655000000000003e-05 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317088+0000", 2024-09-16T10:24:41.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3278e-05 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317177+0000", 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8486999999999995e-05 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51073 8.a 8:53b2a9fa:::benchmark_data_smithi063_101290_object51072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.761 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.32526668999998, 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.957136019, 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51073 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.762 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368028+0000", 2024-09-16T10:24:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5939999999999999e-06 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368053+0000", 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5130000000000002e-05 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368066+0000", 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2397e-05 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368074+0000", 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2789999999999998e-06 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368143+0000", 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9558e-05 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368154+0000", 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0869999999999999e-05 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317228+0000", 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.949073273 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317346+0000", 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011878400000000001 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317477+0000", 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013062600000000001 2024-09-16T10:24:41.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322458+0000", 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049812980000000003 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322513+0000", 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4561999999999997e-05 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325080+0000", 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025674819999999998 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325125+0000", 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5065000000000003e-05 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325163+0000", 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7101000000000003e-05 2024-09-16T10:24:41.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51087 8.a 8:51205303:::benchmark_data_smithi063_101290_object51086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.31430915700003, 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94635271099999996, 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.769 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51087 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378986+0000", 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9850000000000001e-06 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379013+0000", 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6911e-05 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379029+0000", 2024-09-16T10:24:41.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5928999999999999e-05 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379037+0000", 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7100000000000007e-06 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379115+0000", 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8634999999999995e-05 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379128+0000", 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2903e-05 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319060+0000", 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93993237200000002 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319187+0000", 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012621 2024-09-16T10:24:41.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319358+0000", 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017167699999999999 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322553+0000", 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003194625 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325067+0000", 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025139170000000001 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325170+0000", 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102948 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325265+0000", 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5570999999999997e-05 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325337+0000", 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1317999999999997e-05 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51105 8.a 8:558fcaa8:::benchmark_data_smithi063_101290_object51104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.30041690199999, 2024-09-16T10:24:41.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93274633699999998, 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51105 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:24:41.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392878+0000", 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.516e-06 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392900+0000", 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2575e-05 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392913+0000", 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2196e-05 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392920+0000", 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7170000000000008e-06 2024-09-16T10:24:41.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393546+0000", 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00062592699999999997 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393555+0000", 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7809999999999994e-06 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320780+0000", 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92722495100000002 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320895+0000", 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115375 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.321051+0000", 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155482 2024-09-16T10:24:41.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325345+0000", 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042937840000000001 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325384+0000", 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9261999999999998e-05 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325522+0000", 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0001383 2024-09-16T10:24:41.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325584+0000", 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-05 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325623+0000", 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8940999999999999e-05 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.783 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51170 8.a 8:55be5a3b:::benchmark_data_smithi063_101290_object51169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.24446902099999, 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87700012299999996, 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51170 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.784 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448826+0000", 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7749999999999999e-06 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448849+0000", 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3298999999999998e-05 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448867+0000", 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7467999999999999e-05 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448874+0000", 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-06 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448936+0000", 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1383999999999994e-05 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448947+0000", 2024-09-16T10:24:41.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1756000000000001e-05 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322695+0000", 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87374773699999997 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322825+0000", 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000129579 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322994+0000", 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016874899999999999 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325349+0000", 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002355514 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325668+0000", 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000319034 2024-09-16T10:24:41.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325723+0000", 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5376e-05 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325775+0000", 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1924e-05 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325824+0000", 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8918000000000002e-05 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51179 8.a 8:5e35a76f:::benchmark_data_smithi063_101290_object51178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.18017369, 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82446452000000003, 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.791 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51179 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513122+0000", 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5579999999999999e-06 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513149+0000", 2024-09-16T10:24:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7101e-05 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513166+0000", 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7017e-05 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513175+0000", 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6209999999999993e-06 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513268+0000", 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3287000000000003e-05 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513279+0000", 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0889e-05 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325865+0000", 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.812586527 2024-09-16T10:24:41.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325989+0000", 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123347 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.326103+0000", 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011419199999999999 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331852+0000", 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005749466 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331901+0000", 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8854999999999999e-05 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337506+0000", 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005604876 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337547+0000", 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0961999999999998e-05 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337584+0000", 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6822000000000001e-05 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51188 8.a 8:5734c940:::benchmark_data_smithi063_101290_object51187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 329.17015754800002, 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.814521685, 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51188 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.799 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523137+0000", 2024-09-16T10:24:41.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.804e-06 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523159+0000", 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1447e-05 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523171+0000", 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2347000000000001e-05 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523179+0000", 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6310000000000003e-06 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523240+0000", 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1150999999999998e-05 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523248+0000", 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5190000000000003e-06 2024-09-16T10:24:41.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.327963+0000", 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.80471533100000003 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328066+0000", 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103287 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328230+0000", 2024-09-16T10:24:41.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163521 2024-09-16T10:24:41.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:41.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/593768945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:41.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:41 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1181/135882 objects degraded (0.869%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:41.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:41 smithi165 ceph-mon[32696]: osdmap e346: 8 total, 8 up, 8 in 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331938+0000", 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037081639999999999 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331973+0000", 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5355999999999998e-05 2024-09-16T10:24:41.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337597+0000", 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056237809999999996 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337637+0000", 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.019e-05 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337657+0000", 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0156000000000001e-05 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51286 8.1 8:897e7f6e:::benchmark_data_smithi063_101290_object51285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 327.37626658800002, 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.983161858, 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.808 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51286 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317028+0000", 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.141e-06 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317054+0000", 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6186e-05 2024-09-16T10:24:41.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317065+0000", 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.08e-05 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317072+0000", 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3520000000000001e-06 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317688+0000", 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061638600000000002 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317698+0000", 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.346e-06 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292346+0000", 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97464779999999995 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292464+0000", 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118582 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292563+0000", 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9047000000000002e-05 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296570+0000", 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040065379999999996 2024-09-16T10:24:41.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296637+0000", 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7547999999999998e-05 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300071+0000", 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003433298 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300139+0000", 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8702999999999996e-05 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300188+0000", 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9131000000000001e-05 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51312 8.1 8:8f097c26:::benchmark_data_smithi063_101290_object51311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:24:41.815 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 327.35723293699999, 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96891611600000005, 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51312 2024-09-16T10:24:41.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336063+0000", 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3949999999999999e-06 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336103+0000", 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0754000000000003e-05 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336138+0000", 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4662000000000001e-05 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336160+0000", 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1752000000000001e-05 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336277+0000", 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117586 2024-09-16T10:24:41.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336288+0000", 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0852999999999999e-05 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294143+0000", 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95785426600000001 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294236+0000", 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3010000000000006e-05 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294323+0000", 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7619000000000003e-05 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298376+0000", 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004053175 2024-09-16T10:24:41.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300258+0000", 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018817300000000001 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304863+0000", 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046049539999999996 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3490000000000004e-05 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304976+0000", 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9870000000000002e-05 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51316 8.1 8:858e6b14:::benchmark_data_smithi063_101290_object51315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 327.35591296000001, 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96772945600000004, 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51316 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337381+0000", 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.192e-06 2024-09-16T10:24:41.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337441+0000", 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9988000000000002e-05 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337452+0000", 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0857e-05 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.826 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337460+0000", 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2649999999999999e-06 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337520+0000", 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0377999999999997e-05 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337530+0000", 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0416e-05 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295200+0000", 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95766991499999998 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295307+0000", 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010660199999999999 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295446+0000", 2024-09-16T10:24:41.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013906699999999999 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300014+0000", 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045680249999999999 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300329+0000", 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031521199999999999 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305018+0000", 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046893569999999999 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305073+0000", 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4243000000000002e-05 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305110+0000", 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6939e-05 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51363 8.1 8:83ceb830:::benchmark_data_smithi063_101290_object51362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 327.25438693500001, 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.86632090799999995, 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.831 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51363 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438908+0000", 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.454e-06 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438931+0000", 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3197000000000002e-05 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438945+0000", 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3703000000000001e-05 2024-09-16T10:24:41.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438952+0000", 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5259999999999996e-06 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439023+0000", 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0457000000000007e-05 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439034+0000", 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1141000000000001e-05 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296689+0000", 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.85765570300000005 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296811+0000", 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121229 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296957+0000", 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014621999999999999 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300288+0000", 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033314 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300372+0000", 2024-09-16T10:24:41.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3277999999999998e-05 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305147+0000", 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047752990000000002 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305194+0000", 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6789999999999998e-05 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305227+0000", 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3510999999999998e-05 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51388 8.1 8:8d4128ae:::benchmark_data_smithi063_101290_object51387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 327.22967210100001, 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.84172315399999997, 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.838 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51388 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463623+0000", 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463644+0000", 2024-09-16T10:24:41.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0937999999999998e-05 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463654+0000", 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0390999999999999e-05 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463661+0000", 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2400000000000001e-06 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463714+0000", 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3059999999999997e-05 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463722+0000", 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4769999999999997e-06 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298409+0000", 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.83468682000000005 2024-09-16T10:24:41.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298629+0000", 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220258 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298774+0000", 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145012 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300340+0000", 2024-09-16T10:24:41.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001566352 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302114+0000", 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001773518 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305261+0000", 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003147721 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305310+0000", 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9051000000000002e-05 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305344+0000", 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3751000000000003e-05 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51396 8.1 8:8a8d0cb2:::benchmark_data_smithi063_101290_object51395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:24:41.845 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 327.21087999999997, 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82304446200000003, 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51396 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482414+0000", 2024-09-16T10:24:41.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2300000000000001e-06 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482433+0000", 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8876e-05 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482445+0000", 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2170999999999999e-05 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482453+0000", 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8939999999999993e-06 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482504+0000", 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0698000000000001e-05 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482512+0000", 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1319999999999994e-06 2024-09-16T10:24:41.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300411+0000", 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.81789900699999996 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300560+0000", 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014893799999999999 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:24:41.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300718+0000", 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015829700000000001 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302254+0000", 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015359429999999999 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302509+0000", 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025458499999999999 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305379+0000", 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028699329999999999 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305426+0000", 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7330000000000003e-05 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305458+0000", 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1427999999999998e-05 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.852 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.853 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.853 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:24:41.853 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:24:41.853 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:24:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:41.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:41.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:41.956 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:42.004 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:42.106 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:42.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:24:42.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:42.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:42.376 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.418 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:42.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:42.512 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:42 smithi063 ceph-mon[29993]: pgmap v593: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.31k op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:42.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:42.569 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:42.671 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:42.723 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:42.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:42 smithi165 ceph-mon[32696]: pgmap v593: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.31k op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:42.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:42.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:42.884 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:42 smithi063 ceph-mon[39470]: pgmap v593: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.31k op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:42.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:42.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:42.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:43.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:43.126 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:43.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:43.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:43.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:43.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:43.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:43.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:43.401 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:43.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:43.503 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:43.534 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:43.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:43.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:43.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:43.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:43.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:43.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:43.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:43.908 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:43.943 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:43.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:44.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:44.084 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:44.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:44.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:44.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:44.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:44.310 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:44.344 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:44.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:44.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:24:44.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:44.488 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:44.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:44.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:44.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:44.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:44.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:44 smithi063 ceph-mon[39470]: pgmap v594: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 625 op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 21 objects/s recovering 2024-09-16T10:24:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:44 smithi063 ceph-mon[29993]: pgmap v594: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 625 op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 21 objects/s recovering 2024-09-16T10:24:44.721 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:44.767 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:44.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:44.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:44 smithi165 ceph-mon[32696]: pgmap v594: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 625 op/s; 1181/135882 objects degraded (0.869%); 1184/135882 objects misplaced (0.871%); 0 B/s, 21 objects/s recovering 2024-09-16T10:24:44.869 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:44.910 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:44.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:45.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:45.052 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:45.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:45.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:45.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:45.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:45.285 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:45.308 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:45.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:45.410 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:45.447 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:45.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:45.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:45.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:45.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:45.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:45.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:45.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:45.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:45.857 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:45.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:45.959 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:46.003 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:46.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:46.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:46.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:46.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:46.359 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:46.363 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:24:46.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:46.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:46.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:46.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:46.561 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:46.565 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:46.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:46.664 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:46 smithi063 ceph-mon[39470]: pgmap v595: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 1000/118170 objects degraded (0.846%); 1003/118170 objects misplaced (0.849%); 0 B/s, 44 objects/s recovering 2024-09-16T10:24:46.664 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:46 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1000/118170 objects degraded (0.846%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:46.666 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:46 smithi063 ceph-mon[29993]: pgmap v595: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 1000/118170 objects degraded (0.846%); 1003/118170 objects misplaced (0.849%); 0 B/s, 44 objects/s recovering 2024-09-16T10:24:46.666 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:46 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1000/118170 objects degraded (0.846%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:46.703 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:46 smithi165 ceph-mon[32696]: pgmap v595: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 1000/118170 objects degraded (0.846%); 1003/118170 objects misplaced (0.849%); 0 B/s, 44 objects/s recovering 2024-09-16T10:24:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:46 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1000/118170 objects degraded (0.846%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:46.802 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:24:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=cleanup t=2024-09-16T10:24:46.752006371Z level=info msg="Completed cleanup jobs" duration=4.122677ms 2024-09-16T10:24:46.806 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.863 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:46.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:46.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:46.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:46.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:47.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:47.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:47.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:47.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:47.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:47.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:47.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:47.302 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:24:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=plugins.update.checker t=2024-09-16T10:24:46.893848869Z level=info msg="Update check succeeded" duration=58.291202ms 2024-09-16T10:24:47.302 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:24:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=grafana.update.checker t=2024-09-16T10:24:46.912761714Z level=info msg="Update check succeeded" duration=74.078366ms 2024-09-16T10:24:47.302 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:24:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:24:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:24:47.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:47.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:47.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:47.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:47.513 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:47.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:47.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:47.656 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:47.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:47.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:47 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:24:47.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:47 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:24:47.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:47 smithi063 ceph-mon[29993]: osdmap e347: 8 total, 8 up, 8 in 2024-09-16T10:24:47.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:47 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:24:47.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:47 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:24:47.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:47 smithi063 ceph-mon[39470]: osdmap e347: 8 total, 8 up, 8 in 2024-09-16T10:24:47.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:47.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:47.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:47 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:24:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:47 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:24:47.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:47 smithi165 ceph-mon[32696]: osdmap e347: 8 total, 8 up, 8 in 2024-09-16T10:24:47.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:47.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:47.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:48.015 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:48.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:48.123 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:24:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:24:48.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:48.164 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:24:48.164 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:24:48.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:48.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:48.201 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:24:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:24:48.241 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:24:48.241 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:24:48.274 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:24:48.274 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:24:48.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:48.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:48.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:48.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:48.320 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:24:48.320 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:24:48.362 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:24:48.363 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:24:48.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:24:48.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:24:48.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:48.461 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:48.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:no valid command found; 10 closest matches: 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:0 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:1 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:2 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:abort 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:assert 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:bench [] [] [] [] 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs files list 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:bluefs stats 2024-09-16T10:24:48.528 INFO:teuthology.orchestra.run.smithi063.stderr:bluestore allocator dump block 2024-09-16T10:24:48.529 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T10:24:48.551 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:24:48.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:48.616 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:48.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[29993]: pgmap v597: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 1000/118170 objects degraded (0.846%); 1003/118170 objects misplaced (0.849%); 0 B/s, 22 objects/s recovering 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2057236461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4240998194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[39470]: pgmap v597: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 1000/118170 objects degraded (0.846%); 1003/118170 objects misplaced (0.849%); 0 B/s, 22 objects/s recovering 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2057236461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4240998194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:48.687 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:48 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:24:48.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:48.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:48.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:48.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:48.905 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:48.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:49.008 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:48 smithi165 ceph-mon[32696]: pgmap v597: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 1000/118170 objects degraded (0.846%); 1003/118170 objects misplaced (0.849%); 0 B/s, 22 objects/s recovering 2024-09-16T10:24:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2057236461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:24:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4240998194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:48 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:24:49.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:24:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:24:49.058 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:49.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:49.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:49.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:49.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:49.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:49.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:49.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:49.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:49.462 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:49.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:49.564 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:49.610 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:49.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:49.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:49.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:49.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:49.841 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:49.876 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:49.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2057236461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: osdmap e348: 8 total, 8 up, 8 in 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.905 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:24:49.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2057236461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: osdmap e348: 8 total, 8 up, 8 in 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.907 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:49.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:49.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:24:49.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:24:49.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:49 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:49.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:50.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2057236461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:24:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: osdmap e348: 8 total, 8 up, 8 in 2024-09-16T10:24:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T10:24:50.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:50.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:24:50.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:24:50.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:49 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:24:50.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:50.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:50.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:50.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:50.365 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.403 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:50.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:50.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:50.543 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:50.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:50.645 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:50.684 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:24:50.744 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T10:24:50.781 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:50.786 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:50.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:50.918 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:50 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi165 to 3248M 2024-09-16T10:24:50.918 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:50 smithi063 ceph-mon[29993]: Adjusting osd_memory_target on smithi063 to 3248M 2024-09-16T10:24:50.918 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:50 smithi063 ceph-mon[29993]: pgmap v599: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 45 objects/s recovering 2024-09-16T10:24:50.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:50 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi165 to 3248M 2024-09-16T10:24:50.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:50 smithi063 ceph-mon[39470]: Adjusting osd_memory_target on smithi063 to 3248M 2024-09-16T10:24:50.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:50 smithi063 ceph-mon[39470]: pgmap v599: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 45 objects/s recovering 2024-09-16T10:24:50.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:50.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:50.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:51.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:50 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi165 to 3248M 2024-09-16T10:24:51.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:50 smithi165 ceph-mon[32696]: Adjusting osd_memory_target on smithi063 to 3248M 2024-09-16T10:24:51.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:50 smithi165 ceph-mon[32696]: pgmap v599: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 45 objects/s recovering 2024-09-16T10:24:51.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:51.116 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:51.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:51.221 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:51.261 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:51.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:51.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:51.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:51.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:51.421 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:24:51.421 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:24:51.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:51.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:51.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:51.551 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:24:51.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:51.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:51.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:51.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:51.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:51.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:51.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:51.925 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:51.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:51 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 815/101625 objects degraded (0.802%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:51 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 815/101625 objects degraded (0.802%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:52.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:52.035 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:24:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:51 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 815/101625 objects degraded (0.802%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:52.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:52.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:52.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:52.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:52.295 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:24:52.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.344 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:52.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:52.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:52.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:52.479 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:52.553 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:52.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:52.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:52.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:52.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:52.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:52.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:52 smithi063 ceph-mon[29993]: pgmap v600: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 45 objects/s recovering 2024-09-16T10:24:52.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:52 smithi063 ceph-mon[39470]: pgmap v600: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 45 objects/s recovering 2024-09-16T10:24:52.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:52.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:52.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:52 smithi165 ceph-mon[32696]: pgmap v600: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 45 objects/s recovering 2024-09-16T10:24:53.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:53.108 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:53.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:53.209 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:53.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:53.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:53.342 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:53.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:53.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:53.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:53.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:53.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:53.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:53.618 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:53.661 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:53.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:53.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:53.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:53.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:53.846 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:53.846 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:53.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:53.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:53.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:54.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:24:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:54.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:54.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:54.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:54.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:54.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:54.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:54.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:54.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:54.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:54.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:54.683 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.718 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:54.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:54.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:54.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:54.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:54 smithi063 ceph-mon[29993]: pgmap v601: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 23 objects/s recovering 2024-09-16T10:24:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:54 smithi063 ceph-mon[39470]: pgmap v601: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 23 objects/s recovering 2024-09-16T10:24:54.985 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:54.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:54 smithi165 ceph-mon[32696]: pgmap v601: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 2.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s; 815/101625 objects degraded (0.802%); 818/101625 objects misplaced (0.805%); 0 B/s, 23 objects/s recovering 2024-09-16T10:24:55.087 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:55.128 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:55.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:55.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:55.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:55.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:55.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:55.402 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:55.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:55.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:55.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:55.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:55.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:55.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:55.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:55.745 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:24:55.745 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:24:55.745 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T10:24:55.746 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.293615 2024-09-16T10:24:55.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 0.29361495500127 2024-09-16T10:24:55.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:55.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:55.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:55.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:55.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:55.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:56.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:56.068 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:56.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:56.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:56.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:56.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:56.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:56.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:56.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:56.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:56.354 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:24:56.355 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:24:56.386 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:56.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:24:56.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:24:56.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:56.440 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:24:56.440 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:24:56.441 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:24:56.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:56.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:56.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:56.486 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:24:56.486 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:24:56.517 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:24:56.517 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:24:56.553 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:24:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:24:56.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:24:56.589 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:24:56.589 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:24:56.592 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:24:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:56.610 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:56.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:24:56.626 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:24:56.626 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:24:56.711 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:56.753 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:56.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:56.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:56.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:56.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[29993]: pgmap v602: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 628/84735 objects degraded (0.741%); 631/84735 objects misplaced (0.745%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2689028082' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[29993]: from='client.45884 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 628/84735 objects degraded (0.741%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:56.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3099312068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:56.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[39470]: pgmap v602: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 628/84735 objects degraded (0.741%); 631/84735 objects misplaced (0.745%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:56.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2689028082' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:56.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[39470]: from='client.45884 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:56.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 628/84735 objects degraded (0.741%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:56.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3099312068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:56.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:57.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:56 smithi165 ceph-mon[32696]: pgmap v602: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 628/84735 objects degraded (0.741%); 631/84735 objects misplaced (0.745%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2689028082' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:56 smithi165 ceph-mon[32696]: from='client.45884 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:24:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:56 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 628/84735 objects degraded (0.741%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:24:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3099312068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:24:57.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.157 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:57.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:57.258 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:57.292 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:57.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:57.421 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:24:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:24:57.441 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:57.544 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:57.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[29993]: from='client.45884 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:24:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[29993]: osdmap e349: 8 total, 8 up, 8 in 2024-09-16T10:24:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[29993]: 6.11 scrub starts 2024-09-16T10:24:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[29993]: 6.11 scrub ok 2024-09-16T10:24:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[39470]: from='client.45884 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:24:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[39470]: osdmap e349: 8 total, 8 up, 8 in 2024-09-16T10:24:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[39470]: 6.11 scrub starts 2024-09-16T10:24:57.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:57 smithi063 ceph-mon[39470]: 6.11 scrub ok 2024-09-16T10:24:57.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:57.813 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:57.851 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:57.953 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:24:57.996 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:57.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:24:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:57 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:24:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:57 smithi165 ceph-mon[32696]: from='client.45884 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:24:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:57 smithi165 ceph-mon[32696]: osdmap e349: 8 total, 8 up, 8 in 2024-09-16T10:24:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:57 smithi165 ceph-mon[32696]: 6.11 scrub starts 2024-09-16T10:24:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:57 smithi165 ceph-mon[32696]: 6.11 scrub ok 2024-09-16T10:24:58.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:58.140 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:58.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:58.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:58.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:58.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:58.371 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:58.418 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:58.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:58.432 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:24:58.432 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:24:58.432 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:24:58.432 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:24:58.433 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:24:58.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:58.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:58.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:58.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[29993]: pgmap v604: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 992 op/s; 628/84735 objects degraded (0.741%); 631/84735 objects misplaced (0.745%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:58.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[29993]: 5.e scrub starts 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[29993]: 5.e scrub ok 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[29993]: 2.2 scrub starts 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[29993]: 2.2 scrub ok 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[39470]: pgmap v604: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 992 op/s; 628/84735 objects degraded (0.741%); 631/84735 objects misplaced (0.745%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[39470]: 5.e scrub starts 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[39470]: 5.e scrub ok 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[39470]: 2.2 scrub starts 2024-09-16T10:24:58.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:58 smithi063 ceph-mon[39470]: 2.2 scrub ok 2024-09-16T10:24:58.652 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:58.694 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:58.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:58.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:24:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:58.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:58.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:24:58.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:24:58.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:58 smithi165 ceph-mon[32696]: pgmap v604: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 992 op/s; 628/84735 objects degraded (0.741%); 631/84735 objects misplaced (0.745%); 0 B/s, 42 objects/s recovering 2024-09-16T10:24:58.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:58 smithi165 ceph-mon[32696]: 5.e scrub starts 2024-09-16T10:24:58.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:58 smithi165 ceph-mon[32696]: 5.e scrub ok 2024-09-16T10:24:58.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:58 smithi165 ceph-mon[32696]: 2.2 scrub starts 2024-09-16T10:24:58.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:58 smithi165 ceph-mon[32696]: 2.2 scrub ok 2024-09-16T10:24:58.954 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:24:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:24:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:58.978 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:58.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:24:59.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:59.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:59.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:59.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:59.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:59.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:59.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:59.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:59.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:59.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:24:59.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:24:59.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:59.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:24:59.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:24:59.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:24:59.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:59.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:24:59.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:24:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:59.814 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:24:59.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:24:59.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:24:59.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:24:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:24:59.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:24:59.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:59.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:59 smithi063 ceph-mon[29993]: 9.f scrub starts 2024-09-16T10:24:59.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:24:59 smithi063 ceph-mon[29993]: 9.f scrub ok 2024-09-16T10:24:59.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:24:59.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:59 smithi063 ceph-mon[39470]: 9.f scrub starts 2024-09-16T10:24:59.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:24:59 smithi063 ceph-mon[39470]: 9.f scrub ok 2024-09-16T10:25:00.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:00.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:59 smithi165 ceph-mon[32696]: 9.f scrub starts 2024-09-16T10:25:00.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:24:59 smithi165 ceph-mon[32696]: 9.f scrub ok 2024-09-16T10:25:00.059 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:00.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:00.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:00.082 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:00.170 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:00.185 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:25:00.223 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:00.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:00.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:00.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:00.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:00.449 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:00.494 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:00.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:00.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:00.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:00.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:00.669 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.7 primary-affinity to 0.293615 (8192422) 2024-09-16T10:25:00.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:00.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:00.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:00.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:00.876 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:00.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: pgmap v605: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 628/70572 objects degraded (0.890%); 631/70572 objects misplaced (0.894%); 0 B/s, 18 objects/s recovering 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: osdmap e350: 8 total, 8 up, 8 in 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: 3.5 scrub starts 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: 3.5 scrub ok 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1534286453' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127} : dispatch 2024-09-16T10:25:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[39470]: from='client.45893 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127} : dispatch 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: pgmap v605: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 628/70572 objects degraded (0.890%); 631/70572 objects misplaced (0.894%); 0 B/s, 18 objects/s recovering 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: osdmap e350: 8 total, 8 up, 8 in 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: 3.5 scrub starts 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: 3.5 scrub ok 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1534286453' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127} : dispatch 2024-09-16T10:25:00.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:00 smithi063 ceph-mon[29993]: from='client.45893 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127} : dispatch 2024-09-16T10:25:00.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:01.012 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: pgmap v605: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.4 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 628/70572 objects degraded (0.890%); 631/70572 objects misplaced (0.894%); 0 B/s, 18 objects/s recovering 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: osdmap e350: 8 total, 8 up, 8 in 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: 3.5 scrub starts 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: 3.5 scrub ok 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1534286453' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127} : dispatch 2024-09-16T10:25:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:00 smithi165 ceph-mon[32696]: from='client.45893 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127} : dispatch 2024-09-16T10:25:01.114 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:01.153 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:01.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:01.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:01.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.418 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:01.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:01.519 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:01.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.682 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:01.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:01.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:01.805 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:01.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:01.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:01.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: from='client.45893 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127}]': finished 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: osdmap e351: 8 total, 8 up, 8 in 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: 1.12 scrub ok 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 628/70572 objects degraded (0.890%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: osdmap e352: 8 total, 8 up, 8 in 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:25:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: from='client.45893 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127}]': finished 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: osdmap e351: 8 total, 8 up, 8 in 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: 1.12 scrub ok 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 628/70572 objects degraded (0.890%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: osdmap e352: 8 total, 8 up, 8 in 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:25:01.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:01 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:25:02.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: from='client.45893 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.29361495500127}]': finished 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: osdmap e351: 8 total, 8 up, 8 in 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: 1.12 scrub ok 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 628/70572 objects degraded (0.890%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: osdmap e352: 8 total, 8 up, 8 in 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:25:02.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:01 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:25:02.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:02.171 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.208 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:02.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:02.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:02.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:02.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:02.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:02.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:02.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:02.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:02.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:02.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.856 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:02.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:02.937 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[29993]: pgmap v609: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 372/67260 objects degraded (0.553%); 375/67260 objects misplaced (0.558%); 0 B/s, 54 objects/s recovering 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[29993]: 1.d scrub ok 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[29993]: 3.18 scrub starts 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[29993]: 3.18 scrub ok 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[39470]: pgmap v609: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 372/67260 objects degraded (0.553%); 375/67260 objects misplaced (0.558%); 0 B/s, 54 objects/s recovering 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:25:02.938 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[39470]: 1.d scrub ok 2024-09-16T10:25:02.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:25:02.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:25:02.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[39470]: 3.18 scrub starts 2024-09-16T10:25:02.939 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:02 smithi063 ceph-mon[39470]: 3.18 scrub ok 2024-09-16T10:25:02.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:02.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:02.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:02 smithi165 ceph-mon[32696]: pgmap v609: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 372/67260 objects degraded (0.553%); 375/67260 objects misplaced (0.558%); 0 B/s, 54 objects/s recovering 2024-09-16T10:25:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:02 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:25:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:02 smithi165 ceph-mon[32696]: 1.d scrub ok 2024-09-16T10:25:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:02 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:25:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:02 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:25:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:02 smithi165 ceph-mon[32696]: 3.18 scrub starts 2024-09-16T10:25:03.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:02 smithi165 ceph-mon[32696]: 3.18 scrub ok 2024-09-16T10:25:03.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:03.114 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:03.115 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:03.116 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:03.116 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:03.116 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:03.116 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:03.116 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:03.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:03.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:03.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:03.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:03.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:03.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:03.404 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:03.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:03.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:03.552 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:03.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:03.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:03.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:03.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:03.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:03.828 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:03.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:03.897 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4260878752' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4260878752' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:25:03.898 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:03 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:25:03.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:03.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:03.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:03 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:25:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:03 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:25:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:03 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4260878752' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:03 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:25:04.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:03 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:25:04.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:04.099 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:04.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:04.201 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:04.242 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:04.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:04.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:04.381 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:04.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:04.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:04.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:04.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:04.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:04.658 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:04.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:04.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:04.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:04.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:04.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:04 smithi063 ceph-mon[29993]: pgmap v610: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 372/66216 objects degraded (0.562%); 375/66216 objects misplaced (0.566%); 0 B/s, 42 objects/s recovering 2024-09-16T10:25:04.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:04 smithi063 ceph-mon[29993]: osdmap e353: 8 total, 8 up, 8 in 2024-09-16T10:25:04.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:04 smithi063 ceph-mon[29993]: 9.3 scrub starts 2024-09-16T10:25:04.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:04 smithi063 ceph-mon[39470]: pgmap v610: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 372/66216 objects degraded (0.562%); 375/66216 objects misplaced (0.566%); 0 B/s, 42 objects/s recovering 2024-09-16T10:25:04.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:04 smithi063 ceph-mon[39470]: osdmap e353: 8 total, 8 up, 8 in 2024-09-16T10:25:04.806 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:04 smithi063 ceph-mon[39470]: 9.3 scrub starts 2024-09-16T10:25:04.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:04.913 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:04.913 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:04.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:04.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:05.024 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:04 smithi165 ceph-mon[32696]: pgmap v610: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 166 active+clean; 1.3 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 372/66216 objects degraded (0.562%); 375/66216 objects misplaced (0.566%); 0 B/s, 42 objects/s recovering 2024-09-16T10:25:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:04 smithi165 ceph-mon[32696]: osdmap e353: 8 total, 8 up, 8 in 2024-09-16T10:25:05.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:04 smithi165 ceph-mon[32696]: 9.3 scrub starts 2024-09-16T10:25:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:05.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:05.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:05.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:05.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:05.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:05.345 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:05.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:05.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:05.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:05.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:05.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:05.546 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:25:05.576 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:05.605 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:05.648 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:05.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:05.692 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:25:05.692 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:25:05.692 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[29993]: 1.8 scrub starts 2024-09-16T10:25:05.693 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[29993]: 1.8 scrub ok 2024-09-16T10:25:05.693 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[29993]: 9.3 scrub ok 2024-09-16T10:25:05.693 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[29993]: 9.1 scrub starts 2024-09-16T10:25:05.750 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:05.800 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:05.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:05.902 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:05.950 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:25:05.951 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:25:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:25:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[39470]: 1.8 scrub starts 2024-09-16T10:25:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[39470]: 1.8 scrub ok 2024-09-16T10:25:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[39470]: 9.3 scrub ok 2024-09-16T10:25:05.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:05 smithi063 ceph-mon[39470]: 9.1 scrub starts 2024-09-16T10:25:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:05.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:05 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:25:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:05 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:25:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:05 smithi165 ceph-mon[32696]: 1.8 scrub starts 2024-09-16T10:25:06.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:05 smithi165 ceph-mon[32696]: 1.8 scrub ok 2024-09-16T10:25:06.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:05 smithi165 ceph-mon[32696]: 9.3 scrub ok 2024-09-16T10:25:06.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:05 smithi165 ceph-mon[32696]: 9.1 scrub starts 2024-09-16T10:25:06.054 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:06.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:06.194 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:06.220 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:06.296 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:06.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:06.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:06.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:06.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:06.567 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.591 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:06.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:06.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:06.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:06.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:06.839 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:06.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: pgmap v612: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 1.0 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 93/51318 objects degraded (0.181%); 93/51318 objects misplaced (0.181%); 0 B/s, 49 objects/s recovering 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: 3.e scrub starts 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: 3.e scrub ok 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: 9.1 scrub ok 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 93/51318 objects degraded (0.181%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:25:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: osdmap e354: 8 total, 8 up, 8 in 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: 9.2 scrub starts 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/723141961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: from='client.45905 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[29993]: 9.2 scrub ok 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: pgmap v612: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 1.0 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 93/51318 objects degraded (0.181%); 93/51318 objects misplaced (0.181%); 0 B/s, 49 objects/s recovering 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: 3.e scrub starts 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: 3.e scrub ok 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: 9.1 scrub ok 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:25:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 93/51318 objects degraded (0.181%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: osdmap e354: 8 total, 8 up, 8 in 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: 9.2 scrub starts 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/723141961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: from='client.45905 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:06.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:06 smithi063 ceph-mon[39470]: 9.2 scrub ok 2024-09-16T10:25:06.963 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:25:06.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:06.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: pgmap v612: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 1.0 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 93/51318 objects degraded (0.181%); 93/51318 objects misplaced (0.181%); 0 B/s, 49 objects/s recovering 2024-09-16T10:25:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: 3.e scrub starts 2024-09-16T10:25:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: 3.e scrub ok 2024-09-16T10:25:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:25:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: 9.1 scrub ok 2024-09-16T10:25:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:25:07.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 93/51318 objects degraded (0.181%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: osdmap e354: 8 total, 8 up, 8 in 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: 9.2 scrub starts 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/723141961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: from='client.45905 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:07.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:06 smithi165 ceph-mon[32696]: 9.2 scrub ok 2024-09-16T10:25:07.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:07.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:07.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:07.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:07.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:07.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:07.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:07.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:07.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:07.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:25:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:25:07.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:07.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:07.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:07.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:07.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:07.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:07.741 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:07.778 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:07.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:07.880 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:07.902 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:07.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:07.951 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:07.951 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[29993]: 3.19 scrub starts 2024-09-16T10:25:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[29993]: 3.19 scrub ok 2024-09-16T10:25:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:25:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[29993]: from='client.45905 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:07.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[29993]: osdmap e355: 8 total, 8 up, 8 in 2024-09-16T10:25:07.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[39470]: 3.19 scrub starts 2024-09-16T10:25:07.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[39470]: 3.19 scrub ok 2024-09-16T10:25:07.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:25:07.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[39470]: from='client.45905 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:07.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:07 smithi063 ceph-mon[39470]: osdmap e355: 8 total, 8 up, 8 in 2024-09-16T10:25:08.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:08.035 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:08.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:07 smithi165 ceph-mon[32696]: 3.19 scrub starts 2024-09-16T10:25:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:07 smithi165 ceph-mon[32696]: 3.19 scrub ok 2024-09-16T10:25:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:07 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:25:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:07 smithi165 ceph-mon[32696]: from='client.45905 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:08.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:07 smithi165 ceph-mon[32696]: osdmap e355: 8 total, 8 up, 8 in 2024-09-16T10:25:08.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:08.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:08.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:08.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:08.290 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:08.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:08.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:08.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:08.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:08.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:08.559 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:08.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:08.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:08.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:08.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:08.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:08.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:08.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:08.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[29993]: pgmap v615: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 1.0 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 93/51318 objects degraded (0.181%); 93/51318 objects misplaced (0.181%); 0 B/s, 46 objects/s recovering 2024-09-16T10:25:08.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[29993]: 4.14 scrub starts 2024-09-16T10:25:08.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[29993]: 4.14 scrub ok 2024-09-16T10:25:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:08.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:08.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[39470]: pgmap v615: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 1.0 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 93/51318 objects degraded (0.181%); 93/51318 objects misplaced (0.181%); 0 B/s, 46 objects/s recovering 2024-09-16T10:25:08.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[39470]: 4.14 scrub starts 2024-09-16T10:25:08.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[39470]: 4.14 scrub ok 2024-09-16T10:25:08.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:08.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:09.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:09.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:25:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:25:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:08 smithi165 ceph-mon[32696]: pgmap v615: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 1.0 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 93/51318 objects degraded (0.181%); 93/51318 objects misplaced (0.181%); 0 B/s, 46 objects/s recovering 2024-09-16T10:25:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:08 smithi165 ceph-mon[32696]: 4.14 scrub starts 2024-09-16T10:25:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:08 smithi165 ceph-mon[32696]: 4.14 scrub ok 2024-09-16T10:25:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:09.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:09.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:09.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:09.229 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:09.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:09.331 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:09.368 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:09.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:09.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:09.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:09.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:09.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:09.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:09.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:09.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:25:09.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:09.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:09.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:09.858 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:25:09.858 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:25:09.859 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-16T10:25:09.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:25:09.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:09.909 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:09.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:09.910 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:25:09.910 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 4.1c scrub starts 2024-09-16T10:25:09.910 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 4.1c scrub ok 2024-09-16T10:25:09.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:25:09.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 3.14 scrub starts 2024-09-16T10:25:09.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 3.14 scrub ok 2024-09-16T10:25:09.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:09.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:25:09.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:25:09.911 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 4.1c scrub starts 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 4.1c scrub ok 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 3.14 scrub starts 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 3.14 scrub ok 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:25:09.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:09 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:25:10.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:10.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:10.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 4.1c scrub starts 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 4.1c scrub ok 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 3.14 scrub starts 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 3.14 scrub ok 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:25:10.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:09 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:25:10.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:10.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:10.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:10.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:10.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:10.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:10.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:10.428 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:10.462 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:10.466 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:10.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:10.511 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:10.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:10.615 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:10.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:10.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:10.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:10.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:10.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:10.888 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: pgmap v616: 168 pgs: 168 active+clean; 809 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.49k op/s; 0 B/s, 61 objects/s recovering 2024-09-16T10:25:10.888 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: 5.a scrub starts 2024-09-16T10:25:10.888 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: 5.a scrub ok 2024-09-16T10:25:10.888 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:25:10.888 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:25:10.888 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 93/51318 objects degraded (0.181%), 1 pg degraded, 1 pg undersized) 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: 1.10 scrub ok 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[29993]: 9.c scrub starts 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: pgmap v616: 168 pgs: 168 active+clean; 809 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.49k op/s; 0 B/s, 61 objects/s recovering 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: 5.a scrub starts 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: 5.a scrub ok 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 93/51318 objects degraded (0.181%), 1 pg degraded, 1 pg undersized) 2024-09-16T10:25:10.889 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:25:10.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:25:10.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: 1.10 scrub ok 2024-09-16T10:25:10.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:10 smithi063 ceph-mon[39470]: 9.c scrub starts 2024-09-16T10:25:10.898 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:10.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:10.999 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:11.041 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:11.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: pgmap v616: 168 pgs: 168 active+clean; 809 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.49k op/s; 0 B/s, 61 objects/s recovering 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: 5.a scrub starts 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: 5.a scrub ok 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 93/51318 objects degraded (0.181%), 1 pg degraded, 1 pg undersized) 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: 1.10 scrub ok 2024-09-16T10:25:11.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:10 smithi165 ceph-mon[32696]: 9.c scrub starts 2024-09-16T10:25:11.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:11.189 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:11.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:11.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:11.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:11.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:11.343 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:11.343 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:11.377 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:11.378 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:11.414 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:11.415 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:11.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:11.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:11.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:11.470 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:11.501 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:11.502 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:11.541 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:11.541 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:11.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:11.578 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:11.578 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:11.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:11.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:11.618 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:11.618 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:11.697 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:11.736 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:11.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:11.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:11.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:11.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:11.937 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:25:11.940 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:25:11.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:12.010 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:12.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:12.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: 2.1 scrub starts 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: 2.1 scrub ok 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: 4.f scrub starts 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: 4.f scrub ok 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: 9.c scrub ok 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: osdmap e356: 8 total, 8 up, 8 in 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4051056268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:12.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[29993]: pgmap v618: 168 pgs: 168 active+clean; 773 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 759 op/s; 0 B/s, 15 objects/s recovering 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: 2.1 scrub starts 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: 2.1 scrub ok 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: 4.f scrub starts 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: 4.f scrub ok 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: 9.c scrub ok 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: osdmap e356: 8 total, 8 up, 8 in 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4051056268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:12.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:11 smithi063 ceph-mon[39470]: pgmap v618: 168 pgs: 168 active+clean; 773 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 759 op/s; 0 B/s, 15 objects/s recovering 2024-09-16T10:25:12.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:12.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: 2.1 scrub starts 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: 2.1 scrub ok 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: 4.f scrub starts 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: 4.f scrub ok 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: 9.c scrub ok 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: osdmap e356: 8 total, 8 up, 8 in 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4051056268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:11 smithi165 ceph-mon[32696]: pgmap v618: 168 pgs: 168 active+clean; 773 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 759 op/s; 0 B/s, 15 objects/s recovering 2024-09-16T10:25:12.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:25:12.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:12.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:12.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:12.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:12.652 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:12.700 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:12.802 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:12.846 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:12.948 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:12.984 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:12.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:13.086 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:13.121 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:13.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[29993]: 6.15 scrub starts 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[29993]: 6.15 scrub ok 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[29993]: from='client.16062 v1:172.21.15.63:0/2946519383' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[29993]: 5.0 scrub starts 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[29993]: 5.0 scrub ok 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[29993]: 5.11 scrub starts 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[29993]: 5.11 scrub ok 2024-09-16T10:25:13.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[39470]: 6.15 scrub starts 2024-09-16T10:25:13.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[39470]: 6.15 scrub ok 2024-09-16T10:25:13.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[39470]: from='client.16062 v1:172.21.15.63:0/2946519383' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:13.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[39470]: 5.0 scrub starts 2024-09-16T10:25:13.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[39470]: 5.0 scrub ok 2024-09-16T10:25:13.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[39470]: 5.11 scrub starts 2024-09-16T10:25:13.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:12 smithi063 ceph-mon[39470]: 5.11 scrub ok 2024-09-16T10:25:13.224 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:13.265 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:13.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:12 smithi165 ceph-mon[32696]: 6.15 scrub starts 2024-09-16T10:25:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:12 smithi165 ceph-mon[32696]: 6.15 scrub ok 2024-09-16T10:25:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:12 smithi165 ceph-mon[32696]: from='client.16062 v1:172.21.15.63:0/2946519383' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:12 smithi165 ceph-mon[32696]: 5.0 scrub starts 2024-09-16T10:25:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:12 smithi165 ceph-mon[32696]: 5.0 scrub ok 2024-09-16T10:25:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:12 smithi165 ceph-mon[32696]: 5.11 scrub starts 2024-09-16T10:25:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:12 smithi165 ceph-mon[32696]: 5.11 scrub ok 2024-09-16T10:25:13.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:13.404 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:13.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:13.473 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-16T10:25:11.292396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":810031667,"num_objects":12551,"num_object_clones":0,"num_object_copies":37653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12551,"num_whiteouts":0,"num_read":1154,"num_read_kb":906,"num_write":220828,"num_write_kb":7438638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":165023154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74129,"ondisk_log_size":74129,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6124704,"kb_used_data":2830664,"kb_used_omap":12,"kb_used_meta":3293875,"kb_avail":743836512,"statfs":{"total":767960285184,"available":761688588288,"internally_reserved":0,"allocated":2898599936,"data_stored":2894682177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3372928561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-298516480,"num_objects":-4555,"num_object_clones":0,"num_object_copies":-13665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":-93,"num_objects_unfound":0,"num_objects_dirty":-4555,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003102"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":654,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.961110+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:25:06.961110+0000","last_peered":"2024-09-16T10:25:06.961110+0000","last_clean":"2024-09-16T10:25:06.961110+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:25:06.961110+0000","last_undegraded":"2024-09-16T10:25:06.961110+0000","last_fullsized":"2024-09-16T10:25:06.961110+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":702,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.023127+0000","last_change":"2024-09-16T10:25:04.357048+0000","last_active":"2024-09-16T10:25:07.023127+0000","last_peered":"2024-09-16T10:25:07.023127+0000","last_clean":"2024-09-16T10:25:07.023127+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:25:07.023127+0000","last_undegraded":"2024-09-16T10:25:07.023127+0000","last_fullsized":"2024-09-16T10:25:07.023127+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:25:04.356955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.356955+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:13.131130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":632,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959021+0000","last_change":"2024-09-16T10:22:02.564509+0000","last_active":"2024-09-16T10:25:06.959021+0000","last_peered":"2024-09-16T10:25:06.959021+0000","last_clean":"2024-09-16T10:25:06.959021+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:25:06.959021+0000","last_undegraded":"2024-09-16T10:25:06.959021+0000","last_fullsized":"2024-09-16T10:25:06.959021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.564458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.564458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":627,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024014+0000","last_change":"2024-09-16T10:22:35.847447+0000","last_active":"2024-09-16T10:25:07.024014+0000","last_peered":"2024-09-16T10:25:07.024014+0000","last_clean":"2024-09-16T10:25:07.024014+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:25:07.024014+0000","last_undegraded":"2024-09-16T10:25:07.024014+0000","last_fullsized":"2024-09-16T10:25:07.024014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:35.847382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:35.847382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":619,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.016937+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:25:07.016937+0000","last_peered":"2024-09-16T10:25:07.016937+0000","last_clean":"2024-09-16T10:25:07.016937+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:25:07.016937+0000","last_undegraded":"2024-09-16T10:25:07.016937+0000","last_fullsized":"2024-09-16T10:25:07.016937+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:42.600819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":654,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.016561+0000","last_change":"2024-09-16T10:22:30.653658+0000","last_active":"2024-09-16T10:25:07.016561+0000","last_peered":"2024-09-16T10:25:07.016561+0000","last_clean":"2024-09-16T10:25:07.016561+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:25:07.016561+0000","last_undegraded":"2024-09-16T10:25:07.016561+0000","last_fullsized":"2024-09-16T10:25:07.016561+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:30.653580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:30.653580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":724,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.016408+0000","last_change":"2024-09-16T10:24:37.612269+0000","last_active":"2024-09-16T10:25:07.016408+0000","last_peered":"2024-09-16T10:25:07.016408+0000","last_clean":"2024-09-16T10:25:07.016408+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:25:07.016408+0000","last_undegraded":"2024-09-16T10:25:07.016408+0000","last_fullsized":"2024-09-16T10:25:07.016408+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:37.612167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.612167+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.984288+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":631,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.023042+0000","last_change":"2024-09-16T10:24:21.153205+0000","last_active":"2024-09-16T10:25:07.023042+0000","last_peered":"2024-09-16T10:25:07.023042+0000","last_clean":"2024-09-16T10:25:07.023042+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:25:07.023042+0000","last_undegraded":"2024-09-16T10:25:07.023042+0000","last_fullsized":"2024-09-16T10:25:07.023042+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:21.153131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:21.153131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.430119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":648,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014556+0000","last_change":"2024-09-16T10:23:00.881755+0000","last_active":"2024-09-16T10:25:07.014556+0000","last_peered":"2024-09-16T10:25:07.014556+0000","last_clean":"2024-09-16T10:25:07.014556+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:25:07.014556+0000","last_undegraded":"2024-09-16T10:25:07.014556+0000","last_fullsized":"2024-09-16T10:25:07.014556+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:00.881704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:00.881704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:11.814303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":668,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024360+0000","last_change":"2024-09-16T10:24:23.258943+0000","last_active":"2024-09-16T10:25:07.024360+0000","last_peered":"2024-09-16T10:25:07.024360+0000","last_clean":"2024-09-16T10:25:07.024360+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:25:07.024360+0000","last_undegraded":"2024-09-16T10:25:07.024360+0000","last_fullsized":"2024-09-16T10:25:07.024360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:24:23.258886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.258886+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:34.336667+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":632,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014115+0000","last_change":"2024-09-16T10:24:25.902680+0000","last_active":"2024-09-16T10:25:07.014115+0000","last_peered":"2024-09-16T10:25:07.014115+0000","last_clean":"2024-09-16T10:25:07.014115+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:25:07.014115+0000","last_undegraded":"2024-09-16T10:25:07.014115+0000","last_fullsized":"2024-09-16T10:25:07.014115+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:25.902605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:25.902605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:27.186568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":627,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960743+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:25:06.960743+0000","last_peered":"2024-09-16T10:25:06.960743+0000","last_clean":"2024-09-16T10:25:06.960743+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:25:06.960743+0000","last_undegraded":"2024-09-16T10:25:06.960743+0000","last_fullsized":"2024-09-16T10:25:06.960743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":608,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014169+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:25:07.014169+0000","last_peered":"2024-09-16T10:25:07.014169+0000","last_clean":"2024-09-16T10:25:07.014169+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:25:07.014169+0000","last_undegraded":"2024-09-16T10:25:07.014169+0000","last_fullsized":"2024-09-16T10:25:07.014169+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.823305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":656,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.023325+0000","last_change":"2024-09-16T10:24:37.134728+0000","last_active":"2024-09-16T10:25:07.023325+0000","last_peered":"2024-09-16T10:25:07.023325+0000","last_clean":"2024-09-16T10:25:07.023325+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:25:07.023325+0000","last_undegraded":"2024-09-16T10:25:07.023325+0000","last_fullsized":"2024-09-16T10:25:07.023325+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:41.767033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":632,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.955909+0000","last_change":"2024-09-16T10:23:46.668994+0000","last_active":"2024-09-16T10:25:06.955909+0000","last_peered":"2024-09-16T10:25:06.955909+0000","last_clean":"2024-09-16T10:25:06.955909+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:25:06.955909+0000","last_undegraded":"2024-09-16T10:25:06.955909+0000","last_fullsized":"2024-09-16T10:25:06.955909+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:46.668949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.668949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":643,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013709+0000","last_change":"2024-09-16T10:23:42.856401+0000","last_active":"2024-09-16T10:25:07.013709+0000","last_peered":"2024-09-16T10:25:07.013709+0000","last_clean":"2024-09-16T10:25:07.013709+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:25:07.013709+0000","last_undegraded":"2024-09-16T10:25:07.013709+0000","last_fullsized":"2024-09-16T10:25:07.013709+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.856353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.856353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":620,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025492+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:25:07.025492+0000","last_peered":"2024-09-16T10:25:07.025492+0000","last_clean":"2024-09-16T10:25:07.025492+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:25:07.025492+0000","last_undegraded":"2024-09-16T10:25:07.025492+0000","last_fullsized":"2024-09-16T10:25:07.025492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:56.358685+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":631,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960894+0000","last_change":"2024-09-16T10:22:27.210541+0000","last_active":"2024-09-16T10:25:06.960894+0000","last_peered":"2024-09-16T10:25:06.960894+0000","last_clean":"2024-09-16T10:25:06.960894+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:25:06.960894+0000","last_undegraded":"2024-09-16T10:25:06.960894+0000","last_fullsized":"2024-09-16T10:25:06.960894+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":201,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.473527+0000","last_change":"2024-09-16T10:25:08.473527+0000","last_active":"2024-09-16T10:25:08.473527+0000","last_peered":"2024-09-16T10:25:08.473527+0000","last_clean":"2024-09-16T10:25:08.473527+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:25:08.473527+0000","last_undegraded":"2024-09-16T10:25:08.473527+0000","last_fullsized":"2024-09-16T10:25:08.473527+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:25:08.473402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.473402+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.786201+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":624,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959766+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:25:06.959766+0000","last_peered":"2024-09-16T10:25:06.959766+0000","last_clean":"2024-09-16T10:25:06.959766+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:25:06.959766+0000","last_undegraded":"2024-09-16T10:25:06.959766+0000","last_fullsized":"2024-09-16T10:25:06.959766+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":577,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.019416+0000","last_change":"2024-09-16T10:21:57.358479+0000","last_active":"2024-09-16T10:25:07.019416+0000","last_peered":"2024-09-16T10:25:07.019416+0000","last_clean":"2024-09-16T10:25:07.019416+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:25:07.019416+0000","last_undegraded":"2024-09-16T10:25:07.019416+0000","last_fullsized":"2024-09-16T10:25:07.019416+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.358418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.358418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:10.143780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":637,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024504+0000","last_change":"2024-09-16T10:23:08.721407+0000","last_active":"2024-09-16T10:25:07.024504+0000","last_peered":"2024-09-16T10:25:07.024504+0000","last_clean":"2024-09-16T10:25:07.024504+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:25:07.024504+0000","last_undegraded":"2024-09-16T10:25:07.024504+0000","last_fullsized":"2024-09-16T10:25:07.024504+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:08.721339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:08.721339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":694,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018811+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:25:07.018811+0000","last_peered":"2024-09-16T10:25:07.018811+0000","last_clean":"2024-09-16T10:25:07.018811+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:25:07.018811+0000","last_undegraded":"2024-09-16T10:25:07.018811+0000","last_fullsized":"2024-09-16T10:25:07.018811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":653,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024763+0000","last_change":"2024-09-16T10:24:35.186387+0000","last_active":"2024-09-16T10:25:07.024763+0000","last_peered":"2024-09-16T10:25:07.024763+0000","last_clean":"2024-09-16T10:25:07.024763+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:25:07.024763+0000","last_undegraded":"2024-09-16T10:25:07.024763+0000","last_fullsized":"2024-09-16T10:25:07.024763+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:35.186332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:35.186332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.208711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":32,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013918+0000","last_change":"2024-09-16T10:25:00.947987+0000","last_active":"2024-09-16T10:25:07.013918+0000","last_peered":"2024-09-16T10:25:07.013918+0000","last_clean":"2024-09-16T10:25:07.013918+0000","last_became_active":"2024-09-16T10:25:00.947649+0000","last_became_peered":"2024-09-16T10:25:00.947649+0000","last_unstale":"2024-09-16T10:25:07.013918+0000","last_undegraded":"2024-09-16T10:25:07.013918+0000","last_fullsized":"2024-09-16T10:25:07.013918+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:23:09.749746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:09.749746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":639,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959156+0000","last_change":"2024-09-16T10:25:06.652209+0000","last_active":"2024-09-16T10:25:06.959156+0000","last_peered":"2024-09-16T10:25:06.959156+0000","last_clean":"2024-09-16T10:25:06.959156+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:25:06.959156+0000","last_undegraded":"2024-09-16T10:25:06.959156+0000","last_fullsized":"2024-09-16T10:25:06.959156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:25:06.652146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:06.652146+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.202612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":666,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960590+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:25:06.960590+0000","last_peered":"2024-09-16T10:25:06.960590+0000","last_clean":"2024-09-16T10:25:06.960590+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:25:06.960590+0000","last_undegraded":"2024-09-16T10:25:06.960590+0000","last_fullsized":"2024-09-16T10:25:06.960590+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":774,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965451+0000","last_change":"2024-09-16T10:18:09.206537+0000","last_active":"2024-09-16T10:25:06.965451+0000","last_peered":"2024-09-16T10:25:06.965451+0000","last_clean":"2024-09-16T10:25:06.965451+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:25:06.965451+0000","last_undegraded":"2024-09-16T10:25:06.965451+0000","last_fullsized":"2024-09-16T10:25:06.965451+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:18:09.206478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:09.206478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"355'13412","reported_seq":13820,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.939565+0000","last_change":"2024-09-16T10:25:06.532996+0000","last_active":"2024-09-16T10:25:08.939565+0000","last_peered":"2024-09-16T10:25:08.939565+0000","last_clean":"2024-09-16T10:25:08.939565+0000","last_became_active":"2024-09-16T10:22:05.900726+0000","last_became_peered":"2024-09-16T10:22:05.900726+0000","last_unstale":"2024-09-16T10:25:08.939565+0000","last_undegraded":"2024-09-16T10:25:08.939565+0000","last_fullsized":"2024-09-16T10:25:08.939565+0000","mapping_epoch":284,"log_start":"334'9138","ondisk_log_start":"334'9138","created":276,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'13296","last_scrub_stamp":"2024-09-16T10:25:06.532889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:25:06.532889+0000","objects_scrubbed":876,"log_size":4274,"log_dups_size":0,"ondisk_log_size":4274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:10.986602+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807360,"num_objects":760,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13412,"num_write_kb":453504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":485,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013587+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:25:07.013587+0000","last_peered":"2024-09-16T10:25:07.013587+0000","last_clean":"2024-09-16T10:25:07.013587+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:25:07.013587+0000","last_undegraded":"2024-09-16T10:25:07.013587+0000","last_fullsized":"2024-09-16T10:25:07.013587+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:47.799329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":664,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959773+0000","last_change":"2024-09-16T10:17:49.440554+0000","last_active":"2024-09-16T10:25:06.959773+0000","last_peered":"2024-09-16T10:25:06.959773+0000","last_clean":"2024-09-16T10:25:06.959773+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:25:06.959773+0000","last_undegraded":"2024-09-16T10:25:06.959773+0000","last_fullsized":"2024-09-16T10:25:06.959773+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-16T10:17:49.440451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:49.440451+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":640,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018147+0000","last_change":"2024-09-16T10:23:03.779125+0000","last_active":"2024-09-16T10:25:07.018147+0000","last_peered":"2024-09-16T10:25:07.018147+0000","last_clean":"2024-09-16T10:25:07.018147+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:25:07.018147+0000","last_undegraded":"2024-09-16T10:25:07.018147+0000","last_fullsized":"2024-09-16T10:25:07.018147+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:03.779044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:03.779044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":771,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025642+0000","last_change":"2024-09-16T10:24:57.408285+0000","last_active":"2024-09-16T10:25:07.025642+0000","last_peered":"2024-09-16T10:25:07.025642+0000","last_clean":"2024-09-16T10:25:07.025642+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:25:07.025642+0000","last_undegraded":"2024-09-16T10:25:07.025642+0000","last_fullsized":"2024-09-16T10:25:07.025642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:57.408217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.408217+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:09.573273+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":629,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014296+0000","last_change":"2024-09-16T10:24:15.859366+0000","last_active":"2024-09-16T10:25:07.014296+0000","last_peered":"2024-09-16T10:25:07.014296+0000","last_clean":"2024-09-16T10:25:07.014296+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:25:07.014296+0000","last_undegraded":"2024-09-16T10:25:07.014296+0000","last_fullsized":"2024-09-16T10:25:07.014296+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:15.859299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.859299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:41.356094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"355'13822","reported_seq":14083,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.986125+0000","last_change":"2024-09-16T10:25:01.038303+0000","last_active":"2024-09-16T10:25:08.986125+0000","last_peered":"2024-09-16T10:25:08.986125+0000","last_clean":"2024-09-16T10:25:08.986125+0000","last_became_active":"2024-09-16T10:25:00.970440+0000","last_became_peered":"2024-09-16T10:25:00.970440+0000","last_unstale":"2024-09-16T10:25:08.986125+0000","last_undegraded":"2024-09-16T10:25:08.986125+0000","last_fullsized":"2024-09-16T10:25:08.986125+0000","mapping_epoch":351,"log_start":"334'9504","ondisk_log_start":"334'9504","created":276,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'7863","last_scrub_stamp":"2024-09-16T10:23:40.271499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.271499+0000","objects_scrubbed":6759,"log_size":4318,"log_dups_size":0,"ondisk_log_size":4318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48693272,"num_objects":744,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13822,"num_write_kb":466049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":537,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025345+0000","last_change":"2024-09-16T10:24:28.146701+0000","last_active":"2024-09-16T10:25:07.025345+0000","last_peered":"2024-09-16T10:25:07.025345+0000","last_clean":"2024-09-16T10:25:07.025345+0000","last_became_active":"2024-09-16T10:22:56.409926+0000","last_became_peered":"2024-09-16T10:22:56.409926+0000","last_unstale":"2024-09-16T10:25:07.025345+0000","last_undegraded":"2024-09-16T10:25:07.025345+0000","last_fullsized":"2024-09-16T10:25:07.025345+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.146633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.146633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:31.245684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":121,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.022719+0000","last_change":"2024-09-16T10:23:42.023903+0000","last_active":"2024-09-16T10:25:07.022719+0000","last_peered":"2024-09-16T10:25:07.022719+0000","last_clean":"2024-09-16T10:25:07.022719+0000","last_became_active":"2024-09-16T10:23:41.951456+0000","last_became_peered":"2024-09-16T10:23:41.951456+0000","last_unstale":"2024-09-16T10:25:07.022719+0000","last_undegraded":"2024-09-16T10:25:07.022719+0000","last_fullsized":"2024-09-16T10:25:07.022719+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:23:13.871560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.871560+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":605,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.019875+0000","last_change":"2024-09-16T10:23:39.072937+0000","last_active":"2024-09-16T10:25:07.019875+0000","last_peered":"2024-09-16T10:25:07.019875+0000","last_clean":"2024-09-16T10:25:07.019875+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:25:07.019875+0000","last_undegraded":"2024-09-16T10:25:07.019875+0000","last_fullsized":"2024-09-16T10:25:07.019875+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:39.070281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:39.070281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":370,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.964398+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:25:06.964398+0000","last_peered":"2024-09-16T10:25:06.964398+0000","last_clean":"2024-09-16T10:25:06.964398+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:25:06.964398+0000","last_undegraded":"2024-09-16T10:25:06.964398+0000","last_fullsized":"2024-09-16T10:25:06.964398+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":650,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024271+0000","last_change":"2024-09-16T10:24:22.206462+0000","last_active":"2024-09-16T10:25:07.024271+0000","last_peered":"2024-09-16T10:25:07.024271+0000","last_clean":"2024-09-16T10:25:07.024271+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:25:07.024271+0000","last_undegraded":"2024-09-16T10:25:07.024271+0000","last_fullsized":"2024-09-16T10:25:07.024271+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:22.206414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:22.206414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.375025+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"355'13550","reported_seq":13720,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.966408+0000","last_change":"2024-09-16T10:21:56.237148+0000","last_active":"2024-09-16T10:25:08.966408+0000","last_peered":"2024-09-16T10:25:08.966408+0000","last_clean":"2024-09-16T10:25:08.966408+0000","last_became_active":"2024-09-16T10:21:56.236822+0000","last_became_peered":"2024-09-16T10:21:56.236822+0000","last_unstale":"2024-09-16T10:25:08.966408+0000","last_undegraded":"2024-09-16T10:25:08.966408+0000","last_fullsized":"2024-09-16T10:25:08.966408+0000","mapping_epoch":276,"log_start":"334'9261","ondisk_log_start":"334'9261","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48758784,"num_objects":744,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":457408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":529,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018955+0000","last_change":"2024-09-16T10:24:28.343296+0000","last_active":"2024-09-16T10:25:07.018955+0000","last_peered":"2024-09-16T10:25:07.018955+0000","last_clean":"2024-09-16T10:25:07.018955+0000","last_became_active":"2024-09-16T10:22:56.446950+0000","last_became_peered":"2024-09-16T10:22:56.446950+0000","last_unstale":"2024-09-16T10:25:07.018955+0000","last_undegraded":"2024-09-16T10:25:07.018955+0000","last_fullsized":"2024-09-16T10:25:07.018955+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.343066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.343066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:40.703096+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":686,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024341+0000","last_change":"2024-09-16T10:24:59.344438+0000","last_active":"2024-09-16T10:25:07.024341+0000","last_peered":"2024-09-16T10:25:07.024341+0000","last_clean":"2024-09-16T10:25:07.024341+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:25:07.024341+0000","last_undegraded":"2024-09-16T10:25:07.024341+0000","last_fullsized":"2024-09-16T10:25:07.024341+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:24:59.344346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.344346+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:05.786389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":633,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960887+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:25:06.960887+0000","last_peered":"2024-09-16T10:25:06.960887+0000","last_clean":"2024-09-16T10:25:06.960887+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:25:06.960887+0000","last_undegraded":"2024-09-16T10:25:06.960887+0000","last_fullsized":"2024-09-16T10:25:06.960887+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":774,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.017765+0000","last_change":"2024-09-16T10:22:02.357682+0000","last_active":"2024-09-16T10:25:07.017765+0000","last_peered":"2024-09-16T10:25:07.017765+0000","last_clean":"2024-09-16T10:25:07.017765+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:25:07.017765+0000","last_undegraded":"2024-09-16T10:25:07.017765+0000","last_fullsized":"2024-09-16T10:25:07.017765+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:22:02.357616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.357616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:08.090604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":480,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.964907+0000","last_change":"2024-09-16T10:25:00.948343+0000","last_active":"2024-09-16T10:25:06.964907+0000","last_peered":"2024-09-16T10:25:06.964907+0000","last_clean":"2024-09-16T10:25:06.964907+0000","last_became_active":"2024-09-16T10:25:00.946867+0000","last_became_peered":"2024-09-16T10:25:00.946867+0000","last_unstale":"2024-09-16T10:25:06.964907+0000","last_undegraded":"2024-09-16T10:25:06.964907+0000","last_fullsized":"2024-09-16T10:25:06.964907+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"355'13842","reported_seq":14034,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.512518+0000","last_change":"2024-09-16T10:24:59.221124+0000","last_active":"2024-09-16T10:25:08.512518+0000","last_peered":"2024-09-16T10:25:08.512518+0000","last_clean":"2024-09-16T10:25:08.512518+0000","last_became_active":"2024-09-16T10:21:56.238825+0000","last_became_peered":"2024-09-16T10:21:56.238825+0000","last_unstale":"2024-09-16T10:25:08.512518+0000","last_undegraded":"2024-09-16T10:25:08.512518+0000","last_fullsized":"2024-09-16T10:25:08.512518+0000","mapping_epoch":276,"log_start":"334'9343","ondisk_log_start":"334'9343","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'13291","last_scrub_stamp":"2024-09-16T10:24:59.220858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.220858+0000","objects_scrubbed":1373,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:16.125056+0000","scrub_duration":441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52559872,"num_objects":802,"num_object_clones":0,"num_object_copies":2406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13842,"num_write_kb":468608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":578,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965254+0000","last_change":"2024-09-16T10:24:31.617241+0000","last_active":"2024-09-16T10:25:06.965254+0000","last_peered":"2024-09-16T10:25:06.965254+0000","last_clean":"2024-09-16T10:25:06.965254+0000","last_became_active":"2024-09-16T10:22:56.438108+0000","last_became_peered":"2024-09-16T10:22:56.438108+0000","last_unstale":"2024-09-16T10:25:06.965254+0000","last_undegraded":"2024-09-16T10:25:06.965254+0000","last_fullsized":"2024-09-16T10:25:06.965254+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.617190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.617190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.684308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":675,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960561+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:25:06.960561+0000","last_peered":"2024-09-16T10:25:06.960561+0000","last_clean":"2024-09-16T10:25:06.960561+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:25:06.960561+0000","last_undegraded":"2024-09-16T10:25:06.960561+0000","last_fullsized":"2024-09-16T10:25:06.960561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":655,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014696+0000","last_change":"2024-09-16T10:24:59.820337+0000","last_active":"2024-09-16T10:25:07.014696+0000","last_peered":"2024-09-16T10:25:07.014696+0000","last_clean":"2024-09-16T10:25:07.014696+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:25:07.014696+0000","last_undegraded":"2024-09-16T10:25:07.014696+0000","last_fullsized":"2024-09-16T10:25:07.014696+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:59.820273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.820273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.866898+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":763,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960953+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:25:06.960953+0000","last_peered":"2024-09-16T10:25:06.960953+0000","last_clean":"2024-09-16T10:25:06.960953+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:25:06.960953+0000","last_undegraded":"2024-09-16T10:25:06.960953+0000","last_fullsized":"2024-09-16T10:25:06.960953+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":642,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.961058+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:25:06.961058+0000","last_peered":"2024-09-16T10:25:06.961058+0000","last_clean":"2024-09-16T10:25:06.961058+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:25:06.961058+0000","last_undegraded":"2024-09-16T10:25:06.961058+0000","last_fullsized":"2024-09-16T10:25:06.961058+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"355'14164","reported_seq":14380,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.238325+0000","last_change":"2024-09-16T10:25:00.979711+0000","last_active":"2024-09-16T10:25:08.238325+0000","last_peered":"2024-09-16T10:25:08.238325+0000","last_clean":"2024-09-16T10:25:08.238325+0000","last_became_active":"2024-09-16T10:25:00.959191+0000","last_became_peered":"2024-09-16T10:25:00.959191+0000","last_unstale":"2024-09-16T10:25:08.238325+0000","last_undegraded":"2024-09-16T10:25:08.238325+0000","last_fullsized":"2024-09-16T10:25:08.238325+0000","mapping_epoch":351,"log_start":"334'9684","ondisk_log_start":"334'9684","created":276,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11867","last_scrub_stamp":"2024-09-16T10:24:35.105311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:24:35.105311+0000","objects_scrubbed":3142,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:53.928566+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55181312,"num_objects":842,"num_object_clones":0,"num_object_copies":2526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14164,"num_write_kb":480192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":583,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965054+0000","last_change":"2024-09-16T10:24:30.609351+0000","last_active":"2024-09-16T10:25:06.965054+0000","last_peered":"2024-09-16T10:25:06.965054+0000","last_clean":"2024-09-16T10:25:06.965054+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:25:06.965054+0000","last_undegraded":"2024-09-16T10:25:06.965054+0000","last_fullsized":"2024-09-16T10:25:06.965054+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:30.609302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:30.609302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:41.777744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":687,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960585+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:25:06.960585+0000","last_peered":"2024-09-16T10:25:06.960585+0000","last_clean":"2024-09-16T10:25:06.960585+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:25:06.960585+0000","last_undegraded":"2024-09-16T10:25:06.960585+0000","last_fullsized":"2024-09-16T10:25:06.960585+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":655,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.961168+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:25:06.961168+0000","last_peered":"2024-09-16T10:25:06.961168+0000","last_clean":"2024-09-16T10:25:06.961168+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:25:06.961168+0000","last_undegraded":"2024-09-16T10:25:06.961168+0000","last_fullsized":"2024-09-16T10:25:06.961168+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":640,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.964671+0000","last_change":"2024-09-16T10:22:28.126110+0000","last_active":"2024-09-16T10:25:06.964671+0000","last_peered":"2024-09-16T10:25:06.964671+0000","last_clean":"2024-09-16T10:25:06.964671+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:25:06.964671+0000","last_undegraded":"2024-09-16T10:25:06.964671+0000","last_fullsized":"2024-09-16T10:25:06.964671+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:28.126042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:28.126042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":635,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959016+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:25:06.959016+0000","last_peered":"2024-09-16T10:25:06.959016+0000","last_clean":"2024-09-16T10:25:06.959016+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:25:06.959016+0000","last_undegraded":"2024-09-16T10:25:06.959016+0000","last_fullsized":"2024-09-16T10:25:06.959016+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"355'13983","reported_seq":14153,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.292586+0000","last_change":"2024-09-16T10:21:56.233704+0000","last_active":"2024-09-16T10:25:09.292586+0000","last_peered":"2024-09-16T10:25:09.292586+0000","last_clean":"2024-09-16T10:25:09.292586+0000","last_became_active":"2024-09-16T10:21:56.233167+0000","last_became_peered":"2024-09-16T10:21:56.233167+0000","last_unstale":"2024-09-16T10:25:09.292586+0000","last_undegraded":"2024-09-16T10:25:09.292586+0000","last_fullsized":"2024-09-16T10:25:09.292586+0000","mapping_epoch":276,"log_start":"329'8595","ondisk_log_start":"329'8595","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5388,"log_dups_size":0,"ondisk_log_size":5388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50921472,"num_objects":777,"num_object_clones":0,"num_object_copies":2331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13983,"num_write_kb":472320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":529,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013505+0000","last_change":"2024-09-16T10:24:29.815138+0000","last_active":"2024-09-16T10:25:07.013505+0000","last_peered":"2024-09-16T10:25:07.013505+0000","last_clean":"2024-09-16T10:25:07.013505+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:25:07.013505+0000","last_undegraded":"2024-09-16T10:25:07.013505+0000","last_fullsized":"2024-09-16T10:25:07.013505+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:29.814966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:29.814966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:30.793164+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":908,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959607+0000","last_change":"2024-09-16T10:22:04.554377+0000","last_active":"2024-09-16T10:25:06.959607+0000","last_peered":"2024-09-16T10:25:06.959607+0000","last_clean":"2024-09-16T10:25:06.959607+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:25:06.959607+0000","last_undegraded":"2024-09-16T10:25:06.959607+0000","last_fullsized":"2024-09-16T10:25:06.959607+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'30","last_scrub_stamp":"2024-09-16T10:22:04.554260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.554260+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":648,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.023767+0000","last_change":"2024-09-16T10:24:13.097213+0000","last_active":"2024-09-16T10:25:07.023767+0000","last_peered":"2024-09-16T10:25:07.023767+0000","last_clean":"2024-09-16T10:25:07.023767+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:25:07.023767+0000","last_undegraded":"2024-09-16T10:25:07.023767+0000","last_fullsized":"2024-09-16T10:25:07.023767+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:13.097133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:13.097133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:38.312420+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":643,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014023+0000","last_change":"2024-09-16T10:24:57.812754+0000","last_active":"2024-09-16T10:25:07.014023+0000","last_peered":"2024-09-16T10:25:07.014023+0000","last_clean":"2024-09-16T10:25:07.014023+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:25:07.014023+0000","last_undegraded":"2024-09-16T10:25:07.014023+0000","last_fullsized":"2024-09-16T10:25:07.014023+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:24:57.812701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.812701+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:59.841010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":616,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960484+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:25:06.960484+0000","last_peered":"2024-09-16T10:25:06.960484+0000","last_clean":"2024-09-16T10:25:06.960484+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:25:06.960484+0000","last_undegraded":"2024-09-16T10:25:06.960484+0000","last_fullsized":"2024-09-16T10:25:06.960484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":636,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018936+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:25:07.018936+0000","last_peered":"2024-09-16T10:25:07.018936+0000","last_clean":"2024-09-16T10:25:07.018936+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:25:07.018936+0000","last_undegraded":"2024-09-16T10:25:07.018936+0000","last_fullsized":"2024-09-16T10:25:07.018936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"355'13442","reported_seq":13612,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.517644+0000","last_change":"2024-09-16T10:21:56.236854+0000","last_active":"2024-09-16T10:25:08.517644+0000","last_peered":"2024-09-16T10:25:08.517644+0000","last_clean":"2024-09-16T10:25:08.517644+0000","last_became_active":"2024-09-16T10:21:56.236183+0000","last_became_peered":"2024-09-16T10:21:56.236183+0000","last_unstale":"2024-09-16T10:25:08.517644+0000","last_undegraded":"2024-09-16T10:25:08.517644+0000","last_fullsized":"2024-09-16T10:25:08.517644+0000","mapping_epoch":276,"log_start":"333'8962","ondisk_log_start":"333'8962","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:41.854826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51380224,"num_objects":784,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13442,"num_write_kb":455232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":102,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959700+0000","last_change":"2024-09-16T10:25:02.365887+0000","last_active":"2024-09-16T10:25:06.959700+0000","last_peered":"2024-09-16T10:25:06.959700+0000","last_clean":"2024-09-16T10:25:06.959700+0000","last_became_active":"2024-09-16T10:25:00.955466+0000","last_became_peered":"2024-09-16T10:25:00.955466+0000","last_unstale":"2024-09-16T10:25:06.959700+0000","last_undegraded":"2024-09-16T10:25:06.959700+0000","last_fullsized":"2024-09-16T10:25:06.959700+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.096159+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":646,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960472+0000","last_change":"2024-09-16T10:17:50.451840+0000","last_active":"2024-09-16T10:25:06.960472+0000","last_peered":"2024-09-16T10:25:06.960472+0000","last_clean":"2024-09-16T10:25:06.960472+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:25:06.960472+0000","last_undegraded":"2024-09-16T10:25:06.960472+0000","last_fullsized":"2024-09-16T10:25:06.960472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-16T10:17:50.451738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:50.451738+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":639,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959522+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:25:06.959522+0000","last_peered":"2024-09-16T10:25:06.959522+0000","last_clean":"2024-09-16T10:25:06.959522+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:25:06.959522+0000","last_undegraded":"2024-09-16T10:25:06.959522+0000","last_fullsized":"2024-09-16T10:25:06.959522+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":163,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962418+0000","last_change":"2024-09-16T10:22:02.917475+0000","last_active":"2024-09-16T10:25:06.962418+0000","last_peered":"2024-09-16T10:25:06.962418+0000","last_clean":"2024-09-16T10:25:06.962418+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:25:06.962418+0000","last_undegraded":"2024-09-16T10:25:06.962418+0000","last_fullsized":"2024-09-16T10:25:06.962418+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:58.199062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:19:58.199062+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":167,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962931+0000","last_change":"2024-09-16T10:22:29.905776+0000","last_active":"2024-09-16T10:25:06.962931+0000","last_peered":"2024-09-16T10:25:06.962931+0000","last_clean":"2024-09-16T10:25:06.962931+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:25:06.962931+0000","last_undegraded":"2024-09-16T10:25:06.962931+0000","last_fullsized":"2024-09-16T10:25:06.962931+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":20,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965046+0000","last_change":"2024-09-16T10:25:00.950449+0000","last_active":"2024-09-16T10:25:06.965046+0000","last_peered":"2024-09-16T10:25:06.965046+0000","last_clean":"2024-09-16T10:25:06.965046+0000","last_became_active":"2024-09-16T10:25:00.950144+0000","last_became_peered":"2024-09-16T10:25:00.950144+0000","last_unstale":"2024-09-16T10:25:06.965046+0000","last_undegraded":"2024-09-16T10:25:06.965046+0000","last_fullsized":"2024-09-16T10:25:06.965046+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.105178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.105178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"355'13595","reported_seq":25106,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.818418+0000","last_change":"2024-09-16T10:25:07.008445+0000","last_active":"2024-09-16T10:25:08.818418+0000","last_peered":"2024-09-16T10:25:08.818418+0000","last_clean":"2024-09-16T10:25:08.818418+0000","last_became_active":"2024-09-16T10:25:06.965441+0000","last_became_peered":"2024-09-16T10:25:06.965441+0000","last_unstale":"2024-09-16T10:25:08.818418+0000","last_undegraded":"2024-09-16T10:25:08.818418+0000","last_fullsized":"2024-09-16T10:25:08.818418+0000","mapping_epoch":354,"log_start":"327'8111","ondisk_log_start":"327'8111","created":276,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49741824,"num_objects":759,"num_object_clones":0,"num_object_copies":2277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13595,"num_write_kb":459328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3776,"num_bytes_recovered":145489920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":517,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960370+0000","last_change":"2024-09-16T10:24:27.887030+0000","last_active":"2024-09-16T10:25:06.960370+0000","last_peered":"2024-09-16T10:25:06.960370+0000","last_clean":"2024-09-16T10:25:06.960370+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:25:06.960370+0000","last_undegraded":"2024-09-16T10:25:06.960370+0000","last_fullsized":"2024-09-16T10:25:06.960370+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:27.886961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:27.886961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.749309+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":730,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960687+0000","last_change":"2024-09-16T10:22:24.213089+0000","last_active":"2024-09-16T10:25:06.960687+0000","last_peered":"2024-09-16T10:25:06.960687+0000","last_clean":"2024-09-16T10:25:06.960687+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:25:06.960687+0000","last_undegraded":"2024-09-16T10:25:06.960687+0000","last_fullsized":"2024-09-16T10:25:06.960687+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:22:24.213008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.213008+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":656,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960003+0000","last_change":"2024-09-16T10:22:26.226703+0000","last_active":"2024-09-16T10:25:06.960003+0000","last_peered":"2024-09-16T10:25:06.960003+0000","last_clean":"2024-09-16T10:25:06.960003+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:25:06.960003+0000","last_undegraded":"2024-09-16T10:25:06.960003+0000","last_fullsized":"2024-09-16T10:25:06.960003+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.226632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.226632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":70,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.029797+0000","last_change":"2024-09-16T10:25:00.956734+0000","last_active":"2024-09-16T10:25:07.029797+0000","last_peered":"2024-09-16T10:25:07.029797+0000","last_clean":"2024-09-16T10:25:07.029797+0000","last_became_active":"2024-09-16T10:25:00.956304+0000","last_became_peered":"2024-09-16T10:25:00.956304+0000","last_unstale":"2024-09-16T10:25:07.029797+0000","last_undegraded":"2024-09-16T10:25:07.029797+0000","last_fullsized":"2024-09-16T10:25:07.029797+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:23:41.597682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.597682+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":633,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014974+0000","last_change":"2024-09-16T10:24:16.815896+0000","last_active":"2024-09-16T10:25:07.014974+0000","last_peered":"2024-09-16T10:25:07.014974+0000","last_clean":"2024-09-16T10:25:07.014974+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:25:07.014974+0000","last_undegraded":"2024-09-16T10:25:07.014974+0000","last_fullsized":"2024-09-16T10:25:07.014974+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:16.815831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:16.815831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:30.569240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":637,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959262+0000","last_change":"2024-09-16T10:21:58.547682+0000","last_active":"2024-09-16T10:25:06.959262+0000","last_peered":"2024-09-16T10:25:06.959262+0000","last_clean":"2024-09-16T10:25:06.959262+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:25:06.959262+0000","last_undegraded":"2024-09-16T10:25:06.959262+0000","last_fullsized":"2024-09-16T10:25:06.959262+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.547634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.547634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"355'13837","reported_seq":14081,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.969335+0000","last_change":"2024-09-16T10:23:45.220344+0000","last_active":"2024-09-16T10:25:08.969335+0000","last_peered":"2024-09-16T10:25:08.969335+0000","last_clean":"2024-09-16T10:25:08.969335+0000","last_became_active":"2024-09-16T10:21:56.236070+0000","last_became_peered":"2024-09-16T10:21:56.236070+0000","last_unstale":"2024-09-16T10:25:08.969335+0000","last_undegraded":"2024-09-16T10:25:08.969335+0000","last_fullsized":"2024-09-16T10:25:08.969335+0000","mapping_epoch":276,"log_start":"334'9584","ondisk_log_start":"334'9584","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8222","last_scrub_stamp":"2024-09-16T10:23:45.219929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:23:45.219929+0000","objects_scrubbed":6450,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50921472,"num_objects":777,"num_object_clones":0,"num_object_copies":2331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13837,"num_write_kb":467648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"328'28","reported_seq":799,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965223+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:25:06.965223+0000","last_peered":"2024-09-16T10:25:06.965223+0000","last_clean":"2024-09-16T10:25:06.965223+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:25:06.965223+0000","last_undegraded":"2024-09-16T10:25:06.965223+0000","last_fullsized":"2024-09-16T10:25:06.965223+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":605,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018548+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:25:07.018548+0000","last_peered":"2024-09-16T10:25:07.018548+0000","last_clean":"2024-09-16T10:25:07.018548+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:25:07.018548+0000","last_undegraded":"2024-09-16T10:25:07.018548+0000","last_fullsized":"2024-09-16T10:25:07.018548+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.609525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":102,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.961277+0000","last_change":"2024-09-16T10:25:01.375800+0000","last_active":"2024-09-16T10:25:06.961277+0000","last_peered":"2024-09-16T10:25:06.961277+0000","last_clean":"2024-09-16T10:25:06.961277+0000","last_became_active":"2024-09-16T10:25:00.955671+0000","last_became_peered":"2024-09-16T10:25:00.955671+0000","last_unstale":"2024-09-16T10:25:06.961277+0000","last_undegraded":"2024-09-16T10:25:06.961277+0000","last_fullsized":"2024-09-16T10:25:06.961277+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:09.067810+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":492,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962922+0000","last_change":"2024-09-16T10:25:00.947058+0000","last_active":"2024-09-16T10:25:06.962922+0000","last_peered":"2024-09-16T10:25:06.962922+0000","last_clean":"2024-09-16T10:25:06.962922+0000","last_became_active":"2024-09-16T10:25:00.946698+0000","last_became_peered":"2024-09-16T10:25:00.946698+0000","last_unstale":"2024-09-16T10:25:06.962922+0000","last_undegraded":"2024-09-16T10:25:06.962922+0000","last_fullsized":"2024-09-16T10:25:06.962922+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:16.615588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:16.615588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:20.583422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":629,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014867+0000","last_change":"2024-09-16T10:24:33.905860+0000","last_active":"2024-09-16T10:25:07.014867+0000","last_peered":"2024-09-16T10:25:07.014867+0000","last_clean":"2024-09-16T10:25:07.014867+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:25:07.014867+0000","last_undegraded":"2024-09-16T10:25:07.014867+0000","last_fullsized":"2024-09-16T10:25:07.014867+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.905785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.905785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:01.164716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":524,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959182+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:25:06.959182+0000","last_peered":"2024-09-16T10:25:06.959182+0000","last_clean":"2024-09-16T10:25:06.959182+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:25:06.959182+0000","last_undegraded":"2024-09-16T10:25:06.959182+0000","last_fullsized":"2024-09-16T10:25:06.959182+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"355'13939","reported_seq":14133,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.949488+0000","last_change":"2024-09-16T10:25:04.890736+0000","last_active":"2024-09-16T10:25:08.949488+0000","last_peered":"2024-09-16T10:25:08.949488+0000","last_clean":"2024-09-16T10:25:08.949488+0000","last_became_active":"2024-09-16T10:25:00.962918+0000","last_became_peered":"2024-09-16T10:25:00.962918+0000","last_unstale":"2024-09-16T10:25:08.949488+0000","last_undegraded":"2024-09-16T10:25:08.949488+0000","last_fullsized":"2024-09-16T10:25:08.949488+0000","mapping_epoch":351,"log_start":"334'9705","ondisk_log_start":"334'9705","created":276,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'13728","last_scrub_stamp":"2024-09-16T10:25:04.888668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.888668+0000","objects_scrubbed":1030,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:26.751470+0000","scrub_duration":496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52625408,"num_objects":803,"num_object_clones":0,"num_object_copies":2409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13939,"num_write_kb":471744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":686,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025606+0000","last_change":"2024-09-16T10:24:34.142042+0000","last_active":"2024-09-16T10:25:07.025606+0000","last_peered":"2024-09-16T10:25:07.025606+0000","last_clean":"2024-09-16T10:25:07.025606+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:25:07.025606+0000","last_undegraded":"2024-09-16T10:25:07.025606+0000","last_fullsized":"2024-09-16T10:25:07.025606+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:24:34.141958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.141958+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.626228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":646,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024701+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:25:07.024701+0000","last_peered":"2024-09-16T10:25:07.024701+0000","last_clean":"2024-09-16T10:25:07.024701+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:25:07.024701+0000","last_undegraded":"2024-09-16T10:25:07.024701+0000","last_fullsized":"2024-09-16T10:25:07.024701+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.235293+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":641,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014780+0000","last_change":"2024-09-16T10:25:05.858177+0000","last_active":"2024-09-16T10:25:07.014780+0000","last_peered":"2024-09-16T10:25:07.014780+0000","last_clean":"2024-09-16T10:25:07.014780+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:25:07.014780+0000","last_undegraded":"2024-09-16T10:25:07.014780+0000","last_fullsized":"2024-09-16T10:25:07.014780+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.858129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.858129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:12.657363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":576,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018750+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:25:07.018750+0000","last_peered":"2024-09-16T10:25:07.018750+0000","last_clean":"2024-09-16T10:25:07.018750+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:25:07.018750+0000","last_undegraded":"2024-09-16T10:25:07.018750+0000","last_fullsized":"2024-09-16T10:25:07.018750+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:56.465984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"355'13620","reported_seq":13989,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.964410+0000","last_change":"2024-09-16T10:22:03.098658+0000","last_active":"2024-09-16T10:25:08.964410+0000","last_peered":"2024-09-16T10:25:08.964410+0000","last_clean":"2024-09-16T10:25:08.964410+0000","last_became_active":"2024-09-16T10:22:02.896249+0000","last_became_peered":"2024-09-16T10:22:02.896249+0000","last_unstale":"2024-09-16T10:25:08.964410+0000","last_undegraded":"2024-09-16T10:25:08.964410+0000","last_fullsized":"2024-09-16T10:25:08.964410+0000","mapping_epoch":281,"log_start":"334'9316","ondisk_log_start":"334'9316","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49414144,"num_objects":754,"num_object_clones":0,"num_object_copies":2262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13620,"num_write_kb":459968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":492,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013388+0000","last_change":"2024-09-16T10:25:04.820847+0000","last_active":"2024-09-16T10:25:07.013388+0000","last_peered":"2024-09-16T10:25:07.013388+0000","last_clean":"2024-09-16T10:25:07.013388+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:25:07.013388+0000","last_undegraded":"2024-09-16T10:25:07.013388+0000","last_fullsized":"2024-09-16T10:25:07.013388+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:04.820775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.820775+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.996474+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":735,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025284+0000","last_change":"2024-09-16T10:23:09.748114+0000","last_active":"2024-09-16T10:25:07.025284+0000","last_peered":"2024-09-16T10:25:07.025284+0000","last_clean":"2024-09-16T10:25:07.025284+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:25:07.025284+0000","last_undegraded":"2024-09-16T10:25:07.025284+0000","last_fullsized":"2024-09-16T10:25:07.025284+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:23:09.748031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:09.748031+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":632,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965091+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:25:06.965091+0000","last_peered":"2024-09-16T10:25:06.965091+0000","last_clean":"2024-09-16T10:25:06.965091+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:25:06.965091+0000","last_undegraded":"2024-09-16T10:25:06.965091+0000","last_fullsized":"2024-09-16T10:25:06.965091+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":625,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959268+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:25:06.959268+0000","last_peered":"2024-09-16T10:25:06.959268+0000","last_clean":"2024-09-16T10:25:06.959268+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:25:06.959268+0000","last_undegraded":"2024-09-16T10:25:06.959268+0000","last_fullsized":"2024-09-16T10:25:06.959268+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":167,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962660+0000","last_change":"2024-09-16T10:22:05.694026+0000","last_active":"2024-09-16T10:25:06.962660+0000","last_peered":"2024-09-16T10:25:06.962660+0000","last_clean":"2024-09-16T10:25:06.962660+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:25:06.962660+0000","last_undegraded":"2024-09-16T10:25:06.962660+0000","last_fullsized":"2024-09-16T10:25:06.962660+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.693972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.693972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"355'13531","reported_seq":13717,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.963142+0000","last_change":"2024-09-16T10:25:05.724671+0000","last_active":"2024-09-16T10:25:08.963142+0000","last_peered":"2024-09-16T10:25:08.963142+0000","last_clean":"2024-09-16T10:25:08.963142+0000","last_became_active":"2024-09-16T10:21:56.913186+0000","last_became_peered":"2024-09-16T10:21:56.913186+0000","last_unstale":"2024-09-16T10:25:08.963142+0000","last_undegraded":"2024-09-16T10:25:08.963142+0000","last_fullsized":"2024-09-16T10:25:08.963142+0000","mapping_epoch":276,"log_start":"334'9218","ondisk_log_start":"334'9218","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'13403","last_scrub_stamp":"2024-09-16T10:25:05.724398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.724398+0000","objects_scrubbed":837,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:12.152754+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45940736,"num_objects":701,"num_object_clones":0,"num_object_copies":2103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13531,"num_write_kb":455424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":295,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.016852+0000","last_change":"2024-09-16T10:25:01.481822+0000","last_active":"2024-09-16T10:25:07.016852+0000","last_peered":"2024-09-16T10:25:07.016852+0000","last_clean":"2024-09-16T10:25:07.016852+0000","last_became_active":"2024-09-16T10:25:00.955829+0000","last_became_peered":"2024-09-16T10:25:00.955829+0000","last_unstale":"2024-09-16T10:25:07.016852+0000","last_undegraded":"2024-09-16T10:25:07.016852+0000","last_fullsized":"2024-09-16T10:25:07.016852+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.481748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.481748+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.563183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":650,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024926+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:25:07.024926+0000","last_peered":"2024-09-16T10:25:07.024926+0000","last_clean":"2024-09-16T10:25:07.024926+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:25:07.024926+0000","last_undegraded":"2024-09-16T10:25:07.024926+0000","last_fullsized":"2024-09-16T10:25:07.024926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:49.328998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":624,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.017437+0000","last_change":"2024-09-16T10:24:25.364236+0000","last_active":"2024-09-16T10:25:07.017437+0000","last_peered":"2024-09-16T10:25:07.017437+0000","last_clean":"2024-09-16T10:25:07.017437+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:25:07.017437+0000","last_undegraded":"2024-09-16T10:25:07.017437+0000","last_fullsized":"2024-09-16T10:25:07.017437+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:25.364176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:25.364176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.649906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":209,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962744+0000","last_change":"2024-09-16T10:22:24.799901+0000","last_active":"2024-09-16T10:25:06.962744+0000","last_peered":"2024-09-16T10:25:06.962744+0000","last_clean":"2024-09-16T10:25:06.962744+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:25:06.962744+0000","last_undegraded":"2024-09-16T10:25:06.962744+0000","last_fullsized":"2024-09-16T10:25:06.962744+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":642,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025304+0000","last_change":"2024-09-16T10:22:31.886830+0000","last_active":"2024-09-16T10:25:07.025304+0000","last_peered":"2024-09-16T10:25:07.025304+0000","last_clean":"2024-09-16T10:25:07.025304+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:25:07.025304+0000","last_undegraded":"2024-09-16T10:25:07.025304+0000","last_fullsized":"2024-09-16T10:25:07.025304+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:31.886666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:31.886666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":204,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024836+0000","last_change":"2024-09-16T10:24:32.158409+0000","last_active":"2024-09-16T10:25:07.024836+0000","last_peered":"2024-09-16T10:25:07.024836+0000","last_clean":"2024-09-16T10:25:07.024836+0000","last_became_active":"2024-09-16T10:23:31.480326+0000","last_became_peered":"2024-09-16T10:23:31.480326+0000","last_unstale":"2024-09-16T10:25:07.024836+0000","last_undegraded":"2024-09-16T10:25:07.024836+0000","last_fullsized":"2024-09-16T10:25:07.024836+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.158356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.158356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.417298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"355'13670","reported_seq":14100,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.498840+0000","last_change":"2024-09-16T10:23:45.812894+0000","last_active":"2024-09-16T10:25:08.498840+0000","last_peered":"2024-09-16T10:25:08.498840+0000","last_clean":"2024-09-16T10:25:08.498840+0000","last_became_active":"2024-09-16T10:22:02.898446+0000","last_became_peered":"2024-09-16T10:22:02.898446+0000","last_unstale":"2024-09-16T10:25:08.498840+0000","last_undegraded":"2024-09-16T10:25:08.498840+0000","last_fullsized":"2024-09-16T10:25:08.498840+0000","mapping_epoch":281,"log_start":"333'9133","ondisk_log_start":"333'9133","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8189","last_scrub_stamp":"2024-09-16T10:23:45.812693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:23:45.812693+0000","objects_scrubbed":6357,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53084160,"num_objects":810,"num_object_clones":0,"num_object_copies":2430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13670,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":5898240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":645,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960485+0000","last_change":"2024-09-16T10:17:04.087026+0000","last_active":"2024-09-16T10:25:06.960485+0000","last_peered":"2024-09-16T10:25:06.960485+0000","last_clean":"2024-09-16T10:25:06.960485+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:25:06.960485+0000","last_undegraded":"2024-09-16T10:25:06.960485+0000","last_fullsized":"2024-09-16T10:25:06.960485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:17:04.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:04.086856+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":641,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014232+0000","last_change":"2024-09-16T10:21:57.915828+0000","last_active":"2024-09-16T10:25:07.014232+0000","last_peered":"2024-09-16T10:25:07.014232+0000","last_clean":"2024-09-16T10:25:07.014232+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:25:07.014232+0000","last_undegraded":"2024-09-16T10:25:07.014232+0000","last_fullsized":"2024-09-16T10:25:07.014232+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.915773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.915773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":159,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962480+0000","last_change":"2024-09-16T10:22:02.896967+0000","last_active":"2024-09-16T10:25:06.962480+0000","last_peered":"2024-09-16T10:25:06.962480+0000","last_clean":"2024-09-16T10:25:06.962480+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:25:06.962480+0000","last_undegraded":"2024-09-16T10:25:06.962480+0000","last_fullsized":"2024-09-16T10:25:06.962480+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:18.748377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:18.748377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":657,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959908+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:25:06.959908+0000","last_peered":"2024-09-16T10:25:06.959908+0000","last_clean":"2024-09-16T10:25:06.959908+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:25:06.959908+0000","last_undegraded":"2024-09-16T10:25:06.959908+0000","last_fullsized":"2024-09-16T10:25:06.959908+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":595,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965466+0000","last_change":"2024-09-16T10:24:33.610134+0000","last_active":"2024-09-16T10:25:06.965466+0000","last_peered":"2024-09-16T10:25:06.965466+0000","last_clean":"2024-09-16T10:25:06.965466+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:25:06.965466+0000","last_undegraded":"2024-09-16T10:25:06.965466+0000","last_fullsized":"2024-09-16T10:25:06.965466+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:33.610076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.610076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.151277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"355'13963","reported_seq":14205,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.516897+0000","last_change":"2024-09-16T10:23:46.521966+0000","last_active":"2024-09-16T10:25:08.516897+0000","last_peered":"2024-09-16T10:25:08.516897+0000","last_clean":"2024-09-16T10:25:08.516897+0000","last_became_active":"2024-09-16T10:21:56.234401+0000","last_became_peered":"2024-09-16T10:21:56.234401+0000","last_unstale":"2024-09-16T10:25:08.516897+0000","last_undegraded":"2024-09-16T10:25:08.516897+0000","last_fullsized":"2024-09-16T10:25:08.516897+0000","mapping_epoch":276,"log_start":"334'9471","ondisk_log_start":"334'9471","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'8392","last_scrub_stamp":"2024-09-16T10:23:46.521867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.521867+0000","objects_scrubbed":6487,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:09.980599+0000","scrub_duration":1665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56819712,"num_objects":867,"num_object_clones":0,"num_object_copies":2601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13963,"num_write_kb":474560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":699,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.964668+0000","last_change":"2024-09-16T10:22:26.050158+0000","last_active":"2024-09-16T10:25:06.964668+0000","last_peered":"2024-09-16T10:25:06.964668+0000","last_clean":"2024-09-16T10:25:06.964668+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:25:06.964668+0000","last_undegraded":"2024-09-16T10:25:06.964668+0000","last_fullsized":"2024-09-16T10:25:06.964668+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:22:26.050070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.050070+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014454+0000","last_change":"2024-09-16T10:25:00.947814+0000","last_active":"2024-09-16T10:25:07.014454+0000","last_peered":"2024-09-16T10:25:07.014454+0000","last_clean":"2024-09-16T10:25:07.014454+0000","last_became_active":"2024-09-16T10:25:00.947502+0000","last_became_peered":"2024-09-16T10:25:00.947502+0000","last_unstale":"2024-09-16T10:25:07.014454+0000","last_undegraded":"2024-09-16T10:25:07.014454+0000","last_fullsized":"2024-09-16T10:25:07.014454+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.528600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.528600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.404499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":167,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962748+0000","last_change":"2024-09-16T10:22:26.818522+0000","last_active":"2024-09-16T10:25:06.962748+0000","last_peered":"2024-09-16T10:25:06.962748+0000","last_clean":"2024-09-16T10:25:06.962748+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:25:06.962748+0000","last_undegraded":"2024-09-16T10:25:06.962748+0000","last_fullsized":"2024-09-16T10:25:06.962748+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.818439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.818439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":496,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962239+0000","last_change":"2024-09-16T10:25:00.954355+0000","last_active":"2024-09-16T10:25:06.962239+0000","last_peered":"2024-09-16T10:25:06.962239+0000","last_clean":"2024-09-16T10:25:06.962239+0000","last_became_active":"2024-09-16T10:25:00.954013+0000","last_became_peered":"2024-09-16T10:25:00.954013+0000","last_unstale":"2024-09-16T10:25:06.962239+0000","last_undegraded":"2024-09-16T10:25:06.962239+0000","last_fullsized":"2024-09-16T10:25:06.962239+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.651643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.651643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":547,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018416+0000","last_change":"2024-09-16T10:24:31.490967+0000","last_active":"2024-09-16T10:25:07.018416+0000","last_peered":"2024-09-16T10:25:07.018416+0000","last_clean":"2024-09-16T10:25:07.018416+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:25:07.018416+0000","last_undegraded":"2024-09-16T10:25:07.018416+0000","last_fullsized":"2024-09-16T10:25:07.018416+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.490890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.490890+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:53.134247+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"355'13822","reported_seq":13992,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.975720+0000","last_change":"2024-09-16T10:21:56.235273+0000","last_active":"2024-09-16T10:25:08.975720+0000","last_peered":"2024-09-16T10:25:08.975720+0000","last_clean":"2024-09-16T10:25:08.975720+0000","last_became_active":"2024-09-16T10:21:56.234987+0000","last_became_peered":"2024-09-16T10:21:56.234987+0000","last_unstale":"2024-09-16T10:25:08.975720+0000","last_undegraded":"2024-09-16T10:25:08.975720+0000","last_fullsized":"2024-09-16T10:25:08.975720+0000","mapping_epoch":276,"log_start":"334'9448","ondisk_log_start":"334'9448","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13822,"num_write_kb":465920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":660,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024566+0000","last_change":"2024-09-16T10:23:41.811222+0000","last_active":"2024-09-16T10:25:07.024566+0000","last_peered":"2024-09-16T10:25:07.024566+0000","last_clean":"2024-09-16T10:25:07.024566+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:25:07.024566+0000","last_undegraded":"2024-09-16T10:25:07.024566+0000","last_fullsized":"2024-09-16T10:25:07.024566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:41.811140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.811140+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:09.290875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":27,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025760+0000","last_change":"2024-09-16T10:25:05.405432+0000","last_active":"2024-09-16T10:25:07.025760+0000","last_peered":"2024-09-16T10:25:07.025760+0000","last_clean":"2024-09-16T10:25:07.025760+0000","last_became_active":"2024-09-16T10:25:00.959190+0000","last_became_peered":"2024-09-16T10:25:00.959190+0000","last_unstale":"2024-09-16T10:25:07.025760+0000","last_undegraded":"2024-09-16T10:25:07.025760+0000","last_fullsized":"2024-09-16T10:25:07.025760+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.405338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.405338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.454239+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":167,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962341+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:25:06.962341+0000","last_peered":"2024-09-16T10:25:06.962341+0000","last_clean":"2024-09-16T10:25:06.962341+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:25:06.962341+0000","last_undegraded":"2024-09-16T10:25:06.962341+0000","last_fullsized":"2024-09-16T10:25:06.962341+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":628,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.017024+0000","last_change":"2024-09-16T10:24:20.366257+0000","last_active":"2024-09-16T10:25:07.017024+0000","last_peered":"2024-09-16T10:25:07.017024+0000","last_clean":"2024-09-16T10:25:07.017024+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:25:07.017024+0000","last_undegraded":"2024-09-16T10:25:07.017024+0000","last_fullsized":"2024-09-16T10:25:07.017024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:20.366122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:20.366122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:26.488192+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":444,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025097+0000","last_change":"2024-09-16T10:25:01.368064+0000","last_active":"2024-09-16T10:25:07.025097+0000","last_peered":"2024-09-16T10:25:07.025097+0000","last_clean":"2024-09-16T10:25:07.025097+0000","last_became_active":"2024-09-16T10:25:00.960443+0000","last_became_peered":"2024-09-16T10:25:00.960443+0000","last_unstale":"2024-09-16T10:25:07.025097+0000","last_undegraded":"2024-09-16T10:25:07.025097+0000","last_fullsized":"2024-09-16T10:25:07.025097+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.367979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.367979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.643741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"355'13914","reported_seq":14083,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.317494+0000","last_change":"2024-09-16T10:21:56.236925+0000","last_active":"2024-09-16T10:25:09.317494+0000","last_peered":"2024-09-16T10:25:09.317494+0000","last_clean":"2024-09-16T10:25:09.317494+0000","last_became_active":"2024-09-16T10:21:56.236619+0000","last_became_peered":"2024-09-16T10:21:56.236619+0000","last_unstale":"2024-09-16T10:25:09.317494+0000","last_undegraded":"2024-09-16T10:25:09.317494+0000","last_fullsized":"2024-09-16T10:25:09.317494+0000","mapping_epoch":276,"log_start":"329'8507","ondisk_log_start":"329'8507","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:21:55.210123+0000","last_clean_scrub_stamp":"2024-09-16T10:21:55.210123+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45219840,"num_objects":690,"num_object_clones":0,"num_object_copies":2070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13914,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":664,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013528+0000","last_change":"2024-09-16T10:24:36.805155+0000","last_active":"2024-09-16T10:25:07.013528+0000","last_peered":"2024-09-16T10:25:07.013528+0000","last_clean":"2024-09-16T10:25:07.013528+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:25:07.013528+0000","last_undegraded":"2024-09-16T10:25:07.013528+0000","last_fullsized":"2024-09-16T10:25:07.013528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:36.805084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:36.805084+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:01.683712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":29,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025966+0000","last_change":"2024-09-16T10:25:00.952412+0000","last_active":"2024-09-16T10:25:07.025966+0000","last_peered":"2024-09-16T10:25:07.025966+0000","last_clean":"2024-09-16T10:25:07.025966+0000","last_became_active":"2024-09-16T10:25:00.952061+0000","last_became_peered":"2024-09-16T10:25:00.952061+0000","last_unstale":"2024-09-16T10:25:07.025966+0000","last_undegraded":"2024-09-16T10:25:07.025966+0000","last_fullsized":"2024-09-16T10:25:07.025966+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:24:24.486825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:24.486825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:47.489444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"352'9","reported_seq":47,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.964497+0000","last_change":"2024-09-16T10:25:00.946719+0000","last_active":"2024-09-16T10:25:06.964497+0000","last_peered":"2024-09-16T10:25:06.964497+0000","last_clean":"2024-09-16T10:25:06.964497+0000","last_became_active":"2024-09-16T10:25:00.946371+0000","last_became_peered":"2024-09-16T10:25:00.946371+0000","last_unstale":"2024-09-16T10:25:06.964497+0000","last_undegraded":"2024-09-16T10:25:06.964497+0000","last_fullsized":"2024-09-16T10:25:06.964497+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:23:10.768339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:23:10.768339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":637,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013743+0000","last_change":"2024-09-16T10:24:34.863064+0000","last_active":"2024-09-16T10:25:07.013743+0000","last_peered":"2024-09-16T10:25:07.013743+0000","last_clean":"2024-09-16T10:25:07.013743+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:25:07.013743+0000","last_undegraded":"2024-09-16T10:25:07.013743+0000","last_fullsized":"2024-09-16T10:25:07.013743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:34.862992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.862992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.781279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":212,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013953+0000","last_change":"2024-09-16T10:25:00.825815+0000","last_active":"2024-09-16T10:25:07.013953+0000","last_peered":"2024-09-16T10:25:07.013953+0000","last_clean":"2024-09-16T10:25:07.013953+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:25:07.013953+0000","last_undegraded":"2024-09-16T10:25:07.013953+0000","last_fullsized":"2024-09-16T10:25:07.013953+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:00.825743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:00.825743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:15.796512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":623,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965214+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:25:06.965214+0000","last_peered":"2024-09-16T10:25:06.965214+0000","last_clean":"2024-09-16T10:25:06.965214+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:25:06.965214+0000","last_undegraded":"2024-09-16T10:25:06.965214+0000","last_fullsized":"2024-09-16T10:25:06.965214+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":645,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.019728+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:25:07.019728+0000","last_peered":"2024-09-16T10:25:07.019728+0000","last_clean":"2024-09-16T10:25:07.019728+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:25:07.019728+0000","last_undegraded":"2024-09-16T10:25:07.019728+0000","last_fullsized":"2024-09-16T10:25:07.019728+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:52.492565+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":624,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014571+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:25:07.014571+0000","last_peered":"2024-09-16T10:25:07.014571+0000","last_clean":"2024-09-16T10:25:07.014571+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:25:07.014571+0000","last_undegraded":"2024-09-16T10:25:07.014571+0000","last_fullsized":"2024-09-16T10:25:07.014571+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.836191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":612,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.956079+0000","last_change":"2024-09-16T10:23:03.747587+0000","last_active":"2024-09-16T10:25:06.956079+0000","last_peered":"2024-09-16T10:25:06.956079+0000","last_clean":"2024-09-16T10:25:06.956079+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:25:06.956079+0000","last_undegraded":"2024-09-16T10:25:06.956079+0000","last_fullsized":"2024-09-16T10:25:06.956079+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:03.747523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:03.747523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:07.661807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":533,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959753+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:25:06.959753+0000","last_peered":"2024-09-16T10:25:06.959753+0000","last_clean":"2024-09-16T10:25:06.959753+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:25:06.959753+0000","last_undegraded":"2024-09-16T10:25:06.959753+0000","last_fullsized":"2024-09-16T10:25:06.959753+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.154627+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":652,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960783+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:25:06.960783+0000","last_peered":"2024-09-16T10:25:06.960783+0000","last_clean":"2024-09-16T10:25:06.960783+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:25:06.960783+0000","last_undegraded":"2024-09-16T10:25:06.960783+0000","last_fullsized":"2024-09-16T10:25:06.960783+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.024438+0000","last_change":"2024-09-16T10:25:00.944900+0000","last_active":"2024-09-16T10:25:07.024438+0000","last_peered":"2024-09-16T10:25:07.024438+0000","last_clean":"2024-09-16T10:25:07.024438+0000","last_became_active":"2024-09-16T10:25:00.944574+0000","last_became_peered":"2024-09-16T10:25:00.944574+0000","last_unstale":"2024-09-16T10:25:07.024438+0000","last_undegraded":"2024-09-16T10:25:07.024438+0000","last_fullsized":"2024-09-16T10:25:07.024438+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:37.608524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.608524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.990047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":623,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.019501+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:25:07.019501+0000","last_peered":"2024-09-16T10:25:07.019501+0000","last_clean":"2024-09-16T10:25:07.019501+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:25:07.019501+0000","last_undegraded":"2024-09-16T10:25:07.019501+0000","last_fullsized":"2024-09-16T10:25:07.019501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":159,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.963096+0000","last_change":"2024-09-16T10:22:02.903139+0000","last_active":"2024-09-16T10:25:06.963096+0000","last_peered":"2024-09-16T10:25:06.963096+0000","last_clean":"2024-09-16T10:25:06.963096+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:25:06.963096+0000","last_undegraded":"2024-09-16T10:25:06.963096+0000","last_fullsized":"2024-09-16T10:25:06.963096+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:19.951829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:19.951829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":194,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.013291+0000","last_change":"2024-09-16T10:23:01.854762+0000","last_active":"2024-09-16T10:25:07.013291+0000","last_peered":"2024-09-16T10:25:07.013291+0000","last_clean":"2024-09-16T10:25:07.013291+0000","last_became_active":"2024-09-16T10:23:00.902731+0000","last_became_peered":"2024-09-16T10:23:00.902731+0000","last_unstale":"2024-09-16T10:25:07.013291+0000","last_undegraded":"2024-09-16T10:25:07.013291+0000","last_fullsized":"2024-09-16T10:25:07.013291+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.854706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.854706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":645,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014556+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:25:07.014556+0000","last_peered":"2024-09-16T10:25:07.014556+0000","last_clean":"2024-09-16T10:25:07.014556+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:25:07.014556+0000","last_undegraded":"2024-09-16T10:25:07.014556+0000","last_fullsized":"2024-09-16T10:25:07.014556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.812083+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":639,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959963+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:25:06.959963+0000","last_peered":"2024-09-16T10:25:06.959963+0000","last_clean":"2024-09-16T10:25:06.959963+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:25:06.959963+0000","last_undegraded":"2024-09-16T10:25:06.959963+0000","last_fullsized":"2024-09-16T10:25:06.959963+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":808,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.016486+0000","last_change":"2024-09-16T10:22:24.619056+0000","last_active":"2024-09-16T10:25:07.016486+0000","last_peered":"2024-09-16T10:25:07.016486+0000","last_clean":"2024-09-16T10:25:07.016486+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:25:07.016486+0000","last_undegraded":"2024-09-16T10:25:07.016486+0000","last_fullsized":"2024-09-16T10:25:07.016486+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.618991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.618991+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":649,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.025777+0000","last_change":"2024-09-16T10:23:38.791832+0000","last_active":"2024-09-16T10:25:07.025777+0000","last_peered":"2024-09-16T10:25:07.025777+0000","last_clean":"2024-09-16T10:25:07.025777+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:25:07.025777+0000","last_undegraded":"2024-09-16T10:25:07.025777+0000","last_fullsized":"2024-09-16T10:25:07.025777+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:38.791750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:38.791750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":208,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960079+0000","last_change":"2024-09-16T10:24:32.496216+0000","last_active":"2024-09-16T10:25:06.960079+0000","last_peered":"2024-09-16T10:25:06.960079+0000","last_clean":"2024-09-16T10:25:06.960079+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:25:06.960079+0000","last_undegraded":"2024-09-16T10:25:06.960079+0000","last_fullsized":"2024-09-16T10:25:06.960079+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.495461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.495461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:01.094890+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":673,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.470676+0000","last_change":"2024-09-16T10:25:08.470676+0000","last_active":"2024-09-16T10:25:08.470676+0000","last_peered":"2024-09-16T10:25:08.470676+0000","last_clean":"2024-09-16T10:25:08.470676+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:25:08.470676+0000","last_undegraded":"2024-09-16T10:25:08.470676+0000","last_fullsized":"2024-09-16T10:25:08.470676+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:25:08.470600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.470600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:17.849780+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":646,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.955945+0000","last_change":"2024-09-16T10:24:23.469834+0000","last_active":"2024-09-16T10:25:06.955945+0000","last_peered":"2024-09-16T10:25:06.955945+0000","last_clean":"2024-09-16T10:25:06.955945+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:25:06.955945+0000","last_undegraded":"2024-09-16T10:25:06.955945+0000","last_fullsized":"2024-09-16T10:25:06.955945+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:23.469779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.469779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:46.236915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":779,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014427+0000","last_change":"2024-09-16T10:24:26.942679+0000","last_active":"2024-09-16T10:25:07.014427+0000","last_peered":"2024-09-16T10:25:07.014427+0000","last_clean":"2024-09-16T10:25:07.014427+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:25:07.014427+0000","last_undegraded":"2024-09-16T10:25:07.014427+0000","last_fullsized":"2024-09-16T10:25:07.014427+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:26.942604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.942604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.074891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":619,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960356+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:25:06.960356+0000","last_peered":"2024-09-16T10:25:06.960356+0000","last_clean":"2024-09-16T10:25:06.960356+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:25:06.960356+0000","last_undegraded":"2024-09-16T10:25:06.960356+0000","last_fullsized":"2024-09-16T10:25:06.960356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":669,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.023427+0000","last_change":"2024-09-16T10:23:11.758636+0000","last_active":"2024-09-16T10:25:07.023427+0000","last_peered":"2024-09-16T10:25:07.023427+0000","last_clean":"2024-09-16T10:25:07.023427+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:25:07.023427+0000","last_undegraded":"2024-09-16T10:25:07.023427+0000","last_fullsized":"2024-09-16T10:25:07.023427+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:23:11.758543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:11.758543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":90,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:08.657378+0000","last_change":"2024-09-16T10:25:08.657378+0000","last_active":"2024-09-16T10:25:08.657378+0000","last_peered":"2024-09-16T10:25:08.657378+0000","last_clean":"2024-09-16T10:25:08.657378+0000","last_became_active":"2024-09-16T10:23:19.734145+0000","last_became_peered":"2024-09-16T10:23:19.734145+0000","last_unstale":"2024-09-16T10:25:08.657378+0000","last_undegraded":"2024-09-16T10:25:08.657378+0000","last_fullsized":"2024-09-16T10:25:08.657378+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.565118+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":624,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959424+0000","last_change":"2024-09-16T10:18:43.697372+0000","last_active":"2024-09-16T10:25:06.959424+0000","last_peered":"2024-09-16T10:25:06.959424+0000","last_clean":"2024-09-16T10:25:06.959424+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:25:06.959424+0000","last_undegraded":"2024-09-16T10:25:06.959424+0000","last_fullsized":"2024-09-16T10:25:06.959424+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:43.697295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:43.697295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":619,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.016578+0000","last_change":"2024-09-16T10:24:57.507342+0000","last_active":"2024-09-16T10:25:07.016578+0000","last_peered":"2024-09-16T10:25:07.016578+0000","last_clean":"2024-09-16T10:25:07.016578+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:25:07.016578+0000","last_undegraded":"2024-09-16T10:25:07.016578+0000","last_fullsized":"2024-09-16T10:25:07.016578+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:57.507281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.507281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:06.240579+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":687,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960647+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:25:06.960647+0000","last_peered":"2024-09-16T10:25:06.960647+0000","last_clean":"2024-09-16T10:25:06.960647+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:25:06.960647+0000","last_undegraded":"2024-09-16T10:25:06.960647+0000","last_fullsized":"2024-09-16T10:25:06.960647+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018954+0000","last_change":"2024-09-16T10:25:00.955337+0000","last_active":"2024-09-16T10:25:07.018954+0000","last_peered":"2024-09-16T10:25:07.018954+0000","last_clean":"2024-09-16T10:25:07.018954+0000","last_became_active":"2024-09-16T10:25:00.954747+0000","last_became_peered":"2024-09-16T10:25:00.954747+0000","last_unstale":"2024-09-16T10:25:07.018954+0000","last_undegraded":"2024-09-16T10:25:07.018954+0000","last_fullsized":"2024-09-16T10:25:07.018954+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:25.105215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.105215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":588,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.017107+0000","last_change":"2024-09-16T10:23:47.232905+0000","last_active":"2024-09-16T10:25:07.017107+0000","last_peered":"2024-09-16T10:25:07.017107+0000","last_clean":"2024-09-16T10:25:07.017107+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:25:07.017107+0000","last_undegraded":"2024-09-16T10:25:07.017107+0000","last_fullsized":"2024-09-16T10:25:07.017107+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:47.232834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.232834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":603,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960797+0000","last_change":"2024-09-16T10:17:45.206938+0000","last_active":"2024-09-16T10:25:06.960797+0000","last_peered":"2024-09-16T10:25:06.960797+0000","last_clean":"2024-09-16T10:25:06.960797+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:25:06.960797+0000","last_undegraded":"2024-09-16T10:25:06.960797+0000","last_fullsized":"2024-09-16T10:25:06.960797+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:17:45.206858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:17:45.206858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":538,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962830+0000","last_change":"2024-09-16T10:24:32.986890+0000","last_active":"2024-09-16T10:25:06.962830+0000","last_peered":"2024-09-16T10:25:06.962830+0000","last_clean":"2024-09-16T10:25:06.962830+0000","last_became_active":"2024-09-16T10:23:31.473535+0000","last_became_peered":"2024-09-16T10:23:31.473535+0000","last_unstale":"2024-09-16T10:25:06.962830+0000","last_undegraded":"2024-09-16T10:25:06.962830+0000","last_fullsized":"2024-09-16T10:25:06.962830+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.984796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.984796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:52.923327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":642,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.016943+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:25:07.016943+0000","last_peered":"2024-09-16T10:25:07.016943+0000","last_clean":"2024-09-16T10:25:07.016943+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:25:07.016943+0000","last_undegraded":"2024-09-16T10:25:07.016943+0000","last_fullsized":"2024-09-16T10:25:07.016943+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.420414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":640,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.014689+0000","last_change":"2024-09-16T10:23:13.803633+0000","last_active":"2024-09-16T10:25:07.014689+0000","last_peered":"2024-09-16T10:25:07.014689+0000","last_clean":"2024-09-16T10:25:07.014689+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:25:07.014689+0000","last_undegraded":"2024-09-16T10:25:07.014689+0000","last_fullsized":"2024-09-16T10:25:07.014689+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:13.803568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.803568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:49.863054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":632,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.955189+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:25:06.955189+0000","last_peered":"2024-09-16T10:25:06.955189+0000","last_clean":"2024-09-16T10:25:06.955189+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:25:06.955189+0000","last_undegraded":"2024-09-16T10:25:06.955189+0000","last_fullsized":"2024-09-16T10:25:06.955189+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:53.389988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":635,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.017306+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:25:07.017306+0000","last_peered":"2024-09-16T10:25:07.017306+0000","last_clean":"2024-09-16T10:25:07.017306+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:25:07.017306+0000","last_undegraded":"2024-09-16T10:25:07.017306+0000","last_fullsized":"2024-09-16T10:25:07.017306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":673,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.017822+0000","last_change":"2024-09-16T10:24:15.357656+0000","last_active":"2024-09-16T10:25:07.017822+0000","last_peered":"2024-09-16T10:25:07.017822+0000","last_clean":"2024-09-16T10:25:07.017822+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:25:07.017822+0000","last_undegraded":"2024-09-16T10:25:07.017822+0000","last_fullsized":"2024-09-16T10:25:07.017822+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:24:15.357188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.357188+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:30.553120+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":629,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.961119+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:25:06.961119+0000","last_peered":"2024-09-16T10:25:06.961119+0000","last_clean":"2024-09-16T10:25:06.961119+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:25:06.961119+0000","last_undegraded":"2024-09-16T10:25:06.961119+0000","last_fullsized":"2024-09-16T10:25:06.961119+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":623,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.964658+0000","last_change":"2024-09-16T10:18:12.243030+0000","last_active":"2024-09-16T10:25:06.964658+0000","last_peered":"2024-09-16T10:25:06.964658+0000","last_clean":"2024-09-16T10:25:06.964658+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:25:06.964658+0000","last_undegraded":"2024-09-16T10:25:06.964658+0000","last_fullsized":"2024-09-16T10:25:06.964658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:12.242951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:12.242951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":251,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.962407+0000","last_change":"2024-09-16T10:25:00.946850+0000","last_active":"2024-09-16T10:25:06.962407+0000","last_peered":"2024-09-16T10:25:06.962407+0000","last_clean":"2024-09-16T10:25:06.962407+0000","last_became_active":"2024-09-16T10:25:00.946511+0000","last_became_peered":"2024-09-16T10:25:00.946511+0000","last_unstale":"2024-09-16T10:25:06.962407+0000","last_undegraded":"2024-09-16T10:25:06.962407+0000","last_fullsized":"2024-09-16T10:25:06.962407+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:26.432557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.432557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.603536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":620,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.959480+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:25:06.959480+0000","last_peered":"2024-09-16T10:25:06.959480+0000","last_clean":"2024-09-16T10:25:06.959480+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:25:06.959480+0000","last_undegraded":"2024-09-16T10:25:06.959480+0000","last_fullsized":"2024-09-16T10:25:06.959480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":639,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:07.018064+0000","last_change":"2024-09-16T10:25:02.446887+0000","last_active":"2024-09-16T10:25:07.018064+0000","last_peered":"2024-09-16T10:25:07.018064+0000","last_clean":"2024-09-16T10:25:07.018064+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:25:07.018064+0000","last_undegraded":"2024-09-16T10:25:07.018064+0000","last_fullsized":"2024-09-16T10:25:07.018064+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:02.446742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.446742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:02.607831+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":667,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.965514+0000","last_change":"2024-09-16T10:18:42.145880+0000","last_active":"2024-09-16T10:25:06.965514+0000","last_peered":"2024-09-16T10:25:06.965514+0000","last_clean":"2024-09-16T10:25:06.965514+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:25:06.965514+0000","last_undegraded":"2024-09-16T10:25:06.965514+0000","last_fullsized":"2024-09-16T10:25:06.965514+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:18:42.145798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:42.145798+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":638,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-16T10:25:06.960499+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:25:06.960499+0000","last_peered":"2024-09-16T10:25:06.960499+0000","last_clean":"2024-09-16T10:25:06.960499+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:25:06.960499+0000","last_undegraded":"2024-09-16T10:25:06.960499+0000","last_fullsized":"2024-09-16T10:25:06.960499+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":809435160,"num_objects":12352,"num_object_clones":0,"num_object_copies":37056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220106,"num_write_kb":7438593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4067,"num_bytes_recovered":164495360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2750427136,"data_stored":2750414920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73126,"ondisk_log_size":73126,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234126,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":937688,"kb_used_data":501120,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92807464,"statfs":{"total":95995035648,"available":95034843136,"internally_reserved":0,"allocated":513146880,"data_stored":512665816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":447019466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528466,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550216,"kb_used_data":158068,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93194936,"statfs":{"total":95995035648,"available":95431614464,"internally_reserved":0,"allocated":161861632,"data_stored":161392174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790104,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1095128,"kb_used_data":671048,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92650024,"statfs":{"total":95995035648,"available":94873624576,"internally_reserved":0,"allocated":687153152,"data_stored":686665244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986333,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":507496,"kb_used_data":165332,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93237656,"statfs":{"total":95995035648,"available":95475359744,"internally_reserved":0,"allocated":169299968,"data_stored":168798783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215266,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":933604,"kb_used_data":415124,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92811548,"statfs":{"total":95995035648,"available":95039025152,"internally_reserved":0,"allocated":425086976,"data_stored":424587174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":530905544},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411496,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":475284,"kb_used_data":205308,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":93269868,"statfs":{"total":95995035648,"available":95508344832,"internally_reserved":0,"allocated":210235392,"data_stored":209758557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019700,"kb_used_data":505380,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92725452,"statfs":{"total":95995035648,"available":94950862848,"internally_reserved":0,"allocated":517509120,"data_stored":516992095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738545,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605588,"kb_used_data":209284,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93139564,"statfs":{"total":95995035648,"available":95374913536,"internally_reserved":0,"allocated":214306816,"data_stored":213822334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197001216,"data_stored":197001216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":498143232,"data_stored":498139160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202309632,"data_stored":202309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":399572992,"data_stored":399572992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":158531584,"data_stored":158531584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":666505216,"data_stored":666501144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":151257088,"data_stored":151257088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":477106176,"data_stored":477102104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:13.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-09-16T10:25:13.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:13.527 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:25:13.527 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:25:13.527 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:25:13.527 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:25:13.527 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:25:13.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:13.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:13.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:13.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:13.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:13.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:13.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:13.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:13.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:13.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:13.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:14.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:14.082 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:14.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:14.184 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:14.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:14.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:13 smithi165 ceph-mon[32696]: pgmap v619: 168 pgs: 168 active+clean; 748 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 696 B/s rd, 0 B/s wr, 673 op/s; 0 B/s, 12 objects/s recovering 2024-09-16T10:25:14.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:14.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:14.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:13 smithi063 ceph-mon[29993]: pgmap v619: 168 pgs: 168 active+clean; 748 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 696 B/s rd, 0 B/s wr, 673 op/s; 0 B/s, 12 objects/s recovering 2024-09-16T10:25:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:13 smithi063 ceph-mon[39470]: pgmap v619: 168 pgs: 168 active+clean; 748 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 696 B/s rd, 0 B/s wr, 673 op/s; 0 B/s, 12 objects/s recovering 2024-09-16T10:25:14.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:14.488 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:14.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:14.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:25:14.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:14.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:14.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:14.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:14.779 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:14.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:14.881 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:14.920 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:14.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:15.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:25:15.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:15.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:15.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:15.117 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:15.133 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:15.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:15.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:15.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:15.206 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 6.14 scrub starts 2024-09-16T10:25:15.206 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 6.14 scrub ok 2024-09-16T10:25:15.206 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 4.7 scrub starts 2024-09-16T10:25:15.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 4.7 scrub ok 2024-09-16T10:25:15.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:25:15.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:25:15.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 3.1a scrub starts 2024-09-16T10:25:15.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[29993]: 3.1a scrub ok 2024-09-16T10:25:15.208 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 6.14 scrub starts 2024-09-16T10:25:15.208 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 6.14 scrub ok 2024-09-16T10:25:15.208 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 4.7 scrub starts 2024-09-16T10:25:15.208 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 4.7 scrub ok 2024-09-16T10:25:15.208 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:25:15.209 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:25:15.209 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 3.1a scrub starts 2024-09-16T10:25:15.209 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:14 smithi063 ceph-mon[39470]: 3.1a scrub ok 2024-09-16T10:25:15.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 6.14 scrub starts 2024-09-16T10:25:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 6.14 scrub ok 2024-09-16T10:25:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 4.7 scrub starts 2024-09-16T10:25:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 4.7 scrub ok 2024-09-16T10:25:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:25:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:25:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 3.1a scrub starts 2024-09-16T10:25:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:14 smithi165 ceph-mon[32696]: 3.1a scrub ok 2024-09-16T10:25:15.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:15.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:15.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:15.461 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:15.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:15.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:15.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:15.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:15.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:15.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:15.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:15.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:15.873 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:15.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:15.975 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:16.018 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:16.120 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: 6.f scrub starts 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: 6.f scrub ok 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: 5.12 scrub starts 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: 5.12 scrub ok 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: pgmap v620: 168 pgs: 168 active+clean; 376 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 0 B/s, 11 objects/s recovering 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: 6.1f scrub starts 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: 6.1f scrub ok 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3681699147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: 6.f scrub starts 2024-09-16T10:25:16.161 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: 6.f scrub ok 2024-09-16T10:25:16.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: 5.12 scrub starts 2024-09-16T10:25:16.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: 5.12 scrub ok 2024-09-16T10:25:16.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: pgmap v620: 168 pgs: 168 active+clean; 376 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 0 B/s, 11 objects/s recovering 2024-09-16T10:25:16.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: 6.1f scrub starts 2024-09-16T10:25:16.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: 6.1f scrub ok 2024-09-16T10:25:16.162 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3681699147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:16.167 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:16.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:16.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:16.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: 6.f scrub starts 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: 6.f scrub ok 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: 5.12 scrub starts 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: 5.12 scrub ok 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: pgmap v620: 168 pgs: 168 active+clean; 376 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 0 B/s, 11 objects/s recovering 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: 6.1f scrub starts 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: 6.1f scrub ok 2024-09-16T10:25:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3681699147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:16.315 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:16.316 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:16.317 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:16.317 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:16.317 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:16.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:16.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:16.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:16.558 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:16.660 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:16.696 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:16.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:16.833 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:16.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:16.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:16.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:16.980 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:17.002 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 1 pgp_num to 21 2024-09-16T10:25:17.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:17.119 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:17.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:17.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:17.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:17.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 9.b scrub starts 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 9.b scrub ok 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/499404772' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: from='client.25996 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:25:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:25:17.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 5.1f scrub starts 2024-09-16T10:25:17.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:17 smithi165 ceph-mon[32696]: 5.1f scrub ok 2024-09-16T10:25:17.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:17.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:17.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:25:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 9.b scrub starts 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 9.b scrub ok 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/499404772' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: from='client.25996 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:25:17.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 5.1f scrub starts 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[29993]: 5.1f scrub ok 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 9.b scrub starts 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 9.b scrub ok 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/499404772' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: from='client.25996 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:25:17.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 5.1f scrub starts 2024-09-16T10:25:17.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:17 smithi063 ceph-mon[39470]: 5.1f scrub ok 2024-09-16T10:25:17.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:17.538 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:17.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:17.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:17.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:17.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:17.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:17.816 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:17.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:17.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:17.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:17.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:18.050 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:18.088 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: 4.b scrub starts 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: 4.b scrub ok 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: from='client.25996 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: osdmap e357: 8 total, 8 up, 8 in 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: pgmap v622: 168 pgs: 168 active+clean; 376 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 866 op/s 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: 4.13 scrub starts 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: 4.13 scrub ok 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: 3.1e scrub starts 2024-09-16T10:25:18.174 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[29993]: 3.1e scrub ok 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: 4.b scrub starts 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: 4.b scrub ok 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: from='client.25996 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: osdmap e357: 8 total, 8 up, 8 in 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: pgmap v622: 168 pgs: 168 active+clean; 376 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 866 op/s 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: 4.13 scrub starts 2024-09-16T10:25:18.175 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: 4.13 scrub ok 2024-09-16T10:25:18.176 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: 3.1e scrub starts 2024-09-16T10:25:18.176 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:18 smithi063 ceph-mon[39470]: 3.1e scrub ok 2024-09-16T10:25:18.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:18.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:18.219 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: 4.b scrub starts 2024-09-16T10:25:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: 4.b scrub ok 2024-09-16T10:25:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:25:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:25:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: from='client.25996 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-16T10:25:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: osdmap e357: 8 total, 8 up, 8 in 2024-09-16T10:25:18.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: pgmap v622: 168 pgs: 168 active+clean; 376 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 866 op/s 2024-09-16T10:25:18.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: 4.13 scrub starts 2024-09-16T10:25:18.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: 4.13 scrub ok 2024-09-16T10:25:18.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: 3.1e scrub starts 2024-09-16T10:25:18.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-mon[32696]: 3.1e scrub ok 2024-09-16T10:25:18.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.358 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:18.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:18.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:18.485 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:18.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:18.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:18.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:18.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:18.780 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Removed 116228 objects 2024-09-16T10:25:18.780 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Clean up completed and total clean up time :107.351 2024-09-16T10:25:18.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.874 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:18.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:18.975 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:18.998 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:18.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:19.036 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:25:19.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:25:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:25:19.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:19.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:19.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:19.150 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:19.150 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:19.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:19.249 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:19.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:19 smithi165 ceph-mon[32696]: 6.10 scrub starts 2024-09-16T10:25:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:19 smithi165 ceph-mon[32696]: 6.10 scrub ok 2024-09-16T10:25:19.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:19 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:25:19.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:19 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:25:19.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:19.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2117062483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:25:19.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:19.374 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:19.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[29993]: 6.10 scrub starts 2024-09-16T10:25:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[29993]: 6.10 scrub ok 2024-09-16T10:25:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:25:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:25:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2117062483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:25:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[39470]: 6.10 scrub starts 2024-09-16T10:25:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[39470]: 6.10 scrub ok 2024-09-16T10:25:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:25:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:25:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:19.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2117062483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:25:19.475 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:19.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:19.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:19.548 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:25:19.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:19.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:19.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:19.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:19.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:19.854 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:19.876 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:19.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:19.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:20.000 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:20.102 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:20.124 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:20.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[29993]: 6.1c scrub starts 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[29993]: 6.1c scrub ok 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2117062483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[29993]: osdmap e358: 8 total, 8 up, 8 in 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[29993]: pgmap v624: 168 pgs: 168 active+clean; 51 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[39470]: 6.1c scrub starts 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[39470]: 6.1c scrub ok 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2117062483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:25:20.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[39470]: osdmap e358: 8 total, 8 up, 8 in 2024-09-16T10:25:20.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:20 smithi063 ceph-mon[39470]: pgmap v624: 168 pgs: 168 active+clean; 51 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-16T10:25:20.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:20.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:20 smithi165 ceph-mon[32696]: 6.1c scrub starts 2024-09-16T10:25:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:20 smithi165 ceph-mon[32696]: 6.1c scrub ok 2024-09-16T10:25:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:20 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2117062483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:25:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:20 smithi165 ceph-mon[32696]: osdmap e358: 8 total, 8 up, 8 in 2024-09-16T10:25:20.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:20 smithi165 ceph-mon[32696]: pgmap v624: 168 pgs: 168 active+clean; 51 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-16T10:25:20.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:20.376 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:20.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:20.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:20.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:20.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:20.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:25:20.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:20.755 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:20.756 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:20.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:20.879 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:20.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:20.962 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:20.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:21.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:21.104 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:21.126 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:21.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:21.251 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:21.252 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:21.252 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:21.270 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:21.270 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:21.286 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:21.286 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:21.303 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:21.303 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:21.319 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:21.320 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:21.337 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:21.337 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:21.352 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:21.353 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:21.354 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:21.370 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:21.371 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:21.375 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:21.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:21.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:21.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:21.626 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:21.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:21.752 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:21.854 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:21.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:21.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:21.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:25:21.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:22.004 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:22.043 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:22.043 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:25:22.043 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:25:22.043 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:22.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:21 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:25:22.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:21 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:25:22.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:21 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4143618048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:22.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:21 smithi165 ceph-mon[32696]: pgmap v625: 168 pgs: 168 active+clean; 9.4 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-16T10:25:22.105 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:22.130 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:22.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:25:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:25:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4143618048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:22.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[29993]: pgmap v625: 168 pgs: 168 active+clean; 9.4 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-16T10:25:22.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:25:22.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:25:22.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4143618048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:22.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:21 smithi063 ceph-mon[39470]: pgmap v625: 168 pgs: 168 active+clean; 9.4 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-16T10:25:22.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:22.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:22.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:22.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:22.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:22.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:22.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:22.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:22.700 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-16T10:25:22.701 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-16T10:25:22.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:22.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:22.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:25:22.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:22.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:22.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:22.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:22.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:23.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:22 smithi165 ceph-mon[32696]: osdmap e359: 8 total, 8 up, 8 in 2024-09-16T10:25:23.067 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:23.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:23.121 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:23.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:22 smithi063 ceph-mon[29993]: osdmap e359: 8 total, 8 up, 8 in 2024-09-16T10:25:23.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:22 smithi063 ceph-mon[39470]: osdmap e359: 8 total, 8 up, 8 in 2024-09-16T10:25:23.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:23.228 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:25:23.228 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:25:23.228 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-16T10:25:23.228 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T10:25:23.228 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T10:25:23.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-16T10:25:23.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:23.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:23.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:23.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:23.471 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:23.493 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:23.569 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:23.569 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:25:23.569 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:25:23.570 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:25:23.570 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:23.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:23.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:23.719 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:23.741 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:23.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:23.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:23.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:23.997 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:23.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:24.021 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:24.033 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:24.049 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:24.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:24.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:24.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:24.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:25:24.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:24.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:24.245 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:24.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:24.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:24.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:24.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:24.464 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:24.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:24.493 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:24.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:24.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:24.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:24.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:24.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:24.725 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:24 smithi063 ceph-mon[29993]: pgmap v627: 168 pgs: 168 active+clean; 6.3 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 940 op/s 2024-09-16T10:25:24.725 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:24.725 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:24.725 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:24 smithi063 ceph-mon[39470]: pgmap v627: 168 pgs: 168 active+clean; 6.3 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 940 op/s 2024-09-16T10:25:24.725 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:24.725 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:24.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:24.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:24.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:24.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:24.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:24.926 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:24 smithi165 ceph-mon[32696]: pgmap v627: 168 pgs: 168 active+clean; 6.3 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 940 op/s 2024-09-16T10:25:24.926 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:24.926 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:24.926 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:25:24.926 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T10:25:24.927 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:25:24.954 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_2' removed 2024-09-16T10:25:24.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:25.004 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:25.105 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:25.128 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70291 7.1 7:8090c1a5:::benchmark_data_smithi063_73086_object70290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:25:25.161 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 593.25992215600002, 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12180724800000001, 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70291 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898186+0000", 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898188+0000", 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-16T10:25:25.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898240+0000", 2024-09-16T10:25:25.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1740999999999998e-05 2024-09-16T10:25:25.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898249+0000", 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4490000000000005e-06 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898254+0000", 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2569999999999998e-06 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898300+0000", 2024-09-16T10:25:25.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5454000000000002e-05 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898383+0000", 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3005999999999998e-05 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.898476+0000", 2024-09-16T10:25:25.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3029000000000007e-05 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.905816+0000", 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073403699999999997 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906673+0000", 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085627000000000001 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019900+0000", 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11322701 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019950+0000", 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0862999999999997e-05 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019994+0000", 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3169e-05 2024-09-16T10:25:25.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70294 7.1 7:8c6bcbeb:::benchmark_data_smithi063_73086_object70293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 593.25785894299997, 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119864807, 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:25.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70294 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900250+0000", 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900252+0000", 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1119999999999999e-06 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900304+0000", 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1965999999999997e-05 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900313+0000", 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4690000000000003e-06 2024-09-16T10:25:25.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900320+0000", 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.404e-06 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900758+0000", 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000438023 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900864+0000", 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106434 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900994+0000", 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129567 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906642+0000", 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056482299999999997 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906681+0000", 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9443000000000002e-05 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020041+0000", 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113360006 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020084+0000", 2024-09-16T10:25:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2623999999999997e-05 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020114+0000", 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0528999999999999e-05 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:70297 7.1 7:8aa306c3:::benchmark_data_smithi063_73086_object70296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 593.25759051499995, 2024-09-16T10:25:25.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119738053, 2024-09-16T10:25:25.176 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.176 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.176 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 70297 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:25:25.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900518+0000", 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900519+0000", 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1799999999999999e-06 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900602+0000", 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2973000000000007e-05 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900612+0000", 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3230000000000004e-06 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.900618+0000", 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9489999999999999e-06 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903224+0000", 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026055459999999998 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903382+0000", 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015748800000000001 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903501+0000", 2024-09-16T10:25:25.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119727 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906766+0000", 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032648949999999999 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019591+0000", 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112824517 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020144+0000", 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055325600000000002 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020224+0000", 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9878999999999999e-05 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.020256+0000", 2024-09-16T10:25:25.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2320000000000002e-05 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70298 7.3 e106/81 7:c8a39fda:::benchmark_data_smithi063_73086_object70297:head v 106'4332, mlcod=106'4328)", 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 593.25567563799996, 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.117018094, 2024-09-16T10:25:25.183 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902433+0000", 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902434+0000", 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5570000000000001e-06 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902490+0000", 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5559000000000001e-05 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902501+0000", 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0879999999999999e-05 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902508+0000", 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.204e-06 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902555+0000", 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6430000000000001e-05 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.902566+0000", 2024-09-16T10:25:25.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1922999999999999e-05 2024-09-16T10:25:25.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906782+0000", 2024-09-16T10:25:25.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042155300000000003 2024-09-16T10:25:25.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.019451+0000", 2024-09-16T10:25:25.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112669011 2024-09-16T10:25:25.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.190 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70300 7.8 e106/81 7:10588a9d:::benchmark_data_smithi063_73086_object70299:head v 106'4365, mlcod=106'4363)", 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 593.25449997299995, 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118272198, 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:25:25.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903609+0000", 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903610+0000", 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0640000000000001e-06 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903703+0000", 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3548000000000006e-05 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903721+0000", 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8036000000000001e-05 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903727+0000", 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.156e-06 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903787+0000", 2024-09-16T10:25:25.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9658999999999999e-05 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.903816+0000", 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8988000000000001e-05 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.021844+0000", 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11802747700000001 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.021881+0000", 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7270000000000001e-05 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:70303 7.8 e106/81 7:176924a2:::benchmark_data_smithi063_73086_object70302:head v 106'4366, mlcod=106'4363)", 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 593.25147002999995, 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115509458, 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.196 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.197 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:25:25.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906639+0000", 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906640+0000", 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.215e-06 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906678+0000", 2024-09-16T10:25:25.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8124000000000003e-05 2024-09-16T10:25:25.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906691+0000", 2024-09-16T10:25:25.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3244999999999999e-05 2024-09-16T10:25:25.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906698+0000", 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2949999999999996e-06 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906760+0000", 2024-09-16T10:25:25.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1163999999999999e-05 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:31.906778+0000", 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8695e-05 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.022123+0000", 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115344601 2024-09-16T10:25:25.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:32.022148+0000", 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5119000000000001e-05 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174929 7.1 7:83c2bca2:::benchmark_data_smithi063_73086_object55650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 491.289631063, 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12126697, 2024-09-16T10:25:25.202 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174929 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868477+0000", 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868479+0000", 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2869999999999999e-06 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868482+0000", 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6799999999999999e-06 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868491+0000", 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4889999999999995e-06 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868498+0000", 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9450000000000004e-06 2024-09-16T10:25:25.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868544+0000", 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5828000000000002e-05 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.868647+0000", 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102902 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:25:25.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.869028+0000", 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038142800000000001 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.871769+0000", 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027412370000000001 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.871892+0000", 2024-09-16T10:25:25.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012271299999999999 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989639+0000", 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.117746694 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989694+0000", 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5634000000000003e-05 2024-09-16T10:25:25.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.989744+0000", 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0133000000000003e-05 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:201826 7.4 e146/81 7:2bb12e82:::benchmark_data_smithi063_73086_object82547:head v 146'12527, mlcod=146'12525)", 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:25:25.209 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 463.79458838199997, 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13052909300000001, 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:25:25.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363520+0000", 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363522+0000", 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.686e-06 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363529+0000", 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5209999999999997e-06 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363541+0000", 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.163e-05 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363549+0000", 2024-09-16T10:25:25.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2870000000000004e-06 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363609+0000", 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0059000000000002e-05 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363629+0000", 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9953999999999999e-05 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.494008+0000", 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13037901800000001 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.494049+0000", 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0938e-05 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.216 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:201829 7.1 7:86c5e7dd:::benchmark_data_smithi063_73086_object82550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 463.79424156900001, 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13543617699999999, 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 201829 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:25:25.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363867+0000", 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363869+0000", 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.748e-06 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.363874+0000", 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4380000000000003e-06 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489286+0000", 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12541170800000001 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489301+0000", 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4732999999999999e-05 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489378+0000", 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7651000000000006e-05 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489473+0000", 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4846999999999995e-05 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489580+0000", 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106932 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.497200+0000", 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076199780000000003 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.497235+0000", 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4955e-05 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499237+0000", 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020019410000000001 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499273+0000", 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6531e-05 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499303+0000", 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9714999999999999e-05 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.224 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44750.0:201820 7.9 e146/81 7:907729a7:::benchmark_data_smithi063_73086_object82541:head v 146'12370, mlcod=146'12367)", 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 463.78964392099999, 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13076132600000001, 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368465+0000", 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368466+0000", 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5910000000000001e-06 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.368474+0000", 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4989999999999997e-06 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489290+0000", 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120816008 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489300+0000", 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0530000000000001e-05 2024-09-16T10:25:25.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489358+0000", 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7380999999999998e-05 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.489379+0000", 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1596000000000001e-05 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499198+0000", 2024-09-16T10:25:25.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098188400000000006 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:41.499226+0000", 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7881e-05 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:231096 8.2 8:4cceeac5:::benchmark_data_smithi063_101290_object103776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 240.403387149, 2024-09-16T10:25:25.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115767351, 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 231096 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:25:25.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:25:25.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754723+0000", 2024-09-16T10:25:25.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754727+0000", 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2300000000000002e-06 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754734+0000", 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7850000000000003e-06 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754739+0000", 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7979999999999996e-06 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754843+0000", 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010442000000000001 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754957+0000", 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113748 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.755066+0000", 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109279 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758427+0000", 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033604519999999999 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758995+0000", 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056835100000000003 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870387+0000", 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11139234100000001 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870444+0000", 2024-09-16T10:25:25.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6450999999999999e-05 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870489+0000", 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4863000000000001e-05 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212280 9.e 9:7a91a301:::benchmark_data_smithi063_124717_object96049:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.487962982999999, 2024-09-16T10:25:25.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30939322400000002, 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212280 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670148+0000", 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.283e-06 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670154+0000", 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8370000000000002e-06 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670167+0000", 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3243e-05 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670176+0000", 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5969999999999992e-06 2024-09-16T10:25:25.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670244+0000", 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8726999999999994e-05 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:25:25.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670254+0000", 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9210000000000004e-06 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.978436+0000", 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30818199899999998 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979539+0000", 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001102617 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212292 9.e 9:780c7a09:::benchmark_data_smithi063_124717_object96061:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.487911262000001, 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31685680300000002, 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.244 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.244 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212292 2024-09-16T10:25:25.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.244 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:25:25.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:25:25.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670198+0000", 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9599999999999999e-07 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670201+0000", 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1130000000000001e-06 2024-09-16T10:25:25.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670207+0000", 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2689999999999996e-06 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670213+0000", 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.164e-06 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670284+0000", 2024-09-16T10:25:25.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1136000000000005e-05 2024-09-16T10:25:25.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:25:25.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670291+0000", 2024-09-16T10:25:25.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9210000000000003e-06 2024-09-16T10:25:25.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:25.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979884+0000", 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30959349200000003 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980142+0000", 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025797 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980215+0000", 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3054999999999998e-05 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985118+0000", 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049030380000000002 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985153+0000", 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4496999999999999e-05 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986997+0000", 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018447450000000001 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987028+0000", 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.099e-05 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987054+0000", 2024-09-16T10:25:25.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5717000000000001e-05 2024-09-16T10:25:25.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.253 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.253 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212294 9.e 9:724d3208:::benchmark_data_smithi063_124717_object96063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:25:25.254 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:25:25.254 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.487879493000001, 2024-09-16T10:25:25.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.32240128099999998, 2024-09-16T10:25:25.254 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.254 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.254 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.255 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.255 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.255 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212294 2024-09-16T10:25:25.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.255 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:25:25.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:25:25.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670230+0000", 2024-09-16T10:25:25.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7999999999999995e-07 2024-09-16T10:25:25.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670233+0000", 2024-09-16T10:25:25.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9919999999999999e-06 2024-09-16T10:25:25.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670239+0000", 2024-09-16T10:25:25.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3779999999999998e-06 2024-09-16T10:25:25.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670245+0000", 2024-09-16T10:25:25.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-16T10:25:25.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670316+0000", 2024-09-16T10:25:25.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1147000000000002e-05 2024-09-16T10:25:25.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:25:25.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670323+0000", 2024-09-16T10:25:25.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7209999999999997e-06 2024-09-16T10:25:25.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981661+0000", 2024-09-16T10:25:25.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31133887799999999 2024-09-16T10:25:25.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981742+0000", 2024-09-16T10:25:25.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0136000000000007e-05 2024-09-16T10:25:25.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981814+0000", 2024-09-16T10:25:25.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2065999999999997e-05 2024-09-16T10:25:25.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985187+0000", 2024-09-16T10:25:25.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003373048 2024-09-16T10:25:25.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985193+0000", 2024-09-16T10:25:25.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7170000000000002e-06 2024-09-16T10:25:25.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992481+0000", 2024-09-16T10:25:25.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072876870000000002 2024-09-16T10:25:25.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992565+0000", 2024-09-16T10:25:25.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3726000000000004e-05 2024-09-16T10:25:25.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992630+0000", 2024-09-16T10:25:25.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5581999999999997e-05 2024-09-16T10:25:25.269 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.270 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.270 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.270 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212314 9.e 9:73a00d32:::benchmark_data_smithi063_124717_object96083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:25:25.270 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:25:25.271 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.462832617, 2024-09-16T10:25:25.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29758563799999999, 2024-09-16T10:25:25.271 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.271 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.271 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.271 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.272 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212314 2024-09-16T10:25:25.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.272 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:25:25.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:25:25.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695278+0000", 2024-09-16T10:25:25.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9800000000000001e-06 2024-09-16T10:25:25.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695284+0000", 2024-09-16T10:25:25.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9349999999999999e-06 2024-09-16T10:25:25.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695295+0000", 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0849000000000001e-05 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695303+0000", 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2130000000000008e-06 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695359+0000", 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5859999999999997e-05 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695366+0000", 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6580000000000002e-06 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982882+0000", 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.287515939 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982949+0000", 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6370000000000003e-05 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.983024+0000", 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5723000000000007e-05 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985197+0000", 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002172195 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986274+0000", 2024-09-16T10:25:25.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001077733 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992764+0000", 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064893520000000003 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992820+0000", 2024-09-16T10:25:25.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5903000000000003e-05 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992862+0000", 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1928000000000002e-05 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.282 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212323 9.e 9:7cb576e6:::benchmark_data_smithi063_124717_object96092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.457338486000001, 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29218721399999997, 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212323 2024-09-16T10:25:25.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:25:25.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700773+0000", 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5299999999999999e-06 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700779+0000", 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1399999999999997e-06 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700801+0000", 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1812999999999999e-05 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700809+0000", 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1929999999999993e-06 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700877+0000", 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8199000000000002e-05 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700889+0000", 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1904e-05 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984122+0000", 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28323306799999998 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984181+0000", 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8847000000000001e-05 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984241+0000", 2024-09-16T10:25:25.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0056000000000002e-05 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986252+0000", 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020111399999999998 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986461+0000", 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000209273 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992908+0000", 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006446523 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992935+0000", 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7016999999999999e-05 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992957+0000", 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2510999999999998e-05 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212342 9.e 9:700843e6:::benchmark_data_smithi063_124717_object96111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.437798287, 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27274648600000001, 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.291 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212342 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:25:25.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720312+0000", 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0949999999999998e-06 2024-09-16T10:25:25.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720318+0000", 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1059999999999996e-06 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720333+0000", 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4173e-05 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720340+0000", 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0389999999999999e-06 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720405+0000", 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5079000000000005e-05 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:25:25.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720415+0000", 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0351e-05 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985207+0000", 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.264792321 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985273+0000", 2024-09-16T10:25:25.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5215999999999994e-05 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985329+0000", 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6403000000000002e-05 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986392+0000", 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010631939999999999 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986576+0000", 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018346000000000001 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992994+0000", 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064186979999999996 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993027+0000", 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2262999999999999e-05 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993057+0000", 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0088000000000001e-05 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212688 9.e 9:7ab865c0:::benchmark_data_smithi063_124717_object96457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.937111944000002, 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12657935300000001, 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.300 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212688 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220999+0000", 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221004+0000", 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1880000000000001e-06 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221017+0000", 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2802e-05 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221023+0000", 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8789999999999997e-06 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221090+0000", 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6854999999999996e-05 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221200+0000", 2024-09-16T10:25:25.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010959399999999999 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221304+0000", 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104456 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.223708+0000", 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024038449999999999 2024-09-16T10:25:25.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.224936+0000", 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012280119999999999 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347460+0000", 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122523584 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347521+0000", 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1199999999999997e-05 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347576+0000", 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4972000000000002e-05 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:213104 9.e 9:709eca86:::benchmark_data_smithi063_124717_object96873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.413458708, 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12242840100000001, 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.308 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213104 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:25:25.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744652+0000", 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0219999999999999e-06 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744657+0000", 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3020000000000002e-06 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744668+0000", 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.118e-05 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744674+0000", 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6500000000000001e-06 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744728+0000", 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3705000000000002e-05 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744820+0000", 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2052000000000006e-05 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744897+0000", 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7200999999999995e-05 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.746741+0000", 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018441250000000001 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.747036+0000", 2024-09-16T10:25:25.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029480100000000003 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.866958+0000", 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11992201700000001 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867027+0000", 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9314999999999996e-05 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867078+0000", 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1031e-05 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:227208 9.e 9:7c14d3da:::benchmark_data_smithi063_124717_object110977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:14.472938+0000", 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.685170812999999, 2024-09-16T10:25:25.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060126210999999999, 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 227208 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.472938+0000", 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.472938+0000", 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.472939+0000", 2024-09-16T10:25:25.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5899999999999995e-07 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.472942+0000", 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4750000000000002e-06 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.472948+0000", 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.302e-06 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.472954+0000", 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8839999999999999e-06 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.475391+0000", 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002436966 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.475480+0000", 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9250000000000001e-05 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.475577+0000", 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6794999999999997e-05 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:25.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.477702+0000", 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021249400000000001 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.478424+0000", 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072232600000000002 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.532964+0000", 2024-09-16T10:25:25.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054539488999999997 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.533020+0000", 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6069000000000001e-05 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:14.533064+0000", 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3955999999999997e-05 2024-09-16T10:25:25.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:25.323 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:25.323 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:25.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:25.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:25.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:25.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:25.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:25.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:25.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:25.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:25.845 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:25.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:25:25.927 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1716024600' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1716024600' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[29993]: osdmap e360: 8 total, 8 up, 8 in 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1716024600' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1716024600' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:25:25.928 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:25 smithi063 ceph-mon[39470]: osdmap e360: 8 total, 8 up, 8 in 2024-09-16T10:25:25.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:25.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:25.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:26.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:25 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:25:26.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:25 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:25:26.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1716024600' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:25:26.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1716024600' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:25:26.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:25 smithi165 ceph-mon[32696]: osdmap e360: 8 total, 8 up, 8 in 2024-09-16T10:25:26.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:25:26.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:26.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:26.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:26.245 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-16T10:25:26.246 INFO:tasks.radosbench:Beginning radosbench... 2024-09-16T10:25:26.247 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-16T10:25:26.247 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-16T10:25:26.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:26.319 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:26.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.383 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:26.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:26.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:26.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:26.601 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:25:26.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:26.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:26.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:26 smithi165 ceph-mon[32696]: pgmap v629: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-09-16T10:25:26.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3041338426' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:26.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:26 smithi165 ceph-mon[32696]: from='client.45950 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:26.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:26.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:26.851 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:26.852 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:25:26.852 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:25:26.852 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:26.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:26.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:26.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:26 smithi063 ceph-mon[29993]: pgmap v629: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-09-16T10:25:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3041338426' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:26 smithi063 ceph-mon[29993]: from='client.45950 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:26 smithi063 ceph-mon[39470]: pgmap v629: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-09-16T10:25:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3041338426' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:26 smithi063 ceph-mon[39470]: from='client.45950 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:26.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:26.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:26.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:27.063 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:27.100 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:27.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:27.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:27.248 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:27.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:27.374 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:27.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:25:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:25:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:25:27.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:27.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:25:27.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:27.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.624 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:27.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:27.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:27.730 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:27.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:27 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:27 smithi165 ceph-mon[32696]: from='client.45950 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:27 smithi165 ceph-mon[32696]: osdmap e361: 8 total, 8 up, 8 in 2024-09-16T10:25:27.840 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:27 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:27.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:27 smithi063 ceph-mon[29993]: from='client.45950 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:27.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:27 smithi063 ceph-mon[29993]: osdmap e361: 8 total, 8 up, 8 in 2024-09-16T10:25:27.841 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:27 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:27.841 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:27 smithi063 ceph-mon[39470]: from='client.45950 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:27.841 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:27 smithi063 ceph-mon[39470]: osdmap e361: 8 total, 8 up, 8 in 2024-09-16T10:25:27.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:27.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:27.949 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:27.949 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:27.958 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:27.958 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:27.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:27.976 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:27.977 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:27.988 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:27.996 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:27.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:27.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:27.998 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:28.014 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:28.015 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:28.039 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:28.039 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:28.056 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:28.056 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:28.073 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:28.073 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:28.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:28.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:28.211 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:28.211 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:28.224 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.249 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:28.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:28.352 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:28.374 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:28.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:28.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:28.599 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:28.622 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:28.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:28.746 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:28.848 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:28.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:28 smithi165 ceph-mon[32696]: pgmap v631: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:25:28.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2481199725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:28.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:25:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:25:28.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:28 smithi063 ceph-mon[29993]: pgmap v631: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:25:28.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2481199725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:28.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:28 smithi063 ceph-mon[39470]: pgmap v631: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:25:28.856 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2481199725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:28.872 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:28.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:25:28.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:28.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:28.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:29.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:29.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:29.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:29.160 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:29.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:29.256 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:29.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:29.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:29.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:29.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:29.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:29.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:29.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:29.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:29.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:29.542 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:25:29.607 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:29.614 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_3' created 2024-09-16T10:25:29.634 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:29.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:29.688 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:29.689 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:29.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:29.759 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:29.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:29.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:29.885 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:29.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:29.918 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:29.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:29 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:29 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1330437293' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-16T10:25:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:29 smithi063 ceph-mon[29993]: from='client.45962 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-16T10:25:29.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:29 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:29.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:29 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1330437293' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-16T10:25:29.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:29 smithi063 ceph-mon[39470]: from='client.45962 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-16T10:25:29.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:29.995 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:29 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:29.995 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:29 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1330437293' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-16T10:25:29.995 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:29 smithi165 ceph-mon[32696]: from='client.45962 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-16T10:25:30.010 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:30.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:30.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:30.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:30.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:30.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:25:30.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:30.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:30.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:30.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:30.567 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:30.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:30.610 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:25:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:30.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:30.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:30.757 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:30.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-16T10:25:30.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:30.881 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:30.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:30.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:30.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:25:30.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[29993]: pgmap v632: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:25:30.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[29993]: from='client.45962 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-16T10:25:30.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[29993]: osdmap e362: 8 total, 8 up, 8 in 2024-09-16T10:25:30.947 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3593062360' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:25:30.947 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[29993]: from='client.45968 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:25:30.947 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[39470]: pgmap v632: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:25:30.947 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[39470]: from='client.45962 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-16T10:25:30.947 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[39470]: osdmap e362: 8 total, 8 up, 8 in 2024-09-16T10:25:30.947 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3593062360' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:25:30.947 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:30 smithi063 ceph-mon[39470]: from='client.45968 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:25:30.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:30 smithi165 ceph-mon[32696]: pgmap v632: 152 pgs: 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:25:30.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:30 smithi165 ceph-mon[32696]: from='client.45962 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-16T10:25:30.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:30 smithi165 ceph-mon[32696]: osdmap e362: 8 total, 8 up, 8 in 2024-09-16T10:25:30.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3593062360' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:25:30.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:30 smithi165 ceph-mon[32696]: from='client.45968 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:25:30.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.006 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:31.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:31.077 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:31.077 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:31.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:31.130 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:31.232 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:31.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:31.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:31.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.503 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:31.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:31.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:31.610 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:31.628 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:31.632 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:31.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:31.756 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:31 smithi063 ceph-mon[29993]: from='client.45968 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:25:31.756 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:31 smithi063 ceph-mon[29993]: osdmap e363: 8 total, 8 up, 8 in 2024-09-16T10:25:31.756 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:31 smithi063 ceph-mon[29993]: osdmap e364: 8 total, 8 up, 8 in 2024-09-16T10:25:31.756 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:31 smithi063 ceph-mon[39470]: from='client.45968 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:25:31.756 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:31 smithi063 ceph-mon[39470]: osdmap e363: 8 total, 8 up, 8 in 2024-09-16T10:25:31.756 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:31 smithi063 ceph-mon[39470]: osdmap e364: 8 total, 8 up, 8 in 2024-09-16T10:25:31.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:31 smithi165 ceph-mon[32696]: from='client.45968 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:25:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:31 smithi165 ceph-mon[32696]: osdmap e363: 8 total, 8 up, 8 in 2024-09-16T10:25:31.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:31 smithi165 ceph-mon[32696]: osdmap e364: 8 total, 8 up, 8 in 2024-09-16T10:25:31.809 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:25:31.809 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:25:31.810 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:25:31.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:25:31.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:31.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:25:31.883 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:31.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:31.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:32.008 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:32.110 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:32.118 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:32.134 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:32.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:32.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:32.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:32.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:32.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:32.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:32.530 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:32.530 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:32.547 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:32.547 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:32.565 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:32.585 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:32.586 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:32.606 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:32.606 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:32.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:32.625 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:32.625 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:32.633 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:32.642 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:32.642 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:32.643 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:32.643 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:25:32.643 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:25:32.643 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:25:32.643 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:32.658 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:32.659 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:32.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:32.758 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: pgmap v636: 168 pgs: 16 unknown, 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:25:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:25:32.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:25:32.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:25:32.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: osdmap e365: 8 total, 8 up, 8 in 2024-09-16T10:25:32.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:25:32.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:25:32.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2510196217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:32.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/978869499' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:25:32.847 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: pgmap v636: 168 pgs: 16 unknown, 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:25:32.847 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:25:32.847 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:25:32.847 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: osdmap e365: 8 total, 8 up, 8 in 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2510196217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/978869499' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: pgmap v636: 168 pgs: 16 unknown, 152 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:25:32.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: osdmap e365: 8 total, 8 up, 8 in 2024-09-16T10:25:32.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:25:32.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:25:32.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2510196217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:32.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/978869499' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:25:32.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:32.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:32.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:32.959 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-16T10:25:32.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.006 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:33.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:33.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.131 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:33.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:33.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:33.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:33.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:33.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:33.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:25:33.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:25:33.488 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:25:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 366, 2024-09-16T10:25:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:25:33.488 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:25:32.956659+0000", 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:22:01.866838+0000", 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:25:33.489 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:25:33.490 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 10, 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:25:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:25:33.492 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:25:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:25:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:25:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:25:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:25:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:25:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:25:33.493 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 21, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 21, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 21, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 21, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 21, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.b", 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 275, 2024-09-16T10:25:33.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 274, 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 274, 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "357", 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "275", 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "306", 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:25:33.495 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:25:33.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:25:33.497 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.498 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.8999999761581421, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.8999999761581421, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.8999999761581421, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.8999999761581421, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:25:33.499 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:25:33.500 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:25:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:25:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:25:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:25:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:25:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:25:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:25:33.501 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:25:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:25:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:25:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:25:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:25:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:25:33.502 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:25:33.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:25:33.504 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:25:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:25:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:25:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:25:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:25:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:25:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:25:33.505 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:25:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:25:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:25:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:25:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:25:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:25:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:25:33.506 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:25:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:25:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:25:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:25:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:25:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:25:33.508 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:25:33.509 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:25:33.510 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:25:33.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:25:33.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:25:33.513 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:25:33.514 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:25:33.515 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:25:33.516 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:25:33.517 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:25:33.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:25:33.519 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:25:33.520 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:25:33.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:25:33.522 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.5, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.5, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.5, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.5, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:25:33.524 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:25:33.525 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:25:33.526 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:25:33.527 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:25:33.528 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:25:33.529 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:25:33.530 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:25:33.531 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:25:33.532 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:25:33.533 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.534 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.535 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 10, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_3", 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:25:29.286237+0000", 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:25:33.537 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:25:33.538 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "366", 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:25:33.539 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:25:33.540 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:25:33.541 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 3, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 3, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 3, 2024-09-16T10:25:33.542 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 3, 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:25:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.551 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:25:33.552 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.553 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.554 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:25:33.555 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.556 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 351, 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.557 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.558 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.559 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.560 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.561 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.563 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.564 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:25:33.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.566 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:25:33.567 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.569 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:25:33.570 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0.293609619140625, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:25:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:25:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:25:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.577 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:16.926902+0000", 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:31.981645+0000", 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:24:21.747854+0000", 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:25:33.579 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:20.945644+0000", 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.580 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:24:46.426238+0000", 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:03.025290+0000", 2024-09-16T10:25:33.581 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:24.228279+0000", 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:25:33.582 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:05.915019+0000", 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [ 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.0", 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:25:33.583 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.10", 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 6, 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.14", 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 1 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.585 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.13", 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.587 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:25:33.588 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:25:33.589 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:25:33.590 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:33.591 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:25:33.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:25:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:33.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:33.689 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:33.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:33.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/978869499' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[29993]: osdmap e366: 8 total, 8 up, 8 in 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[29993]: 3.8 scrub starts 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[29993]: 3.8 scrub ok 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3216537399' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/978869499' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[39470]: osdmap e366: 8 total, 8 up, 8 in 2024-09-16T10:25:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[39470]: 3.8 scrub starts 2024-09-16T10:25:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[39470]: 3.8 scrub ok 2024-09-16T10:25:33.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3216537399' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:25:33.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:33.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:33.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:33.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:33.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:33 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:25:33.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:33 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:25:33.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/978869499' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:25:33.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:33 smithi165 ceph-mon[32696]: osdmap e366: 8 total, 8 up, 8 in 2024-09-16T10:25:33.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:33 smithi165 ceph-mon[32696]: 3.8 scrub starts 2024-09-16T10:25:33.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:33 smithi165 ceph-mon[32696]: 3.8 scrub ok 2024-09-16T10:25:33.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3216537399' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:25:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:33.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:33.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:34.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:34.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:34.167 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90054 7.c 7:3cae452b:::benchmark_data_smithi063_73086_object90053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 584.65147277100004, 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12751799899999999, 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.168 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90054 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513437+0000", 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513440+0000", 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.452e-06 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513492+0000", 2024-09-16T10:25:34.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1950999999999998e-05 2024-09-16T10:25:34.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513503+0000", 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1232e-05 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513509+0000", 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0680000000000002e-06 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513571+0000", 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2434999999999994e-05 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513694+0000", 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012256499999999999 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.513865+0000", 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017080599999999999 2024-09-16T10:25:34.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.517203+0000", 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033382860000000002 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.518058+0000", 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085528199999999996 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640875+0000", 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12281645400000001 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640919+0000", 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.401e-05 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.640955+0000", 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6458000000000002e-05 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:90065 7.d 7:bffa18df:::benchmark_data_smithi063_73086_object90064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 584.64303934400004, 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124785905, 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90065 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521871+0000", 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521873+0000", 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.497e-06 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521903+0000", 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9407000000000002e-05 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521916+0000", 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3597999999999999e-05 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521924+0000", 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.521991+0000", 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7318999999999997e-05 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522105+0000", 2024-09-16T10:25:34.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113467 2024-09-16T10:25:34.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:25:34.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.522205+0000", 2024-09-16T10:25:34.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9829999999999998e-05 2024-09-16T10:25:34.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525304+0000", 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030995089999999999 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.525567+0000", 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026322999999999999 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646542+0000", 2024-09-16T10:25:34.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120974217 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646604+0000", 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2543000000000001e-05 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:15:49.646657+0000", 2024-09-16T10:25:34.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2438000000000002e-05 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174956 7.7 7:eae3f4a6:::benchmark_data_smithi063_73086_object55677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 500.27522128700002, 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12889425400000001, 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.184 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174956 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:25:34.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889689+0000", 2024-09-16T10:25:34.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889690+0000", 2024-09-16T10:25:34.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.562e-06 2024-09-16T10:25:34.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889695+0000", 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8149999999999996e-06 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889708+0000", 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2943e-05 2024-09-16T10:25:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889715+0000", 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.272e-06 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889765+0000", 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9412000000000001e-05 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889881+0000", 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116789 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.889985+0000", 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103195 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895390+0000", 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054051689999999996 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991131+0000", 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095740960999999999 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018515+0000", 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027384196 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018547+0000", 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1973e-05 2024-09-16T10:25:34.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018583+0000", 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5967000000000003e-05 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44750.0:174957 7.7 7:e71673f7:::benchmark_data_smithi063_73086_object55678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 500.27256144199998, 2024-09-16T10:25:34.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.126386786, 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44750", 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/543552058", 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 174957 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:25:34.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892349+0000", 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892351+0000", 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0310000000000001e-06 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892356+0000", 2024-09-16T10:25:34.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2689999999999999e-06 2024-09-16T10:25:34.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892368+0000", 2024-09-16T10:25:34.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1848000000000001e-05 2024-09-16T10:25:34.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892376+0000", 2024-09-16T10:25:34.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0649999999999999e-06 2024-09-16T10:25:34.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892738+0000", 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036210000000000002 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892862+0000", 2024-09-16T10:25:34.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012463099999999999 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.892975+0000", 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112526 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.895478+0000", 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002503211 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:13.991158+0000", 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095680116999999995 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018680+0000", 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027522007000000001 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018705+0000", 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4802000000000001e-05 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:17:14.018735+0000", 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0179e-05 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:50103 8.5 8:a9c61bbe:::benchmark_data_smithi063_101290_object50102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:11.534001+0000", 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 382.63090919699999, 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113298935, 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 50103 2024-09-16T10:25:34.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534001+0000", 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534001+0000", 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534002+0000", 2024-09-16T10:25:34.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3170000000000001e-06 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534032+0000", 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9859999999999999e-05 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534042+0000", 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0040999999999999e-05 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.534048+0000", 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2790000000000004e-06 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.537093+0000", 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003044593 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.537248+0000", 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155484 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.540637+0000", 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033885410000000001 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.547818+0000", 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071807579999999998 2024-09-16T10:25:34.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647081+0000", 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099263245 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647209+0000", 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000127782 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647261+0000", 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2624999999999998e-05 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:11.647300+0000", 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.841e-05 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:130414 8.9 8:968d8679:::benchmark_data_smithi063_101290_object3094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:20:13.782273+0000", 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.38263669999998, 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115477528, 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 130414 2024-09-16T10:25:34.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782273+0000", 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782273+0000", 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782275+0000", 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.733e-06 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782280+0000", 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.006e-06 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782290+0000", 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8400000000000007e-06 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782296+0000", 2024-09-16T10:25:34.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782342+0000", 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5874999999999997e-05 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782443+0000", 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100833 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:25:34.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.782618+0000", 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017475600000000001 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.785644+0000", 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030258749999999999 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.785923+0000", 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027986599999999998 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.897607+0000", 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111683594 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.897694+0000", 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7159999999999999e-05 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:20:13.897751+0000", 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6598000000000003e-05 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.216 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:213868 8.9 8:9bfe1c7f:::benchmark_data_smithi063_101290_object86548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:13.085724+0000", 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 261.07918626100002, 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13858646499999999, 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213868 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085724+0000", 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085724+0000", 2024-09-16T10:25:34.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085726+0000", 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.804e-06 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085731+0000", 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3329999999999996e-06 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085743+0000", 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2173e-05 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085749+0000", 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0549999999999996e-06 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085809+0000", 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0038999999999999e-05 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085897+0000", 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7892999999999993e-05 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:25:34.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.085969+0000", 2024-09-16T10:25:34.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2039000000000006e-05 2024-09-16T10:25:34.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.087832+0000", 2024-09-16T10:25:34.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018625429999999999 2024-09-16T10:25:34.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.088125+0000", 2024-09-16T10:25:34.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029382799999999999 2024-09-16T10:25:34.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224207+0000", 2024-09-16T10:25:34.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13608161899999999 2024-09-16T10:25:34.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224261+0000", 2024-09-16T10:25:34.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4091000000000001e-05 2024-09-16T10:25:34.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224310+0000", 2024-09-16T10:25:34.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9048000000000002e-05 2024-09-16T10:25:34.226 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.226 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.226 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.227 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:213877 8.9 8:96066085:::benchmark_data_smithi063_101290_object86557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-16T10:25:34.227 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:13.089496+0000", 2024-09-16T10:25:34.227 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 261.07541355000001, 2024-09-16T10:25:34.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.134949242, 2024-09-16T10:25:34.228 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.228 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.228 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:25:34.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:34.229 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213877 2024-09-16T10:25:34.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.229 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089496+0000", 2024-09-16T10:25:34.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089496+0000", 2024-09-16T10:25:34.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089499+0000", 2024-09-16T10:25:34.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1950000000000002e-06 2024-09-16T10:25:34.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089504+0000", 2024-09-16T10:25:34.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5160000000000002e-06 2024-09-16T10:25:34.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089518+0000", 2024-09-16T10:25:34.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.398e-05 2024-09-16T10:25:34.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089525+0000", 2024-09-16T10:25:34.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999996e-06 2024-09-16T10:25:34.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089569+0000", 2024-09-16T10:25:34.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3733000000000003e-05 2024-09-16T10:25:34.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089647+0000", 2024-09-16T10:25:34.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8411000000000005e-05 2024-09-16T10:25:34.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:25:34.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.089716+0000", 2024-09-16T10:25:34.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9054999999999995e-05 2024-09-16T10:25:34.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.091677+0000", 2024-09-16T10:25:34.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019602700000000001 2024-09-16T10:25:34.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.092671+0000", 2024-09-16T10:25:34.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00099401099999999994 2024-09-16T10:25:34.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224358+0000", 2024-09-16T10:25:34.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.131687784 2024-09-16T10:25:34.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224406+0000", 2024-09-16T10:25:34.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7942000000000003e-05 2024-09-16T10:25:34.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:13.224446+0000", 2024-09-16T10:25:34.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9314999999999999e-05 2024-09-16T10:25:34.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.241 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.242 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:231103 8.9 8:90e59f9e:::benchmark_data_smithi063_101290_object103783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:25:34.242 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:24.761238+0000", 2024-09-16T10:25:34.242 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 249.403671745, 2024-09-16T10:25:34.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111702601, 2024-09-16T10:25:34.242 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.243 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.243 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:25:34.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:25:34.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:34.243 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 231103 2024-09-16T10:25:34.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.244 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761238+0000", 2024-09-16T10:25:34.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761238+0000", 2024-09-16T10:25:34.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761239+0000", 2024-09-16T10:25:34.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0729999999999999e-06 2024-09-16T10:25:34.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761243+0000", 2024-09-16T10:25:34.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.67e-06 2024-09-16T10:25:34.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761249+0000", 2024-09-16T10:25:34.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8810000000000001e-06 2024-09-16T10:25:34.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761253+0000", 2024-09-16T10:25:34.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5079999999999999e-06 2024-09-16T10:25:34.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761305+0000", 2024-09-16T10:25:34.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2107000000000001e-05 2024-09-16T10:25:34.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761419+0000", 2024-09-16T10:25:34.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113292 2024-09-16T10:25:34.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:25:34.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.761540+0000", 2024-09-16T10:25:34.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121296 2024-09-16T10:25:34.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:25:34.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.763633+0000", 2024-09-16T10:25:34.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020926930000000001 2024-09-16T10:25:34.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.763827+0000", 2024-09-16T10:25:34.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000194539 2024-09-16T10:25:34.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:34.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.872878+0000", 2024-09-16T10:25:34.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109050915 2024-09-16T10:25:34.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.872914+0000", 2024-09-16T10:25:34.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5453000000000001e-05 2024-09-16T10:25:34.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.872941+0000", 2024-09-16T10:25:34.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7174000000000001e-05 2024-09-16T10:25:34.256 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.256 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.257 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.257 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:224049 9.1 e356/276 9:8948ad32:::benchmark_data_smithi063_124717_object107818:head v 356'13747, mlcod=356'13744)", 2024-09-16T10:25:34.257 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:12.024185+0000", 2024-09-16T10:25:34.257 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.140725025999998, 2024-09-16T10:25:34.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063477990000000003, 2024-09-16T10:25:34.258 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.258 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.258 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.258 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024185+0000", 2024-09-16T10:25:34.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024185+0000", 2024-09-16T10:25:34.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024186+0000", 2024-09-16T10:25:34.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1400000000000001e-06 2024-09-16T10:25:34.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024192+0000", 2024-09-16T10:25:34.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3679999999999998e-06 2024-09-16T10:25:34.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024202+0000", 2024-09-16T10:25:34.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6360000000000006e-06 2024-09-16T10:25:34.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024208+0000", 2024-09-16T10:25:34.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3990000000000001e-06 2024-09-16T10:25:34.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024256+0000", 2024-09-16T10:25:34.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7645999999999997e-05 2024-09-16T10:25:34.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.024274+0000", 2024-09-16T10:25:34.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8085e-05 2024-09-16T10:25:34.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.030494+0000", 2024-09-16T10:25:34.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062194140000000004 2024-09-16T10:25:34.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:12.030533+0000", 2024-09-16T10:25:34.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9110999999999999e-05 2024-09-16T10:25:34.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.267 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.268 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:226495 9.1 e356/276 9:86f0fa28:::benchmark_data_smithi063_124717_object110264:head v 356'13879, mlcod=356'13877)", 2024-09-16T10:25:34.268 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:13.893345+0000", 2024-09-16T10:25:34.268 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.271565469999999, 2024-09-16T10:25:34.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080767730000000006, 2024-09-16T10:25:34.268 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.269 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.269 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.269 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893345+0000", 2024-09-16T10:25:34.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893345+0000", 2024-09-16T10:25:34.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893346+0000", 2024-09-16T10:25:34.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4869999999999999e-06 2024-09-16T10:25:34.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893354+0000", 2024-09-16T10:25:34.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6760000000000007e-06 2024-09-16T10:25:34.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893368+0000", 2024-09-16T10:25:34.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4531e-05 2024-09-16T10:25:34.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893375+0000", 2024-09-16T10:25:34.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489999999999997e-06 2024-09-16T10:25:34.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893433+0000", 2024-09-16T10:25:34.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8263000000000001e-05 2024-09-16T10:25:34.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.893452+0000", 2024-09-16T10:25:34.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9083999999999999e-05 2024-09-16T10:25:34.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.901134+0000", 2024-09-16T10:25:34.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076825879999999997 2024-09-16T10:25:34.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.901421+0000", 2024-09-16T10:25:34.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028679500000000001 2024-09-16T10:25:34.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.278 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.278 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:226527 9.b e356/354 9:d5f34aa7:::benchmark_data_smithi063_124717_object110296:head v 356'13987, mlcod=356'13985)", 2024-09-16T10:25:34.279 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:13.919028+0000", 2024-09-16T10:25:34.279 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.245881536999999, 2024-09-16T10:25:34.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006448507, 2024-09-16T10:25:34.279 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.279 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.279 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.280 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919028+0000", 2024-09-16T10:25:34.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919028+0000", 2024-09-16T10:25:34.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919030+0000", 2024-09-16T10:25:34.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-16T10:25:34.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919036+0000", 2024-09-16T10:25:34.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4439999999999996e-06 2024-09-16T10:25:34.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919050+0000", 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.413e-05 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919057+0000", 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999996e-06 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919112+0000", 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4617000000000002e-05 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.919136+0000", 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4532000000000002e-05 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.925398+0000", 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062615140000000001 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:13.925477+0000", 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8979000000000004e-05 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:227908 9.0 e356/281 9:0d4563de:::benchmark_data_smithi063_124717_object111677:head v 356'14082, mlcod=356'14080)", 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:15.099864+0000", 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.065045783999999, 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069914160000000003, 2024-09-16T10:25:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099864+0000", 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099864+0000", 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099866+0000", 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1730000000000002e-06 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099882+0000", 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6056e-05 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099913+0000", 2024-09-16T10:25:34.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0735000000000003e-05 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099918+0000", 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9710000000000003e-06 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099972+0000", 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4113999999999997e-05 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.099995+0000", 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3019000000000001e-05 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.106809+0000", 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068138649999999997 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.106856+0000", 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6483000000000002e-05 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:228066 9.0 e356/281 9:0fac2325:::benchmark_data_smithi063_124717_object111835:head v 356'14094, mlcod=356'14091)", 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:15.251994+0000", 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.912915817999998, 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067497060000000003, 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.251994+0000", 2024-09-16T10:25:34.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.251994+0000", 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.251995+0000", 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7100000000000002e-07 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.251999+0000", 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4039999999999997e-06 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.252005+0000", 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9490000000000001e-06 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.252010+0000", 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.113e-06 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.254426+0000", 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024156759999999998 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.254447+0000", 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1274999999999999e-05 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.258687+0000", 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042397099999999998 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.258744+0000", 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7108000000000003e-05 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:228261 9.b e356/354 9:d1d21313:::benchmark_data_smithi063_124717_object112030:head v 356'14093, mlcod=356'14091)", 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:15.402175+0000", 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.762734576, 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069029080000000001, 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.295 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.402175+0000", 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.402175+0000", 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.402176+0000", 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0139999999999999e-06 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.402183+0000", 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1349999999999997e-06 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.402192+0000", 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9219999999999992e-06 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.402199+0000", 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8669999999999996e-06 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.403592+0000", 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013924429999999999 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.403611+0000", 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9403999999999999e-05 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.409037+0000", 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054258290000000001 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.409078+0000", 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1294000000000001e-05 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:228468 9.b e356/354 9:de40585d:::benchmark_data_smithi063_124717_object112237:head v 356'14106, mlcod=356'14102)", 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:15.595621+0000", 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.569288606000001, 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006349431, 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.595621+0000", 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.595621+0000", 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.595622+0000", 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.595625+0000", 2024-09-16T10:25:34.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5659999999999998e-06 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.595630+0000", 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9239999999999998e-06 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.595635+0000", 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3499999999999999e-06 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.598760+0000", 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031257059999999998 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.598778+0000", 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8108999999999999e-05 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.601918+0000", 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031399900000000001 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.601971+0000", 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2404999999999997e-05 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45596.0:228697 9.b e356/354 9:d6bc95ac:::benchmark_data_smithi063_124717_object112466:head v 356'14116, mlcod=356'14113)", 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:15.765764+0000", 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.399146057999999, 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071711869999999999, 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:25:34.304 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765764+0000", 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765764+0000", 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765765+0000", 2024-09-16T10:25:34.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5099999999999998e-07 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765770+0000", 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0470000000000001e-06 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765778+0000", 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3720000000000005e-06 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:25:34.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765783+0000", 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1789999999999999e-06 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765855+0000", 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2101999999999995e-05 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.765875+0000", 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9606000000000002e-05 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.772864+0000", 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006988457 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:15.772935+0000", 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1573000000000001e-05 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:25:34.308 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:34.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:34.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:34.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:34.458 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:34.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:34.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:34.586 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:34.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:34.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:34.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:34.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:34.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:34.833 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:34.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:34.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:34.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: pgmap v639: 168 pgs: 13 unknown, 155 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: 5.1b scrub starts 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: 5.1b scrub ok 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: pgmap v639: 168 pgs: 13 unknown, 155 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: 5.1b scrub starts 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: 5.1b scrub ok 2024-09-16T10:25:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:25:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:25:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:25:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:25:34.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:34 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:25:34.958 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:34.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: pgmap v639: 168 pgs: 13 unknown, 155 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: 5.1b scrub starts 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: 5.1b scrub ok 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:25:35.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:34 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:25:35.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:25:35.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:35.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:35.120 INFO:tasks.thrashosds.thrasher:j is {'epoch': 366, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:25:32.956659+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:22:01.866838+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 275, 'last_epoch_started': 274, 'last_epoch_clean': 274, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '357', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '275', 'last_force_op_resend_preluminous': '306', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 1, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-16T10:25:29.286237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '366', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 351, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.293609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:16.926902+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:31.981645+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:24:21.747854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:20.945644+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:24:46.426238+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:03.025290+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:24.228279+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:05.915019+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.0', 'osds': [4, 6, 2]}], 'pg_upmap_items': [{'pgid': '3.10', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:25:35.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:25:35.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:35.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:35.282 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:35.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:35.345 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:35.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:35.469 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:35.538 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:35.568 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:35.571 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:35.594 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:35.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:35 smithi165 ceph-mon[32696]: 6.8 scrub starts 2024-09-16T10:25:35.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:35 smithi165 ceph-mon[32696]: 6.8 scrub ok 2024-09-16T10:25:35.666 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:35 smithi165 ceph-mon[32696]: osdmap e367: 8 total, 8 up, 8 in 2024-09-16T10:25:35.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:35.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:35.768 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:35.768 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:25:35.768 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:25:35.768 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:35.771 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:35 smithi063 ceph-mon[29993]: 6.8 scrub starts 2024-09-16T10:25:35.771 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:35 smithi063 ceph-mon[29993]: 6.8 scrub ok 2024-09-16T10:25:35.771 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:35 smithi063 ceph-mon[29993]: osdmap e367: 8 total, 8 up, 8 in 2024-09-16T10:25:35.771 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:35 smithi063 ceph-mon[39470]: 6.8 scrub starts 2024-09-16T10:25:35.771 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:35 smithi063 ceph-mon[39470]: 6.8 scrub ok 2024-09-16T10:25:35.771 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:35 smithi063 ceph-mon[39470]: osdmap e367: 8 total, 8 up, 8 in 2024-09-16T10:25:35.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:35.845 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:35.946 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:35.971 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:35.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:36.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:36.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:36.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:36.197 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:36.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:36.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:36.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:36.346 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:36.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:36.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:36.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:36.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:36.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:25:36.521 DEBUG:teuthology.orchestra.run.smithi063:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-16T10:25:36.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:25:36.526 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-16T10:25:36.526 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-16T10:25:36.572 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:36.594 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:36.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:36.621 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:25:36.623 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:25:36.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:hints = 1 2024-09-16T10:25:36.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-16T10:25:36.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object prefix: benchmark_data_smithi063_149043 2024-09-16T10:25:36.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:25:36.690 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:25:36.696 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: pgmap v641: 168 pgs: 168 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 4.a scrub starts 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 4.a scrub ok 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 5.4 scrub starts 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 5.4 scrub ok 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:25:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: osdmap e368: 8 total, 8 up, 8 in 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2005847478' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[29993]: from='client.45992 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: pgmap v641: 168 pgs: 168 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:25:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 4.a scrub starts 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 4.a scrub ok 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 5.4 scrub starts 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 5.4 scrub ok 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: osdmap e368: 8 total, 8 up, 8 in 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2005847478' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:36.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:36 smithi063 ceph-mon[39470]: from='client.45992 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:36.742 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:36.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: pgmap v641: 168 pgs: 168 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:25:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 4.a scrub starts 2024-09-16T10:25:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 4.a scrub ok 2024-09-16T10:25:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 5.4 scrub starts 2024-09-16T10:25:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 5.4 scrub ok 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: osdmap e368: 8 total, 8 up, 8 in 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2005847478' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:36.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:36 smithi165 ceph-mon[32696]: from='client.45992 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:36.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:36.863 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:36.881 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:36.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:36.969 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:25:36.982 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:37.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:37.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:37.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:37.170 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:37.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:37.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:37.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:37.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:37.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:37.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:37.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:37.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:25:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:25:37.514 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:25:37.515 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:25:37.515 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:25:37.515 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:25:37.515 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:25:37.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:37.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:37.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:37.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:37.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 1 16 3408 3392 211.988 212 0.00330418 0.00470341 2024-09-16T10:25:37.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:37.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:37.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:37.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: from='client.16143 v1:172.21.15.63:0/3441082946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: 5.19 scrub starts 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: 5.19 scrub ok 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: from='client.45992 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: osdmap e369: 8 total, 8 up, 8 in 2024-09-16T10:25:37.929 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[39470]: 5.5 scrub starts 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: from='client.16143 v1:172.21.15.63:0/3441082946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: 5.19 scrub starts 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: 5.19 scrub ok 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: from='client.45992 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: osdmap e369: 8 total, 8 up, 8 in 2024-09-16T10:25:37.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:37 smithi063 ceph-mon[29993]: 5.5 scrub starts 2024-09-16T10:25:37.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:37.988 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2024-09-16T10:25:35.946643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1178,"num_read_kb":930,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":527794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1003,"ondisk_log_size":1003,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":490,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3540176,"kb_used_data":207244,"kb_used_omap":12,"kb_used_meta":3332723,"kb_avail":746421040,"statfs":{"total":767960285184,"available":764335144960,"internally_reserved":0,"allocated":212217856,"data_stored":207845389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3412708913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.648531"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":676,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.295064+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:25:33.295064+0000","last_peered":"2024-09-16T10:25:33.295064+0000","last_clean":"2024-09-16T10:25:33.295064+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:25:33.295064+0000","last_undegraded":"2024-09-16T10:25:33.295064+0000","last_fullsized":"2024-09-16T10:25:33.295064+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":724,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959697+0000","last_change":"2024-09-16T10:25:04.357048+0000","last_active":"2024-09-16T10:25:32.959697+0000","last_peered":"2024-09-16T10:25:32.959697+0000","last_clean":"2024-09-16T10:25:32.959697+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:25:32.959697+0000","last_undegraded":"2024-09-16T10:25:32.959697+0000","last_fullsized":"2024-09-16T10:25:32.959697+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:25:04.356955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.356955+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:13.131130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":654,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961060+0000","last_change":"2024-09-16T10:22:02.564509+0000","last_active":"2024-09-16T10:25:32.961060+0000","last_peered":"2024-09-16T10:25:32.961060+0000","last_clean":"2024-09-16T10:25:32.961060+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:25:32.961060+0000","last_undegraded":"2024-09-16T10:25:32.961060+0000","last_fullsized":"2024-09-16T10:25:32.961060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:02.564458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:02.564458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":657,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959895+0000","last_change":"2024-09-16T10:25:08.485862+0000","last_active":"2024-09-16T10:25:32.959895+0000","last_peered":"2024-09-16T10:25:32.959895+0000","last_clean":"2024-09-16T10:25:32.959895+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:25:32.959895+0000","last_undegraded":"2024-09-16T10:25:32.959895+0000","last_fullsized":"2024-09-16T10:25:32.959895+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.485783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.485783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:08.732498+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":640,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046462+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:25:33.046462+0000","last_peered":"2024-09-16T10:25:33.046462+0000","last_clean":"2024-09-16T10:25:33.046462+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:25:33.046462+0000","last_undegraded":"2024-09-16T10:25:33.046462+0000","last_fullsized":"2024-09-16T10:25:33.046462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:42.600819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":683,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046637+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:25:33.046637+0000","last_peered":"2024-09-16T10:25:33.046637+0000","last_clean":"2024-09-16T10:25:33.046637+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:25:33.046637+0000","last_undegraded":"2024-09-16T10:25:33.046637+0000","last_fullsized":"2024-09-16T10:25:33.046637+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.018649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":745,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046318+0000","last_change":"2024-09-16T10:24:37.612269+0000","last_active":"2024-09-16T10:25:33.046318+0000","last_peered":"2024-09-16T10:25:33.046318+0000","last_clean":"2024-09-16T10:25:33.046318+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:25:33.046318+0000","last_undegraded":"2024-09-16T10:25:33.046318+0000","last_fullsized":"2024-09-16T10:25:33.046318+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:37.612167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.612167+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.984288+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959684+0000","last_change":"2024-09-16T10:24:21.153205+0000","last_active":"2024-09-16T10:25:32.959684+0000","last_peered":"2024-09-16T10:25:32.959684+0000","last_clean":"2024-09-16T10:25:32.959684+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:25:32.959684+0000","last_undegraded":"2024-09-16T10:25:32.959684+0000","last_fullsized":"2024-09-16T10:25:32.959684+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:21.153131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:21.153131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.430119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":676,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261735+0000","last_change":"2024-09-16T10:25:13.988034+0000","last_active":"2024-09-16T10:25:32.261735+0000","last_peered":"2024-09-16T10:25:32.261735+0000","last_clean":"2024-09-16T10:25:32.261735+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:25:32.261735+0000","last_undegraded":"2024-09-16T10:25:32.261735+0000","last_fullsized":"2024-09-16T10:25:32.261735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.987974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.987974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.738929+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":690,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960205+0000","last_change":"2024-09-16T10:24:23.258943+0000","last_active":"2024-09-16T10:25:32.960205+0000","last_peered":"2024-09-16T10:25:32.960205+0000","last_clean":"2024-09-16T10:25:32.960205+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:25:32.960205+0000","last_undegraded":"2024-09-16T10:25:32.960205+0000","last_fullsized":"2024-09-16T10:25:32.960205+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:24:23.258886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.258886+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:34.336667+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":652,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261612+0000","last_change":"2024-09-16T10:24:25.902680+0000","last_active":"2024-09-16T10:25:32.261612+0000","last_peered":"2024-09-16T10:25:32.261612+0000","last_clean":"2024-09-16T10:25:32.261612+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:25:32.261612+0000","last_undegraded":"2024-09-16T10:25:32.261612+0000","last_fullsized":"2024-09-16T10:25:32.261612+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:25.902605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:25.902605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":649,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.293931+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:25:33.293931+0000","last_peered":"2024-09-16T10:25:33.293931+0000","last_clean":"2024-09-16T10:25:33.293931+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:25:33.293931+0000","last_undegraded":"2024-09-16T10:25:33.293931+0000","last_fullsized":"2024-09-16T10:25:33.293931+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":628,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.260966+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:25:32.260966+0000","last_peered":"2024-09-16T10:25:32.260966+0000","last_clean":"2024-09-16T10:25:32.260966+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:25:32.260966+0000","last_undegraded":"2024-09-16T10:25:32.260966+0000","last_fullsized":"2024-09-16T10:25:32.260966+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.823305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":678,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959972+0000","last_change":"2024-09-16T10:24:37.134728+0000","last_active":"2024-09-16T10:25:32.959972+0000","last_peered":"2024-09-16T10:25:32.959972+0000","last_clean":"2024-09-16T10:25:32.959972+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:25:32.959972+0000","last_undegraded":"2024-09-16T10:25:32.959972+0000","last_fullsized":"2024-09-16T10:25:32.959972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:41.767033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":661,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.278872+0000","last_change":"2024-09-16T10:25:15.369074+0000","last_active":"2024-09-16T10:25:33.278872+0000","last_peered":"2024-09-16T10:25:33.278872+0000","last_clean":"2024-09-16T10:25:33.278872+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:25:33.278872+0000","last_undegraded":"2024-09-16T10:25:33.278872+0000","last_fullsized":"2024-09-16T10:25:33.278872+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.369005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.369005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.347891+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":663,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261220+0000","last_change":"2024-09-16T10:23:42.856401+0000","last_active":"2024-09-16T10:25:32.261220+0000","last_peered":"2024-09-16T10:25:32.261220+0000","last_clean":"2024-09-16T10:25:32.261220+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:25:32.261220+0000","last_undegraded":"2024-09-16T10:25:32.261220+0000","last_fullsized":"2024-09-16T10:25:32.261220+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.856353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.856353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:25.927067+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":642,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960705+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:25:32.960705+0000","last_peered":"2024-09-16T10:25:32.960705+0000","last_clean":"2024-09-16T10:25:32.960705+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:25:32.960705+0000","last_undegraded":"2024-09-16T10:25:32.960705+0000","last_fullsized":"2024-09-16T10:25:32.960705+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:56.358685+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294765+0000","last_change":"2024-09-16T10:22:27.210541+0000","last_active":"2024-09-16T10:25:33.294765+0000","last_peered":"2024-09-16T10:25:33.294765+0000","last_clean":"2024-09-16T10:25:33.294765+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:25:33.294765+0000","last_undegraded":"2024-09-16T10:25:33.294765+0000","last_fullsized":"2024-09-16T10:25:33.294765+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":223,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961205+0000","last_change":"2024-09-16T10:25:08.473527+0000","last_active":"2024-09-16T10:25:32.961205+0000","last_peered":"2024-09-16T10:25:32.961205+0000","last_clean":"2024-09-16T10:25:32.961205+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:25:32.961205+0000","last_undegraded":"2024-09-16T10:25:32.961205+0000","last_fullsized":"2024-09-16T10:25:32.961205+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:25:08.473402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.473402+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.786201+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962013+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:25:32.962013+0000","last_peered":"2024-09-16T10:25:32.962013+0000","last_clean":"2024-09-16T10:25:32.962013+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:25:32.962013+0000","last_undegraded":"2024-09-16T10:25:32.962013+0000","last_fullsized":"2024-09-16T10:25:32.962013+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.737702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":606,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046463+0000","last_change":"2024-09-16T10:25:15.468418+0000","last_active":"2024-09-16T10:25:33.046463+0000","last_peered":"2024-09-16T10:25:33.046463+0000","last_clean":"2024-09-16T10:25:33.046463+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:25:33.046463+0000","last_undegraded":"2024-09-16T10:25:33.046463+0000","last_fullsized":"2024-09-16T10:25:33.046463+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.468271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.468271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.583811+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":667,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960301+0000","last_change":"2024-09-16T10:25:14.407660+0000","last_active":"2024-09-16T10:25:32.960301+0000","last_peered":"2024-09-16T10:25:32.960301+0000","last_clean":"2024-09-16T10:25:32.960301+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:25:32.960301+0000","last_undegraded":"2024-09-16T10:25:32.960301+0000","last_fullsized":"2024-09-16T10:25:32.960301+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.063208+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":715,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.045702+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:25:33.045702+0000","last_peered":"2024-09-16T10:25:33.045702+0000","last_clean":"2024-09-16T10:25:33.045702+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:25:33.045702+0000","last_undegraded":"2024-09-16T10:25:33.045702+0000","last_fullsized":"2024-09-16T10:25:33.045702+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.381310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":675,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960247+0000","last_change":"2024-09-16T10:24:35.186387+0000","last_active":"2024-09-16T10:25:32.960247+0000","last_peered":"2024-09-16T10:25:32.960247+0000","last_clean":"2024-09-16T10:25:32.960247+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:25:32.960247+0000","last_undegraded":"2024-09-16T10:25:32.960247+0000","last_fullsized":"2024-09-16T10:25:32.960247+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:35.186332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:35.186332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.208711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":60,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.260685+0000","last_change":"2024-09-16T10:25:18.973817+0000","last_active":"2024-09-16T10:25:32.260685+0000","last_peered":"2024-09-16T10:25:32.260685+0000","last_clean":"2024-09-16T10:25:32.260685+0000","last_became_active":"2024-09-16T10:25:00.947649+0000","last_became_peered":"2024-09-16T10:25:00.947649+0000","last_unstale":"2024-09-16T10:25:32.260685+0000","last_undegraded":"2024-09-16T10:25:32.260685+0000","last_fullsized":"2024-09-16T10:25:32.260685+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:44.946516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":661,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961196+0000","last_change":"2024-09-16T10:25:06.652209+0000","last_active":"2024-09-16T10:25:32.961196+0000","last_peered":"2024-09-16T10:25:32.961196+0000","last_clean":"2024-09-16T10:25:32.961196+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:25:32.961196+0000","last_undegraded":"2024-09-16T10:25:32.961196+0000","last_fullsized":"2024-09-16T10:25:32.961196+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:25:06.652146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:06.652146+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.202612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":688,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.295201+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:25:33.295201+0000","last_peered":"2024-09-16T10:25:33.295201+0000","last_clean":"2024-09-16T10:25:33.295201+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:25:33.295201+0000","last_undegraded":"2024-09-16T10:25:33.295201+0000","last_fullsized":"2024-09-16T10:25:33.295201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":807,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959546+0000","last_change":"2024-09-16T10:25:16.538777+0000","last_active":"2024-09-16T10:25:31.959546+0000","last_peered":"2024-09-16T10:25:31.959546+0000","last_clean":"2024-09-16T10:25:31.959546+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:25:31.959546+0000","last_undegraded":"2024-09-16T10:25:31.959546+0000","last_fullsized":"2024-09-16T10:25:31.959546+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:16.538706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.538706+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.979675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959810+0000","last_change":"2024-09-16T10:25:30.947944+0000","last_active":"2024-09-16T10:25:32.959810+0000","last_peered":"2024-09-16T10:25:32.959810+0000","last_clean":"2024-09-16T10:25:32.959810+0000","last_became_active":"2024-09-16T10:25:30.947200+0000","last_became_peered":"2024-09-16T10:25:30.947200+0000","last_unstale":"2024-09-16T10:25:32.959810+0000","last_undegraded":"2024-09-16T10:25:32.959810+0000","last_fullsized":"2024-09-16T10:25:32.959810+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.630601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":506,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.260948+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:25:32.260948+0000","last_peered":"2024-09-16T10:25:32.260948+0000","last_clean":"2024-09-16T10:25:32.260948+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:25:32.260948+0000","last_undegraded":"2024-09-16T10:25:32.260948+0000","last_fullsized":"2024-09-16T10:25:32.260948+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.039248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":694,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962279+0000","last_change":"2024-09-16T10:25:10.700157+0000","last_active":"2024-09-16T10:25:32.962279+0000","last_peered":"2024-09-16T10:25:32.962279+0000","last_clean":"2024-09-16T10:25:32.962279+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:25:32.962279+0000","last_undegraded":"2024-09-16T10:25:32.962279+0000","last_fullsized":"2024-09-16T10:25:32.962279+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:25:10.699735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:10.699735+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:35.612246+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":669,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.377037+0000","last_change":"2024-09-16T10:25:33.377037+0000","last_active":"2024-09-16T10:25:33.377037+0000","last_peered":"2024-09-16T10:25:33.377037+0000","last_clean":"2024-09-16T10:25:33.377037+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:25:33.377037+0000","last_undegraded":"2024-09-16T10:25:33.377037+0000","last_fullsized":"2024-09-16T10:25:33.377037+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:41.874798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":798,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960769+0000","last_change":"2024-09-16T10:24:57.408285+0000","last_active":"2024-09-16T10:25:32.960769+0000","last_peered":"2024-09-16T10:25:32.960769+0000","last_clean":"2024-09-16T10:25:32.960769+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:25:32.960769+0000","last_undegraded":"2024-09-16T10:25:32.960769+0000","last_fullsized":"2024-09-16T10:25:32.960769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:57.408217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.408217+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:09.573273+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":649,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261477+0000","last_change":"2024-09-16T10:24:15.859366+0000","last_active":"2024-09-16T10:25:32.261477+0000","last_peered":"2024-09-16T10:25:32.261477+0000","last_clean":"2024-09-16T10:25:32.261477+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:25:32.261477+0000","last_undegraded":"2024-09-16T10:25:32.261477+0000","last_fullsized":"2024-09-16T10:25:32.261477+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:15.859299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.859299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:41.356094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960568+0000","last_change":"2024-09-16T10:25:30.619893+0000","last_active":"2024-09-16T10:25:32.960568+0000","last_peered":"2024-09-16T10:25:32.960568+0000","last_clean":"2024-09-16T10:25:32.960568+0000","last_became_active":"2024-09-16T10:25:30.619574+0000","last_became_peered":"2024-09-16T10:25:30.619574+0000","last_unstale":"2024-09-16T10:25:32.960568+0000","last_undegraded":"2024-09-16T10:25:32.960568+0000","last_fullsized":"2024-09-16T10:25:32.960568+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.565846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":560,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960623+0000","last_change":"2024-09-16T10:24:28.146701+0000","last_active":"2024-09-16T10:25:32.960623+0000","last_peered":"2024-09-16T10:25:32.960623+0000","last_clean":"2024-09-16T10:25:32.960623+0000","last_became_active":"2024-09-16T10:22:56.409926+0000","last_became_peered":"2024-09-16T10:22:56.409926+0000","last_unstale":"2024-09-16T10:25:32.960623+0000","last_undegraded":"2024-09-16T10:25:32.960623+0000","last_fullsized":"2024-09-16T10:25:32.960623+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.146633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.146633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.409351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":143,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959575+0000","last_change":"2024-09-16T10:23:42.023903+0000","last_active":"2024-09-16T10:25:32.959575+0000","last_peered":"2024-09-16T10:25:32.959575+0000","last_clean":"2024-09-16T10:25:32.959575+0000","last_became_active":"2024-09-16T10:23:41.951456+0000","last_became_peered":"2024-09-16T10:23:41.951456+0000","last_unstale":"2024-09-16T10:25:32.959575+0000","last_undegraded":"2024-09-16T10:25:32.959575+0000","last_fullsized":"2024-09-16T10:25:32.959575+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:23:13.871560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.871560+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.206554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":626,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046844+0000","last_change":"2024-09-16T10:23:39.072937+0000","last_active":"2024-09-16T10:25:33.046844+0000","last_peered":"2024-09-16T10:25:33.046844+0000","last_clean":"2024-09-16T10:25:33.046844+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:25:33.046844+0000","last_undegraded":"2024-09-16T10:25:33.046844+0000","last_fullsized":"2024-09-16T10:25:33.046844+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:39.070281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:39.070281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":391,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958583+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:25:31.958583+0000","last_peered":"2024-09-16T10:25:31.958583+0000","last_clean":"2024-09-16T10:25:31.958583+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:25:31.958583+0000","last_undegraded":"2024-09-16T10:25:31.958583+0000","last_fullsized":"2024-09-16T10:25:31.958583+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.558332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":672,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960149+0000","last_change":"2024-09-16T10:24:22.206462+0000","last_active":"2024-09-16T10:25:32.960149+0000","last_peered":"2024-09-16T10:25:32.960149+0000","last_clean":"2024-09-16T10:25:32.960149+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:25:32.960149+0000","last_undegraded":"2024-09-16T10:25:32.960149+0000","last_fullsized":"2024-09-16T10:25:32.960149+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:22.206414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:22.206414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.375025+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959387+0000","last_change":"2024-09-16T10:25:31.163385+0000","last_active":"2024-09-16T10:25:31.959387+0000","last_peered":"2024-09-16T10:25:31.959387+0000","last_clean":"2024-09-16T10:25:31.959387+0000","last_became_active":"2024-09-16T10:25:31.162906+0000","last_became_peered":"2024-09-16T10:25:31.162906+0000","last_unstale":"2024-09-16T10:25:31.959387+0000","last_undegraded":"2024-09-16T10:25:31.959387+0000","last_fullsized":"2024-09-16T10:25:31.959387+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.426084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":551,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046833+0000","last_change":"2024-09-16T10:24:28.343296+0000","last_active":"2024-09-16T10:25:33.046833+0000","last_peered":"2024-09-16T10:25:33.046833+0000","last_clean":"2024-09-16T10:25:33.046833+0000","last_became_active":"2024-09-16T10:22:56.446950+0000","last_became_peered":"2024-09-16T10:22:56.446950+0000","last_unstale":"2024-09-16T10:25:33.046833+0000","last_undegraded":"2024-09-16T10:25:33.046833+0000","last_fullsized":"2024-09-16T10:25:33.046833+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.343066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.343066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":708,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959770+0000","last_change":"2024-09-16T10:24:59.344438+0000","last_active":"2024-09-16T10:25:32.959770+0000","last_peered":"2024-09-16T10:25:32.959770+0000","last_clean":"2024-09-16T10:25:32.959770+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:25:32.959770+0000","last_undegraded":"2024-09-16T10:25:32.959770+0000","last_fullsized":"2024-09-16T10:25:32.959770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:24:59.344346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.344346+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:05.786389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":655,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.295344+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:25:33.295344+0000","last_peered":"2024-09-16T10:25:33.295344+0000","last_clean":"2024-09-16T10:25:33.295344+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:25:33.295344+0000","last_undegraded":"2024-09-16T10:25:33.295344+0000","last_fullsized":"2024-09-16T10:25:33.295344+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.295227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":808,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.045792+0000","last_change":"2024-09-16T10:25:12.476210+0000","last_active":"2024-09-16T10:25:33.045792+0000","last_peered":"2024-09-16T10:25:33.045792+0000","last_clean":"2024-09-16T10:25:33.045792+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:25:33.045792+0000","last_undegraded":"2024-09-16T10:25:33.045792+0000","last_fullsized":"2024-09-16T10:25:33.045792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:12.476137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.476137+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:36.834759+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":500,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959536+0000","last_change":"2024-09-16T10:25:00.948343+0000","last_active":"2024-09-16T10:25:31.959536+0000","last_peered":"2024-09-16T10:25:31.959536+0000","last_clean":"2024-09-16T10:25:31.959536+0000","last_became_active":"2024-09-16T10:25:00.946867+0000","last_became_peered":"2024-09-16T10:25:00.946867+0000","last_unstale":"2024-09-16T10:25:31.959536+0000","last_undegraded":"2024-09-16T10:25:31.959536+0000","last_fullsized":"2024-09-16T10:25:31.959536+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.423640+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960541+0000","last_change":"2024-09-16T10:25:30.949013+0000","last_active":"2024-09-16T10:25:32.960541+0000","last_peered":"2024-09-16T10:25:32.960541+0000","last_clean":"2024-09-16T10:25:32.960541+0000","last_became_active":"2024-09-16T10:25:30.948309+0000","last_became_peered":"2024-09-16T10:25:30.948309+0000","last_unstale":"2024-09-16T10:25:32.960541+0000","last_undegraded":"2024-09-16T10:25:32.960541+0000","last_fullsized":"2024-09-16T10:25:32.960541+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.205764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":599,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959359+0000","last_change":"2024-09-16T10:24:31.617241+0000","last_active":"2024-09-16T10:25:31.959359+0000","last_peered":"2024-09-16T10:25:31.959359+0000","last_clean":"2024-09-16T10:25:31.959359+0000","last_became_active":"2024-09-16T10:22:56.438108+0000","last_became_peered":"2024-09-16T10:22:56.438108+0000","last_unstale":"2024-09-16T10:25:31.959359+0000","last_undegraded":"2024-09-16T10:25:31.959359+0000","last_fullsized":"2024-09-16T10:25:31.959359+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.617190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.617190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.711210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":697,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962375+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:25:32.962375+0000","last_peered":"2024-09-16T10:25:32.962375+0000","last_clean":"2024-09-16T10:25:32.962375+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:25:32.962375+0000","last_undegraded":"2024-09-16T10:25:32.962375+0000","last_fullsized":"2024-09-16T10:25:32.962375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":675,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261764+0000","last_change":"2024-09-16T10:24:59.820337+0000","last_active":"2024-09-16T10:25:32.261764+0000","last_peered":"2024-09-16T10:25:32.261764+0000","last_clean":"2024-09-16T10:25:32.261764+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:25:32.261764+0000","last_undegraded":"2024-09-16T10:25:32.261764+0000","last_fullsized":"2024-09-16T10:25:32.261764+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:59.820273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.820273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.866898+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":790,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294505+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:25:33.294505+0000","last_peered":"2024-09-16T10:25:33.294505+0000","last_clean":"2024-09-16T10:25:33.294505+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:25:33.294505+0000","last_undegraded":"2024-09-16T10:25:33.294505+0000","last_fullsized":"2024-09-16T10:25:33.294505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":664,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294105+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:25:33.294105+0000","last_peered":"2024-09-16T10:25:33.294105+0000","last_clean":"2024-09-16T10:25:33.294105+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:25:33.294105+0000","last_undegraded":"2024-09-16T10:25:33.294105+0000","last_fullsized":"2024-09-16T10:25:33.294105+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959266+0000","last_change":"2024-09-16T10:25:30.621332+0000","last_active":"2024-09-16T10:25:31.959266+0000","last_peered":"2024-09-16T10:25:31.959266+0000","last_clean":"2024-09-16T10:25:31.959266+0000","last_became_active":"2024-09-16T10:25:30.620945+0000","last_became_peered":"2024-09-16T10:25:30.620945+0000","last_unstale":"2024-09-16T10:25:31.959266+0000","last_undegraded":"2024-09-16T10:25:31.959266+0000","last_fullsized":"2024-09-16T10:25:31.959266+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.036991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":604,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958992+0000","last_change":"2024-09-16T10:24:30.609351+0000","last_active":"2024-09-16T10:25:31.958992+0000","last_peered":"2024-09-16T10:25:31.958992+0000","last_clean":"2024-09-16T10:25:31.958992+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:25:31.958992+0000","last_undegraded":"2024-09-16T10:25:31.958992+0000","last_fullsized":"2024-09-16T10:25:31.958992+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:30.609302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:30.609302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:47.839505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":709,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294575+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:25:33.294575+0000","last_peered":"2024-09-16T10:25:33.294575+0000","last_clean":"2024-09-16T10:25:33.294575+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:25:33.294575+0000","last_undegraded":"2024-09-16T10:25:33.294575+0000","last_fullsized":"2024-09-16T10:25:33.294575+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":677,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962145+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:25:32.962145+0000","last_peered":"2024-09-16T10:25:32.962145+0000","last_clean":"2024-09-16T10:25:32.962145+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:25:32.962145+0000","last_undegraded":"2024-09-16T10:25:32.962145+0000","last_fullsized":"2024-09-16T10:25:32.962145+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":668,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958860+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:25:31.958860+0000","last_peered":"2024-09-16T10:25:31.958860+0000","last_clean":"2024-09-16T10:25:31.958860+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:25:31.958860+0000","last_undegraded":"2024-09-16T10:25:31.958860+0000","last_fullsized":"2024-09-16T10:25:31.958860+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.230724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":657,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960263+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:25:32.960263+0000","last_peered":"2024-09-16T10:25:32.960263+0000","last_clean":"2024-09-16T10:25:32.960263+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:25:32.960263+0000","last_undegraded":"2024-09-16T10:25:32.960263+0000","last_fullsized":"2024-09-16T10:25:32.960263+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.861279+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959231+0000","last_change":"2024-09-16T10:25:30.622174+0000","last_active":"2024-09-16T10:25:31.959231+0000","last_peered":"2024-09-16T10:25:31.959231+0000","last_clean":"2024-09-16T10:25:31.959231+0000","last_became_active":"2024-09-16T10:25:30.621742+0000","last_became_peered":"2024-09-16T10:25:30.621742+0000","last_unstale":"2024-09-16T10:25:31.959231+0000","last_undegraded":"2024-09-16T10:25:31.959231+0000","last_fullsized":"2024-09-16T10:25:31.959231+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:46.549823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":550,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261011+0000","last_change":"2024-09-16T10:24:29.815138+0000","last_active":"2024-09-16T10:25:32.261011+0000","last_peered":"2024-09-16T10:25:32.261011+0000","last_clean":"2024-09-16T10:25:32.261011+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:25:32.261011+0000","last_undegraded":"2024-09-16T10:25:32.261011+0000","last_fullsized":"2024-09-16T10:25:32.261011+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:29.814966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:29.814966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.438495+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":943,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.966865+0000","last_change":"2024-09-16T10:25:33.966865+0000","last_active":"2024-09-16T10:25:33.966865+0000","last_peered":"2024-09-16T10:25:33.966865+0000","last_clean":"2024-09-16T10:25:33.966865+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:25:33.966865+0000","last_undegraded":"2024-09-16T10:25:33.966865+0000","last_fullsized":"2024-09-16T10:25:33.966865+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:25:33.966775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.966775+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:45.795898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":670,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960288+0000","last_change":"2024-09-16T10:24:13.097213+0000","last_active":"2024-09-16T10:25:32.960288+0000","last_peered":"2024-09-16T10:25:32.960288+0000","last_clean":"2024-09-16T10:25:32.960288+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:25:32.960288+0000","last_undegraded":"2024-09-16T10:25:32.960288+0000","last_fullsized":"2024-09-16T10:25:32.960288+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:13.097133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:13.097133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:38.312420+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":663,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261496+0000","last_change":"2024-09-16T10:24:57.812754+0000","last_active":"2024-09-16T10:25:32.261496+0000","last_peered":"2024-09-16T10:25:32.261496+0000","last_clean":"2024-09-16T10:25:32.261496+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:25:32.261496+0000","last_undegraded":"2024-09-16T10:25:32.261496+0000","last_fullsized":"2024-09-16T10:25:32.261496+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:24:57.812701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.812701+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:59.841010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":638,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294955+0000","last_change":"2024-09-16T10:20:26.114180+0000","last_active":"2024-09-16T10:25:33.294955+0000","last_peered":"2024-09-16T10:25:33.294955+0000","last_clean":"2024-09-16T10:25:33.294955+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:25:33.294955+0000","last_undegraded":"2024-09-16T10:25:33.294955+0000","last_fullsized":"2024-09-16T10:25:33.294955+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:26.114104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:26.114104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":657,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046406+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:25:33.046406+0000","last_peered":"2024-09-16T10:25:33.046406+0000","last_clean":"2024-09-16T10:25:33.046406+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:25:33.046406+0000","last_undegraded":"2024-09-16T10:25:33.046406+0000","last_fullsized":"2024-09-16T10:25:33.046406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.440379+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261206+0000","last_change":"2024-09-16T10:25:30.948418+0000","last_active":"2024-09-16T10:25:32.261206+0000","last_peered":"2024-09-16T10:25:32.261206+0000","last_clean":"2024-09-16T10:25:32.261206+0000","last_became_active":"2024-09-16T10:25:30.947760+0000","last_became_peered":"2024-09-16T10:25:30.947760+0000","last_unstale":"2024-09-16T10:25:32.261206+0000","last_undegraded":"2024-09-16T10:25:32.261206+0000","last_fullsized":"2024-09-16T10:25:32.261206+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:54.306648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":125,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294247+0000","last_change":"2024-09-16T10:25:02.365887+0000","last_active":"2024-09-16T10:25:33.294247+0000","last_peered":"2024-09-16T10:25:33.294247+0000","last_clean":"2024-09-16T10:25:33.294247+0000","last_became_active":"2024-09-16T10:25:00.955466+0000","last_became_peered":"2024-09-16T10:25:00.955466+0000","last_unstale":"2024-09-16T10:25:33.294247+0000","last_undegraded":"2024-09-16T10:25:33.294247+0000","last_fullsized":"2024-09-16T10:25:33.294247+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.133909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":676,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962342+0000","last_change":"2024-09-16T10:25:13.649143+0000","last_active":"2024-09-16T10:25:32.962342+0000","last_peered":"2024-09-16T10:25:32.962342+0000","last_clean":"2024-09-16T10:25:32.962342+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:25:32.962342+0000","last_undegraded":"2024-09-16T10:25:32.962342+0000","last_fullsized":"2024-09-16T10:25:32.962342+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:25:13.648981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.648981+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.641189+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":661,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961909+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:25:32.961909+0000","last_peered":"2024-09-16T10:25:32.961909+0000","last_clean":"2024-09-16T10:25:32.961909+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:25:32.961909+0000","last_undegraded":"2024-09-16T10:25:32.961909+0000","last_fullsized":"2024-09-16T10:25:32.961909+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:26.834421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":193,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960540+0000","last_change":"2024-09-16T10:25:10.324756+0000","last_active":"2024-09-16T10:25:32.960540+0000","last_peered":"2024-09-16T10:25:32.960540+0000","last_clean":"2024-09-16T10:25:32.960540+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:25:32.960540+0000","last_undegraded":"2024-09-16T10:25:32.960540+0000","last_fullsized":"2024-09-16T10:25:32.960540+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:25:10.324680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:25:10.324680+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.799924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":189,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960692+0000","last_change":"2024-09-16T10:22:29.905776+0000","last_active":"2024-09-16T10:25:32.960692+0000","last_peered":"2024-09-16T10:25:32.960692+0000","last_clean":"2024-09-16T10:25:32.960692+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:25:32.960692+0000","last_undegraded":"2024-09-16T10:25:32.960692+0000","last_fullsized":"2024-09-16T10:25:32.960692+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.460922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":48,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.463597+0000","last_change":"2024-09-16T10:25:32.463597+0000","last_active":"2024-09-16T10:25:32.463597+0000","last_peered":"2024-09-16T10:25:32.463597+0000","last_clean":"2024-09-16T10:25:32.463597+0000","last_became_active":"2024-09-16T10:25:00.950144+0000","last_became_peered":"2024-09-16T10:25:00.950144+0000","last_unstale":"2024-09-16T10:25:32.463597+0000","last_undegraded":"2024-09-16T10:25:32.463597+0000","last_fullsized":"2024-09-16T10:25:32.463597+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.360271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294150+0000","last_change":"2024-09-16T10:25:30.950233+0000","last_active":"2024-09-16T10:25:33.294150+0000","last_peered":"2024-09-16T10:25:33.294150+0000","last_clean":"2024-09-16T10:25:33.294150+0000","last_became_active":"2024-09-16T10:25:30.948708+0000","last_became_peered":"2024-09-16T10:25:30.948708+0000","last_unstale":"2024-09-16T10:25:33.294150+0000","last_undegraded":"2024-09-16T10:25:33.294150+0000","last_fullsized":"2024-09-16T10:25:33.294150+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:35.650968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":540,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960869+0000","last_change":"2024-09-16T10:24:27.887030+0000","last_active":"2024-09-16T10:25:32.960869+0000","last_peered":"2024-09-16T10:25:32.960869+0000","last_clean":"2024-09-16T10:25:32.960869+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:25:32.960869+0000","last_undegraded":"2024-09-16T10:25:32.960869+0000","last_fullsized":"2024-09-16T10:25:32.960869+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:27.886961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:27.886961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.710539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":752,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.295252+0000","last_change":"2024-09-16T10:22:24.213089+0000","last_active":"2024-09-16T10:25:33.295252+0000","last_peered":"2024-09-16T10:25:33.295252+0000","last_clean":"2024-09-16T10:25:33.295252+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:25:33.295252+0000","last_undegraded":"2024-09-16T10:25:33.295252+0000","last_fullsized":"2024-09-16T10:25:33.295252+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:22:24.213008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.213008+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":678,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294156+0000","last_change":"2024-09-16T10:22:26.226703+0000","last_active":"2024-09-16T10:25:33.294156+0000","last_peered":"2024-09-16T10:25:33.294156+0000","last_clean":"2024-09-16T10:25:33.294156+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:25:33.294156+0000","last_undegraded":"2024-09-16T10:25:33.294156+0000","last_fullsized":"2024-09-16T10:25:33.294156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.226632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.226632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":117,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959734+0000","last_change":"2024-09-16T10:25:00.956734+0000","last_active":"2024-09-16T10:25:32.959734+0000","last_peered":"2024-09-16T10:25:32.959734+0000","last_clean":"2024-09-16T10:25:32.959734+0000","last_became_active":"2024-09-16T10:25:00.956304+0000","last_became_peered":"2024-09-16T10:25:00.956304+0000","last_unstale":"2024-09-16T10:25:32.959734+0000","last_undegraded":"2024-09-16T10:25:32.959734+0000","last_fullsized":"2024-09-16T10:25:32.959734+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:23:41.597682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.597682+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":653,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.262274+0000","last_change":"2024-09-16T10:24:16.815896+0000","last_active":"2024-09-16T10:25:32.262274+0000","last_peered":"2024-09-16T10:25:32.262274+0000","last_clean":"2024-09-16T10:25:32.262274+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:25:32.262274+0000","last_undegraded":"2024-09-16T10:25:32.262274+0000","last_fullsized":"2024-09-16T10:25:32.262274+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:16.815831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:16.815831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":659,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961777+0000","last_change":"2024-09-16T10:21:58.547682+0000","last_active":"2024-09-16T10:25:32.961777+0000","last_peered":"2024-09-16T10:25:32.961777+0000","last_clean":"2024-09-16T10:25:32.961777+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:25:32.961777+0000","last_undegraded":"2024-09-16T10:25:32.961777+0000","last_fullsized":"2024-09-16T10:25:32.961777+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.547634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.547634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959371+0000","last_change":"2024-09-16T10:25:30.622105+0000","last_active":"2024-09-16T10:25:31.959371+0000","last_peered":"2024-09-16T10:25:31.959371+0000","last_clean":"2024-09-16T10:25:31.959371+0000","last_became_active":"2024-09-16T10:25:30.621779+0000","last_became_peered":"2024-09-16T10:25:30.621779+0000","last_unstale":"2024-09-16T10:25:31.959371+0000","last_undegraded":"2024-09-16T10:25:31.959371+0000","last_fullsized":"2024-09-16T10:25:31.959371+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:33.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"328'28","reported_seq":824,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959388+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:25:31.959388+0000","last_peered":"2024-09-16T10:25:31.959388+0000","last_clean":"2024-09-16T10:25:31.959388+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:25:31.959388+0000","last_undegraded":"2024-09-16T10:25:31.959388+0000","last_fullsized":"2024-09-16T10:25:31.959388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:26.575852+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":626,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046661+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:25:33.046661+0000","last_peered":"2024-09-16T10:25:33.046661+0000","last_clean":"2024-09-16T10:25:33.046661+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:25:33.046661+0000","last_undegraded":"2024-09-16T10:25:33.046661+0000","last_fullsized":"2024-09-16T10:25:33.046661+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.609525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":125,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294308+0000","last_change":"2024-09-16T10:25:01.375800+0000","last_active":"2024-09-16T10:25:33.294308+0000","last_peered":"2024-09-16T10:25:33.294308+0000","last_clean":"2024-09-16T10:25:33.294308+0000","last_became_active":"2024-09-16T10:25:00.955671+0000","last_became_peered":"2024-09-16T10:25:00.955671+0000","last_unstale":"2024-09-16T10:25:33.294308+0000","last_undegraded":"2024-09-16T10:25:33.294308+0000","last_fullsized":"2024-09-16T10:25:33.294308+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.945066+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":514,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960871+0000","last_change":"2024-09-16T10:25:00.947058+0000","last_active":"2024-09-16T10:25:32.960871+0000","last_peered":"2024-09-16T10:25:32.960871+0000","last_clean":"2024-09-16T10:25:32.960871+0000","last_became_active":"2024-09-16T10:25:00.946698+0000","last_became_peered":"2024-09-16T10:25:00.946698+0000","last_unstale":"2024-09-16T10:25:32.960871+0000","last_undegraded":"2024-09-16T10:25:32.960871+0000","last_fullsized":"2024-09-16T10:25:32.960871+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:16.615588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:16.615588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":649,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.262118+0000","last_change":"2024-09-16T10:24:33.905860+0000","last_active":"2024-09-16T10:25:32.262118+0000","last_peered":"2024-09-16T10:25:32.262118+0000","last_clean":"2024-09-16T10:25:32.262118+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:25:32.262118+0000","last_undegraded":"2024-09-16T10:25:32.262118+0000","last_fullsized":"2024-09-16T10:25:32.262118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.905785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.905785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:01.164716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958667+0000","last_change":"2024-09-16T10:25:30.621715+0000","last_active":"2024-09-16T10:25:31.958667+0000","last_peered":"2024-09-16T10:25:31.958667+0000","last_clean":"2024-09-16T10:25:31.958667+0000","last_became_active":"2024-09-16T10:25:30.621428+0000","last_became_peered":"2024-09-16T10:25:30.621428+0000","last_unstale":"2024-09-16T10:25:31.958667+0000","last_undegraded":"2024-09-16T10:25:31.958667+0000","last_fullsized":"2024-09-16T10:25:31.958667+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:52.108854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":548,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960582+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:25:32.960582+0000","last_peered":"2024-09-16T10:25:32.960582+0000","last_clean":"2024-09-16T10:25:32.960582+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:25:32.960582+0000","last_undegraded":"2024-09-16T10:25:32.960582+0000","last_fullsized":"2024-09-16T10:25:32.960582+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":708,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960803+0000","last_change":"2024-09-16T10:24:34.142042+0000","last_active":"2024-09-16T10:25:32.960803+0000","last_peered":"2024-09-16T10:25:32.960803+0000","last_clean":"2024-09-16T10:25:32.960803+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:25:32.960803+0000","last_undegraded":"2024-09-16T10:25:32.960803+0000","last_fullsized":"2024-09-16T10:25:32.960803+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:24:34.141958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.141958+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.626228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":668,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960062+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:25:32.960062+0000","last_peered":"2024-09-16T10:25:32.960062+0000","last_clean":"2024-09-16T10:25:32.960062+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:25:32.960062+0000","last_undegraded":"2024-09-16T10:25:32.960062+0000","last_fullsized":"2024-09-16T10:25:32.960062+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.235293+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":661,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261984+0000","last_change":"2024-09-16T10:25:05.858177+0000","last_active":"2024-09-16T10:25:32.261984+0000","last_peered":"2024-09-16T10:25:32.261984+0000","last_clean":"2024-09-16T10:25:32.261984+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:25:32.261984+0000","last_undegraded":"2024-09-16T10:25:32.261984+0000","last_fullsized":"2024-09-16T10:25:32.261984+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.858129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.858129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:12.657363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":598,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046728+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:25:33.046728+0000","last_peered":"2024-09-16T10:25:33.046728+0000","last_clean":"2024-09-16T10:25:33.046728+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:25:33.046728+0000","last_undegraded":"2024-09-16T10:25:33.046728+0000","last_fullsized":"2024-09-16T10:25:33.046728+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:56.465984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960506+0000","last_change":"2024-09-16T10:25:31.161519+0000","last_active":"2024-09-16T10:25:32.960506+0000","last_peered":"2024-09-16T10:25:32.960506+0000","last_clean":"2024-09-16T10:25:32.960506+0000","last_became_active":"2024-09-16T10:25:31.161181+0000","last_became_peered":"2024-09-16T10:25:31.161181+0000","last_unstale":"2024-09-16T10:25:32.960506+0000","last_undegraded":"2024-09-16T10:25:32.960506+0000","last_fullsized":"2024-09-16T10:25:32.960506+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:42.844942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":513,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.260363+0000","last_change":"2024-09-16T10:25:04.820847+0000","last_active":"2024-09-16T10:25:32.260363+0000","last_peered":"2024-09-16T10:25:32.260363+0000","last_clean":"2024-09-16T10:25:32.260363+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:25:32.260363+0000","last_undegraded":"2024-09-16T10:25:32.260363+0000","last_fullsized":"2024-09-16T10:25:32.260363+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:04.820775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.820775+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.871951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":757,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960474+0000","last_change":"2024-09-16T10:23:09.748114+0000","last_active":"2024-09-16T10:25:32.960474+0000","last_peered":"2024-09-16T10:25:32.960474+0000","last_clean":"2024-09-16T10:25:32.960474+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:25:32.960474+0000","last_undegraded":"2024-09-16T10:25:32.960474+0000","last_fullsized":"2024-09-16T10:25:32.960474+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:23:09.748031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:09.748031+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.221819+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":652,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959185+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:25:31.959185+0000","last_peered":"2024-09-16T10:25:31.959185+0000","last_clean":"2024-09-16T10:25:31.959185+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:25:31.959185+0000","last_undegraded":"2024-09-16T10:25:31.959185+0000","last_fullsized":"2024-09-16T10:25:31.959185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.468841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":647,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961870+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:25:32.961870+0000","last_peered":"2024-09-16T10:25:32.961870+0000","last_clean":"2024-09-16T10:25:32.961870+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:25:32.961870+0000","last_undegraded":"2024-09-16T10:25:32.961870+0000","last_fullsized":"2024-09-16T10:25:32.961870+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.814189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":197,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960928+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:25:32.960928+0000","last_peered":"2024-09-16T10:25:32.960928+0000","last_clean":"2024-09-16T10:25:32.960928+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:25:32.960928+0000","last_undegraded":"2024-09-16T10:25:32.960928+0000","last_fullsized":"2024-09-16T10:25:32.960928+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.866924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962315+0000","last_change":"2024-09-16T10:25:30.613792+0000","last_active":"2024-09-16T10:25:32.962315+0000","last_peered":"2024-09-16T10:25:32.962315+0000","last_clean":"2024-09-16T10:25:32.962315+0000","last_became_active":"2024-09-16T10:25:30.613465+0000","last_became_peered":"2024-09-16T10:25:30.613465+0000","last_unstale":"2024-09-16T10:25:32.962315+0000","last_undegraded":"2024-09-16T10:25:32.962315+0000","last_fullsized":"2024-09-16T10:25:32.962315+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.984379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":317,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.045865+0000","last_change":"2024-09-16T10:25:01.481822+0000","last_active":"2024-09-16T10:25:33.045865+0000","last_peered":"2024-09-16T10:25:33.045865+0000","last_clean":"2024-09-16T10:25:33.045865+0000","last_became_active":"2024-09-16T10:25:00.955829+0000","last_became_peered":"2024-09-16T10:25:00.955829+0000","last_unstale":"2024-09-16T10:25:33.045865+0000","last_undegraded":"2024-09-16T10:25:33.045865+0000","last_fullsized":"2024-09-16T10:25:33.045865+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.481748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.481748+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:13.239898+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":672,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960406+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:25:32.960406+0000","last_peered":"2024-09-16T10:25:32.960406+0000","last_clean":"2024-09-16T10:25:32.960406+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:25:32.960406+0000","last_undegraded":"2024-09-16T10:25:32.960406+0000","last_fullsized":"2024-09-16T10:25:32.960406+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:49.328998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":645,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046592+0000","last_change":"2024-09-16T10:24:25.364236+0000","last_active":"2024-09-16T10:25:33.046592+0000","last_peered":"2024-09-16T10:25:33.046592+0000","last_clean":"2024-09-16T10:25:33.046592+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:25:33.046592+0000","last_undegraded":"2024-09-16T10:25:33.046592+0000","last_fullsized":"2024-09-16T10:25:33.046592+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:25.364176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:25.364176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.649906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":236,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961444+0000","last_change":"2024-09-16T10:22:24.799901+0000","last_active":"2024-09-16T10:25:32.961444+0000","last_peered":"2024-09-16T10:25:32.961444+0000","last_clean":"2024-09-16T10:25:32.961444+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:25:32.961444+0000","last_undegraded":"2024-09-16T10:25:32.961444+0000","last_fullsized":"2024-09-16T10:25:32.961444+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.520234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":672,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960647+0000","last_change":"2024-09-16T10:25:09.348134+0000","last_active":"2024-09-16T10:25:32.960647+0000","last_peered":"2024-09-16T10:25:32.960647+0000","last_clean":"2024-09-16T10:25:32.960647+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:25:32.960647+0000","last_undegraded":"2024-09-16T10:25:32.960647+0000","last_fullsized":"2024-09-16T10:25:32.960647+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:09.348090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.348090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.427158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961238+0000","last_change":"2024-09-16T10:25:31.161545+0000","last_active":"2024-09-16T10:25:32.961238+0000","last_peered":"2024-09-16T10:25:32.961238+0000","last_clean":"2024-09-16T10:25:32.961238+0000","last_became_active":"2024-09-16T10:25:31.161230+0000","last_became_peered":"2024-09-16T10:25:31.161230+0000","last_unstale":"2024-09-16T10:25:32.961238+0000","last_undegraded":"2024-09-16T10:25:32.961238+0000","last_fullsized":"2024-09-16T10:25:32.961238+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.886610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":228,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960649+0000","last_change":"2024-09-16T10:24:32.158409+0000","last_active":"2024-09-16T10:25:32.960649+0000","last_peered":"2024-09-16T10:25:32.960649+0000","last_clean":"2024-09-16T10:25:32.960649+0000","last_became_active":"2024-09-16T10:23:31.480326+0000","last_became_peered":"2024-09-16T10:23:31.480326+0000","last_unstale":"2024-09-16T10:25:32.960649+0000","last_undegraded":"2024-09-16T10:25:32.960649+0000","last_fullsized":"2024-09-16T10:25:32.960649+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.158356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.158356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.670555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":675,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294562+0000","last_change":"2024-09-16T10:25:16.446472+0000","last_active":"2024-09-16T10:25:33.294562+0000","last_peered":"2024-09-16T10:25:33.294562+0000","last_clean":"2024-09-16T10:25:33.294562+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:25:33.294562+0000","last_undegraded":"2024-09-16T10:25:33.294562+0000","last_fullsized":"2024-09-16T10:25:33.294562+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:25:16.446381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.446381+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:31.794942+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":669,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261123+0000","last_change":"2024-09-16T10:25:08.916850+0000","last_active":"2024-09-16T10:25:32.261123+0000","last_peered":"2024-09-16T10:25:32.261123+0000","last_clean":"2024-09-16T10:25:32.261123+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:25:32.261123+0000","last_undegraded":"2024-09-16T10:25:32.261123+0000","last_fullsized":"2024-09-16T10:25:32.261123+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.916778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.916778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.226842+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":189,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960112+0000","last_change":"2024-09-16T10:25:09.346506+0000","last_active":"2024-09-16T10:25:32.960112+0000","last_peered":"2024-09-16T10:25:32.960112+0000","last_clean":"2024-09-16T10:25:32.960112+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:25:32.960112+0000","last_undegraded":"2024-09-16T10:25:32.960112+0000","last_fullsized":"2024-09-16T10:25:32.960112+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:09.346437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.346437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.598246+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":679,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294134+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:25:33.294134+0000","last_peered":"2024-09-16T10:25:33.294134+0000","last_clean":"2024-09-16T10:25:33.294134+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:25:33.294134+0000","last_undegraded":"2024-09-16T10:25:33.294134+0000","last_fullsized":"2024-09-16T10:25:33.294134+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.267128+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":19,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294746+0000","last_change":"2024-09-16T10:25:31.162408+0000","last_active":"2024-09-16T10:25:33.294746+0000","last_peered":"2024-09-16T10:25:33.294746+0000","last_clean":"2024-09-16T10:25:33.294746+0000","last_became_active":"2024-09-16T10:25:31.162120+0000","last_became_peered":"2024-09-16T10:25:31.162120+0000","last_unstale":"2024-09-16T10:25:33.294746+0000","last_undegraded":"2024-09-16T10:25:33.294746+0000","last_fullsized":"2024-09-16T10:25:33.294746+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.047456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":617,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959577+0000","last_change":"2024-09-16T10:24:33.610134+0000","last_active":"2024-09-16T10:25:31.959577+0000","last_peered":"2024-09-16T10:25:31.959577+0000","last_clean":"2024-09-16T10:25:31.959577+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:25:31.959577+0000","last_undegraded":"2024-09-16T10:25:31.959577+0000","last_fullsized":"2024-09-16T10:25:31.959577+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:33.610076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.610076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:37.833560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":719,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958940+0000","last_change":"2024-09-16T10:22:26.050158+0000","last_active":"2024-09-16T10:25:31.958940+0000","last_peered":"2024-09-16T10:25:31.958940+0000","last_clean":"2024-09-16T10:25:31.958940+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:25:31.958940+0000","last_undegraded":"2024-09-16T10:25:31.958940+0000","last_fullsized":"2024-09-16T10:25:31.958940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:22:26.050070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.050070+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261653+0000","last_change":"2024-09-16T10:25:00.947814+0000","last_active":"2024-09-16T10:25:32.261653+0000","last_peered":"2024-09-16T10:25:32.261653+0000","last_clean":"2024-09-16T10:25:32.261653+0000","last_became_active":"2024-09-16T10:25:00.947502+0000","last_became_peered":"2024-09-16T10:25:00.947502+0000","last_unstale":"2024-09-16T10:25:32.261653+0000","last_undegraded":"2024-09-16T10:25:32.261653+0000","last_fullsized":"2024-09-16T10:25:32.261653+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.528600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.528600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.404499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":189,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960734+0000","last_change":"2024-09-16T10:22:26.818522+0000","last_active":"2024-09-16T10:25:32.960734+0000","last_peered":"2024-09-16T10:25:32.960734+0000","last_clean":"2024-09-16T10:25:32.960734+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:25:32.960734+0000","last_undegraded":"2024-09-16T10:25:32.960734+0000","last_fullsized":"2024-09-16T10:25:32.960734+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.818439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.818439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.481964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":518,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960038+0000","last_change":"2024-09-16T10:25:00.954355+0000","last_active":"2024-09-16T10:25:32.960038+0000","last_peered":"2024-09-16T10:25:32.960038+0000","last_clean":"2024-09-16T10:25:32.960038+0000","last_became_active":"2024-09-16T10:25:00.954013+0000","last_became_peered":"2024-09-16T10:25:00.954013+0000","last_unstale":"2024-09-16T10:25:32.960038+0000","last_undegraded":"2024-09-16T10:25:32.960038+0000","last_fullsized":"2024-09-16T10:25:32.960038+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.651643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.651643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959595+0000","last_change":"2024-09-16T10:25:31.163086+0000","last_active":"2024-09-16T10:25:31.959595+0000","last_peered":"2024-09-16T10:25:31.959595+0000","last_clean":"2024-09-16T10:25:31.959595+0000","last_became_active":"2024-09-16T10:25:31.162769+0000","last_became_peered":"2024-09-16T10:25:31.162769+0000","last_unstale":"2024-09-16T10:25:31.959595+0000","last_undegraded":"2024-09-16T10:25:31.959595+0000","last_fullsized":"2024-09-16T10:25:31.959595+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.638366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":570,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046237+0000","last_change":"2024-09-16T10:24:31.490967+0000","last_active":"2024-09-16T10:25:33.046237+0000","last_peered":"2024-09-16T10:25:33.046237+0000","last_clean":"2024-09-16T10:25:33.046237+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:25:33.046237+0000","last_undegraded":"2024-09-16T10:25:33.046237+0000","last_fullsized":"2024-09-16T10:25:33.046237+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.490890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.490890+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:52.214164+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":682,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959932+0000","last_change":"2024-09-16T10:23:41.811222+0000","last_active":"2024-09-16T10:25:32.959932+0000","last_peered":"2024-09-16T10:25:32.959932+0000","last_clean":"2024-09-16T10:25:32.959932+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:25:32.959932+0000","last_undegraded":"2024-09-16T10:25:32.959932+0000","last_fullsized":"2024-09-16T10:25:32.959932+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:41.811140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.811140+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.191555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":49,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960956+0000","last_change":"2024-09-16T10:25:05.405432+0000","last_active":"2024-09-16T10:25:32.960956+0000","last_peered":"2024-09-16T10:25:32.960956+0000","last_clean":"2024-09-16T10:25:32.960956+0000","last_became_active":"2024-09-16T10:25:00.959190+0000","last_became_peered":"2024-09-16T10:25:00.959190+0000","last_unstale":"2024-09-16T10:25:32.960956+0000","last_undegraded":"2024-09-16T10:25:32.960956+0000","last_fullsized":"2024-09-16T10:25:32.960956+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.405338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.405338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.454239+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":189,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960123+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:25:32.960123+0000","last_peered":"2024-09-16T10:25:32.960123+0000","last_clean":"2024-09-16T10:25:32.960123+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:25:32.960123+0000","last_undegraded":"2024-09-16T10:25:32.960123+0000","last_fullsized":"2024-09-16T10:25:32.960123+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.415920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":649,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046026+0000","last_change":"2024-09-16T10:24:20.366257+0000","last_active":"2024-09-16T10:25:33.046026+0000","last_peered":"2024-09-16T10:25:33.046026+0000","last_clean":"2024-09-16T10:25:33.046026+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:25:33.046026+0000","last_undegraded":"2024-09-16T10:25:33.046026+0000","last_fullsized":"2024-09-16T10:25:33.046026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:20.366122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:20.366122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":16,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261484+0000","last_change":"2024-09-16T10:25:30.621276+0000","last_active":"2024-09-16T10:25:32.261484+0000","last_peered":"2024-09-16T10:25:32.261484+0000","last_clean":"2024-09-16T10:25:32.261484+0000","last_became_active":"2024-09-16T10:25:30.620971+0000","last_became_peered":"2024-09-16T10:25:30.620971+0000","last_unstale":"2024-09-16T10:25:32.261484+0000","last_undegraded":"2024-09-16T10:25:32.261484+0000","last_fullsized":"2024-09-16T10:25:32.261484+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:33.975706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":467,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960423+0000","last_change":"2024-09-16T10:25:01.368064+0000","last_active":"2024-09-16T10:25:32.960423+0000","last_peered":"2024-09-16T10:25:32.960423+0000","last_clean":"2024-09-16T10:25:32.960423+0000","last_became_active":"2024-09-16T10:25:00.960443+0000","last_became_peered":"2024-09-16T10:25:00.960443+0000","last_unstale":"2024-09-16T10:25:32.960423+0000","last_undegraded":"2024-09-16T10:25:32.960423+0000","last_fullsized":"2024-09-16T10:25:32.960423+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.367979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.367979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:11.835419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":684,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.260511+0000","last_change":"2024-09-16T10:24:36.805155+0000","last_active":"2024-09-16T10:25:32.260511+0000","last_peered":"2024-09-16T10:25:32.260511+0000","last_clean":"2024-09-16T10:25:32.260511+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:25:32.260511+0000","last_undegraded":"2024-09-16T10:25:32.260511+0000","last_fullsized":"2024-09-16T10:25:32.260511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:36.805084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:36.805084+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:01.683712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":51,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960757+0000","last_change":"2024-09-16T10:25:00.952412+0000","last_active":"2024-09-16T10:25:32.960757+0000","last_peered":"2024-09-16T10:25:32.960757+0000","last_clean":"2024-09-16T10:25:32.960757+0000","last_became_active":"2024-09-16T10:25:00.952061+0000","last_became_peered":"2024-09-16T10:25:00.952061+0000","last_unstale":"2024-09-16T10:25:32.960757+0000","last_undegraded":"2024-09-16T10:25:32.960757+0000","last_fullsized":"2024-09-16T10:25:32.960757+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:24:24.486825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:24.486825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:47.489444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"352'9","reported_seq":72,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958658+0000","last_change":"2024-09-16T10:25:00.946719+0000","last_active":"2024-09-16T10:25:31.958658+0000","last_peered":"2024-09-16T10:25:31.958658+0000","last_clean":"2024-09-16T10:25:31.958658+0000","last_became_active":"2024-09-16T10:25:00.946371+0000","last_became_peered":"2024-09-16T10:25:00.946371+0000","last_unstale":"2024-09-16T10:25:31.958658+0000","last_undegraded":"2024-09-16T10:25:31.958658+0000","last_fullsized":"2024-09-16T10:25:31.958658+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:23:10.768339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:23:10.768339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":657,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261101+0000","last_change":"2024-09-16T10:24:34.863064+0000","last_active":"2024-09-16T10:25:32.261101+0000","last_peered":"2024-09-16T10:25:32.261101+0000","last_clean":"2024-09-16T10:25:32.261101+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:25:32.261101+0000","last_undegraded":"2024-09-16T10:25:32.261101+0000","last_fullsized":"2024-09-16T10:25:32.261101+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:34.862992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.862992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.781279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":233,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261324+0000","last_change":"2024-09-16T10:25:00.825815+0000","last_active":"2024-09-16T10:25:32.261324+0000","last_peered":"2024-09-16T10:25:32.261324+0000","last_clean":"2024-09-16T10:25:32.261324+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:25:32.261324+0000","last_undegraded":"2024-09-16T10:25:32.261324+0000","last_fullsized":"2024-09-16T10:25:32.261324+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:00.825743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:00.825743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:11.827743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":644,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958978+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:25:31.958978+0000","last_peered":"2024-09-16T10:25:31.958978+0000","last_clean":"2024-09-16T10:25:31.958978+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:25:31.958978+0000","last_undegraded":"2024-09-16T10:25:31.958978+0000","last_fullsized":"2024-09-16T10:25:31.958978+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.558479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":666,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046753+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:25:33.046753+0000","last_peered":"2024-09-16T10:25:33.046753+0000","last_clean":"2024-09-16T10:25:33.046753+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:25:33.046753+0000","last_undegraded":"2024-09-16T10:25:33.046753+0000","last_fullsized":"2024-09-16T10:25:33.046753+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:52.492565+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":644,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261738+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:25:32.261738+0000","last_peered":"2024-09-16T10:25:32.261738+0000","last_clean":"2024-09-16T10:25:32.261738+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:25:32.261738+0000","last_undegraded":"2024-09-16T10:25:32.261738+0000","last_fullsized":"2024-09-16T10:25:32.261738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.836191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":641,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.279526+0000","last_change":"2024-09-16T10:25:11.324922+0000","last_active":"2024-09-16T10:25:33.279526+0000","last_peered":"2024-09-16T10:25:33.279526+0000","last_clean":"2024-09-16T10:25:33.279526+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:25:33.279526+0000","last_undegraded":"2024-09-16T10:25:33.279526+0000","last_fullsized":"2024-09-16T10:25:33.279526+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:33.037219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":557,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960679+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:25:32.960679+0000","last_peered":"2024-09-16T10:25:32.960679+0000","last_clean":"2024-09-16T10:25:32.960679+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:25:32.960679+0000","last_undegraded":"2024-09-16T10:25:32.960679+0000","last_fullsized":"2024-09-16T10:25:32.960679+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.977445+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":674,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.295296+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:25:33.295296+0000","last_peered":"2024-09-16T10:25:33.295296+0000","last_clean":"2024-09-16T10:25:33.295296+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:25:33.295296+0000","last_undegraded":"2024-09-16T10:25:33.295296+0000","last_fullsized":"2024-09-16T10:25:33.295296+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.299672+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":41,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959866+0000","last_change":"2024-09-16T10:25:00.944900+0000","last_active":"2024-09-16T10:25:32.959866+0000","last_peered":"2024-09-16T10:25:32.959866+0000","last_clean":"2024-09-16T10:25:32.959866+0000","last_became_active":"2024-09-16T10:25:00.944574+0000","last_became_peered":"2024-09-16T10:25:00.944574+0000","last_unstale":"2024-09-16T10:25:32.959866+0000","last_undegraded":"2024-09-16T10:25:32.959866+0000","last_fullsized":"2024-09-16T10:25:32.959866+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:37.608524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.608524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.990047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046603+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:25:33.046603+0000","last_peered":"2024-09-16T10:25:33.046603+0000","last_clean":"2024-09-16T10:25:33.046603+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:25:33.046603+0000","last_undegraded":"2024-09-16T10:25:33.046603+0000","last_fullsized":"2024-09-16T10:25:33.046603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.469447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":189,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961296+0000","last_change":"2024-09-16T10:25:13.357255+0000","last_active":"2024-09-16T10:25:32.961296+0000","last_peered":"2024-09-16T10:25:32.961296+0000","last_clean":"2024-09-16T10:25:32.961296+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:25:32.961296+0000","last_undegraded":"2024-09-16T10:25:32.961296+0000","last_fullsized":"2024-09-16T10:25:32.961296+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.501033+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":224,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.260259+0000","last_change":"2024-09-16T10:25:09.961140+0000","last_active":"2024-09-16T10:25:32.260259+0000","last_peered":"2024-09-16T10:25:32.260259+0000","last_clean":"2024-09-16T10:25:32.260259+0000","last_became_active":"2024-09-16T10:23:00.902731+0000","last_became_peered":"2024-09-16T10:23:00.902731+0000","last_unstale":"2024-09-16T10:25:32.260259+0000","last_undegraded":"2024-09-16T10:25:32.260259+0000","last_fullsized":"2024-09-16T10:25:32.260259+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:09.961098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.961098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:18.332815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":665,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261678+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:25:32.261678+0000","last_peered":"2024-09-16T10:25:32.261678+0000","last_clean":"2024-09-16T10:25:32.261678+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:25:32.261678+0000","last_undegraded":"2024-09-16T10:25:32.261678+0000","last_fullsized":"2024-09-16T10:25:32.261678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.812083+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":662,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294376+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:25:33.294376+0000","last_peered":"2024-09-16T10:25:33.294376+0000","last_clean":"2024-09-16T10:25:33.294376+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:25:33.294376+0000","last_undegraded":"2024-09-16T10:25:33.294376+0000","last_fullsized":"2024-09-16T10:25:33.294376+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":843,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046712+0000","last_change":"2024-09-16T10:25:16.464789+0000","last_active":"2024-09-16T10:25:33.046712+0000","last_peered":"2024-09-16T10:25:33.046712+0000","last_clean":"2024-09-16T10:25:33.046712+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:25:33.046712+0000","last_undegraded":"2024-09-16T10:25:33.046712+0000","last_fullsized":"2024-09-16T10:25:33.046712+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:25.206981+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":671,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.960606+0000","last_change":"2024-09-16T10:23:38.791832+0000","last_active":"2024-09-16T10:25:32.960606+0000","last_peered":"2024-09-16T10:25:32.960606+0000","last_clean":"2024-09-16T10:25:32.960606+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:25:32.960606+0000","last_undegraded":"2024-09-16T10:25:32.960606+0000","last_fullsized":"2024-09-16T10:25:32.960606+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:38.791750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:38.791750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:23.207250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":232,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294225+0000","last_change":"2024-09-16T10:24:32.496216+0000","last_active":"2024-09-16T10:25:33.294225+0000","last_peered":"2024-09-16T10:25:33.294225+0000","last_clean":"2024-09-16T10:25:33.294225+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:25:33.294225+0000","last_undegraded":"2024-09-16T10:25:33.294225+0000","last_fullsized":"2024-09-16T10:25:33.294225+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.495461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.495461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.882583+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":695,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046502+0000","last_change":"2024-09-16T10:25:08.470676+0000","last_active":"2024-09-16T10:25:33.046502+0000","last_peered":"2024-09-16T10:25:33.046502+0000","last_clean":"2024-09-16T10:25:33.046502+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:25:33.046502+0000","last_undegraded":"2024-09-16T10:25:33.046502+0000","last_fullsized":"2024-09-16T10:25:33.046502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:25:08.470600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.470600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:17.849780+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":668,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.279441+0000","last_change":"2024-09-16T10:24:23.469834+0000","last_active":"2024-09-16T10:25:33.279441+0000","last_peered":"2024-09-16T10:25:33.279441+0000","last_clean":"2024-09-16T10:25:33.279441+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:25:33.279441+0000","last_undegraded":"2024-09-16T10:25:33.279441+0000","last_fullsized":"2024-09-16T10:25:33.279441+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:23.469779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.469779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:46.236915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":804,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261628+0000","last_change":"2024-09-16T10:24:26.942679+0000","last_active":"2024-09-16T10:25:32.261628+0000","last_peered":"2024-09-16T10:25:32.261628+0000","last_clean":"2024-09-16T10:25:32.261628+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:25:32.261628+0000","last_undegraded":"2024-09-16T10:25:32.261628+0000","last_fullsized":"2024-09-16T10:25:32.261628+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:26.942604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.942604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.074891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":641,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.294984+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:25:33.294984+0000","last_peered":"2024-09-16T10:25:33.294984+0000","last_clean":"2024-09-16T10:25:33.294984+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:25:33.294984+0000","last_undegraded":"2024-09-16T10:25:33.294984+0000","last_fullsized":"2024-09-16T10:25:33.294984+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":700,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959982+0000","last_change":"2024-09-16T10:25:17.300226+0000","last_active":"2024-09-16T10:25:32.959982+0000","last_peered":"2024-09-16T10:25:32.959982+0000","last_clean":"2024-09-16T10:25:32.959982+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:25:32.959982+0000","last_undegraded":"2024-09-16T10:25:32.959982+0000","last_fullsized":"2024-09-16T10:25:32.959982+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:17.300150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.300150+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:42.406783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":112,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961689+0000","last_change":"2024-09-16T10:25:08.657378+0000","last_active":"2024-09-16T10:25:32.961689+0000","last_peered":"2024-09-16T10:25:32.961689+0000","last_clean":"2024-09-16T10:25:32.961689+0000","last_became_active":"2024-09-16T10:23:19.734145+0000","last_became_peered":"2024-09-16T10:23:19.734145+0000","last_unstale":"2024-09-16T10:25:32.961689+0000","last_undegraded":"2024-09-16T10:25:32.961689+0000","last_fullsized":"2024-09-16T10:25:32.961689+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.565118+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":655,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962194+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:25:32.962194+0000","last_peered":"2024-09-16T10:25:32.962194+0000","last_clean":"2024-09-16T10:25:32.962194+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:25:32.962194+0000","last_undegraded":"2024-09-16T10:25:32.962194+0000","last_fullsized":"2024-09-16T10:25:32.962194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.630109+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":641,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.045768+0000","last_change":"2024-09-16T10:24:57.507342+0000","last_active":"2024-09-16T10:25:33.045768+0000","last_peered":"2024-09-16T10:25:33.045768+0000","last_clean":"2024-09-16T10:25:33.045768+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:25:33.045768+0000","last_undegraded":"2024-09-16T10:25:33.045768+0000","last_fullsized":"2024-09-16T10:25:33.045768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:57.507281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.507281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:06.240579+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":710,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.962426+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:25:32.962426+0000","last_peered":"2024-09-16T10:25:32.962426+0000","last_clean":"2024-09-16T10:25:32.962426+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:25:32.962426+0000","last_undegraded":"2024-09-16T10:25:32.962426+0000","last_fullsized":"2024-09-16T10:25:32.962426+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.824736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":40,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046282+0000","last_change":"2024-09-16T10:25:00.955337+0000","last_active":"2024-09-16T10:25:33.046282+0000","last_peered":"2024-09-16T10:25:33.046282+0000","last_clean":"2024-09-16T10:25:33.046282+0000","last_became_active":"2024-09-16T10:25:00.954747+0000","last_became_peered":"2024-09-16T10:25:00.954747+0000","last_unstale":"2024-09-16T10:25:33.046282+0000","last_undegraded":"2024-09-16T10:25:33.046282+0000","last_fullsized":"2024-09-16T10:25:33.046282+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:25.105215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.105215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.465171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":610,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046237+0000","last_change":"2024-09-16T10:23:47.232905+0000","last_active":"2024-09-16T10:25:33.046237+0000","last_peered":"2024-09-16T10:25:33.046237+0000","last_clean":"2024-09-16T10:25:33.046237+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:25:33.046237+0000","last_undegraded":"2024-09-16T10:25:33.046237+0000","last_fullsized":"2024-09-16T10:25:33.046237+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:47.232834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.232834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":634,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.293855+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:25:33.293855+0000","last_peered":"2024-09-16T10:25:33.293855+0000","last_clean":"2024-09-16T10:25:33.293855+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:25:33.293855+0000","last_undegraded":"2024-09-16T10:25:33.293855+0000","last_fullsized":"2024-09-16T10:25:33.293855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.416346+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":562,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959797+0000","last_change":"2024-09-16T10:24:32.986890+0000","last_active":"2024-09-16T10:25:32.959797+0000","last_peered":"2024-09-16T10:25:32.959797+0000","last_clean":"2024-09-16T10:25:32.959797+0000","last_became_active":"2024-09-16T10:23:31.473535+0000","last_became_peered":"2024-09-16T10:23:31.473535+0000","last_unstale":"2024-09-16T10:25:32.959797+0000","last_undegraded":"2024-09-16T10:25:32.959797+0000","last_fullsized":"2024-09-16T10:25:32.959797+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.984796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.984796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:51.261830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":664,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.045900+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:25:33.045900+0000","last_peered":"2024-09-16T10:25:33.045900+0000","last_clean":"2024-09-16T10:25:33.045900+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:25:33.045900+0000","last_undegraded":"2024-09-16T10:25:33.045900+0000","last_fullsized":"2024-09-16T10:25:33.045900+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.420414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.261775+0000","last_change":"2024-09-16T10:23:13.803633+0000","last_active":"2024-09-16T10:25:32.261775+0000","last_peered":"2024-09-16T10:25:32.261775+0000","last_clean":"2024-09-16T10:25:32.261775+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:25:32.261775+0000","last_undegraded":"2024-09-16T10:25:32.261775+0000","last_fullsized":"2024-09-16T10:25:32.261775+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:13.803568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.803568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:49.863054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":654,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.278872+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:25:33.278872+0000","last_peered":"2024-09-16T10:25:33.278872+0000","last_clean":"2024-09-16T10:25:33.278872+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:25:33.278872+0000","last_undegraded":"2024-09-16T10:25:33.278872+0000","last_fullsized":"2024-09-16T10:25:33.278872+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:53.389988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":657,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.046484+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:25:33.046484+0000","last_peered":"2024-09-16T10:25:33.046484+0000","last_clean":"2024-09-16T10:25:33.046484+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:25:33.046484+0000","last_undegraded":"2024-09-16T10:25:33.046484+0000","last_fullsized":"2024-09-16T10:25:33.046484+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.383093+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":695,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.045856+0000","last_change":"2024-09-16T10:24:15.357656+0000","last_active":"2024-09-16T10:25:33.045856+0000","last_peered":"2024-09-16T10:25:33.045856+0000","last_clean":"2024-09-16T10:25:33.045856+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:25:33.045856+0000","last_undegraded":"2024-09-16T10:25:33.045856+0000","last_fullsized":"2024-09-16T10:25:33.045856+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:24:15.357188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.357188+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":651,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.295152+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:25:33.295152+0000","last_peered":"2024-09-16T10:25:33.295152+0000","last_clean":"2024-09-16T10:25:33.295152+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:25:33.295152+0000","last_undegraded":"2024-09-16T10:25:33.295152+0000","last_fullsized":"2024-09-16T10:25:33.295152+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:23.300677+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":652,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.958721+0000","last_change":"2024-09-16T10:25:12.539167+0000","last_active":"2024-09-16T10:25:31.958721+0000","last_peered":"2024-09-16T10:25:31.958721+0000","last_clean":"2024-09-16T10:25:31.958721+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:25:31.958721+0000","last_undegraded":"2024-09-16T10:25:31.958721+0000","last_fullsized":"2024-09-16T10:25:31.958721+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:12.539106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.539106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:36.533748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":273,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.959961+0000","last_change":"2024-09-16T10:25:00.946850+0000","last_active":"2024-09-16T10:25:32.959961+0000","last_peered":"2024-09-16T10:25:32.959961+0000","last_clean":"2024-09-16T10:25:32.959961+0000","last_became_active":"2024-09-16T10:25:00.946511+0000","last_became_peered":"2024-09-16T10:25:00.946511+0000","last_unstale":"2024-09-16T10:25:32.959961+0000","last_undegraded":"2024-09-16T10:25:32.959961+0000","last_fullsized":"2024-09-16T10:25:32.959961+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:26.432557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.432557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:39.603536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":642,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:32.961402+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:25:32.961402+0000","last_peered":"2024-09-16T10:25:32.961402+0000","last_clean":"2024-09-16T10:25:32.961402+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:25:32.961402+0000","last_undegraded":"2024-09-16T10:25:32.961402+0000","last_fullsized":"2024-09-16T10:25:32.961402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.811471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":660,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.045936+0000","last_change":"2024-09-16T10:25:02.446887+0000","last_active":"2024-09-16T10:25:33.045936+0000","last_peered":"2024-09-16T10:25:33.045936+0000","last_clean":"2024-09-16T10:25:33.045936+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:25:33.045936+0000","last_undegraded":"2024-09-16T10:25:33.045936+0000","last_fullsized":"2024-09-16T10:25:33.045936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:02.446742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.446742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:02.607831+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":695,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-16T10:25:31.959628+0000","last_change":"2024-09-16T10:25:18.564978+0000","last_active":"2024-09-16T10:25:31.959628+0000","last_peered":"2024-09-16T10:25:31.959628+0000","last_clean":"2024-09-16T10:25:31.959628+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:25:31.959628+0000","last_undegraded":"2024-09-16T10:25:31.959628+0000","last_fullsized":"2024-09-16T10:25:31.959628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:18.564894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.564894+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.312714+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":660,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-16T10:25:33.295102+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:25:33.295102+0000","last_peered":"2024-09-16T10:25:33.295102+0000","last_clean":"2024-09-16T10:25:33.295102+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:25:33.295102+0000","last_undegraded":"2024-09-16T10:25:33.295102+0000","last_fullsized":"2024-09-16T10:25:33.295102+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.261724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234131,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433548,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93311604,"statfs":{"total":95995035648,"available":95551082496,"internally_reserved":0,"allocated":1171456,"data_stored":638545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":442759626},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528471,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401932,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343220,"statfs":{"total":95995035648,"available":95583457280,"internally_reserved":0,"allocated":1490944,"data_stored":965567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":410057158},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790109,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647828,"kb_used_data":193216,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93097324,"statfs":{"total":95995035648,"available":95331659776,"internally_reserved":0,"allocated":197853184,"data_stored":197313429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986338,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339196,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93405956,"statfs":{"total":95995035648,"available":95647698944,"internally_reserved":0,"allocated":1208320,"data_stored":646323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215271,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":528564,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93216588,"statfs":{"total":95995035648,"available":95453786112,"internally_reserved":0,"allocated":1798144,"data_stored":1237530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":539425224},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411501,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":267236,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93477916,"statfs":{"total":95995035648,"available":95721385984,"internally_reserved":0,"allocated":1445888,"data_stored":908241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":523836,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93221316,"statfs":{"total":95995035648,"available":95458627584,"internally_reserved":0,"allocated":1220608,"data_stored":646843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398036,"kb_used_data":5888,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93347116,"statfs":{"total":95995035648,"available":95587446784,"internally_reserved":0,"allocated":6029312,"data_stored":5488911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:37.996 INFO:tasks.thrashosds.thrasher:Setting 3.e to [7, 3, 4, 5, 1, 0] 2024-09-16T10:25:37.996 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.e', '7', '3', '4', '5', '1', '0'] 2024-09-16T10:25:37.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 3.e 7 3 4 5 1 0 2024-09-16T10:25:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:38.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: from='client.16143 v1:172.21.15.63:0/3441082946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: 5.19 scrub starts 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: 5.19 scrub ok 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: from='client.45992 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: osdmap e369: 8 total, 8 up, 8 in 2024-09-16T10:25:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:37 smithi165 ceph-mon[32696]: 5.5 scrub starts 2024-09-16T10:25:38.109 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:38.149 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:38.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:38.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:38.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.433 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:38.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:38.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:38.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:38.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:25:38.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:38.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 2 16 6731 6715 209.829 207.688 0.00742257 0.00475424 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[29993]: pgmap v644: 168 pgs: 168 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[29993]: 5.5 scrub ok 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[29993]: 5.9 scrub starts 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[29993]: 5.9 scrub ok 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[39470]: pgmap v644: 168 pgs: 168 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[39470]: 5.5 scrub ok 2024-09-16T10:25:38.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[39470]: 5.9 scrub starts 2024-09-16T10:25:38.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[39470]: 5.9 scrub ok 2024-09-16T10:25:38.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:38.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:38.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:38.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:38.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:38.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:38.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:38.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:39.044 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:39.046 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:25:39.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:25:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:25:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:38 smithi165 ceph-mon[32696]: pgmap v644: 168 pgs: 168 active+clean; 583 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:25:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:38 smithi165 ceph-mon[32696]: 5.5 scrub ok 2024-09-16T10:25:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:38 smithi165 ceph-mon[32696]: 5.9 scrub starts 2024-09-16T10:25:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:38 smithi165 ceph-mon[32696]: 5.9 scrub ok 2024-09-16T10:25:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:39.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:39.107 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:39.107 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:39.128 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:39.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:39.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:39.142 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:39.177 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:39.211 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:39.212 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:39.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:39.244 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:39.244 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:39.272 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:39.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:39.276 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:39.277 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:39.308 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:39.308 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:39.342 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:39.342 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:39.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:39.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:39.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:39.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:39.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:39.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:39.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:39.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:39.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:39.684 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:39.684 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:39.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 3 16 9502 9486 197.612 173.188 0.00568116 0.00505208 2024-09-16T10:25:39.776 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:39.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2470322974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:39.814 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:39.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:39.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:39.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:39.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:39.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:39.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2470322974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2470322974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:40.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:40.079 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:40.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:40.181 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:40.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:40.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:40.360 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:40.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:40.461 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:40.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:25:40.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:40.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:40.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:40.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:40.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 4 16 12273 12257 191.502 173.188 0.00162439 0.00515099 2024-09-16T10:25:40.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:40.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:40.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:40.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:40.904 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:40.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:40.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:40.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:40 smithi063 ceph-mon[29993]: pgmap v645: 168 pgs: 168 active+clean; 329 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 55 MiB/s wr, 877 op/s 2024-09-16T10:25:40.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:40 smithi063 ceph-mon[29993]: 5.13 scrub starts 2024-09-16T10:25:40.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:40 smithi063 ceph-mon[29993]: 5.13 scrub ok 2024-09-16T10:25:40.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:40 smithi063 ceph-mon[39470]: pgmap v645: 168 pgs: 168 active+clean; 329 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 55 MiB/s wr, 877 op/s 2024-09-16T10:25:40.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:40 smithi063 ceph-mon[39470]: 5.13 scrub starts 2024-09-16T10:25:40.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:40 smithi063 ceph-mon[39470]: 5.13 scrub ok 2024-09-16T10:25:40.958 INFO:teuthology.orchestra.run.smithi063.stderr:set 3.e pg_upmap_items mapping to [7->3,4->5,1->0] 2024-09-16T10:25:41.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:40 smithi165 ceph-mon[32696]: pgmap v645: 168 pgs: 168 active+clean; 329 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 55 MiB/s wr, 877 op/s 2024-09-16T10:25:41.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:40 smithi165 ceph-mon[32696]: 5.13 scrub starts 2024-09-16T10:25:41.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:40 smithi165 ceph-mon[32696]: 5.13 scrub ok 2024-09-16T10:25:41.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:41.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:25:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:41.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:41.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:41.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:41.208 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:41.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:41.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:41.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:41.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:41.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:41.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:41.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:41.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:41.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:41.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:41.689 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 5 15 14960 14945 186.799 168 0.00313784 0.00534633 2024-09-16T10:25:41.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:41.754 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:41.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:41.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:41.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:41.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:41.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:25:41.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: 1.f scrub ok 2024-09-16T10:25:41.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4146228043' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]} : dispatch 2024-09-16T10:25:41.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: from='client.46019 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]} : dispatch 2024-09-16T10:25:41.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: from='client.46019 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]}]': finished 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: osdmap e370: 8 total, 8 up, 8 in 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: 6.b scrub starts 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[29993]: 6.b scrub ok 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: 1.f scrub ok 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4146228043' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]} : dispatch 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: from='client.46019 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]} : dispatch 2024-09-16T10:25:41.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: from='client.46019 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]}]': finished 2024-09-16T10:25:41.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: osdmap e370: 8 total, 8 up, 8 in 2024-09-16T10:25:41.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: 6.b scrub starts 2024-09-16T10:25:41.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:41 smithi063 ceph-mon[39470]: 6.b scrub ok 2024-09-16T10:25:41.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:42.035 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:42.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: 1.f scrub ok 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4146228043' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]} : dispatch 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: from='client.46019 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]} : dispatch 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: from='client.46019 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.e", "id": [7, 3, 4, 5, 1, 0]}]': finished 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: osdmap e370: 8 total, 8 up, 8 in 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: 6.b scrub starts 2024-09-16T10:25:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:41 smithi165 ceph-mon[32696]: 6.b scrub ok 2024-09-16T10:25:42.108 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:25:42.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:42.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:42.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:42.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:42.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:42.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:42.454 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:42.476 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:42.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:42.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:42.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:42.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:42.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:42.690 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 6 16 17400 17384 181.07 152.438 0.0041034 0.00551608 2024-09-16T10:25:42.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:42.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:42.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:42.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:42.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:42.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[39470]: pgmap v647: 168 pgs: 168 active+clean; 395 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[39470]: osdmap e371: 8 total, 8 up, 8 in 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[39470]: 4.10 scrub starts 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[39470]: 4.10 scrub ok 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/370750253' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[29993]: pgmap v647: 168 pgs: 168 active+clean; 395 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[29993]: osdmap e371: 8 total, 8 up, 8 in 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[29993]: 4.10 scrub starts 2024-09-16T10:25:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[29993]: 4.10 scrub ok 2024-09-16T10:25:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/370750253' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:25:42.979 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:25:42.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:43.015 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:42 smithi165 ceph-mon[32696]: pgmap v647: 168 pgs: 168 active+clean; 395 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-09-16T10:25:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:42 smithi165 ceph-mon[32696]: osdmap e371: 8 total, 8 up, 8 in 2024-09-16T10:25:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:42 smithi165 ceph-mon[32696]: 4.10 scrub starts 2024-09-16T10:25:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:42 smithi165 ceph-mon[32696]: 4.10 scrub ok 2024-09-16T10:25:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/370750253' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:25:43.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:43.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:43.256 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:43.291 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:43.393 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:43.431 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:43.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:43.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:43.670 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:43.690 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 7 15 19960 19945 178.063 160.062 0.010704 0.00560974 2024-09-16T10:25:43.710 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:43.812 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.845 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:43.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:43.946 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:43.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:43 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:43.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/370750253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:25:43.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:43 smithi063 ceph-mon[39470]: osdmap e372: 8 total, 8 up, 8 in 2024-09-16T10:25:43.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:43 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:43.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/370750253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:25:43.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:43 smithi063 ceph-mon[29993]: osdmap e372: 8 total, 8 up, 8 in 2024-09-16T10:25:43.981 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:43.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:44.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:43 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/370750253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:25:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:43 smithi165 ceph-mon[32696]: osdmap e372: 8 total, 8 up, 8 in 2024-09-16T10:25:44.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:44.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:44.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:44.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:44.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:44.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:44.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:44.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:44.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:44.496 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:44.534 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:44.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:44.636 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:44.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:44.690 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 8 15 22259 22244 173.762 143.688 0.00456618 0.00574924 2024-09-16T10:25:44.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:44.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:44.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:44.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:44.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:44.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:44 smithi063 ceph-mon[39470]: pgmap v650: 168 pgs: 168 active+clean; 549 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-16T10:25:44.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:44 smithi063 ceph-mon[29993]: pgmap v650: 168 pgs: 168 active+clean; 549 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-16T10:25:45.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:45.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:44 smithi165 ceph-mon[32696]: pgmap v650: 168 pgs: 168 active+clean; 549 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-16T10:25:45.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:45.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:45.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:45.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:45.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:45.305 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:45.343 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:45.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:45.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:45.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:45.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:45.547 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:25:45.547 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:25:45.582 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:45.623 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:45.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:45.690 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 9 16 24543 24527 170.308 142.688 0.00722042 0.00586608 2024-09-16T10:25:45.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:45.768 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:45.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:45.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:45.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:45.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:46.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:46.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:46.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:46.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:46.177 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:46.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:46.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:46.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:46.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:46.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:46.455 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:46.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:46.558 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:46.595 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:46.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:46.690 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 10 15 27027 27012 168.807 155.312 0.00241003 0.00591886 2024-09-16T10:25:46.698 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:46.734 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:46.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:46.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:46.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:46.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:46.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:46 smithi063 ceph-mon[29993]: pgmap v651: 168 pgs: 168 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-16T10:25:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:46 smithi063 ceph-mon[39470]: pgmap v651: 168 pgs: 168 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-16T10:25:46.975 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:47.014 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:25:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:47.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:47.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:25:47.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:25:47.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:25:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:46 smithi165 ceph-mon[32696]: pgmap v651: 168 pgs: 168 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-16T10:25:47.087 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:47.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:47.166 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:47.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:47.232 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:25:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:25:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:25:47.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:47.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:47.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:47.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:47.461 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:47.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:47.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:47.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:47.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:47.690 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 11 16 29085 29069 165.148 128.562 0.00746539 0.00604982 2024-09-16T10:25:47.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:47.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:47.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:47.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:47.862 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:47.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:47.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:48.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:48.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:48.240 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.287 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:48.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:48.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:25:48.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:48.438 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:48.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:48.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:48.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:48.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 12 16 31014 30998 161.427 120.562 0.00364606 0.00618466 2024-09-16T10:25:48.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:48.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:48.942 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:48.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:48 smithi063 ceph-mon[29993]: pgmap v652: 168 pgs: 168 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 806 B/s rd, 130 MiB/s wr, 2.09k op/s 2024-09-16T10:25:48.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3001969687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:48.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:48 smithi063 ceph-mon[39470]: pgmap v652: 168 pgs: 168 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 806 B/s rd, 130 MiB/s wr, 2.09k op/s 2024-09-16T10:25:48.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3001969687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:48.976 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:48.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:49.047 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:49.047 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:49.047 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:49.047 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:49.047 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:48 smithi165 ceph-mon[32696]: pgmap v652: 168 pgs: 168 active+clean; 1.2 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 806 B/s rd, 130 MiB/s wr, 2.09k op/s 2024-09-16T10:25:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3001969687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:49.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:25:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:25:49.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:49.117 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:49.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:49.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:49.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:49.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:49.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:49.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:49.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:49.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:49.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:49.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:49.714 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 13 16 33155 33139 159.302 133.812 0.00992414 0.00627155 2024-09-16T10:25:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:49.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:49.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:49.902 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:49.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:49.911 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:49.937 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:49.941 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:49.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:49 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:49 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:50.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:50.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:50.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:50.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:49 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:25:50.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:50.180 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:50.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:50.282 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:50.338 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:50.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:50.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:50.494 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:50.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:50.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:50.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:50.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 14 15 34999 34984 156.16 115.312 0.018585 0.00639708 2024-09-16T10:25:50.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:50.766 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:25:50.766 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:25:50.766 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:25:50.766 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:25:50.766 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:25:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:50.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:50.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:50.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:50.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:50.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:50.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:50 smithi063 ceph-mon[29993]: pgmap v653: 168 pgs: 168 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.90k op/s 2024-09-16T10:25:50.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:50 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:25:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:50 smithi063 ceph-mon[39470]: pgmap v653: 168 pgs: 168 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.90k op/s 2024-09-16T10:25:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:50 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:25:51.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:50 smithi165 ceph-mon[32696]: pgmap v653: 168 pgs: 168 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.90k op/s 2024-09-16T10:25:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:50 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:25:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:51.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:51.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:51.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:51.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:51.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:51.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:51.329 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:51.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:51.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:51.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:51.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:51.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:51.607 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:51.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:51.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 15 16 36836 36820 153.398 114.75 0.00989553 0.00651362 2024-09-16T10:25:51.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:51.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:51.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:51.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:51.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:25:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:51.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:51.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:51 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:25:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:51 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:25:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:51 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:51 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:25:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:51 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:25:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:51 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:51.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:52.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:52.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:51 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:25:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:51 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:25:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:51 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:52.116 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:52.163 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:52.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:52.221 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:25:52.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:52.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:52.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:52.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:52.464 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:52.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:52.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:52.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:52.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:52.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 16 16 38637 38621 150.846 112.562 0.0118935 0.00662479 2024-09-16T10:25:52.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:52.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:52 smithi063 ceph-mon[29993]: pgmap v654: 168 pgs: 168 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 165 MiB/s wr, 2.63k op/s 2024-09-16T10:25:52.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:52 smithi063 ceph-mon[39470]: pgmap v654: 168 pgs: 168 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 165 MiB/s wr, 2.63k op/s 2024-09-16T10:25:52.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:52.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:52.833 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:52.883 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:52.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:52.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:53.028 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:53.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:52 smithi165 ceph-mon[32696]: pgmap v654: 168 pgs: 168 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 165 MiB/s wr, 2.63k op/s 2024-09-16T10:25:53.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:53.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:53.210 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:53.233 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:25:53.257 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:53.299 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:25:53.303 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:25:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.315 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:53.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:53.416 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:53.457 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:53.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:53.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:53.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:53.691 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:25:53.692 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 17 16 40281 40265 148.016 102.75 0.0108772 0.00674923 2024-09-16T10:25:53.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:53.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:53.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:53.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/435483851' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:53.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[29993]: from='client.26104 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:53.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:53.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:53.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/435483851' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:53.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[39470]: from='client.26104 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:53.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:53.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:53.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:53.994 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:53.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:53 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/435483851' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:53 smithi165 ceph-mon[32696]: from='client.26104 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:25:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:53 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:25:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:25:54.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:54.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:54.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:54.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:54.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:54.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:54.365 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:54.405 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:54.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:54.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:54.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:54.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:54.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:54.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:54.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:54.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 18 16 42025 42009 145.848 109 0.0104366 0.0068513 2024-09-16T10:25:54.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:54.812 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:54.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:54.914 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[29993]: from='client.16185 v1:172.21.15.63:0/4232481921' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[29993]: pgmap v655: 168 pgs: 168 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 991 B/s rd, 159 MiB/s wr, 2.55k op/s 2024-09-16T10:25:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[29993]: from='client.26104 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[29993]: osdmap e373: 8 total, 8 up, 8 in 2024-09-16T10:25:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[39470]: from='client.16185 v1:172.21.15.63:0/4232481921' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[39470]: pgmap v655: 168 pgs: 168 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 991 B/s rd, 159 MiB/s wr, 2.55k op/s 2024-09-16T10:25:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[39470]: from='client.26104 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:54 smithi063 ceph-mon[39470]: osdmap e373: 8 total, 8 up, 8 in 2024-09-16T10:25:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:54.968 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:54.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:54.977 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-16T10:25:51.314971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2025265691,"num_objects":31093,"num_object_clones":0,"num_object_copies":93279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31093,"num_whiteouts":0,"num_read":1193,"num_read_kb":945,"num_write":31616,"num_write_kb":1977261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":527794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31897,"ondisk_log_size":31897,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9395964,"kb_used_data":5954748,"kb_used_omap":12,"kb_used_meta":3441011,"kb_avail":740565252,"statfs":{"total":767960285184,"available":758338818048,"internally_reserved":0,"allocated":6097661952,"data_stored":6093018233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3523595825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1611202560,"num_objects":24585,"num_object_clones":0,"num_object_copies":73755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24585,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":24585,"num_write_kb":1573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.339831"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":688,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034168+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:25:43.034168+0000","last_peered":"2024-09-16T10:25:43.034168+0000","last_clean":"2024-09-16T10:25:43.034168+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:25:43.034168+0000","last_undegraded":"2024-09-16T10:25:43.034168+0000","last_fullsized":"2024-09-16T10:25:43.034168+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":737,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980053+0000","last_change":"2024-09-16T10:25:04.357048+0000","last_active":"2024-09-16T10:25:42.980053+0000","last_peered":"2024-09-16T10:25:42.980053+0000","last_clean":"2024-09-16T10:25:42.980053+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:25:42.980053+0000","last_undegraded":"2024-09-16T10:25:42.980053+0000","last_fullsized":"2024-09-16T10:25:42.980053+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:25:04.356955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.356955+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:13.131130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985311+0000","last_change":"2024-09-16T10:25:36.940535+0000","last_active":"2024-09-16T10:25:42.985311+0000","last_peered":"2024-09-16T10:25:42.985311+0000","last_clean":"2024-09-16T10:25:42.985311+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:25:42.985311+0000","last_undegraded":"2024-09-16T10:25:42.985311+0000","last_fullsized":"2024-09-16T10:25:42.985311+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.940459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.940459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.230715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.981129+0000","last_change":"2024-09-16T10:25:08.485862+0000","last_active":"2024-09-16T10:25:42.981129+0000","last_peered":"2024-09-16T10:25:42.981129+0000","last_clean":"2024-09-16T10:25:42.981129+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:25:42.981129+0000","last_undegraded":"2024-09-16T10:25:42.981129+0000","last_fullsized":"2024-09-16T10:25:42.981129+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.485783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.485783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:08.732498+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":652,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028535+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:25:43.028535+0000","last_peered":"2024-09-16T10:25:43.028535+0000","last_clean":"2024-09-16T10:25:43.028535+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:25:43.028535+0000","last_undegraded":"2024-09-16T10:25:43.028535+0000","last_fullsized":"2024-09-16T10:25:43.028535+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":696,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031858+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:25:43.031858+0000","last_peered":"2024-09-16T10:25:43.031858+0000","last_clean":"2024-09-16T10:25:43.031858+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:25:43.031858+0000","last_undegraded":"2024-09-16T10:25:43.031858+0000","last_fullsized":"2024-09-16T10:25:43.031858+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.018649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":758,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028638+0000","last_change":"2024-09-16T10:24:37.612269+0000","last_active":"2024-09-16T10:25:43.028638+0000","last_peered":"2024-09-16T10:25:43.028638+0000","last_clean":"2024-09-16T10:25:43.028638+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:25:43.028638+0000","last_undegraded":"2024-09-16T10:25:43.028638+0000","last_fullsized":"2024-09-16T10:25:43.028638+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:37.612167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.612167+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.984288+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.979704+0000","last_change":"2024-09-16T10:24:21.153205+0000","last_active":"2024-09-16T10:25:42.979704+0000","last_peered":"2024-09-16T10:25:42.979704+0000","last_clean":"2024-09-16T10:25:42.979704+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:25:42.979704+0000","last_undegraded":"2024-09-16T10:25:42.979704+0000","last_fullsized":"2024-09-16T10:25:42.979704+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:21.153131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:21.153131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.430119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":690,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985978+0000","last_change":"2024-09-16T10:25:13.988034+0000","last_active":"2024-09-16T10:25:42.985978+0000","last_peered":"2024-09-16T10:25:42.985978+0000","last_clean":"2024-09-16T10:25:42.985978+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:25:42.985978+0000","last_undegraded":"2024-09-16T10:25:42.985978+0000","last_fullsized":"2024-09-16T10:25:42.985978+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.987974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.987974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.738929+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":702,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.981951+0000","last_change":"2024-09-16T10:24:23.258943+0000","last_active":"2024-09-16T10:25:42.981951+0000","last_peered":"2024-09-16T10:25:42.981951+0000","last_clean":"2024-09-16T10:25:42.981951+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:25:42.981951+0000","last_undegraded":"2024-09-16T10:25:42.981951+0000","last_fullsized":"2024-09-16T10:25:42.981951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:24:23.258886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.258886+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985515+0000","last_change":"2024-09-16T10:25:33.834886+0000","last_active":"2024-09-16T10:25:42.985515+0000","last_peered":"2024-09-16T10:25:42.985515+0000","last_clean":"2024-09-16T10:25:42.985515+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:25:42.985515+0000","last_undegraded":"2024-09-16T10:25:42.985515+0000","last_fullsized":"2024-09-16T10:25:42.985515+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.834823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.834823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.729220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":661,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032647+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:25:43.032647+0000","last_peered":"2024-09-16T10:25:43.032647+0000","last_clean":"2024-09-16T10:25:43.032647+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:25:43.032647+0000","last_undegraded":"2024-09-16T10:25:43.032647+0000","last_fullsized":"2024-09-16T10:25:43.032647+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986292+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:25:42.986292+0000","last_peered":"2024-09-16T10:25:42.986292+0000","last_clean":"2024-09-16T10:25:42.986292+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:25:42.986292+0000","last_undegraded":"2024-09-16T10:25:42.986292+0000","last_fullsized":"2024-09-16T10:25:42.986292+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.823305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":690,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.979484+0000","last_change":"2024-09-16T10:24:37.134728+0000","last_active":"2024-09-16T10:25:42.979484+0000","last_peered":"2024-09-16T10:25:42.979484+0000","last_clean":"2024-09-16T10:25:42.979484+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:25:42.979484+0000","last_undegraded":"2024-09-16T10:25:42.979484+0000","last_fullsized":"2024-09-16T10:25:42.979484+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.981147+0000","last_change":"2024-09-16T10:25:15.369074+0000","last_active":"2024-09-16T10:25:42.981147+0000","last_peered":"2024-09-16T10:25:42.981147+0000","last_clean":"2024-09-16T10:25:42.981147+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:25:42.981147+0000","last_undegraded":"2024-09-16T10:25:42.981147+0000","last_fullsized":"2024-09-16T10:25:42.981147+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.369005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.369005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.347891+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":677,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986898+0000","last_change":"2024-09-16T10:23:42.856401+0000","last_active":"2024-09-16T10:25:42.986898+0000","last_peered":"2024-09-16T10:25:42.986898+0000","last_clean":"2024-09-16T10:25:42.986898+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:25:42.986898+0000","last_undegraded":"2024-09-16T10:25:42.986898+0000","last_fullsized":"2024-09-16T10:25:42.986898+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.856353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.856353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:25.927067+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":655,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982398+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:25:42.982398+0000","last_peered":"2024-09-16T10:25:42.982398+0000","last_clean":"2024-09-16T10:25:42.982398+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:25:42.982398+0000","last_undegraded":"2024-09-16T10:25:42.982398+0000","last_fullsized":"2024-09-16T10:25:42.982398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:56.358685+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":665,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.033700+0000","last_change":"2024-09-16T10:22:27.210541+0000","last_active":"2024-09-16T10:25:43.033700+0000","last_peered":"2024-09-16T10:25:43.033700+0000","last_clean":"2024-09-16T10:25:43.033700+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:25:43.033700+0000","last_undegraded":"2024-09-16T10:25:43.033700+0000","last_fullsized":"2024-09-16T10:25:43.033700+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":235,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029624+0000","last_change":"2024-09-16T10:25:08.473527+0000","last_active":"2024-09-16T10:25:43.029624+0000","last_peered":"2024-09-16T10:25:43.029624+0000","last_clean":"2024-09-16T10:25:43.029624+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:25:43.029624+0000","last_undegraded":"2024-09-16T10:25:43.029624+0000","last_fullsized":"2024-09-16T10:25:43.029624+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:25:08.473402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.473402+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.786201+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":658,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013181+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:25:43.013181+0000","last_peered":"2024-09-16T10:25:43.013181+0000","last_clean":"2024-09-16T10:25:43.013181+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:25:43.013181+0000","last_undegraded":"2024-09-16T10:25:43.013181+0000","last_fullsized":"2024-09-16T10:25:43.013181+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.737702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":618,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032247+0000","last_change":"2024-09-16T10:25:15.468418+0000","last_active":"2024-09-16T10:25:43.032247+0000","last_peered":"2024-09-16T10:25:43.032247+0000","last_clean":"2024-09-16T10:25:43.032247+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:25:43.032247+0000","last_undegraded":"2024-09-16T10:25:43.032247+0000","last_fullsized":"2024-09-16T10:25:43.032247+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.468271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.468271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.583811+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":679,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982100+0000","last_change":"2024-09-16T10:25:14.407660+0000","last_active":"2024-09-16T10:25:42.982100+0000","last_peered":"2024-09-16T10:25:42.982100+0000","last_clean":"2024-09-16T10:25:42.982100+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:25:42.982100+0000","last_undegraded":"2024-09-16T10:25:42.982100+0000","last_fullsized":"2024-09-16T10:25:42.982100+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.063208+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":727,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028839+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:25:43.028839+0000","last_peered":"2024-09-16T10:25:43.028839+0000","last_clean":"2024-09-16T10:25:43.028839+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:25:43.028839+0000","last_undegraded":"2024-09-16T10:25:43.028839+0000","last_fullsized":"2024-09-16T10:25:43.028839+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.381310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":687,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982424+0000","last_change":"2024-09-16T10:24:35.186387+0000","last_active":"2024-09-16T10:25:42.982424+0000","last_peered":"2024-09-16T10:25:42.982424+0000","last_clean":"2024-09-16T10:25:42.982424+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:25:42.982424+0000","last_undegraded":"2024-09-16T10:25:42.982424+0000","last_fullsized":"2024-09-16T10:25:42.982424+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:35.186332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:35.186332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.208711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":74,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985217+0000","last_change":"2024-09-16T10:25:18.973817+0000","last_active":"2024-09-16T10:25:42.985217+0000","last_peered":"2024-09-16T10:25:42.985217+0000","last_clean":"2024-09-16T10:25:42.985217+0000","last_became_active":"2024-09-16T10:25:00.947649+0000","last_became_peered":"2024-09-16T10:25:00.947649+0000","last_unstale":"2024-09-16T10:25:42.985217+0000","last_undegraded":"2024-09-16T10:25:42.985217+0000","last_fullsized":"2024-09-16T10:25:42.985217+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:44.946516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.012230+0000","last_change":"2024-09-16T10:25:06.652209+0000","last_active":"2024-09-16T10:25:43.012230+0000","last_peered":"2024-09-16T10:25:43.012230+0000","last_clean":"2024-09-16T10:25:43.012230+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:25:43.012230+0000","last_undegraded":"2024-09-16T10:25:43.012230+0000","last_fullsized":"2024-09-16T10:25:43.012230+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:25:06.652146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:06.652146+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.202612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":701,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034796+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:25:43.034796+0000","last_peered":"2024-09-16T10:25:43.034796+0000","last_clean":"2024-09-16T10:25:43.034796+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:25:43.034796+0000","last_undegraded":"2024-09-16T10:25:43.034796+0000","last_fullsized":"2024-09-16T10:25:43.034796+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":824,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989847+0000","last_change":"2024-09-16T10:25:16.538777+0000","last_active":"2024-09-16T10:25:42.989847+0000","last_peered":"2024-09-16T10:25:42.989847+0000","last_clean":"2024-09-16T10:25:42.989847+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:25:42.989847+0000","last_undegraded":"2024-09-16T10:25:42.989847+0000","last_fullsized":"2024-09-16T10:25:42.989847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:16.538706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.538706+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.979675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"372'1867","reported_seq":1898,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.240071+0000","last_change":"2024-09-16T10:25:30.947944+0000","last_active":"2024-09-16T10:25:48.240071+0000","last_peered":"2024-09-16T10:25:48.240071+0000","last_clean":"2024-09-16T10:25:48.240071+0000","last_became_active":"2024-09-16T10:25:30.947200+0000","last_became_peered":"2024-09-16T10:25:30.947200+0000","last_unstale":"2024-09-16T10:25:48.240071+0000","last_undegraded":"2024-09-16T10:25:48.240071+0000","last_fullsized":"2024-09-16T10:25:48.240071+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1867,"log_dups_size":0,"ondisk_log_size":1867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.630601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122355712,"num_objects":1867,"num_object_clones":0,"num_object_copies":5601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1867,"num_write_kb":119488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":520,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.984657+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:25:42.984657+0000","last_peered":"2024-09-16T10:25:42.984657+0000","last_clean":"2024-09-16T10:25:42.984657+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:25:42.984657+0000","last_undegraded":"2024-09-16T10:25:42.984657+0000","last_fullsized":"2024-09-16T10:25:42.984657+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.039248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":706,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.014335+0000","last_change":"2024-09-16T10:25:10.700157+0000","last_active":"2024-09-16T10:25:43.014335+0000","last_peered":"2024-09-16T10:25:43.014335+0000","last_clean":"2024-09-16T10:25:43.014335+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:25:43.014335+0000","last_undegraded":"2024-09-16T10:25:43.014335+0000","last_fullsized":"2024-09-16T10:25:43.014335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:25:10.699735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:10.699735+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:35.612246+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":681,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031996+0000","last_change":"2024-09-16T10:25:33.377037+0000","last_active":"2024-09-16T10:25:43.031996+0000","last_peered":"2024-09-16T10:25:43.031996+0000","last_clean":"2024-09-16T10:25:43.031996+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:25:43.031996+0000","last_undegraded":"2024-09-16T10:25:43.031996+0000","last_fullsized":"2024-09-16T10:25:43.031996+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:41.874798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":813,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982401+0000","last_change":"2024-09-16T10:24:57.408285+0000","last_active":"2024-09-16T10:25:42.982401+0000","last_peered":"2024-09-16T10:25:42.982401+0000","last_clean":"2024-09-16T10:25:42.982401+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:25:42.982401+0000","last_undegraded":"2024-09-16T10:25:42.982401+0000","last_fullsized":"2024-09-16T10:25:42.982401+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:57.408217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.408217+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:09.573273+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":663,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985545+0000","last_change":"2024-09-16T10:24:15.859366+0000","last_active":"2024-09-16T10:25:42.985545+0000","last_peered":"2024-09-16T10:25:42.985545+0000","last_clean":"2024-09-16T10:25:42.985545+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:25:42.985545+0000","last_undegraded":"2024-09-16T10:25:42.985545+0000","last_fullsized":"2024-09-16T10:25:42.985545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:15.859299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.859299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"372'1845","reported_seq":1876,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.239928+0000","last_change":"2024-09-16T10:25:30.619893+0000","last_active":"2024-09-16T10:25:48.239928+0000","last_peered":"2024-09-16T10:25:48.239928+0000","last_clean":"2024-09-16T10:25:48.239928+0000","last_became_active":"2024-09-16T10:25:30.619574+0000","last_became_peered":"2024-09-16T10:25:30.619574+0000","last_unstale":"2024-09-16T10:25:48.239928+0000","last_undegraded":"2024-09-16T10:25:48.239928+0000","last_fullsized":"2024-09-16T10:25:48.239928+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1845,"log_dups_size":0,"ondisk_log_size":1845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.565846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120913920,"num_objects":1845,"num_object_clones":0,"num_object_copies":5535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1845,"num_write_kb":118080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":572,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982068+0000","last_change":"2024-09-16T10:24:28.146701+0000","last_active":"2024-09-16T10:25:42.982068+0000","last_peered":"2024-09-16T10:25:42.982068+0000","last_clean":"2024-09-16T10:25:42.982068+0000","last_became_active":"2024-09-16T10:22:56.409926+0000","last_became_peered":"2024-09-16T10:22:56.409926+0000","last_unstale":"2024-09-16T10:25:42.982068+0000","last_undegraded":"2024-09-16T10:25:42.982068+0000","last_fullsized":"2024-09-16T10:25:42.982068+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.146633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.146633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":156,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.979520+0000","last_change":"2024-09-16T10:23:42.023903+0000","last_active":"2024-09-16T10:25:42.979520+0000","last_peered":"2024-09-16T10:25:42.979520+0000","last_clean":"2024-09-16T10:25:42.979520+0000","last_became_active":"2024-09-16T10:23:41.951456+0000","last_became_peered":"2024-09-16T10:23:41.951456+0000","last_unstale":"2024-09-16T10:25:42.979520+0000","last_undegraded":"2024-09-16T10:25:42.979520+0000","last_fullsized":"2024-09-16T10:25:42.979520+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:23:13.871560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.871560+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.206554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032717+0000","last_change":"2024-09-16T10:25:36.404433+0000","last_active":"2024-09-16T10:25:43.032717+0000","last_peered":"2024-09-16T10:25:43.032717+0000","last_clean":"2024-09-16T10:25:43.032717+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:25:43.032717+0000","last_undegraded":"2024-09-16T10:25:43.032717+0000","last_fullsized":"2024-09-16T10:25:43.032717+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.404383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.404383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.505459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":405,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.988455+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:25:42.988455+0000","last_peered":"2024-09-16T10:25:42.988455+0000","last_clean":"2024-09-16T10:25:42.988455+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:25:42.988455+0000","last_undegraded":"2024-09-16T10:25:42.988455+0000","last_fullsized":"2024-09-16T10:25:42.988455+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.558332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":684,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.981447+0000","last_change":"2024-09-16T10:24:22.206462+0000","last_active":"2024-09-16T10:25:42.981447+0000","last_peered":"2024-09-16T10:25:42.981447+0000","last_clean":"2024-09-16T10:25:42.981447+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:25:42.981447+0000","last_undegraded":"2024-09-16T10:25:42.981447+0000","last_fullsized":"2024-09-16T10:25:42.981447+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:22.206414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:22.206414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.375025+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"372'1885","reported_seq":1916,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.329797+0000","last_change":"2024-09-16T10:25:31.163385+0000","last_active":"2024-09-16T10:25:48.329797+0000","last_peered":"2024-09-16T10:25:48.329797+0000","last_clean":"2024-09-16T10:25:48.329797+0000","last_became_active":"2024-09-16T10:25:31.162906+0000","last_became_peered":"2024-09-16T10:25:31.162906+0000","last_unstale":"2024-09-16T10:25:48.329797+0000","last_undegraded":"2024-09-16T10:25:48.329797+0000","last_fullsized":"2024-09-16T10:25:48.329797+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1885,"log_dups_size":0,"ondisk_log_size":1885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.032805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123535360,"num_objects":1885,"num_object_clones":0,"num_object_copies":5655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1885,"num_write_kb":120640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":563,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029559+0000","last_change":"2024-09-16T10:24:28.343296+0000","last_active":"2024-09-16T10:25:43.029559+0000","last_peered":"2024-09-16T10:25:43.029559+0000","last_clean":"2024-09-16T10:25:43.029559+0000","last_became_active":"2024-09-16T10:22:56.446950+0000","last_became_peered":"2024-09-16T10:22:56.446950+0000","last_unstale":"2024-09-16T10:25:43.029559+0000","last_undegraded":"2024-09-16T10:25:43.029559+0000","last_fullsized":"2024-09-16T10:25:43.029559+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.343066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.343066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":720,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.979689+0000","last_change":"2024-09-16T10:24:59.344438+0000","last_active":"2024-09-16T10:25:42.979689+0000","last_peered":"2024-09-16T10:25:42.979689+0000","last_clean":"2024-09-16T10:25:42.979689+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:25:42.979689+0000","last_undegraded":"2024-09-16T10:25:42.979689+0000","last_fullsized":"2024-09-16T10:25:42.979689+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:24:59.344346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.344346+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:05.786389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":667,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034851+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:25:43.034851+0000","last_peered":"2024-09-16T10:25:43.034851+0000","last_clean":"2024-09-16T10:25:43.034851+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:25:43.034851+0000","last_undegraded":"2024-09-16T10:25:43.034851+0000","last_fullsized":"2024-09-16T10:25:43.034851+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.295227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":824,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.989763+0000","last_change":"2024-09-16T10:25:12.476210+0000","last_active":"2024-09-16T10:25:43.989763+0000","last_peered":"2024-09-16T10:25:43.989763+0000","last_clean":"2024-09-16T10:25:43.989763+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:25:43.989763+0000","last_undegraded":"2024-09-16T10:25:43.989763+0000","last_fullsized":"2024-09-16T10:25:43.989763+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:12.476137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.476137+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:36.834759+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":514,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989035+0000","last_change":"2024-09-16T10:25:00.948343+0000","last_active":"2024-09-16T10:25:42.989035+0000","last_peered":"2024-09-16T10:25:42.989035+0000","last_clean":"2024-09-16T10:25:42.989035+0000","last_became_active":"2024-09-16T10:25:00.946867+0000","last_became_peered":"2024-09-16T10:25:00.946867+0000","last_unstale":"2024-09-16T10:25:42.989035+0000","last_undegraded":"2024-09-16T10:25:42.989035+0000","last_fullsized":"2024-09-16T10:25:42.989035+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.423640+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"372'1953","reported_seq":1984,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.240111+0000","last_change":"2024-09-16T10:25:30.949013+0000","last_active":"2024-09-16T10:25:48.240111+0000","last_peered":"2024-09-16T10:25:48.240111+0000","last_clean":"2024-09-16T10:25:48.240111+0000","last_became_active":"2024-09-16T10:25:30.948309+0000","last_became_peered":"2024-09-16T10:25:30.948309+0000","last_unstale":"2024-09-16T10:25:48.240111+0000","last_undegraded":"2024-09-16T10:25:48.240111+0000","last_fullsized":"2024-09-16T10:25:48.240111+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1953,"log_dups_size":0,"ondisk_log_size":1953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.205764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127991808,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":124992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":613,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989697+0000","last_change":"2024-09-16T10:24:31.617241+0000","last_active":"2024-09-16T10:25:42.989697+0000","last_peered":"2024-09-16T10:25:42.989697+0000","last_clean":"2024-09-16T10:25:42.989697+0000","last_became_active":"2024-09-16T10:22:56.438108+0000","last_became_peered":"2024-09-16T10:22:56.438108+0000","last_unstale":"2024-09-16T10:25:42.989697+0000","last_undegraded":"2024-09-16T10:25:42.989697+0000","last_fullsized":"2024-09-16T10:25:42.989697+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.617190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.617190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.711210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":709,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013127+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:25:43.013127+0000","last_peered":"2024-09-16T10:25:43.013127+0000","last_clean":"2024-09-16T10:25:43.013127+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:25:43.013127+0000","last_undegraded":"2024-09-16T10:25:43.013127+0000","last_fullsized":"2024-09-16T10:25:43.013127+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":689,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986049+0000","last_change":"2024-09-16T10:24:59.820337+0000","last_active":"2024-09-16T10:25:42.986049+0000","last_peered":"2024-09-16T10:25:42.986049+0000","last_clean":"2024-09-16T10:25:42.986049+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:25:42.986049+0000","last_undegraded":"2024-09-16T10:25:42.986049+0000","last_fullsized":"2024-09-16T10:25:42.986049+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:59.820273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.820273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.866898+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":805,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031547+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:25:43.031547+0000","last_peered":"2024-09-16T10:25:43.031547+0000","last_clean":"2024-09-16T10:25:43.031547+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:25:43.031547+0000","last_undegraded":"2024-09-16T10:25:43.031547+0000","last_fullsized":"2024-09-16T10:25:43.031547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.537119+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":677,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034147+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:25:43.034147+0000","last_peered":"2024-09-16T10:25:43.034147+0000","last_clean":"2024-09-16T10:25:43.034147+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:25:43.034147+0000","last_undegraded":"2024-09-16T10:25:43.034147+0000","last_fullsized":"2024-09-16T10:25:43.034147+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"372'1954","reported_seq":1985,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.321974+0000","last_change":"2024-09-16T10:25:30.621332+0000","last_active":"2024-09-16T10:25:48.321974+0000","last_peered":"2024-09-16T10:25:48.321974+0000","last_clean":"2024-09-16T10:25:48.321974+0000","last_became_active":"2024-09-16T10:25:30.620945+0000","last_became_peered":"2024-09-16T10:25:30.620945+0000","last_unstale":"2024-09-16T10:25:48.321974+0000","last_undegraded":"2024-09-16T10:25:48.321974+0000","last_fullsized":"2024-09-16T10:25:48.321974+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1954,"log_dups_size":0,"ondisk_log_size":1954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:38.539869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128057344,"num_objects":1954,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1954,"num_write_kb":125056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":618,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989014+0000","last_change":"2024-09-16T10:24:30.609351+0000","last_active":"2024-09-16T10:25:42.989014+0000","last_peered":"2024-09-16T10:25:42.989014+0000","last_clean":"2024-09-16T10:25:42.989014+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:25:42.989014+0000","last_undegraded":"2024-09-16T10:25:42.989014+0000","last_fullsized":"2024-09-16T10:25:42.989014+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:30.609302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:30.609302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:47.839505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":721,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.033555+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:25:43.033555+0000","last_peered":"2024-09-16T10:25:43.033555+0000","last_clean":"2024-09-16T10:25:43.033555+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:25:43.033555+0000","last_undegraded":"2024-09-16T10:25:43.033555+0000","last_fullsized":"2024-09-16T10:25:43.033555+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":689,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013353+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:25:43.013353+0000","last_peered":"2024-09-16T10:25:43.013353+0000","last_clean":"2024-09-16T10:25:43.013353+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:25:43.013353+0000","last_undegraded":"2024-09-16T10:25:43.013353+0000","last_fullsized":"2024-09-16T10:25:43.013353+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":682,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989012+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:25:42.989012+0000","last_peered":"2024-09-16T10:25:42.989012+0000","last_clean":"2024-09-16T10:25:42.989012+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:25:42.989012+0000","last_undegraded":"2024-09-16T10:25:42.989012+0000","last_fullsized":"2024-09-16T10:25:42.989012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.230724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985178+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:25:42.985178+0000","last_peered":"2024-09-16T10:25:42.985178+0000","last_clean":"2024-09-16T10:25:42.985178+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:25:42.985178+0000","last_undegraded":"2024-09-16T10:25:42.985178+0000","last_fullsized":"2024-09-16T10:25:42.985178+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.861279+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"372'1887","reported_seq":1917,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.338654+0000","last_change":"2024-09-16T10:25:30.622174+0000","last_active":"2024-09-16T10:25:48.338654+0000","last_peered":"2024-09-16T10:25:48.338654+0000","last_clean":"2024-09-16T10:25:48.338654+0000","last_became_active":"2024-09-16T10:25:30.621742+0000","last_became_peered":"2024-09-16T10:25:30.621742+0000","last_unstale":"2024-09-16T10:25:48.338654+0000","last_undegraded":"2024-09-16T10:25:48.338654+0000","last_fullsized":"2024-09-16T10:25:48.338654+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1887,"log_dups_size":0,"ondisk_log_size":1887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.135457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123666432,"num_objects":1887,"num_object_clones":0,"num_object_copies":5661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1887,"num_write_kb":120768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":564,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986701+0000","last_change":"2024-09-16T10:24:29.815138+0000","last_active":"2024-09-16T10:25:42.986701+0000","last_peered":"2024-09-16T10:25:42.986701+0000","last_clean":"2024-09-16T10:25:42.986701+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:25:42.986701+0000","last_undegraded":"2024-09-16T10:25:42.986701+0000","last_fullsized":"2024-09-16T10:25:42.986701+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:29.814966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:29.814966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.438495+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":958,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013243+0000","last_change":"2024-09-16T10:25:33.966865+0000","last_active":"2024-09-16T10:25:43.013243+0000","last_peered":"2024-09-16T10:25:43.013243+0000","last_clean":"2024-09-16T10:25:43.013243+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:25:43.013243+0000","last_undegraded":"2024-09-16T10:25:43.013243+0000","last_fullsized":"2024-09-16T10:25:43.013243+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:25:33.966775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.966775+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:45.795898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":682,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980058+0000","last_change":"2024-09-16T10:24:13.097213+0000","last_active":"2024-09-16T10:25:42.980058+0000","last_peered":"2024-09-16T10:25:42.980058+0000","last_clean":"2024-09-16T10:25:42.980058+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:25:42.980058+0000","last_undegraded":"2024-09-16T10:25:42.980058+0000","last_fullsized":"2024-09-16T10:25:42.980058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:13.097133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:13.097133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":677,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985584+0000","last_change":"2024-09-16T10:24:57.812754+0000","last_active":"2024-09-16T10:25:42.985584+0000","last_peered":"2024-09-16T10:25:42.985584+0000","last_clean":"2024-09-16T10:25:42.985584+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:25:42.985584+0000","last_undegraded":"2024-09-16T10:25:42.985584+0000","last_fullsized":"2024-09-16T10:25:42.985584+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:24:57.812701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.812701+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:59.841010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":658,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031982+0000","last_change":"2024-09-16T10:25:37.359150+0000","last_active":"2024-09-16T10:25:43.031982+0000","last_peered":"2024-09-16T10:25:43.031982+0000","last_clean":"2024-09-16T10:25:43.031982+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:25:43.031982+0000","last_undegraded":"2024-09-16T10:25:43.031982+0000","last_fullsized":"2024-09-16T10:25:43.031982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:37.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.658889+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029543+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:25:43.029543+0000","last_peered":"2024-09-16T10:25:43.029543+0000","last_clean":"2024-09-16T10:25:43.029543+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:25:43.029543+0000","last_undegraded":"2024-09-16T10:25:43.029543+0000","last_fullsized":"2024-09-16T10:25:43.029543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.440379+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"372'1965","reported_seq":1996,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.511820+0000","last_change":"2024-09-16T10:25:30.948418+0000","last_active":"2024-09-16T10:25:48.511820+0000","last_peered":"2024-09-16T10:25:48.511820+0000","last_clean":"2024-09-16T10:25:48.511820+0000","last_became_active":"2024-09-16T10:25:30.947760+0000","last_became_peered":"2024-09-16T10:25:30.947760+0000","last_unstale":"2024-09-16T10:25:48.511820+0000","last_undegraded":"2024-09-16T10:25:48.511820+0000","last_fullsized":"2024-09-16T10:25:48.511820+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1965,"log_dups_size":0,"ondisk_log_size":1965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.237682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128778240,"num_objects":1965,"num_object_clones":0,"num_object_copies":5895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1965,"num_write_kb":125760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":137,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031980+0000","last_change":"2024-09-16T10:25:02.365887+0000","last_active":"2024-09-16T10:25:43.031980+0000","last_peered":"2024-09-16T10:25:43.031980+0000","last_clean":"2024-09-16T10:25:43.031980+0000","last_became_active":"2024-09-16T10:25:00.955466+0000","last_became_peered":"2024-09-16T10:25:00.955466+0000","last_unstale":"2024-09-16T10:25:43.031980+0000","last_undegraded":"2024-09-16T10:25:43.031980+0000","last_fullsized":"2024-09-16T10:25:43.031980+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.133909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":688,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.014434+0000","last_change":"2024-09-16T10:25:13.649143+0000","last_active":"2024-09-16T10:25:43.014434+0000","last_peered":"2024-09-16T10:25:43.014434+0000","last_clean":"2024-09-16T10:25:43.014434+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:25:43.014434+0000","last_undegraded":"2024-09-16T10:25:43.014434+0000","last_fullsized":"2024-09-16T10:25:43.014434+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:25:13.648981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.648981+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.641189+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.012726+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:25:43.012726+0000","last_peered":"2024-09-16T10:25:43.012726+0000","last_clean":"2024-09-16T10:25:43.012726+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:25:43.012726+0000","last_undegraded":"2024-09-16T10:25:43.012726+0000","last_fullsized":"2024-09-16T10:25:43.012726+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:26.834421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":205,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013533+0000","last_change":"2024-09-16T10:25:10.324756+0000","last_active":"2024-09-16T10:25:43.013533+0000","last_peered":"2024-09-16T10:25:43.013533+0000","last_clean":"2024-09-16T10:25:43.013533+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:25:43.013533+0000","last_undegraded":"2024-09-16T10:25:43.013533+0000","last_fullsized":"2024-09-16T10:25:43.013533+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:25:10.324680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:25:10.324680+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.799924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":201,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013559+0000","last_change":"2024-09-16T10:22:29.905776+0000","last_active":"2024-09-16T10:25:43.013559+0000","last_peered":"2024-09-16T10:25:43.013559+0000","last_clean":"2024-09-16T10:25:43.013559+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:25:43.013559+0000","last_undegraded":"2024-09-16T10:25:43.013559+0000","last_fullsized":"2024-09-16T10:25:43.013559+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.460922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":62,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989042+0000","last_change":"2024-09-16T10:25:32.463597+0000","last_active":"2024-09-16T10:25:42.989042+0000","last_peered":"2024-09-16T10:25:42.989042+0000","last_clean":"2024-09-16T10:25:42.989042+0000","last_became_active":"2024-09-16T10:25:00.950144+0000","last_became_peered":"2024-09-16T10:25:00.950144+0000","last_unstale":"2024-09-16T10:25:42.989042+0000","last_undegraded":"2024-09-16T10:25:42.989042+0000","last_fullsized":"2024-09-16T10:25:42.989042+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.360271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"372'1926","reported_seq":1957,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:49.299557+0000","last_change":"2024-09-16T10:25:30.950233+0000","last_active":"2024-09-16T10:25:49.299557+0000","last_peered":"2024-09-16T10:25:49.299557+0000","last_clean":"2024-09-16T10:25:49.299557+0000","last_became_active":"2024-09-16T10:25:30.948708+0000","last_became_peered":"2024-09-16T10:25:30.948708+0000","last_unstale":"2024-09-16T10:25:49.299557+0000","last_undegraded":"2024-09-16T10:25:49.299557+0000","last_fullsized":"2024-09-16T10:25:49.299557+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1926,"log_dups_size":0,"ondisk_log_size":1926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:35.650968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126222336,"num_objects":1926,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1926,"num_write_kb":123264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":552,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.015414+0000","last_change":"2024-09-16T10:24:27.887030+0000","last_active":"2024-09-16T10:25:43.015414+0000","last_peered":"2024-09-16T10:25:43.015414+0000","last_clean":"2024-09-16T10:25:43.015414+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:25:43.015414+0000","last_undegraded":"2024-09-16T10:25:43.015414+0000","last_fullsized":"2024-09-16T10:25:43.015414+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:27.886961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:27.886961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.710539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":764,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034736+0000","last_change":"2024-09-16T10:22:24.213089+0000","last_active":"2024-09-16T10:25:43.034736+0000","last_peered":"2024-09-16T10:25:43.034736+0000","last_clean":"2024-09-16T10:25:43.034736+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:25:43.034736+0000","last_undegraded":"2024-09-16T10:25:43.034736+0000","last_fullsized":"2024-09-16T10:25:43.034736+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:22:24.213008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.213008+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":690,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032597+0000","last_change":"2024-09-16T10:22:26.226703+0000","last_active":"2024-09-16T10:25:43.032597+0000","last_peered":"2024-09-16T10:25:43.032597+0000","last_clean":"2024-09-16T10:25:43.032597+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:25:43.032597+0000","last_undegraded":"2024-09-16T10:25:43.032597+0000","last_fullsized":"2024-09-16T10:25:43.032597+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.226632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.226632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":144,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:47.097138+0000","last_change":"2024-09-16T10:25:00.956734+0000","last_active":"2024-09-16T10:25:47.097138+0000","last_peered":"2024-09-16T10:25:47.097138+0000","last_clean":"2024-09-16T10:25:47.097138+0000","last_became_active":"2024-09-16T10:25:00.956304+0000","last_became_peered":"2024-09-16T10:25:00.956304+0000","last_unstale":"2024-09-16T10:25:47.097138+0000","last_undegraded":"2024-09-16T10:25:47.097138+0000","last_fullsized":"2024-09-16T10:25:47.097138+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:23:41.597682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.597682+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":54,"num_read_kb":49,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986446+0000","last_change":"2024-09-16T10:25:35.847648+0000","last_active":"2024-09-16T10:25:42.986446+0000","last_peered":"2024-09-16T10:25:42.986446+0000","last_clean":"2024-09-16T10:25:42.986446+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:25:42.986446+0000","last_undegraded":"2024-09-16T10:25:42.986446+0000","last_fullsized":"2024-09-16T10:25:42.986446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.847598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.847598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:00.609111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":679,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.012794+0000","last_change":"2024-09-16T10:25:35.937305+0000","last_active":"2024-09-16T10:25:43.012794+0000","last_peered":"2024-09-16T10:25:43.012794+0000","last_clean":"2024-09-16T10:25:43.012794+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:25:43.012794+0000","last_undegraded":"2024-09-16T10:25:43.012794+0000","last_fullsized":"2024-09-16T10:25:43.012794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.937257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.937257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.310507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"372'1883","reported_seq":1914,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.342191+0000","last_change":"2024-09-16T10:25:30.622105+0000","last_active":"2024-09-16T10:25:48.342191+0000","last_peered":"2024-09-16T10:25:48.342191+0000","last_clean":"2024-09-16T10:25:48.342191+0000","last_became_active":"2024-09-16T10:25:30.621779+0000","last_became_peered":"2024-09-16T10:25:30.621779+0000","last_unstale":"2024-09-16T10:25:48.342191+0000","last_undegraded":"2024-09-16T10:25:48.342191+0000","last_fullsized":"2024-09-16T10:25:48.342191+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1883,"log_dups_size":0,"ondisk_log_size":1883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.268900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123404288,"num_objects":1883,"num_object_clones":0,"num_object_copies":5649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1883,"num_write_kb":120512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"328'28","reported_seq":841,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989147+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:25:42.989147+0000","last_peered":"2024-09-16T10:25:42.989147+0000","last_clean":"2024-09-16T10:25:42.989147+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:25:42.989147+0000","last_undegraded":"2024-09-16T10:25:42.989147+0000","last_fullsized":"2024-09-16T10:25:42.989147+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:26.575852+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":638,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029046+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:25:43.029046+0000","last_peered":"2024-09-16T10:25:43.029046+0000","last_clean":"2024-09-16T10:25:43.029046+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:25:43.029046+0000","last_undegraded":"2024-09-16T10:25:43.029046+0000","last_fullsized":"2024-09-16T10:25:43.029046+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.609525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":137,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034609+0000","last_change":"2024-09-16T10:25:01.375800+0000","last_active":"2024-09-16T10:25:43.034609+0000","last_peered":"2024-09-16T10:25:43.034609+0000","last_clean":"2024-09-16T10:25:43.034609+0000","last_became_active":"2024-09-16T10:25:00.955671+0000","last_became_peered":"2024-09-16T10:25:00.955671+0000","last_unstale":"2024-09-16T10:25:43.034609+0000","last_undegraded":"2024-09-16T10:25:43.034609+0000","last_fullsized":"2024-09-16T10:25:43.034609+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.945066+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":534,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013774+0000","last_change":"2024-09-16T10:25:35.591372+0000","last_active":"2024-09-16T10:25:43.013774+0000","last_peered":"2024-09-16T10:25:43.013774+0000","last_clean":"2024-09-16T10:25:43.013774+0000","last_became_active":"2024-09-16T10:25:00.946698+0000","last_became_peered":"2024-09-16T10:25:00.946698+0000","last_unstale":"2024-09-16T10:25:43.013774+0000","last_undegraded":"2024-09-16T10:25:43.013774+0000","last_fullsized":"2024-09-16T10:25:43.013774+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.591323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.194521+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":663,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986640+0000","last_change":"2024-09-16T10:24:33.905860+0000","last_active":"2024-09-16T10:25:42.986640+0000","last_peered":"2024-09-16T10:25:42.986640+0000","last_clean":"2024-09-16T10:25:42.986640+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:25:42.986640+0000","last_undegraded":"2024-09-16T10:25:42.986640+0000","last_fullsized":"2024-09-16T10:25:42.986640+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.905785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.905785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:01.164716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"372'1900","reported_seq":1931,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.326012+0000","last_change":"2024-09-16T10:25:30.621715+0000","last_active":"2024-09-16T10:25:48.326012+0000","last_peered":"2024-09-16T10:25:48.326012+0000","last_clean":"2024-09-16T10:25:48.326012+0000","last_became_active":"2024-09-16T10:25:30.621428+0000","last_became_peered":"2024-09-16T10:25:30.621428+0000","last_unstale":"2024-09-16T10:25:48.326012+0000","last_undegraded":"2024-09-16T10:25:48.326012+0000","last_fullsized":"2024-09-16T10:25:48.326012+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1900,"log_dups_size":0,"ondisk_log_size":1900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:38.872942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124518400,"num_objects":1900,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":121600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":560,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.012745+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:25:43.012745+0000","last_peered":"2024-09-16T10:25:43.012745+0000","last_clean":"2024-09-16T10:25:43.012745+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:25:43.012745+0000","last_undegraded":"2024-09-16T10:25:43.012745+0000","last_fullsized":"2024-09-16T10:25:43.012745+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":720,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982456+0000","last_change":"2024-09-16T10:24:34.142042+0000","last_active":"2024-09-16T10:25:42.982456+0000","last_peered":"2024-09-16T10:25:42.982456+0000","last_clean":"2024-09-16T10:25:42.982456+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:25:42.982456+0000","last_undegraded":"2024-09-16T10:25:42.982456+0000","last_fullsized":"2024-09-16T10:25:42.982456+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:24:34.141958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.141958+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:57.626228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":680,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980216+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:25:42.980216+0000","last_peered":"2024-09-16T10:25:42.980216+0000","last_clean":"2024-09-16T10:25:42.980216+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:25:42.980216+0000","last_undegraded":"2024-09-16T10:25:42.980216+0000","last_fullsized":"2024-09-16T10:25:42.980216+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.235293+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986231+0000","last_change":"2024-09-16T10:25:05.858177+0000","last_active":"2024-09-16T10:25:42.986231+0000","last_peered":"2024-09-16T10:25:42.986231+0000","last_clean":"2024-09-16T10:25:42.986231+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:25:42.986231+0000","last_undegraded":"2024-09-16T10:25:42.986231+0000","last_fullsized":"2024-09-16T10:25:42.986231+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.858129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.858129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:12.657363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":610,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029480+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:25:43.029480+0000","last_peered":"2024-09-16T10:25:43.029480+0000","last_clean":"2024-09-16T10:25:43.029480+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:25:43.029480+0000","last_undegraded":"2024-09-16T10:25:43.029480+0000","last_fullsized":"2024-09-16T10:25:43.029480+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:56.465984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"372'1966","reported_seq":1997,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.827227+0000","last_change":"2024-09-16T10:25:31.161519+0000","last_active":"2024-09-16T10:25:48.827227+0000","last_peered":"2024-09-16T10:25:48.827227+0000","last_clean":"2024-09-16T10:25:48.827227+0000","last_became_active":"2024-09-16T10:25:31.161181+0000","last_became_peered":"2024-09-16T10:25:31.161181+0000","last_unstale":"2024-09-16T10:25:48.827227+0000","last_undegraded":"2024-09-16T10:25:48.827227+0000","last_fullsized":"2024-09-16T10:25:48.827227+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1966,"log_dups_size":0,"ondisk_log_size":1966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:42.844942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128843776,"num_objects":1966,"num_object_clones":0,"num_object_copies":5898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":125824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":527,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.984771+0000","last_change":"2024-09-16T10:25:04.820847+0000","last_active":"2024-09-16T10:25:42.984771+0000","last_peered":"2024-09-16T10:25:42.984771+0000","last_clean":"2024-09-16T10:25:42.984771+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:25:42.984771+0000","last_undegraded":"2024-09-16T10:25:42.984771+0000","last_fullsized":"2024-09-16T10:25:42.984771+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:04.820775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.820775+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.871951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":769,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980639+0000","last_change":"2024-09-16T10:23:09.748114+0000","last_active":"2024-09-16T10:25:42.980639+0000","last_peered":"2024-09-16T10:25:42.980639+0000","last_clean":"2024-09-16T10:25:42.980639+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:25:42.980639+0000","last_undegraded":"2024-09-16T10:25:42.980639+0000","last_fullsized":"2024-09-16T10:25:42.980639+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:23:09.748031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:09.748031+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.221819+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989590+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:25:42.989590+0000","last_peered":"2024-09-16T10:25:42.989590+0000","last_clean":"2024-09-16T10:25:42.989590+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:25:42.989590+0000","last_undegraded":"2024-09-16T10:25:42.989590+0000","last_fullsized":"2024-09-16T10:25:42.989590+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.468841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":659,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013435+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:25:43.013435+0000","last_peered":"2024-09-16T10:25:43.013435+0000","last_clean":"2024-09-16T10:25:43.013435+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:25:43.013435+0000","last_undegraded":"2024-09-16T10:25:43.013435+0000","last_fullsized":"2024-09-16T10:25:43.013435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.814189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":209,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.014077+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:25:43.014077+0000","last_peered":"2024-09-16T10:25:43.014077+0000","last_clean":"2024-09-16T10:25:43.014077+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:25:43.014077+0000","last_undegraded":"2024-09-16T10:25:43.014077+0000","last_fullsized":"2024-09-16T10:25:43.014077+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.866924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"372'1981","reported_seq":2012,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:49.011576+0000","last_change":"2024-09-16T10:25:30.613792+0000","last_active":"2024-09-16T10:25:49.011576+0000","last_peered":"2024-09-16T10:25:49.011576+0000","last_clean":"2024-09-16T10:25:49.011576+0000","last_became_active":"2024-09-16T10:25:30.613465+0000","last_became_peered":"2024-09-16T10:25:30.613465+0000","last_unstale":"2024-09-16T10:25:49.011576+0000","last_undegraded":"2024-09-16T10:25:49.011576+0000","last_fullsized":"2024-09-16T10:25:49.011576+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1981,"log_dups_size":0,"ondisk_log_size":1981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.984379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129826816,"num_objects":1981,"num_object_clones":0,"num_object_copies":5943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":126784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":329,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028178+0000","last_change":"2024-09-16T10:25:01.481822+0000","last_active":"2024-09-16T10:25:43.028178+0000","last_peered":"2024-09-16T10:25:43.028178+0000","last_clean":"2024-09-16T10:25:43.028178+0000","last_became_active":"2024-09-16T10:25:00.955829+0000","last_became_peered":"2024-09-16T10:25:00.955829+0000","last_unstale":"2024-09-16T10:25:43.028178+0000","last_undegraded":"2024-09-16T10:25:43.028178+0000","last_fullsized":"2024-09-16T10:25:43.028178+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.481748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.481748+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:13.239898+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":684,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982372+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:25:42.982372+0000","last_peered":"2024-09-16T10:25:42.982372+0000","last_clean":"2024-09-16T10:25:42.982372+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:25:42.982372+0000","last_undegraded":"2024-09-16T10:25:42.982372+0000","last_fullsized":"2024-09-16T10:25:42.982372+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:49.328998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":657,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029171+0000","last_change":"2024-09-16T10:24:25.364236+0000","last_active":"2024-09-16T10:25:43.029171+0000","last_peered":"2024-09-16T10:25:43.029171+0000","last_clean":"2024-09-16T10:25:43.029171+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:25:43.029171+0000","last_undegraded":"2024-09-16T10:25:43.029171+0000","last_fullsized":"2024-09-16T10:25:43.029171+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:25.364176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:25.364176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.649906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":251,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.030140+0000","last_change":"2024-09-16T10:22:24.799901+0000","last_active":"2024-09-16T10:25:43.030140+0000","last_peered":"2024-09-16T10:25:43.030140+0000","last_clean":"2024-09-16T10:25:43.030140+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:25:43.030140+0000","last_undegraded":"2024-09-16T10:25:43.030140+0000","last_fullsized":"2024-09-16T10:25:43.030140+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.520234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":684,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982184+0000","last_change":"2024-09-16T10:25:09.348134+0000","last_active":"2024-09-16T10:25:42.982184+0000","last_peered":"2024-09-16T10:25:42.982184+0000","last_clean":"2024-09-16T10:25:42.982184+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:25:42.982184+0000","last_undegraded":"2024-09-16T10:25:42.982184+0000","last_fullsized":"2024-09-16T10:25:42.982184+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:09.348090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.348090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.427158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"372'1871","reported_seq":1900,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.843993+0000","last_change":"2024-09-16T10:25:31.161545+0000","last_active":"2024-09-16T10:25:48.843993+0000","last_peered":"2024-09-16T10:25:48.843993+0000","last_clean":"2024-09-16T10:25:48.843993+0000","last_became_active":"2024-09-16T10:25:31.161230+0000","last_became_peered":"2024-09-16T10:25:31.161230+0000","last_unstale":"2024-09-16T10:25:48.843993+0000","last_undegraded":"2024-09-16T10:25:48.843993+0000","last_fullsized":"2024-09-16T10:25:48.843993+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1871,"log_dups_size":0,"ondisk_log_size":1871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.886610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122617856,"num_objects":1871,"num_object_clones":0,"num_object_copies":5613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1871,"num_write_kb":119744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":240,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.982270+0000","last_change":"2024-09-16T10:24:32.158409+0000","last_active":"2024-09-16T10:25:42.982270+0000","last_peered":"2024-09-16T10:25:42.982270+0000","last_clean":"2024-09-16T10:25:42.982270+0000","last_became_active":"2024-09-16T10:23:31.480326+0000","last_became_peered":"2024-09-16T10:23:31.480326+0000","last_unstale":"2024-09-16T10:25:42.982270+0000","last_undegraded":"2024-09-16T10:25:42.982270+0000","last_fullsized":"2024-09-16T10:25:42.982270+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.158356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.158356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:45.670555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":687,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031599+0000","last_change":"2024-09-16T10:25:16.446472+0000","last_active":"2024-09-16T10:25:43.031599+0000","last_peered":"2024-09-16T10:25:43.031599+0000","last_clean":"2024-09-16T10:25:43.031599+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:25:43.031599+0000","last_undegraded":"2024-09-16T10:25:43.031599+0000","last_fullsized":"2024-09-16T10:25:43.031599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:25:16.446381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.446381+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:31.794942+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":683,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.986428+0000","last_change":"2024-09-16T10:25:08.916850+0000","last_active":"2024-09-16T10:25:42.986428+0000","last_peered":"2024-09-16T10:25:42.986428+0000","last_clean":"2024-09-16T10:25:42.986428+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:25:42.986428+0000","last_undegraded":"2024-09-16T10:25:42.986428+0000","last_fullsized":"2024-09-16T10:25:42.986428+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.916778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.916778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.226842+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":201,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013518+0000","last_change":"2024-09-16T10:25:09.346506+0000","last_active":"2024-09-16T10:25:43.013518+0000","last_peered":"2024-09-16T10:25:43.013518+0000","last_clean":"2024-09-16T10:25:43.013518+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:25:43.013518+0000","last_undegraded":"2024-09-16T10:25:43.013518+0000","last_fullsized":"2024-09-16T10:25:43.013518+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:09.346437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.346437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.598246+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":691,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032338+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:25:43.032338+0000","last_peered":"2024-09-16T10:25:43.032338+0000","last_clean":"2024-09-16T10:25:43.032338+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:25:43.032338+0000","last_undegraded":"2024-09-16T10:25:43.032338+0000","last_fullsized":"2024-09-16T10:25:43.032338+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.267128+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"372'2069","reported_seq":2100,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:49.308234+0000","last_change":"2024-09-16T10:25:31.162408+0000","last_active":"2024-09-16T10:25:49.308234+0000","last_peered":"2024-09-16T10:25:49.308234+0000","last_clean":"2024-09-16T10:25:49.308234+0000","last_became_active":"2024-09-16T10:25:31.162120+0000","last_became_peered":"2024-09-16T10:25:31.162120+0000","last_unstale":"2024-09-16T10:25:49.308234+0000","last_undegraded":"2024-09-16T10:25:49.308234+0000","last_fullsized":"2024-09-16T10:25:49.308234+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.047456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135593984,"num_objects":2069,"num_object_clones":0,"num_object_copies":6207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":132416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":639,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989862+0000","last_change":"2024-09-16T10:25:40.594525+0000","last_active":"2024-09-16T10:25:42.989862+0000","last_peered":"2024-09-16T10:25:42.989862+0000","last_clean":"2024-09-16T10:25:42.989862+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:25:42.989862+0000","last_undegraded":"2024-09-16T10:25:42.989862+0000","last_fullsized":"2024-09-16T10:25:42.989862+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:40.594454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.594454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:45.283077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":741,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989141+0000","last_change":"2024-09-16T10:25:35.488583+0000","last_active":"2024-09-16T10:25:42.989141+0000","last_peered":"2024-09-16T10:25:42.989141+0000","last_clean":"2024-09-16T10:25:42.989141+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:25:42.989141+0000","last_undegraded":"2024-09-16T10:25:42.989141+0000","last_fullsized":"2024-09-16T10:25:42.989141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.463926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":53,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985807+0000","last_change":"2024-09-16T10:25:00.947814+0000","last_active":"2024-09-16T10:25:42.985807+0000","last_peered":"2024-09-16T10:25:42.985807+0000","last_clean":"2024-09-16T10:25:42.985807+0000","last_became_active":"2024-09-16T10:25:00.947502+0000","last_became_peered":"2024-09-16T10:25:00.947502+0000","last_unstale":"2024-09-16T10:25:42.985807+0000","last_undegraded":"2024-09-16T10:25:42.985807+0000","last_fullsized":"2024-09-16T10:25:42.985807+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.528600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.528600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:48.404499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":201,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.014044+0000","last_change":"2024-09-16T10:22:26.818522+0000","last_active":"2024-09-16T10:25:43.014044+0000","last_peered":"2024-09-16T10:25:43.014044+0000","last_clean":"2024-09-16T10:25:43.014044+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:25:43.014044+0000","last_undegraded":"2024-09-16T10:25:43.014044+0000","last_fullsized":"2024-09-16T10:25:43.014044+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.818439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.818439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.481964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":538,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029486+0000","last_change":"2024-09-16T10:25:34.550982+0000","last_active":"2024-09-16T10:25:43.029486+0000","last_peered":"2024-09-16T10:25:43.029486+0000","last_clean":"2024-09-16T10:25:43.029486+0000","last_became_active":"2024-09-16T10:25:00.954013+0000","last_became_peered":"2024-09-16T10:25:00.954013+0000","last_unstale":"2024-09-16T10:25:43.029486+0000","last_undegraded":"2024-09-16T10:25:43.029486+0000","last_fullsized":"2024-09-16T10:25:43.029486+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:34.550928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:34.550928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:04.254803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"372'2001","reported_seq":2032,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.329722+0000","last_change":"2024-09-16T10:25:31.163086+0000","last_active":"2024-09-16T10:25:48.329722+0000","last_peered":"2024-09-16T10:25:48.329722+0000","last_clean":"2024-09-16T10:25:48.329722+0000","last_became_active":"2024-09-16T10:25:31.162769+0000","last_became_peered":"2024-09-16T10:25:31.162769+0000","last_unstale":"2024-09-16T10:25:48.329722+0000","last_undegraded":"2024-09-16T10:25:48.329722+0000","last_fullsized":"2024-09-16T10:25:48.329722+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":2001,"log_dups_size":0,"ondisk_log_size":2001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.363042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131137536,"num_objects":2001,"num_object_clones":0,"num_object_copies":6003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2001,"num_write_kb":128064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":582,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029176+0000","last_change":"2024-09-16T10:24:31.490967+0000","last_active":"2024-09-16T10:25:43.029176+0000","last_peered":"2024-09-16T10:25:43.029176+0000","last_clean":"2024-09-16T10:25:43.029176+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:25:43.029176+0000","last_undegraded":"2024-09-16T10:25:43.029176+0000","last_fullsized":"2024-09-16T10:25:43.029176+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.490890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.490890+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:52.214164+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":694,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980071+0000","last_change":"2024-09-16T10:23:41.811222+0000","last_active":"2024-09-16T10:25:42.980071+0000","last_peered":"2024-09-16T10:25:42.980071+0000","last_clean":"2024-09-16T10:25:42.980071+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:25:42.980071+0000","last_undegraded":"2024-09-16T10:25:42.980071+0000","last_fullsized":"2024-09-16T10:25:42.980071+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:41.811140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.811140+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.191555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031611+0000","last_change":"2024-09-16T10:25:41.990922+0000","last_active":"2024-09-16T10:25:43.031611+0000","last_peered":"2024-09-16T10:25:43.031611+0000","last_clean":"2024-09-16T10:25:43.031611+0000","last_became_active":"2024-09-16T10:25:41.990483+0000","last_became_peered":"2024-09-16T10:25:41.990483+0000","last_unstale":"2024-09-16T10:25:43.031611+0000","last_undegraded":"2024-09-16T10:25:43.031611+0000","last_fullsized":"2024-09-16T10:25:43.031611+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.405338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.405338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.926501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":201,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029625+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:25:43.029625+0000","last_peered":"2024-09-16T10:25:43.029625+0000","last_clean":"2024-09-16T10:25:43.029625+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:25:43.029625+0000","last_undegraded":"2024-09-16T10:25:43.029625+0000","last_fullsized":"2024-09-16T10:25:43.029625+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.415920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028646+0000","last_change":"2024-09-16T10:25:41.478330+0000","last_active":"2024-09-16T10:25:43.028646+0000","last_peered":"2024-09-16T10:25:43.028646+0000","last_clean":"2024-09-16T10:25:43.028646+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:25:43.028646+0000","last_undegraded":"2024-09-16T10:25:43.028646+0000","last_fullsized":"2024-09-16T10:25:43.028646+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:41.478267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:41.478267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:06.996051+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"372'1941","reported_seq":1972,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:48.518647+0000","last_change":"2024-09-16T10:25:30.621276+0000","last_active":"2024-09-16T10:25:48.518647+0000","last_peered":"2024-09-16T10:25:48.518647+0000","last_clean":"2024-09-16T10:25:48.518647+0000","last_became_active":"2024-09-16T10:25:30.620971+0000","last_became_peered":"2024-09-16T10:25:30.620971+0000","last_unstale":"2024-09-16T10:25:48.518647+0000","last_undegraded":"2024-09-16T10:25:48.518647+0000","last_fullsized":"2024-09-16T10:25:48.518647+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.992526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":5823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":479,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980502+0000","last_change":"2024-09-16T10:25:01.368064+0000","last_active":"2024-09-16T10:25:42.980502+0000","last_peered":"2024-09-16T10:25:42.980502+0000","last_clean":"2024-09-16T10:25:42.980502+0000","last_became_active":"2024-09-16T10:25:00.960443+0000","last_became_peered":"2024-09-16T10:25:00.960443+0000","last_unstale":"2024-09-16T10:25:42.980502+0000","last_undegraded":"2024-09-16T10:25:42.980502+0000","last_fullsized":"2024-09-16T10:25:42.980502+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.367979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.367979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:11.835419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":698,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.984925+0000","last_change":"2024-09-16T10:24:36.805155+0000","last_active":"2024-09-16T10:25:42.984925+0000","last_peered":"2024-09-16T10:25:42.984925+0000","last_clean":"2024-09-16T10:25:42.984925+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:25:42.984925+0000","last_undegraded":"2024-09-16T10:25:42.984925+0000","last_fullsized":"2024-09-16T10:25:42.984925+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:36.805084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:36.805084+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:01.683712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":63,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980469+0000","last_change":"2024-09-16T10:25:00.952412+0000","last_active":"2024-09-16T10:25:42.980469+0000","last_peered":"2024-09-16T10:25:42.980469+0000","last_clean":"2024-09-16T10:25:42.980469+0000","last_became_active":"2024-09-16T10:25:00.952061+0000","last_became_peered":"2024-09-16T10:25:00.952061+0000","last_unstale":"2024-09-16T10:25:42.980469+0000","last_undegraded":"2024-09-16T10:25:42.980469+0000","last_fullsized":"2024-09-16T10:25:42.980469+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:24:24.486825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:24.486825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:47.489444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"352'9","reported_seq":97,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.988490+0000","last_change":"2024-09-16T10:25:37.549535+0000","last_active":"2024-09-16T10:25:42.988490+0000","last_peered":"2024-09-16T10:25:42.988490+0000","last_clean":"2024-09-16T10:25:42.988490+0000","last_became_active":"2024-09-16T10:25:00.946371+0000","last_became_peered":"2024-09-16T10:25:00.946371+0000","last_unstale":"2024-09-16T10:25:42.988490+0000","last_undegraded":"2024-09-16T10:25:42.988490+0000","last_fullsized":"2024-09-16T10:25:42.988490+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'9","last_scrub_stamp":"2024-09-16T10:25:37.549468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.549468+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:39.799942+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":671,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985106+0000","last_change":"2024-09-16T10:24:34.863064+0000","last_active":"2024-09-16T10:25:42.985106+0000","last_peered":"2024-09-16T10:25:42.985106+0000","last_clean":"2024-09-16T10:25:42.985106+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:25:42.985106+0000","last_undegraded":"2024-09-16T10:25:42.985106+0000","last_fullsized":"2024-09-16T10:25:42.985106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:34.862992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.862992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.781279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":247,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985316+0000","last_change":"2024-09-16T10:25:00.825815+0000","last_active":"2024-09-16T10:25:42.985316+0000","last_peered":"2024-09-16T10:25:42.985316+0000","last_clean":"2024-09-16T10:25:42.985316+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:25:42.985316+0000","last_undegraded":"2024-09-16T10:25:42.985316+0000","last_fullsized":"2024-09-16T10:25:42.985316+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:00.825743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:00.825743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:11.827743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":658,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989711+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:25:42.989711+0000","last_peered":"2024-09-16T10:25:42.989711+0000","last_clean":"2024-09-16T10:25:42.989711+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:25:42.989711+0000","last_undegraded":"2024-09-16T10:25:42.989711+0000","last_fullsized":"2024-09-16T10:25:42.989711+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.558479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":679,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032613+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:25:43.032613+0000","last_peered":"2024-09-16T10:25:43.032613+0000","last_clean":"2024-09-16T10:25:43.032613+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:25:43.032613+0000","last_undegraded":"2024-09-16T10:25:43.032613+0000","last_fullsized":"2024-09-16T10:25:43.032613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:52.492565+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":659,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.987685+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:25:42.987685+0000","last_peered":"2024-09-16T10:25:42.987685+0000","last_clean":"2024-09-16T10:25:42.987685+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:25:42.987685+0000","last_undegraded":"2024-09-16T10:25:42.987685+0000","last_fullsized":"2024-09-16T10:25:42.987685+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":654,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.981498+0000","last_change":"2024-09-16T10:25:11.324922+0000","last_active":"2024-09-16T10:25:42.981498+0000","last_peered":"2024-09-16T10:25:42.981498+0000","last_clean":"2024-09-16T10:25:42.981498+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:25:42.981498+0000","last_undegraded":"2024-09-16T10:25:42.981498+0000","last_fullsized":"2024-09-16T10:25:42.981498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:33.037219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":569,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.014325+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:25:43.014325+0000","last_peered":"2024-09-16T10:25:43.014325+0000","last_clean":"2024-09-16T10:25:43.014325+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:25:43.014325+0000","last_undegraded":"2024-09-16T10:25:43.014325+0000","last_fullsized":"2024-09-16T10:25:43.014325+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:43.977445+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":687,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034894+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:25:43.034894+0000","last_peered":"2024-09-16T10:25:43.034894+0000","last_clean":"2024-09-16T10:25:43.034894+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:25:43.034894+0000","last_undegraded":"2024-09-16T10:25:43.034894+0000","last_fullsized":"2024-09-16T10:25:43.034894+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.299672+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":53,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.979780+0000","last_change":"2024-09-16T10:25:00.944900+0000","last_active":"2024-09-16T10:25:42.979780+0000","last_peered":"2024-09-16T10:25:42.979780+0000","last_clean":"2024-09-16T10:25:42.979780+0000","last_became_active":"2024-09-16T10:25:00.944574+0000","last_became_peered":"2024-09-16T10:25:00.944574+0000","last_unstale":"2024-09-16T10:25:42.979780+0000","last_undegraded":"2024-09-16T10:25:42.979780+0000","last_fullsized":"2024-09-16T10:25:42.979780+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:37.608524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.608524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":657,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032414+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:25:43.032414+0000","last_peered":"2024-09-16T10:25:43.032414+0000","last_clean":"2024-09-16T10:25:43.032414+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:25:43.032414+0000","last_undegraded":"2024-09-16T10:25:43.032414+0000","last_fullsized":"2024-09-16T10:25:43.032414+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.469447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":201,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029683+0000","last_change":"2024-09-16T10:25:13.357255+0000","last_active":"2024-09-16T10:25:43.029683+0000","last_peered":"2024-09-16T10:25:43.029683+0000","last_clean":"2024-09-16T10:25:43.029683+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:25:43.029683+0000","last_undegraded":"2024-09-16T10:25:43.029683+0000","last_fullsized":"2024-09-16T10:25:43.029683+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.501033+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":238,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.984657+0000","last_change":"2024-09-16T10:25:09.961140+0000","last_active":"2024-09-16T10:25:42.984657+0000","last_peered":"2024-09-16T10:25:42.984657+0000","last_clean":"2024-09-16T10:25:42.984657+0000","last_became_active":"2024-09-16T10:23:00.902731+0000","last_became_peered":"2024-09-16T10:23:00.902731+0000","last_unstale":"2024-09-16T10:25:42.984657+0000","last_undegraded":"2024-09-16T10:25:42.984657+0000","last_fullsized":"2024-09-16T10:25:42.984657+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:09.961098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.961098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:18.332815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":680,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.985936+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:25:42.985936+0000","last_peered":"2024-09-16T10:25:42.985936+0000","last_clean":"2024-09-16T10:25:42.985936+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:25:42.985936+0000","last_undegraded":"2024-09-16T10:25:42.985936+0000","last_fullsized":"2024-09-16T10:25:42.985936+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.812083+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":674,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032288+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:25:43.032288+0000","last_peered":"2024-09-16T10:25:43.032288+0000","last_clean":"2024-09-16T10:25:43.032288+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:25:43.032288+0000","last_undegraded":"2024-09-16T10:25:43.032288+0000","last_fullsized":"2024-09-16T10:25:43.032288+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":858,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029502+0000","last_change":"2024-09-16T10:25:16.464789+0000","last_active":"2024-09-16T10:25:43.029502+0000","last_peered":"2024-09-16T10:25:43.029502+0000","last_clean":"2024-09-16T10:25:43.029502+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:25:43.029502+0000","last_undegraded":"2024-09-16T10:25:43.029502+0000","last_fullsized":"2024-09-16T10:25:43.029502+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:25.206981+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":684,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.980120+0000","last_change":"2024-09-16T10:23:38.791832+0000","last_active":"2024-09-16T10:25:42.980120+0000","last_peered":"2024-09-16T10:25:42.980120+0000","last_clean":"2024-09-16T10:25:42.980120+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:25:42.980120+0000","last_undegraded":"2024-09-16T10:25:42.980120+0000","last_fullsized":"2024-09-16T10:25:42.980120+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:38.791750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:38.791750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:23.207250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":244,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032686+0000","last_change":"2024-09-16T10:24:32.496216+0000","last_active":"2024-09-16T10:25:43.032686+0000","last_peered":"2024-09-16T10:25:43.032686+0000","last_clean":"2024-09-16T10:25:43.032686+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:25:43.032686+0000","last_undegraded":"2024-09-16T10:25:43.032686+0000","last_fullsized":"2024-09-16T10:25:43.032686+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.495461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.495461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.882583+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":707,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029296+0000","last_change":"2024-09-16T10:25:08.470676+0000","last_active":"2024-09-16T10:25:43.029296+0000","last_peered":"2024-09-16T10:25:43.029296+0000","last_clean":"2024-09-16T10:25:43.029296+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:25:43.029296+0000","last_undegraded":"2024-09-16T10:25:43.029296+0000","last_fullsized":"2024-09-16T10:25:43.029296+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:25:08.470600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.470600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:17.849780+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":680,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.981447+0000","last_change":"2024-09-16T10:24:23.469834+0000","last_active":"2024-09-16T10:25:42.981447+0000","last_peered":"2024-09-16T10:25:42.981447+0000","last_clean":"2024-09-16T10:25:42.981447+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:25:42.981447+0000","last_undegraded":"2024-09-16T10:25:42.981447+0000","last_fullsized":"2024-09-16T10:25:42.981447+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:23.469779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.469779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:46.236915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":822,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.539321+0000","last_change":"2024-09-16T10:24:26.942679+0000","last_active":"2024-09-16T10:25:43.539321+0000","last_peered":"2024-09-16T10:25:43.539321+0000","last_clean":"2024-09-16T10:25:43.539321+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:25:43.539321+0000","last_undegraded":"2024-09-16T10:25:43.539321+0000","last_fullsized":"2024-09-16T10:25:43.539321+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:26.942604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.942604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:50.074891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":654,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034413+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:25:43.034413+0000","last_peered":"2024-09-16T10:25:43.034413+0000","last_clean":"2024-09-16T10:25:43.034413+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:25:43.034413+0000","last_undegraded":"2024-09-16T10:25:43.034413+0000","last_fullsized":"2024-09-16T10:25:43.034413+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:47.474897+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":712,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.979585+0000","last_change":"2024-09-16T10:25:17.300226+0000","last_active":"2024-09-16T10:25:42.979585+0000","last_peered":"2024-09-16T10:25:42.979585+0000","last_clean":"2024-09-16T10:25:42.979585+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:25:42.979585+0000","last_undegraded":"2024-09-16T10:25:42.979585+0000","last_fullsized":"2024-09-16T10:25:42.979585+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:17.300150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.300150+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:42.406783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":125,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.012724+0000","last_change":"2024-09-16T10:25:08.657378+0000","last_active":"2024-09-16T10:25:43.012724+0000","last_peered":"2024-09-16T10:25:43.012724+0000","last_clean":"2024-09-16T10:25:43.012724+0000","last_became_active":"2024-09-16T10:23:19.734145+0000","last_became_peered":"2024-09-16T10:23:19.734145+0000","last_unstale":"2024-09-16T10:25:43.012724+0000","last_undegraded":"2024-09-16T10:25:43.012724+0000","last_fullsized":"2024-09-16T10:25:43.012724+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.565118+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":667,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013552+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:25:43.013552+0000","last_peered":"2024-09-16T10:25:43.013552+0000","last_clean":"2024-09-16T10:25:43.013552+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:25:43.013552+0000","last_undegraded":"2024-09-16T10:25:43.013552+0000","last_fullsized":"2024-09-16T10:25:43.013552+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.630109+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":653,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028006+0000","last_change":"2024-09-16T10:24:57.507342+0000","last_active":"2024-09-16T10:25:43.028006+0000","last_peered":"2024-09-16T10:25:43.028006+0000","last_clean":"2024-09-16T10:25:43.028006+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:25:43.028006+0000","last_undegraded":"2024-09-16T10:25:43.028006+0000","last_fullsized":"2024-09-16T10:25:43.028006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:57.507281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.507281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:06.240579+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":722,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013224+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:25:43.013224+0000","last_peered":"2024-09-16T10:25:43.013224+0000","last_clean":"2024-09-16T10:25:43.013224+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:25:43.013224+0000","last_undegraded":"2024-09-16T10:25:43.013224+0000","last_fullsized":"2024-09-16T10:25:43.013224+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.824736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":52,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028994+0000","last_change":"2024-09-16T10:25:00.955337+0000","last_active":"2024-09-16T10:25:43.028994+0000","last_peered":"2024-09-16T10:25:43.028994+0000","last_clean":"2024-09-16T10:25:43.028994+0000","last_became_active":"2024-09-16T10:25:00.954747+0000","last_became_peered":"2024-09-16T10:25:00.954747+0000","last_unstale":"2024-09-16T10:25:43.028994+0000","last_undegraded":"2024-09-16T10:25:43.028994+0000","last_fullsized":"2024-09-16T10:25:43.028994+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:25.105215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.105215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.465171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":630,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029086+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:25:43.029086+0000","last_peered":"2024-09-16T10:25:43.029086+0000","last_clean":"2024-09-16T10:25:43.029086+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:25:43.029086+0000","last_undegraded":"2024-09-16T10:25:43.029086+0000","last_fullsized":"2024-09-16T10:25:43.029086+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.437165+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.032430+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:25:43.032430+0000","last_peered":"2024-09-16T10:25:43.032430+0000","last_clean":"2024-09-16T10:25:43.032430+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:25:43.032430+0000","last_undegraded":"2024-09-16T10:25:43.032430+0000","last_fullsized":"2024-09-16T10:25:43.032430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.416346+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":574,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013625+0000","last_change":"2024-09-16T10:24:32.986890+0000","last_active":"2024-09-16T10:25:43.013625+0000","last_peered":"2024-09-16T10:25:43.013625+0000","last_clean":"2024-09-16T10:25:43.013625+0000","last_became_active":"2024-09-16T10:23:31.473535+0000","last_became_peered":"2024-09-16T10:23:31.473535+0000","last_unstale":"2024-09-16T10:25:43.013625+0000","last_undegraded":"2024-09-16T10:25:43.013625+0000","last_fullsized":"2024-09-16T10:25:43.013625+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.984796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.984796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:51.261830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.028268+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:25:43.028268+0000","last_peered":"2024-09-16T10:25:43.028268+0000","last_clean":"2024-09-16T10:25:43.028268+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:25:43.028268+0000","last_undegraded":"2024-09-16T10:25:43.028268+0000","last_fullsized":"2024-09-16T10:25:43.028268+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:55.420414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":675,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.987829+0000","last_change":"2024-09-16T10:23:13.803633+0000","last_active":"2024-09-16T10:25:42.987829+0000","last_peered":"2024-09-16T10:25:42.987829+0000","last_clean":"2024-09-16T10:25:42.987829+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:25:42.987829+0000","last_undegraded":"2024-09-16T10:25:42.987829+0000","last_fullsized":"2024-09-16T10:25:42.987829+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:13.803568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.803568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:49.863054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.981462+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:25:42.981462+0000","last_peered":"2024-09-16T10:25:42.981462+0000","last_clean":"2024-09-16T10:25:42.981462+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:25:42.981462+0000","last_undegraded":"2024-09-16T10:25:42.981462+0000","last_fullsized":"2024-09-16T10:25:42.981462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:25:53.389988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.029015+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:25:43.029015+0000","last_peered":"2024-09-16T10:25:43.029015+0000","last_clean":"2024-09-16T10:25:43.029015+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:25:43.029015+0000","last_undegraded":"2024-09-16T10:25:43.029015+0000","last_fullsized":"2024-09-16T10:25:43.029015+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.383093+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":715,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031622+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:25:43.031622+0000","last_peered":"2024-09-16T10:25:43.031622+0000","last_clean":"2024-09-16T10:25:43.031622+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:25:43.031622+0000","last_undegraded":"2024-09-16T10:25:43.031622+0000","last_fullsized":"2024-09-16T10:25:43.031622+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:05.043608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":664,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034302+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:25:43.034302+0000","last_peered":"2024-09-16T10:25:43.034302+0000","last_clean":"2024-09-16T10:25:43.034302+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:25:43.034302+0000","last_undegraded":"2024-09-16T10:25:43.034302+0000","last_fullsized":"2024-09-16T10:25:43.034302+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:23.300677+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":666,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.988544+0000","last_change":"2024-09-16T10:25:12.539167+0000","last_active":"2024-09-16T10:25:42.988544+0000","last_peered":"2024-09-16T10:25:42.988544+0000","last_clean":"2024-09-16T10:25:42.988544+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:25:42.988544+0000","last_undegraded":"2024-09-16T10:25:42.988544+0000","last_fullsized":"2024-09-16T10:25:42.988544+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:12.539106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.539106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:36.533748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":285,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.013356+0000","last_change":"2024-09-16T10:25:00.946850+0000","last_active":"2024-09-16T10:25:43.013356+0000","last_peered":"2024-09-16T10:25:43.013356+0000","last_clean":"2024-09-16T10:25:43.013356+0000","last_became_active":"2024-09-16T10:25:00.946511+0000","last_became_peered":"2024-09-16T10:25:00.946511+0000","last_unstale":"2024-09-16T10:25:43.013356+0000","last_undegraded":"2024-09-16T10:25:43.013356+0000","last_fullsized":"2024-09-16T10:25:43.013356+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:26.432557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.432557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":654,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.012879+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:25:43.012879+0000","last_peered":"2024-09-16T10:25:43.012879+0000","last_clean":"2024-09-16T10:25:43.012879+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:25:43.012879+0000","last_undegraded":"2024-09-16T10:25:43.012879+0000","last_fullsized":"2024-09-16T10:25:43.012879+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.811471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.031859+0000","last_change":"2024-09-16T10:25:02.446887+0000","last_active":"2024-09-16T10:25:43.031859+0000","last_peered":"2024-09-16T10:25:43.031859+0000","last_clean":"2024-09-16T10:25:43.031859+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:25:43.031859+0000","last_undegraded":"2024-09-16T10:25:43.031859+0000","last_fullsized":"2024-09-16T10:25:43.031859+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:02.446742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.446742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:02.607831+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":709,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:42.989881+0000","last_change":"2024-09-16T10:25:18.564978+0000","last_active":"2024-09-16T10:25:42.989881+0000","last_peered":"2024-09-16T10:25:42.989881+0000","last_clean":"2024-09-16T10:25:42.989881+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:25:42.989881+0000","last_undegraded":"2024-09-16T10:25:42.989881+0000","last_fullsized":"2024-09-16T10:25:42.989881+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:18.564894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.564894+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.312714+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-16T10:25:43.034630+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:25:43.034630+0000","last_peered":"2024-09-16T10:25:43.034630+0000","last_clean":"2024-09-16T10:25:43.034630+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:25:43.034630+0000","last_undegraded":"2024-09-16T10:25:43.034630+0000","last_fullsized":"2024-09-16T10:25:43.034630+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.261724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2024669184,"num_objects":30894,"num_object_clones":0,"num_object_copies":92682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30894,"num_write_kb":1977216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6133645312,"data_stored":6133645312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30894,"ondisk_log_size":30894,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":49,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234134,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":915100,"kb_used_data":482700,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92830052,"statfs":{"total":95995035648,"available":95057973248,"internally_reserved":0,"allocated":494284800,"data_stored":493715536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":442759626},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528474,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1207892,"kb_used_data":811588,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92537260,"statfs":{"total":95995035648,"available":94758154240,"internally_reserved":0,"allocated":831066112,"data_stored":830504382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":405797318},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790112,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215780,"kb_used_data":761172,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92529372,"statfs":{"total":95995035648,"available":94750076928,"internally_reserved":0,"allocated":779440128,"data_stored":778864020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986341,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1333620,"kb_used_data":977572,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92411532,"statfs":{"total":95995035648,"available":94629408768,"internally_reserved":0,"allocated":1001033728,"data_stored":1000440271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215274,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1178100,"kb_used_data":615204,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92567052,"statfs":{"total":95995035648,"available":94788661248,"internally_reserved":0,"allocated":629968896,"data_stored":629376822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":576387528},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411504,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015964,"kb_used_data":723724,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92729188,"statfs":{"total":95995035648,"available":94954688512,"internally_reserved":0,"allocated":741093376,"data_stored":740524269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":873664,"kb_used_data":359344,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92871488,"statfs":{"total":95995035648,"available":95100403712,"internally_reserved":0,"allocated":367968256,"data_stored":367363031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1655844,"kb_used_data":1223444,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92089308,"statfs":{"total":95995035648,"available":94299451392,"internally_reserved":0,"allocated":1252806656,"data_stored":1252229902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1323499520,"data_stored":1323499520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":387842048,"data_stored":387842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":758054912,"data_stored":758054912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":651034624,"data_stored":651034624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1000079360,"data_stored":1000079360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":640811008,"data_stored":640811008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":875954176,"data_stored":875954176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":496369664,"data_stored":496369664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:54 smithi165 ceph-mon[32696]: from='client.16185 v1:172.21.15.63:0/4232481921' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:25:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:54 smithi165 ceph-mon[32696]: pgmap v655: 168 pgs: 168 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 991 B/s rd, 159 MiB/s wr, 2.55k op/s 2024-09-16T10:25:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:54 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:25:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:54 smithi165 ceph-mon[32696]: from='client.26104 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:25:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:54 smithi165 ceph-mon[32696]: osdmap e373: 8 total, 8 up, 8 in 2024-09-16T10:25:55.070 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:55.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:55.214 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:55.253 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:55.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:55.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:55.395 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:55.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:55.497 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:55.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:55.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:55.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:55.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:55.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 19 16 43876 43860 144.26 115.688 0.0066399 0.00692706 2024-09-16T10:25:55.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:55.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:55.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:55.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:55.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:55.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:55.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:55 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/192295785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:55.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:55 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/192295785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:56.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:56.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:55 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/192295785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:25:56.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:56.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:56.181 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:56.189 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:25:56.190 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:25:56.190 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:25:56.190 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:25:56.190 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:25:56.190 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:25:56.190 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:25:56.190 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:25:56.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:25:56.218 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:56.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:56.320 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:56.359 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:56.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:56.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:56.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:56.597 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:56.636 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:56.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:56.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:25:56.690824+0000 min lat: 0.00115798 max lat: 0.105762 avg lat: 0.00699498 2024-09-16T10:25:56.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:25:56.692 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 20 16 45739 45723 142.869 116.438 0.00694885 0.00699498 2024-09-16T10:25:56.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:56.774 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:56.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:56.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:25:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:56.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:25:57.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:57.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:57.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:57.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:56 smithi165 ceph-mon[32696]: pgmap v657: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1022 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-16T10:25:57.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:56 smithi165 ceph-mon[32696]: osdmap e374: 8 total, 8 up, 8 in 2024-09-16T10:25:57.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:56 smithi063 ceph-mon[29993]: pgmap v657: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1022 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-16T10:25:57.075 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:56 smithi063 ceph-mon[29993]: osdmap e374: 8 total, 8 up, 8 in 2024-09-16T10:25:57.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:56 smithi063 ceph-mon[39470]: pgmap v657: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1022 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-16T10:25:57.075 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:56 smithi063 ceph-mon[39470]: osdmap e374: 8 total, 8 up, 8 in 2024-09-16T10:25:57.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:57.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:57.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:57.191 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:25:57.191 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51238 8.e 8:7236be56:::benchmark_data_smithi063_101290_object51237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 401.92961074999999, 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96343729, 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.192 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51238 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341635+0000", 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341637+0000", 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.621e-06 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.194 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341663+0000", 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6026000000000001e-05 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341674+0000", 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0871e-05 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341682+0000", 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8390000000000007e-06 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341780+0000", 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8233999999999995e-05 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.341791+0000", 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1426999999999999e-05 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296465+0000", 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95467394000000005 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296604+0000", 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013824400000000001 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.296727+0000", 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012366699999999999 2024-09-16T10:25:57.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304672+0000", 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079445339999999996 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304896+0000", 2024-09-16T10:25:57.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000224117 2024-09-16T10:25:57.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.304941+0000", 2024-09-16T10:25:57.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5374999999999998e-05 2024-09-16T10:25:57.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305028+0000", 2024-09-16T10:25:57.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6447999999999996e-05 2024-09-16T10:25:57.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305073+0000", 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4947000000000002e-05 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51239 8.e 8:7166e593:::benchmark_data_smithi063_101290_object51238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:25:57.201 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 401.92919257300002, 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96520916099999998, 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51239 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342054+0000", 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342055+0000", 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.051e-06 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.204 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342073+0000", 2024-09-16T10:25:57.204 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8325000000000002e-05 2024-09-16T10:25:57.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342083+0000", 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9850000000000001e-06 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342090+0000", 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2119999999999997e-06 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342196+0000", 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010593400000000001 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.342244+0000", 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8272000000000001e-05 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299084+0000", 2024-09-16T10:25:57.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95683958300000005 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299216+0000", 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131595 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.299311+0000", 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5445000000000005e-05 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305117+0000", 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058058169999999996 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305157+0000", 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9836999999999998e-05 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307167+0000", 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020101390000000002 2024-09-16T10:25:57.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307221+0000", 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4317000000000002e-05 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307263+0000", 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1649000000000001e-05 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.209 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51244 8.e 8:7a51d24a:::benchmark_data_smithi063_101290_object51243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 401.92597144199999, 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96214958799999994, 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:25:57.210 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51244 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345275+0000", 2024-09-16T10:25:57.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345277+0000", 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9810000000000002e-06 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345305+0000", 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7858999999999999e-05 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345320+0000", 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5268e-05 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345328+0000", 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2900000000000002e-06 2024-09-16T10:25:57.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345404+0000", 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6241000000000004e-05 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.345415+0000", 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0672e-05 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.300950+0000", 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95553457900000005 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301082+0000", 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013229299999999999 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:25:57.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.301242+0000", 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015961099999999999 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307271+0000", 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060297889999999998 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307310+0000", 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8288999999999999e-05 2024-09-16T10:25:57.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307349+0000", 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9499999999999998e-05 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307391+0000", 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1779000000000001e-05 2024-09-16T10:25:57.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.307424+0000", 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3436999999999999e-05 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51260 8.e 8:7df47e01:::benchmark_data_smithi063_101290_object51259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 401.90901052499999, 2024-09-16T10:25:57.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95708149499999995, 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51260 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:25:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362236+0000", 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362238+0000", 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4140000000000001e-06 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362265+0000", 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7183999999999999e-05 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362279+0000", 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3832000000000001e-05 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.221 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362287+0000", 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6990000000000003e-06 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362347+0000", 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0560000000000003e-05 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.362356+0000", 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1990000000000006e-06 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305199+0000", 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94284296700000003 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305311+0000", 2024-09-16T10:25:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011281900000000001 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.305449+0000", 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000137708 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.309902+0000", 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0044530100000000003 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.312979+0000", 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030772859999999998 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317533+0000", 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045533479999999996 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319259+0000", 2024-09-16T10:25:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017263129999999999 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.319317+0000", 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8155999999999997e-05 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9:dc100d6c:::benchmark_data_smithi063_124717_object60003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38964685600001, 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1260944720000001, 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.227 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881601+0000", 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.841e-06 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881621+0000", 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9976e-05 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881629+0000", 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2120000000000003e-06 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881636+0000", 2024-09-16T10:25:57.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.472e-06 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881700+0000", 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4152000000000005e-05 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881707+0000", 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882746+0000", 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001038574 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882755+0000", 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0480000000000004e-06 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007615+0000", 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.124860408 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007694+0000", 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8479000000000005e-05 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9:d2f9f966:::benchmark_data_smithi063_124717_object60060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38959973199999, 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1652373410000001, 2024-09-16T10:25:57.232 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881647+0000", 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7599999999999997e-07 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881671+0000", 2024-09-16T10:25:57.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4284e-05 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881677+0000", 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3619999999999997e-06 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881681+0000", 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1980000000000003e-06 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.235 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881840+0000", 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015907000000000001 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting_for_map not empty", 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881846+0000", 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6250000000000004e-06 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883410+0000", 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001564085 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883417+0000", 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4749999999999996e-06 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008717+0000", 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.125299746 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008819+0000", 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102545 2024-09-16T10:25:57.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008934+0000", 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000114923 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044911+0000", 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035976248000000002 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044965+0000", 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4756000000000002e-05 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045041+0000", 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5828000000000007e-05 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045130+0000", 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9221000000000005e-05 2024-09-16T10:25:57.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045181+0000", 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0691e-05 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046884+0000", 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001702708 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9:de195d7b:::benchmark_data_smithi063_124717_object60066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38955066299999, 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1671491920000001, 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:25:57.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5400000000000001e-07 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881718+0000", 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1948999999999999e-05 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881724+0000", 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7080000000000002e-06 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881728+0000", 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.946e-06 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883442+0000", 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00171441 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883449+0000", 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4309999999999999e-06 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010618+0000", 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1271692849999999 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010714+0000", 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6600000000000003e-05 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010840+0000", 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012543600000000001 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044920+0000", 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034079854 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045083+0000", 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163688 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047061+0000", 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019780269999999998 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047155+0000", 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3529999999999994e-05 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047228+0000", 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2822000000000002e-05 2024-09-16T10:25:57.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048845+0000", 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016169520000000001 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9:d28fbaa4:::benchmark_data_smithi063_124717_object60144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.389493969, 2024-09-16T10:25:57.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.171973098, 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881753+0000", 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8699999999999995e-07 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881772+0000", 2024-09-16T10:25:57.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9159000000000001e-05 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881779+0000", 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.362e-06 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881784+0000", 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2580000000000003e-06 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883478+0000", 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016937180000000001 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883485+0000", 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3000000000000004e-06 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012549+0000", 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.129063785 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.253 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012705+0000", 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015551399999999999 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012901+0000", 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019668299999999999 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046970+0000", 2024-09-16T10:25:57.254 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034068909000000001 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047098+0000", 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012766899999999999 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048912+0000", 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001813797 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051278+0000", 2024-09-16T10:25:57.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023664480000000002 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051344+0000", 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5415999999999998e-05 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053725+0000", 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002381593 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.256 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9:dfbc90fc:::benchmark_data_smithi063_124717_object60211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38945094300001, 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1744191719999999, 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.257 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881796+0000", 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3499999999999996e-07 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881881+0000", 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5458999999999996e-05 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881889+0000", 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9060000000000002e-06 2024-09-16T10:25:57.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881896+0000", 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7830000000000001e-06 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883507+0000", 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016115249999999999 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883514+0000", 2024-09-16T10:25:57.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-06 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.014943+0000", 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1314289719999999 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015122+0000", 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000178982 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015285+0000", 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016376499999999999 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046978+0000", 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031692310000000001 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051202+0000", 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042246860000000001 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053812+0000", 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026092720000000002 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053866+0000", 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4876000000000001e-05 2024-09-16T10:25:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053932+0000", 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5314000000000005e-05 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056214+0000", 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002282634 2024-09-16T10:25:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9:dcccd893:::benchmark_data_smithi063_124717_object60240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38934169999999, 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2232440419999999, 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881905+0000", 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2900000000000003e-07 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881928+0000", 2024-09-16T10:25:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2747000000000001e-05 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881935+0000", 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3150000000000003e-06 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881941+0000", 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1639999999999998e-06 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883538+0000", 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015963939999999999 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883543+0000", 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6540000000000004e-06 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056263+0000", 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.172719834 2024-09-16T10:25:57.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056408+0000", 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014513800000000001 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056584+0000", 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017557299999999999 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064153+0000", 2024-09-16T10:25:57.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0075692820000000001 2024-09-16T10:25:57.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068713+0000", 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045601620000000004 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068767+0000", 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3881000000000001e-05 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101726+0000", 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032958901999999998 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101782+0000", 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5609999999999998e-05 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105149+0000", 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033666569999999999 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9:dbba527e:::benchmark_data_smithi063_124717_object60254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38929738499999, 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2264516890000001, 2024-09-16T10:25:57.274 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.275 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1399999999999997e-07 2024-09-16T10:25:57.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881971+0000", 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1461999999999999e-05 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881979+0000", 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.259e-06 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881985+0000", 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7320000000000003e-06 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883562+0000", 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001577171 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883568+0000", 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7910000000000001e-06 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059019+0000", 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1754510929999999 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059188+0000", 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016928700000000001 2024-09-16T10:25:57.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059360+0000", 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017208100000000001 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068663+0000", 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0093023089999999999 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068831+0000", 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016804999999999999 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082474+0000", 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013643597 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.281 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105299+0000", 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.022824816000000001 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105363+0000", 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4088e-05 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108400+0000", 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003037339 2024-09-16T10:25:57.282 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9:ddb5c386:::benchmark_data_smithi063_124717_object60260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.389254188, 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2293183249999999, 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.283 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:25:57.284 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881993+0000", 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9999999999999997e-07 2024-09-16T10:25:57.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882015+0000", 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1946e-05 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882021+0000", 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9430000000000003e-06 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.286 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882028+0000", 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7599999999999997e-06 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883589+0000", 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001560875 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883595+0000", 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0719999999999996e-06 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061597+0000", 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1780015660000001 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.288 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061781+0000", 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000183909 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.062032+0000", 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025166899999999999 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068729+0000", 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0066966940000000004 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068881+0000", 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000151965 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101683+0000", 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032802338 2024-09-16T10:25:57.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108512+0000", 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0068289229999999998 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108574+0000", 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1630000000000005e-05 2024-09-16T10:25:57.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.111310+0000", 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027364350000000002 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60273 9.b 9:d6970cdd:::benchmark_data_smithi063_124717_object60272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.292 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.3892104, 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2328018119999999, 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60273 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.293 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3499999999999996e-07 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882058+0000", 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1236000000000001e-05 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.295 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882064+0000", 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5230000000000001e-06 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882070+0000", 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4670000000000003e-06 2024-09-16T10:25:57.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883625+0000", 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015556820000000001 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883632+0000", 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6610000000000003e-06 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064190+0000", 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1805575690000001 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064363+0000", 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000173124 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.298 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064567+0000", 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020472000000000001 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068929+0000", 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043613879999999999 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076738+0000", 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0078089300000000004 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105256+0000", 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.028517793 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112019+0000", 2024-09-16T10:25:57.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067630110000000002 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112083+0000", 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4938999999999996e-05 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114838+0000", 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027541340000000001 2024-09-16T10:25:57.301 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60317 9.b 9:d4f738e0:::benchmark_data_smithi063_124717_object60316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.389169662, 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.235504929, 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.302 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60317 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:25:57.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:25:57.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5799999999999999e-07 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882098+0000", 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0758e-05 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882104+0000", 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3239999999999999e-06 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882110+0000", 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5879999999999997e-06 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883724+0000", 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016140670000000001 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.306 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883731+0000", 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068967+0000", 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1852353390000001 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069080+0000", 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113442 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069174+0000", 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3623999999999997e-05 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.308 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079562+0000", 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010388029 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082423+0000", 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028609740000000001 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114907+0000", 2024-09-16T10:25:57.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032484065999999999 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114978+0000", 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0919000000000003e-05 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.115036+0000", 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7878999999999999e-05 2024-09-16T10:25:57.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117581+0000", 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025458019999999998 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60326 9.b 9:d0ff592b:::benchmark_data_smithi063_124717_object60325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:25:57.311 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38912320399999, 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2387679700000001, 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60326 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882124+0000", 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.99e-07 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882145+0000", 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1650000000000001e-05 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882151+0000", 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2110000000000003e-06 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882158+0000", 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6669999999999999e-06 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883760+0000", 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001601781 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883766+0000", 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4019999999999999e-06 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071354+0000", 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1875881050000001 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.316 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071501+0000", 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014622300000000001 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071673+0000", 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000172518 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082372+0000", 2024-09-16T10:25:57.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010698874000000001 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082515+0000", 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000143409 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117647+0000", 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035131837999999999 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117792+0000", 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014443299999999999 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117866+0000", 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4274000000000001e-05 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120891+0000", 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030249859999999999 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60333 9.b 9:d83577cf:::benchmark_data_smithi063_124717_object60332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38908092099999, 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2414839360000001, 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.320 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60333 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882166+0000", 2024-09-16T10:25:57.322 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0100000000000005e-07 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882188+0000", 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2252999999999999e-05 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882221+0000", 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2459999999999998e-05 2024-09-16T10:25:57.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882227+0000", 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.015e-06 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883790+0000", 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015635759999999999 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883797+0000", 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.567e-06 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076778+0000", 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1929816879999999 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076959+0000", 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000180607 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.077150+0000", 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019081600000000001 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082436+0000", 2024-09-16T10:25:57.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0052863240000000002 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082555+0000", 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119305 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120947+0000", 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.038391740000000001 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120989+0000", 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2107999999999997e-05 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.121049+0000", 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9888e-05 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123649+0000", 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025999880000000001 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60362 9.b 9:d2c32f51:::benchmark_data_smithi063_124717_object60361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:25:57.329 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.3890098, 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.244097462, 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60362 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882237+0000", 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0399999999999995e-07 2024-09-16T10:25:57.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882261+0000", 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4269000000000001e-05 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882268+0000", 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8619999999999997e-06 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882274+0000", 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1829999999999999e-06 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883834+0000", 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015597930000000001 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.333 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883840+0000", 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.251e-06 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079597+0000", 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1957566589999999 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079755+0000", 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015786399999999999 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.080192+0000", 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00043717099999999998 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105209+0000", 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.025016842000000001 2024-09-16T10:25:57.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114928+0000", 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0097189919999999992 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123709+0000", 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0087810240000000001 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123762+0000", 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2567e-05 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123828+0000", 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6799999999999997e-05 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126334+0000", 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025054809999999999 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60366 9.b 9:d4ec99e6:::benchmark_data_smithi063_124717_object60365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38896430099999, 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.246752562, 2024-09-16T10:25:57.338 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60366 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.339 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7899999999999998e-07 2024-09-16T10:25:57.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883494+0000", 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012115959999999999 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883505+0000", 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.111e-05 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883513+0000", 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4320000000000002e-06 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883866+0000", 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00035366599999999998 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883873+0000", 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2910000000000004e-06 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099038+0000", 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2151655379999999 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099212+0000", 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017424000000000001 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099404+0000", 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000191366 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108466+0000", 2024-09-16T10:25:57.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0090625159999999996 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117602+0000", 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0091353030000000009 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126413+0000", 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088110150000000002 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126572+0000", 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015944599999999999 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126628+0000", 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5819999999999997e-05 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129034+0000", 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024066439999999999 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60423 9.b 9:db831613:::benchmark_data_smithi063_124717_object60422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.38769984800001, 2024-09-16T10:25:57.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2882874870000001, 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60423 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883548+0000", 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.206e-06 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883603+0000", 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5036e-05 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883614+0000", 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1352000000000001e-05 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883624+0000", 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0519999999999999e-05 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.351 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883899+0000", 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027465699999999998 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883905+0000", 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3119999999999999e-06 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129084+0000", 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.245178672 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129268+0000", 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018391999999999999 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129423+0000", 2024-09-16T10:25:57.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015478000000000001 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136532+0000", 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071096270000000003 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136582+0000", 2024-09-16T10:25:57.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9962e-05 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.152752+0000", 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016169444000000002 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168726+0000", 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015974559999999999 2024-09-16T10:25:57.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168790+0000", 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3474999999999998e-05 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.171834+0000", 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030439640000000001 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60452 9.b 9:d848878f:::benchmark_data_smithi063_124717_object60451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 199.32873137600001, 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.233084683, 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:25:57.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60452 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:25:57.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942516+0000", 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5969999999999999e-06 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942536+0000", 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9970999999999999e-05 2024-09-16T10:25:57.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942548+0000", 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2057e-05 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942556+0000", 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0350000000000001e-06 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942605+0000", 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8770999999999998e-05 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942611+0000", 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1040000000000003e-06 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131546+0000", 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.188934811 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131704+0000", 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015802399999999999 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131894+0000", 2024-09-16T10:25:57.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018940499999999999 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136598+0000", 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047045409999999996 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136646+0000", 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8204999999999997e-05 2024-09-16T10:25:57.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168552+0000", 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031905880999999997 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172759+0000", 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042071809999999999 2024-09-16T10:25:57.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172950+0000", 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019102799999999999 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.175599+0000", 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026490720000000001 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.365 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.366 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.366 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:25:57.366 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:25:57.366 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:25:57.366 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:57.394 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:57.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:57.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:25:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:25:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:25:57.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:57.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:57.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:57.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:57.670 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:57.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:57.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 21 16 47627 47611 141.684 118 0.00634145 0.00705306 2024-09-16T10:25:57.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:57.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:57.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:57.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:57.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:57.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:58.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:58.085 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:58.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:58.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:25:58.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:58.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:25:58.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:58.361 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:58.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:58.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:58.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:58.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:58.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:58.636 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:58.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:58.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:58.691 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 22 16 49479 49463 140.505 115.75 0.00648782 0.00711116 2024-09-16T10:25:58.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:25:58.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:58.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:58.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:58.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:25:58.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:58 smithi063 ceph-mon[39470]: pgmap v659: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-16T10:25:58.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:58 smithi063 ceph-mon[29993]: pgmap v659: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-16T10:25:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:58.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:58.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:25:59.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:25:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:25:59.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:58 smithi165 ceph-mon[32696]: pgmap v659: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-16T10:25:59.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:25:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:59.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:59.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:25:59.197 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:59.234 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:59.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:59.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:59.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:59.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:59.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:25:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:59.523 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:59.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:25:59.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:25:59.658 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:59.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:25:59.692 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 23 16 51105 51089 138.814 101.625 0.0144143 0.00719846 2024-09-16T10:25:59.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:25:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:25:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:59.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:25:59.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:25:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:25:59.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:25:59.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:25:59.986 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:25:59.986 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:25:59.986 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T10:25:59.986 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:26:00.025 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:25:59 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:00.025 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:25:59 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:00.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:00.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:25:59 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:00.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:00.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:00.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:00.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:00.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:00.220 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:00.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:00.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:00.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:00.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:00.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:00.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:00.590 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:00.627 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:00.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:00.692 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 24 16 52593 52577 136.905 93 0.0037409 0.00729903 2024-09-16T10:26:00.729 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:00.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:00.823 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:00.824 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:00.869 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:00.915 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:00.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:00.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:00 smithi063 ceph-mon[29993]: pgmap v660: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-09-16T10:26:00.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:00 smithi063 ceph-mon[39470]: pgmap v660: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-09-16T10:26:01.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:00 smithi165 ceph-mon[32696]: pgmap v660: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-09-16T10:26:01.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:01.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:01.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:01.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:01.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:01.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:01.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:01.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:01.427 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:01.464 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:01.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:01.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:26:01.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:01.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:01.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:01.692 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 25 16 54038 54022 135.041 90.3125 0.00270751 0.00737389 2024-09-16T10:26:01.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:01.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:01.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:01.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:01.866 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:01.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:01.968 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:02.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:02.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:02.151 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:26:02.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:02.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:01 smithi165 ceph-mon[32696]: pgmap v661: 168 pgs: 168 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-16T10:26:02.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:02.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:01 smithi063 ceph-mon[29993]: pgmap v661: 168 pgs: 168 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-16T10:26:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:01 smithi063 ceph-mon[39470]: pgmap v661: 168 pgs: 168 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-16T10:26:02.518 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:02.558 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:02.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:02.687 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:02.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:02.693 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 26 15 55433 55418 133.197 87.25 0.0085799 0.00750134 2024-09-16T10:26:02.717 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:26:02.792 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:02.836 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:02.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:02.976 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:02.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:03.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:03.118 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:03.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:03.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:03.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:03.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:03.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:03.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:03.384 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:03.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:03.393 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:03.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:03.551 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:03.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:03.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:03.665 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:26:03.667 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:26:03.667 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 374, 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:25:55.948868+0000", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:13:27.869667+0000", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:22:01.866838+0000", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5806080, 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "noscrub", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:26:03.668 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 18, 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 10, 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:26:03.669 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:26:03.670 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 21, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 21, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 21, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 21, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 21, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.b", 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 275, 2024-09-16T10:26:03.671 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 274, 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 274, 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "357", 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "275", 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "306", 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:26:03.672 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:26:03.673 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:26:03.674 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.8999999761581421, 2024-09-16T10:26:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.8999999761581421, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.8999999761581421, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.8999999761581421, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:26:03.676 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:26:03.677 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:26:03.678 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:26:03.679 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:26:03.680 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:26:03.681 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.6500000953674316, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.6500000953674316, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:26:03.682 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:26:03.683 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.684 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:26:03.685 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:26:03.686 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:26:03.686 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:26:03.686 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:26:03.686 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:26:03.686 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:26:03.686 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:26:03.687 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:26:03.688 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2, 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2, 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:26:03.689 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:26:03.690 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:26:03.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:26:03.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:26:03.693 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:26:03.694 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:26:03.695 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:26:03.696 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.5, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.5, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.5, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.5, 2024-09-16T10:26:03.697 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:26:03.698 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:26:03.699 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:26:03.700 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:26:03.701 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:26:03.702 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5, 2024-09-16T10:26:03.703 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5, 2024-09-16T10:26:03.704 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:26:03.704 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:26:03.704 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:26:03.705 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:26:03.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:26:03.707 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:26:03.708 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:26:03.709 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:26:03.710 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 10, 2024-09-16T10:26:03.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_3", 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:25:29.286237+0000", 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:26:03.712 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:26:03.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "366", 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:26:03.714 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:26:03.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:26:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 3, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 3, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 3, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 3, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.91170120239257812, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:26:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:26:03.722 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 27 15 56776 56761 131.372 83.9375 0.00916317 0.00760702 2024-09-16T10:26:03.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:26:03.723 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.724 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.726 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.729 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:26:03.731 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.732 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 370, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:26:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:26:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:26:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.741 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.742 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:26:03.744 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.745 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.747 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.748 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.749 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0.293609619140625, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 54, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 322, 2024-09-16T10:26:03.750 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:26:03.752 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2484590937 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/2484590937", 2024-09-16T10:26:03.753 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/2484590937", 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/2484590937", 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/2484590937", 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:16.926902+0000", 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:31.981645+0000", 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:33.942058+0000", 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:20.945644+0000", 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:24:46.426238+0000", 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:03.025290+0000", 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:24.228279+0000", 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:25:05.915019+0000", 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [ 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.0", 2024-09-16T10:26:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.e", 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 7, 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 3 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.762 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.10", 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 6, 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:03.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:03.764 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.14", 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 1 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.13", 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.767 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.768 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:26:03.769 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:26:03.770 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:26:03.771 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:26:03.772 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:03.773 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:26:03.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:03.894 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:03.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:03.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:04.035 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:04.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:04.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:04.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:04.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:04.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:04.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:04.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:04.451 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:04.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:04.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:04.587 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:04.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:04.689 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:04.693 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 28 16 58414 58398 130.334 102.312 0.00506857 0.00766804 2024-09-16T10:26:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:04 smithi063 ceph-mon[29993]: pgmap v662: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-16T10:26:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:04 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3532566732' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:26:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:04 smithi063 ceph-mon[39470]: pgmap v662: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-16T10:26:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:04 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3532566732' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:26:04.727 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:04.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:04 smithi165 ceph-mon[32696]: pgmap v662: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-16T10:26:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:04 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3532566732' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:26:04.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:04.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:04.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:04.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:05.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:05.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:05.112 INFO:tasks.thrashosds.thrasher:j is {'epoch': 374, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:25:55.948868+0000', 'last_up_change': '2024-09-16T10:13:27.869667+0000', 'last_in_change': '2024-09-16T10:22:01.866838+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 275, 'last_epoch_started': 274, 'last_epoch_clean': 274, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '357', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '275', 'last_force_op_resend_preluminous': '306', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 1, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-16T10:25:29.286237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '366', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9117012023925781, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.293609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 2484590937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 2484590937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 2484590937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 2484590937}]}, 'public_addr': '172.21.15.165:6812/2484590937', 'cluster_addr': '172.21.15.165:6813/2484590937', 'heartbeat_back_addr': '172.21.15.165:6815/2484590937', 'heartbeat_front_addr': '172.21.15.165:6814/2484590937', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:16.926902+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:31.981645+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:33.942058+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:20.945644+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:24:46.426238+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:03.025290+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:24.228279+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:25:05.915019+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.0', 'osds': [4, 6, 2]}], 'pg_upmap_items': [{'pgid': '3.e', 'mappings': [{'from': 7, 'to': 3}, {'from': 4, 'to': 5}, {'from': 1, 'to': 0}]}, {'pgid': '3.10', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:26:05.112 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:26:05.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:05.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:05.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:05.375 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:05.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:05.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:05.438 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:05.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:05.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:05.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:05.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:05.693 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 29 16 59872 59856 128.982 91.125 0.00552286 0.00774836 2024-09-16T10:26:05.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:05.822 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:05.924 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:05.961 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:05.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:06.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:06.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:06.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:06.187 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:06.225 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:06.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:06.309 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:06.309 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:06.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:06.338 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:06.338 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:06.353 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:06.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:06.393 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:26:06.422 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:06.422 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:06.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:06.456 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:06.456 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:06.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:06.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:06.489 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:06.490 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:06.525 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:06.525 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:06.557 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:06.558 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:06.588 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:26:06.588 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:26:06.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:06.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:06.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:06.693 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 30 16 61165 61149 127.376 80.8125 0.0103444 0.00784556 2024-09-16T10:26:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[29993]: pgmap v663: 168 pgs: 168 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-16T10:26:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/788969040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[29993]: from='client.46055 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/173420380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[39470]: pgmap v663: 168 pgs: 168 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-16T10:26:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/788969040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[39470]: from='client.46055 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/173420380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:06.721 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:06.760 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:06.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:06 smithi165 ceph-mon[32696]: pgmap v663: 168 pgs: 168 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-16T10:26:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/788969040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:06 smithi165 ceph-mon[32696]: from='client.46055 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/173420380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:06.863 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:06.902 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:06.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:07.004 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:07.040 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:07.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:07.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:07.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:07.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:07.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:07.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:07.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:07.328 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:26:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:26:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:26:07.416 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:07.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:07.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:07.496 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:26:07.496 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:26:07.496 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:26:07.496 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:26:07.497 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:26:07.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:07.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:07.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:07.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:07 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:07.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:07 smithi063 ceph-mon[29993]: from='client.46055 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:26:07.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:07 smithi063 ceph-mon[29993]: osdmap e375: 8 total, 8 up, 8 in 2024-09-16T10:26:07.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:07 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:07.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:07 smithi063 ceph-mon[39470]: from='client.46055 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:26:07.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:07 smithi063 ceph-mon[39470]: osdmap e375: 8 total, 8 up, 8 in 2024-09-16T10:26:07.693 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 31 16 62662 62646 126.285 93.5625 0.00757443 0.00791345 2024-09-16T10:26:07.694 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:07.736 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:07.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:07 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:07 smithi165 ceph-mon[32696]: from='client.46055 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:26:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:07 smithi165 ceph-mon[32696]: osdmap e375: 8 total, 8 up, 8 in 2024-09-16T10:26:07.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:07.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:07.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:07.973 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:08.013 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:08.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:08.153 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:08.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:08.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:08.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:08.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:08.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:08.569 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:08.671 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:08.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 32 16 64216 64200 125.37 97.125 0.00890418 0.00797073 2024-09-16T10:26:08.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:08 smithi063 ceph-mon[29993]: pgmap v665: 168 pgs: 168 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-16T10:26:08.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:08 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:26:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:08 smithi063 ceph-mon[29993]: 1.e scrub ok 2024-09-16T10:26:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:08 smithi063 ceph-mon[39470]: pgmap v665: 168 pgs: 168 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-16T10:26:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:08 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:26:08.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:08 smithi063 ceph-mon[39470]: 1.e scrub ok 2024-09-16T10:26:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.714 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:08.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:08 smithi165 ceph-mon[32696]: pgmap v665: 168 pgs: 168 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-16T10:26:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:08 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:26:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:08 smithi165 ceph-mon[32696]: 1.e scrub ok 2024-09-16T10:26:08.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:08.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:08.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:08.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:08.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:09.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:09.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:09.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:09.224 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:09.254 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:26:09.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:09.287 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:09.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:09.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:26:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:26:09.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:09.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:09.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:09.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[29993]: 3.f scrub starts 2024-09-16T10:26:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[29993]: 3.f scrub ok 2024-09-16T10:26:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[39470]: 3.f scrub starts 2024-09-16T10:26:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[39470]: 3.f scrub ok 2024-09-16T10:26:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:09 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:09.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:09.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:09.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:09.595 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:09.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:09.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 33 16 65691 65675 124.364 92.1875 0.00656249 0.00803639 2024-09-16T10:26:09.704 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:09.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:09 smithi165 ceph-mon[32696]: 3.f scrub starts 2024-09-16T10:26:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:09 smithi165 ceph-mon[32696]: 3.f scrub ok 2024-09-16T10:26:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:09 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:09.806 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:09.844 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:09.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:09.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:09.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:09.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:10.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:10.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:10.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:10.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:10.228 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:10.268 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:10.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:10.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:10.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:10.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:10.438 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:26:10.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:10.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:10.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:26:10.589 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:26:10.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:10.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:10.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 6.18 scrub starts 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 6.18 scrub ok 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 3.d scrub starts 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 3.d scrub ok 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: pgmap v666: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-16T10:26:10.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 5.1c scrub starts 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[29993]: 5.1c scrub ok 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 6.18 scrub starts 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 6.18 scrub ok 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 3.d scrub starts 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 3.d scrub ok 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: pgmap v666: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 5.1c scrub starts 2024-09-16T10:26:10.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:10 smithi063 ceph-mon[39470]: 5.1c scrub ok 2024-09-16T10:26:10.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 34 16 66986 66970 123.087 80.9375 0.015081 0.00811972 2024-09-16T10:26:10.773 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 6.18 scrub starts 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 6.18 scrub ok 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 3.d scrub starts 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 3.d scrub ok 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: pgmap v666: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 5.1c scrub starts 2024-09-16T10:26:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:10 smithi165 ceph-mon[32696]: 5.1c scrub ok 2024-09-16T10:26:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:10.822 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:10.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50949 8.a 8:59c98f0f:::benchmark_data_smithi063_101290_object50948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.58326391000003, 2024-09-16T10:26:10.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0382288049999999, 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50949 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:26:10.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264642+0000", 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264644+0000", 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.978e-06 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264667+0000", 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3649999999999999e-05 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264681+0000", 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3423e-05 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.264688+0000", 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1010000000000003e-06 2024-09-16T10:26:10.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265626+0000", 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00093838999999999999 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.265747+0000", 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120319 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.267052+0000", 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013057990000000001 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.281048+0000", 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013995452 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.862 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283470+0000", 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024224540000000001 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.283482+0000", 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1581e-05 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302818+0000", 2024-09-16T10:26:10.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0193361350000001 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.302871+0000", 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2522999999999998e-05 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50954 8.a 8:57f268c8:::benchmark_data_smithi063_101290_object50953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:26:10.864 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.57990348300001, 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0425692, 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50954 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.865 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268002+0000", 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268004+0000", 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1050000000000002e-06 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.866 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268031+0000", 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6468000000000001e-05 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268041+0000", 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0532e-05 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.268048+0000", 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7259999999999996e-06 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.270471+0000", 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024233789999999998 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275200+0000", 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047289130000000004 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.275360+0000", 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000159531 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.870 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284217+0000", 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088568970000000007 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284226+0000", 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3559999999999999e-06 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.303128+0000", 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0189021700000001 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304057+0000", 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00092923599999999995 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310454+0000", 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063961260000000002 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310571+0000", 2024-09-16T10:26:10.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117761 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50965 8.a 8:5e5d9df4:::benchmark_data_smithi063_101290_object50964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.56944374800003, 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0323928659999999, 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.873 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50965 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278462+0000", 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278464+0000", 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.968e-06 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278491+0000", 2024-09-16T10:26:10.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7061999999999999e-05 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278500+0000", 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4970000000000007e-06 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.278507+0000", 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5139999999999998e-06 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.876 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284249+0000", 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0057418360000000002 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.284255+0000", 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4529999999999999e-06 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304143+0000", 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.019888039 2024-09-16T10:26:10.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304299+0000", 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155651 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.304463+0000", 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016370800000000001 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308275+0000", 2024-09-16T10:26:10.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0038120049999999998 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310595+0000", 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023207620000000001 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310754+0000", 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015848500000000001 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310795+0000", 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1655e-05 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310855+0000", 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9231000000000003e-05 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:50996 8.a 8:5adb42d1:::benchmark_data_smithi063_101290_object50995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.880 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.54281390699998, 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005949921, 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 50996 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.881 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305092+0000", 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305093+0000", 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.779e-06 2024-09-16T10:26:10.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305117+0000", 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3370000000000002e-05 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305127+0000", 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0284e-05 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305134+0000", 2024-09-16T10:26:10.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9659999999999999e-06 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305192+0000", 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7587000000000003e-05 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.305201+0000", 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2590000000000006e-06 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.884 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306679+0000", 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0014777290000001 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306790+0000", 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111675 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.306942+0000", 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015151300000000001 2024-09-16T10:26:10.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310399+0000", 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034570529999999999 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310904+0000", 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00050482900000000004 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.886 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310943+0000", 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9048000000000003e-05 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310974+0000", 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1164999999999997e-05 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311042+0000", 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7663999999999995e-05 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.887 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51001 8.a 8:59406931:::benchmark_data_smithi063_101290_object51000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.53869678799998, 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0021313270000001, 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.888 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51001 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309209+0000", 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309210+0000", 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2589999999999999e-06 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309227+0000", 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6765999999999998e-05 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.890 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309235+0000", 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1969999999999996e-06 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309240+0000", 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6290000000000001e-06 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309279+0000", 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9199999999999997e-05 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.891 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.309285+0000", 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8810000000000001e-06 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308323+0000", 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99903792999999996 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308464+0000", 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014168000000000001 2024-09-16T10:26:10.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.308635+0000", 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000170383 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.310405+0000", 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001770134 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311104+0000", 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069940900000000001 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311178+0000", 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3911000000000004e-05 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311237+0000", 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9113000000000001e-05 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311340+0000", 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102835 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51015 8.a 8:5f39a03a:::benchmark_data_smithi063_101290_object51014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.52239081900001, 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99112391399999999, 2024-09-16T10:26:10.895 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51015 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.896 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325515+0000", 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325516+0000", 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-07 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325537+0000", 2024-09-16T10:26:10.897 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1900999999999999e-05 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325554+0000", 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6937999999999999e-05 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325562+0000", 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3830000000000003e-06 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.898 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325628+0000", 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6418e-05 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.325640+0000", 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2157000000000001e-05 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311387+0000", 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98574625999999999 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.899 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311498+0000", 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111768 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.311624+0000", 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012560199999999999 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316394+0000", 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047700279999999999 2024-09-16T10:26:10.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316457+0000", 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2819999999999998e-05 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316471+0000", 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4236000000000001e-05 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.901 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.902 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316522+0000", 2024-09-16T10:26:10.902 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1001000000000002e-05 2024-09-16T10:26:10.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.902 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.902 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316639+0000", 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116641 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51028 8.a 8:504c9504:::benchmark_data_smithi063_101290_object51027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.903 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.51400882199999, 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98302855600000005, 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51028 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.904 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333897+0000", 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333898+0000", 2024-09-16T10:26:10.905 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.375e-06 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333916+0000", 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8236e-05 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333925+0000", 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.585e-06 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.906 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.333931+0000", 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1070000000000001e-06 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334057+0000", 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012610900000000001 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.334083+0000", 2024-09-16T10:26:10.907 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5327e-05 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.312988+0000", 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97890501699999999 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313110+0000", 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012285099999999999 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.908 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.313254+0000", 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014338900000000001 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316650+0000", 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033961529999999998 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316706+0000", 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5878000000000003e-05 2024-09-16T10:26:10.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316760+0000", 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4502000000000001e-05 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316833+0000", 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2632000000000005e-05 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.910 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316925+0000", 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2394999999999999e-05 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51060 8.a 8:500fc7dc:::benchmark_data_smithi063_101290_object51059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.49100868800002, 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96027958700000005, 2024-09-16T10:26:10.911 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51060 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.912 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356897+0000", 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.913 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356899+0000", 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.821e-06 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356917+0000", 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7972999999999999e-05 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356926+0000", 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9679999999999995e-06 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.914 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356931+0000", 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9180000000000002e-06 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.356997+0000", 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6079000000000002e-05 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.357005+0000", 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7569999999999993e-06 2024-09-16T10:26:10.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314700+0000", 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95769431000000005 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314827+0000", 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012752799999999999 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.314995+0000", 2024-09-16T10:26:10.916 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016789100000000001 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316982+0000", 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001986603 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.316999+0000", 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7319000000000001e-05 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.917 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317035+0000", 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5655000000000003e-05 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317088+0000", 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3278e-05 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317177+0000", 2024-09-16T10:26:10.918 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8486999999999995e-05 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51073 8.a 8:53b2a9fa:::benchmark_data_smithi063_101290_object51072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.47987912500002, 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.957136019, 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.919 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51073 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.920 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368026+0000", 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368028+0000", 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5939999999999999e-06 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368053+0000", 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5130000000000002e-05 2024-09-16T10:26:10.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368066+0000", 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2397e-05 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368074+0000", 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2789999999999998e-06 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.922 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368143+0000", 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9558e-05 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.368154+0000", 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0869999999999999e-05 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317228+0000", 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.949073273 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317346+0000", 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011878400000000001 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.317477+0000", 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013062600000000001 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322458+0000", 2024-09-16T10:26:10.924 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0049812980000000003 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322513+0000", 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4561999999999997e-05 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325080+0000", 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025674819999999998 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.925 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325125+0000", 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5065000000000003e-05 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325163+0000", 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7101000000000003e-05 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.926 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51087 8.a 8:51205303:::benchmark_data_smithi063_101290_object51086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.46892159200002, 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94635271099999996, 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51087 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.927 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378984+0000", 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.378986+0000", 2024-09-16T10:26:10.928 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9850000000000001e-06 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379013+0000", 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6911e-05 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379029+0000", 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5928999999999999e-05 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.929 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379037+0000", 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7100000000000007e-06 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379115+0000", 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8634999999999995e-05 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.379128+0000", 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2903e-05 2024-09-16T10:26:10.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319060+0000", 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93993237200000002 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319187+0000", 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012621 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.319358+0000", 2024-09-16T10:26:10.931 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017167699999999999 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322553+0000", 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003194625 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325067+0000", 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025139170000000001 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325170+0000", 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102948 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325265+0000", 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5570999999999997e-05 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325337+0000", 2024-09-16T10:26:10.933 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1317999999999997e-05 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51105 8.a 8:558fcaa8:::benchmark_data_smithi063_101290_object51104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.45502933699998, 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93274633699999998, 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.934 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51105 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.935 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392876+0000", 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392878+0000", 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.516e-06 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392900+0000", 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2575e-05 2024-09-16T10:26:10.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392913+0000", 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2196e-05 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.392920+0000", 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7170000000000008e-06 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393546+0000", 2024-09-16T10:26:10.937 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00062592699999999997 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.393555+0000", 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7809999999999994e-06 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320780+0000", 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92722495100000002 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.938 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.320895+0000", 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000115375 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.321051+0000", 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155482 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325345+0000", 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042937840000000001 2024-09-16T10:26:10.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325384+0000", 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9261999999999998e-05 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325522+0000", 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0001383 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.940 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325584+0000", 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-05 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325623+0000", 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8940999999999999e-05 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51170 8.a 8:55be5a3b:::benchmark_data_smithi063_101290_object51169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:26:10.941 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.39908145599998, 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87700012299999996, 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51170 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.942 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448824+0000", 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448826+0000", 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7749999999999999e-06 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.943 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448849+0000", 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3298999999999998e-05 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448867+0000", 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7467999999999999e-05 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448874+0000", 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.61e-06 2024-09-16T10:26:10.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448936+0000", 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1383999999999994e-05 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.448947+0000", 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1756000000000001e-05 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.945 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322695+0000", 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.87374773699999997 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322825+0000", 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000129579 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.946 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.322994+0000", 2024-09-16T10:26:10.947 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016874899999999999 2024-09-16T10:26:10.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.947 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.947 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.947 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325349+0000", 2024-09-16T10:26:10.947 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002355514 2024-09-16T10:26:10.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:10.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325668+0000", 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000319034 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325723+0000", 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5376e-05 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325775+0000", 2024-09-16T10:26:10.949 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1924e-05 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325824+0000", 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8918000000000002e-05 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51179 8.a 8:5e35a76f:::benchmark_data_smithi063_101290_object51178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:26:10.950 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.33478612499999, 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82446452000000003, 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51179 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:26:10.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513119+0000", 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513122+0000", 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5579999999999999e-06 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513149+0000", 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7101e-05 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513166+0000", 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7017e-05 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513175+0000", 2024-09-16T10:26:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6209999999999993e-06 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513268+0000", 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3287000000000003e-05 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.513279+0000", 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0889e-05 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325865+0000", 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.812586527 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.325989+0000", 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123347 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.326103+0000", 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011419199999999999 2024-09-16T10:26:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331852+0000", 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005749466 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331901+0000", 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8854999999999999e-05 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337506+0000", 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.005604876 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337547+0000", 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0961999999999998e-05 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337584+0000", 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6822000000000001e-05 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51188 8.a 8:5734c940:::benchmark_data_smithi063_101290_object51187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 418.32476998300001, 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.814521685, 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51188 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523136+0000", 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523137+0000", 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.804e-06 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523159+0000", 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1447e-05 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523171+0000", 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2347000000000001e-05 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523179+0000", 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6310000000000003e-06 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523240+0000", 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1150999999999998e-05 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:12.523248+0000", 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5190000000000003e-06 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.327963+0000", 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.80471533100000003 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328066+0000", 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103287 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.328230+0000", 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163521 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331938+0000", 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037081639999999999 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.331973+0000", 2024-09-16T10:26:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5355999999999998e-05 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337597+0000", 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056237809999999996 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337637+0000", 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.019e-05 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:13.337657+0000", 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0156000000000001e-05 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51286 8.1 8:897e7f6e:::benchmark_data_smithi063_101290_object51285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 416.53087902300001, 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.983161858, 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51286 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317027+0000", 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317028+0000", 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.141e-06 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317054+0000", 2024-09-16T10:26:10.967 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6186e-05 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317065+0000", 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.08e-05 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317072+0000", 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3520000000000001e-06 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.968 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317688+0000", 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061638600000000002 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.317698+0000", 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.346e-06 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.969 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292346+0000", 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97464779999999995 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292464+0000", 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118582 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.292563+0000", 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9047000000000002e-05 2024-09-16T10:26:10.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296570+0000", 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040065379999999996 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296637+0000", 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7547999999999998e-05 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.971 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300071+0000", 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003433298 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300139+0000", 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8702999999999996e-05 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300188+0000", 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9131000000000001e-05 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.972 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51312 8.1 8:8f097c26:::benchmark_data_smithi063_101290_object51311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 416.51184537199998, 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96891611600000005, 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.973 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51312 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336060+0000", 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336063+0000", 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3949999999999999e-06 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336103+0000", 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0754000000000003e-05 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.975 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336138+0000", 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4662000000000001e-05 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336160+0000", 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1752000000000001e-05 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336277+0000", 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117586 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.336288+0000", 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0852999999999999e-05 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294143+0000", 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95785426600000001 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294236+0000", 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3010000000000006e-05 2024-09-16T10:26:10.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.294323+0000", 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7619000000000003e-05 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298376+0000", 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004053175 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.978 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300258+0000", 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018817300000000001 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304863+0000", 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046049539999999996 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3490000000000004e-05 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.304976+0000", 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9870000000000002e-05 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51316 8.1 8:858e6b14:::benchmark_data_smithi063_101290_object51315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 416.510525395, 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96772945600000004, 2024-09-16T10:26:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51316 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337380+0000", 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337381+0000", 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.192e-06 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337441+0000", 2024-09-16T10:26:10.982 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9988000000000002e-05 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337452+0000", 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0857e-05 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337460+0000", 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2649999999999999e-06 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.983 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337520+0000", 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0377999999999997e-05 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.337530+0000", 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0416e-05 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295200+0000", 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95766991499999998 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295307+0000", 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010660199999999999 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.295446+0000", 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013906699999999999 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300014+0000", 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045680249999999999 2024-09-16T10:26:10.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300329+0000", 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031521199999999999 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305018+0000", 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0046893569999999999 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305073+0000", 2024-09-16T10:26:10.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4243000000000002e-05 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305110+0000", 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6939e-05 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51363 8.1 8:83ceb830:::benchmark_data_smithi063_101290_object51362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:26:10.987 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 416.40899937, 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.86632090799999995, 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51363 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.988 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438906+0000", 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438908+0000", 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.454e-06 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.989 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438931+0000", 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3197000000000002e-05 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438945+0000", 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3703000000000001e-05 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.438952+0000", 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5259999999999996e-06 2024-09-16T10:26:10.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439023+0000", 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0457000000000007e-05 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.439034+0000", 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1141000000000001e-05 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.991 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296689+0000", 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.85765570300000005 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296811+0000", 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121229 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.296957+0000", 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014621999999999999 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300288+0000", 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033314 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300372+0000", 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3277999999999998e-05 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305147+0000", 2024-09-16T10:26:10.993 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047752990000000002 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305194+0000", 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6789999999999998e-05 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305227+0000", 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3510999999999998e-05 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:10.994 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51388 8.1 8:8d4128ae:::benchmark_data_smithi063_101290_object51387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 416.384284536, 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.84172315399999997, 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:10.995 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51388 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463621+0000", 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463623+0000", 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463644+0000", 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0937999999999998e-05 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463654+0000", 2024-09-16T10:26:10.997 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0390999999999999e-05 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463661+0000", 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2400000000000001e-06 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463714+0000", 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3059999999999997e-05 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.998 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.463722+0000", 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4769999999999997e-06 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298409+0000", 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.83468682000000005 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298629+0000", 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000220258 2024-09-16T10:26:10.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.298774+0000", 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145012 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300340+0000", 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001566352 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:11.000 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302114+0000", 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001773518 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305261+0000", 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003147721 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305310+0000", 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9051000000000002e-05 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305344+0000", 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3751000000000003e-05 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.15333.0:51396 8.1 8:8a8d0cb2:::benchmark_data_smithi063_101290_object51395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 416.36549243500002, 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.82304446200000003, 2024-09-16T10:26:11.002 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.15333", 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 51396 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482413+0000", 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482414+0000", 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2300000000000001e-06 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:11.004 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482433+0000", 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8876e-05 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482445+0000", 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2170999999999999e-05 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482453+0000", 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8939999999999993e-06 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.005 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:11.006 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482504+0000", 2024-09-16T10:26:11.006 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0698000000000001e-05 2024-09-16T10:26:11.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.006 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:14.482512+0000", 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1319999999999994e-06 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300411+0000", 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.81789900699999996 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.007 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300560+0000", 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014893799999999999 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.300718+0000", 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015829700000000001 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302254+0000", 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015359429999999999 2024-09-16T10:26:11.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.302509+0000", 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025458499999999999 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305379+0000", 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028699329999999999 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305426+0000", 2024-09-16T10:26:11.009 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7330000000000003e-05 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:19:15.305458+0000", 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1427999999999998e-05 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:11.010 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:26:11.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:11.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:11.152 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:11.193 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:11.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:11.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:11.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:11.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:11.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:11.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:11.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:11.574 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:11.613 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:11.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:11.646 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:26:11.680 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: 1.1 scrub ok 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: 1.c scrub starts 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: 1.c scrub ok 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: 4.e scrub starts 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: 4.e scrub ok 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: from='client.16215 v1:172.21.15.63:0/745948469' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[29993]: osdmap e376: 8 total, 8 up, 8 in 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: 1.1 scrub ok 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: 1.c scrub starts 2024-09-16T10:26:11.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: 1.c scrub ok 2024-09-16T10:26:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: 4.e scrub starts 2024-09-16T10:26:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: 4.e scrub ok 2024-09-16T10:26:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: from='client.16215 v1:172.21.15.63:0/745948469' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:26:11.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:11 smithi063 ceph-mon[39470]: osdmap e376: 8 total, 8 up, 8 in 2024-09-16T10:26:11.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 35 16 68381 68365 122.061 87.1875 0.00814576 0.00818726 2024-09-16T10:26:11.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:11.751 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:11.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: 1.1 scrub ok 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: 1.c scrub starts 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: 1.c scrub ok 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: 4.e scrub starts 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: 4.e scrub ok 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: from='client.16215 v1:172.21.15.63:0/745948469' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:26:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:11 smithi165 ceph-mon[32696]: osdmap e376: 8 total, 8 up, 8 in 2024-09-16T10:26:11.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:11.899 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:11.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:12.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:12.038 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:12.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:12.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:12.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:12.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:12.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:12.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:12.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:12.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:12.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:12.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:12.563 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:12.584 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:12.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:12.686 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:12.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 36 16 69732 69716 121.016 84.4375 0.0114448 0.0082584 2024-09-16T10:26:12.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[29993]: 5.15 scrub starts 2024-09-16T10:26:12.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[29993]: 5.15 scrub ok 2024-09-16T10:26:12.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[29993]: pgmap v668: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-16T10:26:12.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:26:12.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:26:12.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[39470]: 5.15 scrub starts 2024-09-16T10:26:12.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[39470]: 5.15 scrub ok 2024-09-16T10:26:12.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[39470]: pgmap v668: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-16T10:26:12.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:26:12.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:12 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:26:12.728 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:12.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:12 smithi165 ceph-mon[32696]: 5.15 scrub starts 2024-09-16T10:26:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:12 smithi165 ceph-mon[32696]: 5.15 scrub ok 2024-09-16T10:26:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:12 smithi165 ceph-mon[32696]: pgmap v668: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-16T10:26:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:12 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:26:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:12 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:26:12.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:12.867 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:12.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:12.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:12.984 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2024-09-16T10:26:09.321057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4141226523,"num_objects":63380,"num_object_clones":0,"num_object_copies":190140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63380,"num_whiteouts":0,"num_read":1213,"num_read_kb":965,"num_write":63903,"num_write_kb":4043629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":527794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64184,"ondisk_log_size":64184,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15656276,"kb_used_data":11980496,"kb_used_omap":12,"kb_used_meta":3675635,"kb_avail":734304940,"statfs":{"total":767960285184,"available":751928258560,"internally_reserved":0,"allocated":12268027904,"data_stored":12263263711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3763850801},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,5,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":987430912,"num_objects":15067,"num_object_clones":0,"num_object_copies":45201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15067,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15067,"num_write_kb":964288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003191"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":692,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963771+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:25:55.963771+0000","last_peered":"2024-09-16T10:25:55.963771+0000","last_clean":"2024-09-16T10:25:55.963771+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:25:55.963771+0000","last_undegraded":"2024-09-16T10:25:55.963771+0000","last_fullsized":"2024-09-16T10:25:55.963771+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.493863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":743,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392805+0000","last_change":"2024-09-16T10:25:04.357048+0000","last_active":"2024-09-16T10:26:06.392805+0000","last_peered":"2024-09-16T10:26:06.392805+0000","last_clean":"2024-09-16T10:26:06.392805+0000","last_became_active":"2024-09-16T10:18:52.836076+0000","last_became_peered":"2024-09-16T10:18:52.836076+0000","last_unstale":"2024-09-16T10:26:06.392805+0000","last_undegraded":"2024-09-16T10:26:06.392805+0000","last_fullsized":"2024-09-16T10:26:06.392805+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:25:04.356955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.356955+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:13.131130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":681,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392917+0000","last_change":"2024-09-16T10:25:36.940535+0000","last_active":"2024-09-16T10:26:06.392917+0000","last_peered":"2024-09-16T10:26:06.392917+0000","last_clean":"2024-09-16T10:26:06.392917+0000","last_became_active":"2024-09-16T10:13:41.107971+0000","last_became_peered":"2024-09-16T10:13:41.107971+0000","last_unstale":"2024-09-16T10:26:06.392917+0000","last_undegraded":"2024-09-16T10:26:06.392917+0000","last_fullsized":"2024-09-16T10:26:06.392917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.940459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.940459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.230715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392864+0000","last_change":"2024-09-16T10:25:08.485862+0000","last_active":"2024-09-16T10:26:06.392864+0000","last_peered":"2024-09-16T10:26:06.392864+0000","last_clean":"2024-09-16T10:26:06.392864+0000","last_became_active":"2024-09-16T10:13:43.112849+0000","last_became_peered":"2024-09-16T10:13:43.112849+0000","last_unstale":"2024-09-16T10:26:06.392864+0000","last_undegraded":"2024-09-16T10:26:06.392864+0000","last_fullsized":"2024-09-16T10:26:06.392864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.485783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.485783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:08.732498+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":658,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414320+0000","last_change":"2024-09-16T10:20:24.255491+0000","last_active":"2024-09-16T10:26:06.414320+0000","last_peered":"2024-09-16T10:26:06.414320+0000","last_clean":"2024-09-16T10:26:06.414320+0000","last_became_active":"2024-09-16T10:13:43.286786+0000","last_became_peered":"2024-09-16T10:13:43.286786+0000","last_unstale":"2024-09-16T10:26:06.414320+0000","last_undegraded":"2024-09-16T10:26:06.414320+0000","last_fullsized":"2024-09-16T10:26:06.414320+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:45.448685+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":702,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415069+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:26:06.415069+0000","last_peered":"2024-09-16T10:26:06.415069+0000","last_clean":"2024-09-16T10:26:06.415069+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:26:06.415069+0000","last_undegraded":"2024-09-16T10:26:06.415069+0000","last_fullsized":"2024-09-16T10:26:06.415069+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.018649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":764,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415451+0000","last_change":"2024-09-16T10:24:37.612269+0000","last_active":"2024-09-16T10:26:06.415451+0000","last_peered":"2024-09-16T10:26:06.415451+0000","last_clean":"2024-09-16T10:26:06.415451+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:26:06.415451+0000","last_undegraded":"2024-09-16T10:26:06.415451+0000","last_fullsized":"2024-09-16T10:26:06.415451+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:37.612167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.612167+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":672,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392771+0000","last_change":"2024-09-16T10:24:21.153205+0000","last_active":"2024-09-16T10:26:06.392771+0000","last_peered":"2024-09-16T10:26:06.392771+0000","last_clean":"2024-09-16T10:26:06.392771+0000","last_became_active":"2024-09-16T10:13:41.121220+0000","last_became_peered":"2024-09-16T10:13:41.121220+0000","last_unstale":"2024-09-16T10:26:06.392771+0000","last_undegraded":"2024-09-16T10:26:06.392771+0000","last_fullsized":"2024-09-16T10:26:06.392771+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:21.153131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:21.153131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:04.341537+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":696,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418914+0000","last_change":"2024-09-16T10:25:13.988034+0000","last_active":"2024-09-16T10:26:06.418914+0000","last_peered":"2024-09-16T10:26:06.418914+0000","last_clean":"2024-09-16T10:26:06.418914+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:26:06.418914+0000","last_undegraded":"2024-09-16T10:26:06.418914+0000","last_fullsized":"2024-09-16T10:26:06.418914+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.987974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.987974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.738929+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":709,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393140+0000","last_change":"2024-09-16T10:24:23.258943+0000","last_active":"2024-09-16T10:26:06.393140+0000","last_peered":"2024-09-16T10:26:06.393140+0000","last_clean":"2024-09-16T10:26:06.393140+0000","last_became_active":"2024-09-16T10:13:39.092181+0000","last_became_peered":"2024-09-16T10:13:39.092181+0000","last_unstale":"2024-09-16T10:26:06.393140+0000","last_undegraded":"2024-09-16T10:26:06.393140+0000","last_fullsized":"2024-09-16T10:26:06.393140+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:24:23.258886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.258886+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:46.299405+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":681,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418240+0000","last_change":"2024-09-16T10:25:33.834886+0000","last_active":"2024-09-16T10:26:06.418240+0000","last_peered":"2024-09-16T10:26:06.418240+0000","last_clean":"2024-09-16T10:26:06.418240+0000","last_became_active":"2024-09-16T10:13:41.111971+0000","last_became_peered":"2024-09-16T10:13:41.111971+0000","last_unstale":"2024-09-16T10:26:06.418240+0000","last_undegraded":"2024-09-16T10:26:06.418240+0000","last_fullsized":"2024-09-16T10:26:06.418240+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.834823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.834823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.729220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":666,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963473+0000","last_change":"2024-09-16T10:22:23.238895+0000","last_active":"2024-09-16T10:25:55.963473+0000","last_peered":"2024-09-16T10:25:55.963473+0000","last_clean":"2024-09-16T10:25:55.963473+0000","last_became_active":"2024-09-16T10:13:43.115474+0000","last_became_peered":"2024-09-16T10:13:43.115474+0000","last_unstale":"2024-09-16T10:25:55.963473+0000","last_undegraded":"2024-09-16T10:25:55.963473+0000","last_fullsized":"2024-09-16T10:25:55.963473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:23.238826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:23.238826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":648,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418512+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:26:06.418512+0000","last_peered":"2024-09-16T10:26:06.418512+0000","last_clean":"2024-09-16T10:26:06.418512+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:26:06.418512+0000","last_undegraded":"2024-09-16T10:26:06.418512+0000","last_fullsized":"2024-09-16T10:26:06.418512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:53.954468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":697,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393037+0000","last_change":"2024-09-16T10:24:37.134728+0000","last_active":"2024-09-16T10:26:06.393037+0000","last_peered":"2024-09-16T10:26:06.393037+0000","last_clean":"2024-09-16T10:26:06.393037+0000","last_became_active":"2024-09-16T10:13:39.092172+0000","last_became_peered":"2024-09-16T10:13:39.092172+0000","last_unstale":"2024-09-16T10:26:06.393037+0000","last_undegraded":"2024-09-16T10:26:06.393037+0000","last_fullsized":"2024-09-16T10:26:06.393037+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:00.275341+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":679,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.412023+0000","last_change":"2024-09-16T10:25:15.369074+0000","last_active":"2024-09-16T10:26:06.412023+0000","last_peered":"2024-09-16T10:26:06.412023+0000","last_clean":"2024-09-16T10:26:06.412023+0000","last_became_active":"2024-09-16T10:13:41.111332+0000","last_became_peered":"2024-09-16T10:13:41.111332+0000","last_unstale":"2024-09-16T10:26:06.412023+0000","last_undegraded":"2024-09-16T10:26:06.412023+0000","last_fullsized":"2024-09-16T10:26:06.412023+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.369005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.369005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.347891+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":684,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417853+0000","last_change":"2024-09-16T10:23:42.856401+0000","last_active":"2024-09-16T10:26:06.417853+0000","last_peered":"2024-09-16T10:26:06.417853+0000","last_clean":"2024-09-16T10:26:06.417853+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:26:06.417853+0000","last_undegraded":"2024-09-16T10:26:06.417853+0000","last_fullsized":"2024-09-16T10:26:06.417853+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.856353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.856353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:25.927067+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":661,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393983+0000","last_change":"2024-09-16T10:19:47.698169+0000","last_active":"2024-09-16T10:26:06.393983+0000","last_peered":"2024-09-16T10:26:06.393983+0000","last_clean":"2024-09-16T10:26:06.393983+0000","last_became_active":"2024-09-16T10:13:43.283929+0000","last_became_peered":"2024-09-16T10:13:43.283929+0000","last_unstale":"2024-09-16T10:26:06.393983+0000","last_undegraded":"2024-09-16T10:26:06.393983+0000","last_fullsized":"2024-09-16T10:26:06.393983+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.254203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":669,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.967664+0000","last_change":"2024-09-16T10:22:27.210541+0000","last_active":"2024-09-16T10:25:55.967664+0000","last_peered":"2024-09-16T10:25:55.967664+0000","last_clean":"2024-09-16T10:25:55.967664+0000","last_became_active":"2024-09-16T10:13:37.097592+0000","last_became_peered":"2024-09-16T10:13:37.097592+0000","last_unstale":"2024-09-16T10:25:55.967664+0000","last_undegraded":"2024-09-16T10:25:55.967664+0000","last_fullsized":"2024-09-16T10:25:55.967664+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":241,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419193+0000","last_change":"2024-09-16T10:25:08.473527+0000","last_active":"2024-09-16T10:26:06.419193+0000","last_peered":"2024-09-16T10:26:06.419193+0000","last_clean":"2024-09-16T10:26:06.419193+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:26:06.419193+0000","last_undegraded":"2024-09-16T10:26:06.419193+0000","last_fullsized":"2024-09-16T10:26:06.419193+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:25:08.473402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.473402+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.786201+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":664,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393342+0000","last_change":"2024-09-16T10:18:32.830616+0000","last_active":"2024-09-16T10:26:06.393342+0000","last_peered":"2024-09-16T10:26:06.393342+0000","last_clean":"2024-09-16T10:26:06.393342+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:26:06.393342+0000","last_undegraded":"2024-09-16T10:26:06.393342+0000","last_fullsized":"2024-09-16T10:26:06.393342+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:18:32.830532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:18:32.830532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.737702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":624,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.416139+0000","last_change":"2024-09-16T10:25:15.468418+0000","last_active":"2024-09-16T10:26:06.416139+0000","last_peered":"2024-09-16T10:26:06.416139+0000","last_clean":"2024-09-16T10:26:06.416139+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:26:06.416139+0000","last_undegraded":"2024-09-16T10:26:06.416139+0000","last_fullsized":"2024-09-16T10:26:06.416139+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.468271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.468271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.583811+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":686,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393238+0000","last_change":"2024-09-16T10:25:14.407660+0000","last_active":"2024-09-16T10:26:06.393238+0000","last_peered":"2024-09-16T10:26:06.393238+0000","last_clean":"2024-09-16T10:26:06.393238+0000","last_became_active":"2024-09-16T10:22:02.899223+0000","last_became_peered":"2024-09-16T10:22:02.899223+0000","last_unstale":"2024-09-16T10:26:06.393238+0000","last_undegraded":"2024-09-16T10:26:06.393238+0000","last_fullsized":"2024-09-16T10:26:06.393238+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.063208+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":733,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414677+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:26:06.414677+0000","last_peered":"2024-09-16T10:26:06.414677+0000","last_clean":"2024-09-16T10:26:06.414677+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:26:06.414677+0000","last_undegraded":"2024-09-16T10:26:06.414677+0000","last_fullsized":"2024-09-16T10:26:06.414677+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.381310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":693,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393170+0000","last_change":"2024-09-16T10:24:35.186387+0000","last_active":"2024-09-16T10:26:06.393170+0000","last_peered":"2024-09-16T10:26:06.393170+0000","last_clean":"2024-09-16T10:26:06.393170+0000","last_became_active":"2024-09-16T10:22:02.907658+0000","last_became_peered":"2024-09-16T10:22:02.907658+0000","last_unstale":"2024-09-16T10:26:06.393170+0000","last_undegraded":"2024-09-16T10:26:06.393170+0000","last_fullsized":"2024-09-16T10:26:06.393170+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:35.186332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:35.186332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":80,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418644+0000","last_change":"2024-09-16T10:25:18.973817+0000","last_active":"2024-09-16T10:26:06.418644+0000","last_peered":"2024-09-16T10:26:06.418644+0000","last_clean":"2024-09-16T10:26:06.418644+0000","last_became_active":"2024-09-16T10:25:00.947649+0000","last_became_peered":"2024-09-16T10:25:00.947649+0000","last_unstale":"2024-09-16T10:26:06.418644+0000","last_undegraded":"2024-09-16T10:26:06.418644+0000","last_fullsized":"2024-09-16T10:26:06.418644+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:44.946516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":680,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393070+0000","last_change":"2024-09-16T10:25:06.652209+0000","last_active":"2024-09-16T10:26:06.393070+0000","last_peered":"2024-09-16T10:26:06.393070+0000","last_clean":"2024-09-16T10:26:06.393070+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:26:06.393070+0000","last_undegraded":"2024-09-16T10:26:06.393070+0000","last_fullsized":"2024-09-16T10:26:06.393070+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:25:06.652146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:06.652146+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.202612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":705,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963591+0000","last_change":"2024-09-16T10:21:34.114224+0000","last_active":"2024-09-16T10:25:55.963591+0000","last_peered":"2024-09-16T10:25:55.963591+0000","last_clean":"2024-09-16T10:25:55.963591+0000","last_became_active":"2024-09-16T10:13:39.107745+0000","last_became_peered":"2024-09-16T10:13:39.107745+0000","last_unstale":"2024-09-16T10:25:55.963591+0000","last_undegraded":"2024-09-16T10:25:55.963591+0000","last_fullsized":"2024-09-16T10:25:55.963591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:50.538123+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":834,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.403301+0000","last_change":"2024-09-16T10:25:16.538777+0000","last_active":"2024-09-16T10:26:06.403301+0000","last_peered":"2024-09-16T10:26:06.403301+0000","last_clean":"2024-09-16T10:26:06.403301+0000","last_became_active":"2024-09-16T10:13:41.115593+0000","last_became_peered":"2024-09-16T10:13:41.115593+0000","last_unstale":"2024-09-16T10:26:06.403301+0000","last_undegraded":"2024-09-16T10:26:06.403301+0000","last_fullsized":"2024-09-16T10:26:06.403301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:16.538706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.538706+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.979675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"375'3941","reported_seq":3978,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.253998+0000","last_change":"2024-09-16T10:25:30.947944+0000","last_active":"2024-09-16T10:26:08.253998+0000","last_peered":"2024-09-16T10:26:08.253998+0000","last_clean":"2024-09-16T10:26:08.253998+0000","last_became_active":"2024-09-16T10:25:30.947200+0000","last_became_peered":"2024-09-16T10:25:30.947200+0000","last_unstale":"2024-09-16T10:26:08.253998+0000","last_undegraded":"2024-09-16T10:26:08.253998+0000","last_fullsized":"2024-09-16T10:26:08.253998+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.630601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258277376,"num_objects":3941,"num_object_clones":0,"num_object_copies":11823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":252224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":526,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417711+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:26:06.417711+0000","last_peered":"2024-09-16T10:26:06.417711+0000","last_clean":"2024-09-16T10:26:06.417711+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:26:06.417711+0000","last_undegraded":"2024-09-16T10:26:06.417711+0000","last_fullsized":"2024-09-16T10:26:06.417711+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.039248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":712,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393383+0000","last_change":"2024-09-16T10:25:10.700157+0000","last_active":"2024-09-16T10:26:06.393383+0000","last_peered":"2024-09-16T10:26:06.393383+0000","last_clean":"2024-09-16T10:26:06.393383+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:26:06.393383+0000","last_undegraded":"2024-09-16T10:26:06.393383+0000","last_fullsized":"2024-09-16T10:26:06.393383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:25:10.699735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:10.699735+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:35.612246+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":687,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415567+0000","last_change":"2024-09-16T10:25:33.377037+0000","last_active":"2024-09-16T10:26:06.415567+0000","last_peered":"2024-09-16T10:26:06.415567+0000","last_clean":"2024-09-16T10:26:06.415567+0000","last_became_active":"2024-09-16T10:13:37.089277+0000","last_became_peered":"2024-09-16T10:13:37.089277+0000","last_unstale":"2024-09-16T10:26:06.415567+0000","last_undegraded":"2024-09-16T10:26:06.415567+0000","last_fullsized":"2024-09-16T10:26:06.415567+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:41.874798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":823,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.394057+0000","last_change":"2024-09-16T10:24:57.408285+0000","last_active":"2024-09-16T10:26:06.394057+0000","last_peered":"2024-09-16T10:26:06.394057+0000","last_clean":"2024-09-16T10:26:06.394057+0000","last_became_active":"2024-09-16T10:13:41.118726+0000","last_became_peered":"2024-09-16T10:13:41.118726+0000","last_unstale":"2024-09-16T10:26:06.394057+0000","last_undegraded":"2024-09-16T10:26:06.394057+0000","last_fullsized":"2024-09-16T10:26:06.394057+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:57.408217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.408217+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:09.573273+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":669,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417976+0000","last_change":"2024-09-16T10:24:15.859366+0000","last_active":"2024-09-16T10:26:06.417976+0000","last_peered":"2024-09-16T10:26:06.417976+0000","last_clean":"2024-09-16T10:26:06.417976+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:26:06.417976+0000","last_undegraded":"2024-09-16T10:26:06.417976+0000","last_fullsized":"2024-09-16T10:26:06.417976+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:15.859299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.859299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:00.003522+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"375'3943","reported_seq":3980,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.270584+0000","last_change":"2024-09-16T10:25:30.619893+0000","last_active":"2024-09-16T10:26:08.270584+0000","last_peered":"2024-09-16T10:26:08.270584+0000","last_clean":"2024-09-16T10:26:08.270584+0000","last_became_active":"2024-09-16T10:25:30.619574+0000","last_became_peered":"2024-09-16T10:25:30.619574+0000","last_unstale":"2024-09-16T10:26:08.270584+0000","last_undegraded":"2024-09-16T10:26:08.270584+0000","last_fullsized":"2024-09-16T10:26:08.270584+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.565846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258408448,"num_objects":3943,"num_object_clones":0,"num_object_copies":11829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3943,"num_write_kb":252352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":578,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393628+0000","last_change":"2024-09-16T10:24:28.146701+0000","last_active":"2024-09-16T10:26:06.393628+0000","last_peered":"2024-09-16T10:26:06.393628+0000","last_clean":"2024-09-16T10:26:06.393628+0000","last_became_active":"2024-09-16T10:22:56.409926+0000","last_became_peered":"2024-09-16T10:22:56.409926+0000","last_unstale":"2024-09-16T10:26:06.393628+0000","last_undegraded":"2024-09-16T10:26:06.393628+0000","last_fullsized":"2024-09-16T10:26:06.393628+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.146633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.146633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:49.227487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":162,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392630+0000","last_change":"2024-09-16T10:23:42.023903+0000","last_active":"2024-09-16T10:26:06.392630+0000","last_peered":"2024-09-16T10:26:06.392630+0000","last_clean":"2024-09-16T10:26:06.392630+0000","last_became_active":"2024-09-16T10:23:41.951456+0000","last_became_peered":"2024-09-16T10:23:41.951456+0000","last_unstale":"2024-09-16T10:26:06.392630+0000","last_undegraded":"2024-09-16T10:26:06.392630+0000","last_fullsized":"2024-09-16T10:26:06.392630+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-16T10:23:13.871560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.871560+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.206554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":652,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.416727+0000","last_change":"2024-09-16T10:25:36.404433+0000","last_active":"2024-09-16T10:26:06.416727+0000","last_peered":"2024-09-16T10:26:06.416727+0000","last_clean":"2024-09-16T10:26:06.416727+0000","last_became_active":"2024-09-16T10:18:33.892790+0000","last_became_peered":"2024-09-16T10:18:33.892790+0000","last_unstale":"2024-09-16T10:26:06.416727+0000","last_undegraded":"2024-09-16T10:26:06.416727+0000","last_fullsized":"2024-09-16T10:26:06.416727+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.404383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.404383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.505459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":411,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402267+0000","last_change":"2024-09-16T10:20:23.549351+0000","last_active":"2024-09-16T10:26:06.402267+0000","last_peered":"2024-09-16T10:26:06.402267+0000","last_clean":"2024-09-16T10:26:06.402267+0000","last_became_active":"2024-09-16T10:19:09.301404+0000","last_became_peered":"2024-09-16T10:19:09.301404+0000","last_unstale":"2024-09-16T10:26:06.402267+0000","last_undegraded":"2024-09-16T10:26:06.402267+0000","last_fullsized":"2024-09-16T10:26:06.402267+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:23.549157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:23.549157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.558332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":690,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393077+0000","last_change":"2024-09-16T10:24:22.206462+0000","last_active":"2024-09-16T10:26:06.393077+0000","last_peered":"2024-09-16T10:26:06.393077+0000","last_clean":"2024-09-16T10:26:06.393077+0000","last_became_active":"2024-09-16T10:22:02.898865+0000","last_became_peered":"2024-09-16T10:22:02.898865+0000","last_unstale":"2024-09-16T10:26:06.393077+0000","last_undegraded":"2024-09-16T10:26:06.393077+0000","last_fullsized":"2024-09-16T10:26:06.393077+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:22.206414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:22.206414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:02.272177+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"375'3876","reported_seq":3913,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.342802+0000","last_change":"2024-09-16T10:25:31.163385+0000","last_active":"2024-09-16T10:26:08.342802+0000","last_peered":"2024-09-16T10:26:08.342802+0000","last_clean":"2024-09-16T10:26:08.342802+0000","last_became_active":"2024-09-16T10:25:31.162906+0000","last_became_peered":"2024-09-16T10:25:31.162906+0000","last_unstale":"2024-09-16T10:26:08.342802+0000","last_undegraded":"2024-09-16T10:26:08.342802+0000","last_fullsized":"2024-09-16T10:26:08.342802+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.032805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254017536,"num_objects":3876,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":248064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":569,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419246+0000","last_change":"2024-09-16T10:24:28.343296+0000","last_active":"2024-09-16T10:26:06.419246+0000","last_peered":"2024-09-16T10:26:06.419246+0000","last_clean":"2024-09-16T10:26:06.419246+0000","last_became_active":"2024-09-16T10:22:56.446950+0000","last_became_peered":"2024-09-16T10:22:56.446950+0000","last_unstale":"2024-09-16T10:26:06.419246+0000","last_undegraded":"2024-09-16T10:26:06.419246+0000","last_fullsized":"2024-09-16T10:26:06.419246+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.343066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.343066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:52.413280+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":727,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392817+0000","last_change":"2024-09-16T10:24:59.344438+0000","last_active":"2024-09-16T10:26:06.392817+0000","last_peered":"2024-09-16T10:26:06.392817+0000","last_clean":"2024-09-16T10:26:06.392817+0000","last_became_active":"2024-09-16T10:13:39.093644+0000","last_became_peered":"2024-09-16T10:13:39.093644+0000","last_unstale":"2024-09-16T10:26:06.392817+0000","last_undegraded":"2024-09-16T10:26:06.392817+0000","last_fullsized":"2024-09-16T10:26:06.392817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:24:59.344346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.344346+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":671,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963945+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:25:55.963945+0000","last_peered":"2024-09-16T10:25:55.963945+0000","last_clean":"2024-09-16T10:25:55.963945+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:25:55.963945+0000","last_undegraded":"2024-09-16T10:25:55.963945+0000","last_fullsized":"2024-09-16T10:25:55.963945+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.295227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":834,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415271+0000","last_change":"2024-09-16T10:25:12.476210+0000","last_active":"2024-09-16T10:26:06.415271+0000","last_peered":"2024-09-16T10:26:06.415271+0000","last_clean":"2024-09-16T10:26:06.415271+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:26:06.415271+0000","last_undegraded":"2024-09-16T10:26:06.415271+0000","last_fullsized":"2024-09-16T10:26:06.415271+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:12.476137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.476137+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:36.834759+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":520,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.403052+0000","last_change":"2024-09-16T10:25:00.948343+0000","last_active":"2024-09-16T10:26:06.403052+0000","last_peered":"2024-09-16T10:26:06.403052+0000","last_clean":"2024-09-16T10:26:06.403052+0000","last_became_active":"2024-09-16T10:25:00.946867+0000","last_became_peered":"2024-09-16T10:25:00.946867+0000","last_unstale":"2024-09-16T10:26:06.403052+0000","last_undegraded":"2024-09-16T10:26:06.403052+0000","last_fullsized":"2024-09-16T10:26:06.403052+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.423640+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"375'4003","reported_seq":4040,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.271259+0000","last_change":"2024-09-16T10:25:30.949013+0000","last_active":"2024-09-16T10:26:08.271259+0000","last_peered":"2024-09-16T10:26:08.271259+0000","last_clean":"2024-09-16T10:26:08.271259+0000","last_became_active":"2024-09-16T10:25:30.948309+0000","last_became_peered":"2024-09-16T10:25:30.948309+0000","last_unstale":"2024-09-16T10:26:08.271259+0000","last_undegraded":"2024-09-16T10:26:08.271259+0000","last_fullsized":"2024-09-16T10:26:08.271259+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.205764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262340608,"num_objects":4003,"num_object_clones":0,"num_object_copies":12009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4003,"num_write_kb":256192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":619,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402879+0000","last_change":"2024-09-16T10:24:31.617241+0000","last_active":"2024-09-16T10:26:06.402879+0000","last_peered":"2024-09-16T10:26:06.402879+0000","last_clean":"2024-09-16T10:26:06.402879+0000","last_became_active":"2024-09-16T10:22:56.438108+0000","last_became_peered":"2024-09-16T10:22:56.438108+0000","last_unstale":"2024-09-16T10:26:06.402879+0000","last_undegraded":"2024-09-16T10:26:06.402879+0000","last_fullsized":"2024-09-16T10:26:06.402879+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.617190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.617190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.510231+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":715,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.412626+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:26:06.412626+0000","last_peered":"2024-09-16T10:26:06.412626+0000","last_clean":"2024-09-16T10:26:06.412626+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:26:06.412626+0000","last_undegraded":"2024-09-16T10:26:06.412626+0000","last_fullsized":"2024-09-16T10:26:06.412626+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:57.162232+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418947+0000","last_change":"2024-09-16T10:24:59.820337+0000","last_active":"2024-09-16T10:26:06.418947+0000","last_peered":"2024-09-16T10:26:06.418947+0000","last_clean":"2024-09-16T10:26:06.418947+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:26:06.418947+0000","last_undegraded":"2024-09-16T10:26:06.418947+0000","last_fullsized":"2024-09-16T10:26:06.418947+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:59.820273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.820273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:07.866898+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":812,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:57.314937+0000","last_change":"2024-09-16T10:20:31.274721+0000","last_active":"2024-09-16T10:25:57.314937+0000","last_peered":"2024-09-16T10:25:57.314937+0000","last_clean":"2024-09-16T10:25:57.314937+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:25:57.314937+0000","last_undegraded":"2024-09-16T10:25:57.314937+0000","last_fullsized":"2024-09-16T10:25:57.314937+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:20:31.274658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.274658+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.537119+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":681,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.964063+0000","last_change":"2024-09-16T10:22:02.904776+0000","last_active":"2024-09-16T10:25:55.964063+0000","last_peered":"2024-09-16T10:25:55.964063+0000","last_clean":"2024-09-16T10:25:55.964063+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:25:55.964063+0000","last_undegraded":"2024-09-16T10:25:55.964063+0000","last_fullsized":"2024-09-16T10:25:55.964063+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:31.081494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.081494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:51.528206+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"375'4047","reported_seq":4084,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.357843+0000","last_change":"2024-09-16T10:25:30.621332+0000","last_active":"2024-09-16T10:26:08.357843+0000","last_peered":"2024-09-16T10:26:08.357843+0000","last_clean":"2024-09-16T10:26:08.357843+0000","last_became_active":"2024-09-16T10:25:30.620945+0000","last_became_peered":"2024-09-16T10:25:30.620945+0000","last_unstale":"2024-09-16T10:26:08.357843+0000","last_undegraded":"2024-09-16T10:26:08.357843+0000","last_fullsized":"2024-09-16T10:26:08.357843+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:38.539869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265224192,"num_objects":4047,"num_object_clones":0,"num_object_copies":12141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4047,"num_write_kb":259008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":624,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402717+0000","last_change":"2024-09-16T10:24:30.609351+0000","last_active":"2024-09-16T10:26:06.402717+0000","last_peered":"2024-09-16T10:26:06.402717+0000","last_clean":"2024-09-16T10:26:06.402717+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:26:06.402717+0000","last_undegraded":"2024-09-16T10:26:06.402717+0000","last_fullsized":"2024-09-16T10:26:06.402717+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:30.609302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:30.609302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:49.574204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":725,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.966824+0000","last_change":"2024-09-16T10:21:27.125848+0000","last_active":"2024-09-16T10:25:55.966824+0000","last_peered":"2024-09-16T10:25:55.966824+0000","last_clean":"2024-09-16T10:25:55.966824+0000","last_became_active":"2024-09-16T10:13:39.105778+0000","last_became_peered":"2024-09-16T10:13:39.105778+0000","last_unstale":"2024-09-16T10:25:55.966824+0000","last_undegraded":"2024-09-16T10:25:55.966824+0000","last_fullsized":"2024-09-16T10:25:55.966824+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:44.535673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":696,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393398+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:26:06.393398+0000","last_peered":"2024-09-16T10:26:06.393398+0000","last_clean":"2024-09-16T10:26:06.393398+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:26:06.393398+0000","last_undegraded":"2024-09-16T10:26:06.393398+0000","last_fullsized":"2024-09-16T10:26:06.393398+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.166054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":688,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402437+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:26:06.402437+0000","last_peered":"2024-09-16T10:26:06.402437+0000","last_clean":"2024-09-16T10:26:06.402437+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:26:06.402437+0000","last_undegraded":"2024-09-16T10:26:06.402437+0000","last_fullsized":"2024-09-16T10:26:06.402437+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.230724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392784+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:26:06.392784+0000","last_peered":"2024-09-16T10:26:06.392784+0000","last_clean":"2024-09-16T10:26:06.392784+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:26:06.392784+0000","last_undegraded":"2024-09-16T10:26:06.392784+0000","last_fullsized":"2024-09-16T10:26:06.392784+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.861279+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"375'4027","reported_seq":4062,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.365720+0000","last_change":"2024-09-16T10:25:30.622174+0000","last_active":"2024-09-16T10:26:08.365720+0000","last_peered":"2024-09-16T10:26:08.365720+0000","last_clean":"2024-09-16T10:26:08.365720+0000","last_became_active":"2024-09-16T10:25:30.621742+0000","last_became_peered":"2024-09-16T10:25:30.621742+0000","last_unstale":"2024-09-16T10:26:08.365720+0000","last_undegraded":"2024-09-16T10:26:08.365720+0000","last_fullsized":"2024-09-16T10:26:08.365720+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.135457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263913472,"num_objects":4027,"num_object_clones":0,"num_object_copies":12081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4027,"num_write_kb":257728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":570,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417703+0000","last_change":"2024-09-16T10:24:29.815138+0000","last_active":"2024-09-16T10:26:06.417703+0000","last_peered":"2024-09-16T10:26:06.417703+0000","last_clean":"2024-09-16T10:26:06.417703+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:26:06.417703+0000","last_undegraded":"2024-09-16T10:26:06.417703+0000","last_fullsized":"2024-09-16T10:26:06.417703+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:29.814966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:29.814966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":968,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393221+0000","last_change":"2024-09-16T10:25:33.966865+0000","last_active":"2024-09-16T10:26:06.393221+0000","last_peered":"2024-09-16T10:26:06.393221+0000","last_clean":"2024-09-16T10:26:06.393221+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:26:06.393221+0000","last_undegraded":"2024-09-16T10:26:06.393221+0000","last_fullsized":"2024-09-16T10:26:06.393221+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:25:33.966775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.966775+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:45.795898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":689,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393473+0000","last_change":"2024-09-16T10:24:13.097213+0000","last_active":"2024-09-16T10:26:06.393473+0000","last_peered":"2024-09-16T10:26:06.393473+0000","last_clean":"2024-09-16T10:26:06.393473+0000","last_became_active":"2024-09-16T10:13:37.092284+0000","last_became_peered":"2024-09-16T10:13:37.092284+0000","last_unstale":"2024-09-16T10:26:06.393473+0000","last_undegraded":"2024-09-16T10:26:06.393473+0000","last_fullsized":"2024-09-16T10:26:06.393473+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:13.097133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:13.097133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:48.251274+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":684,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418492+0000","last_change":"2024-09-16T10:24:57.812754+0000","last_active":"2024-09-16T10:26:06.418492+0000","last_peered":"2024-09-16T10:26:06.418492+0000","last_clean":"2024-09-16T10:26:06.418492+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:26:06.418492+0000","last_undegraded":"2024-09-16T10:26:06.418492+0000","last_fullsized":"2024-09-16T10:26:06.418492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:24:57.812701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.812701+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":662,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.965696+0000","last_change":"2024-09-16T10:25:37.359150+0000","last_active":"2024-09-16T10:25:55.965696+0000","last_peered":"2024-09-16T10:25:55.965696+0000","last_clean":"2024-09-16T10:25:55.965696+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:25:55.965696+0000","last_undegraded":"2024-09-16T10:25:55.965696+0000","last_fullsized":"2024-09-16T10:25:55.965696+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:37.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.658889+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414795+0000","last_change":"2024-09-16T10:22:21.598828+0000","last_active":"2024-09-16T10:26:06.414795+0000","last_peered":"2024-09-16T10:26:06.414795+0000","last_clean":"2024-09-16T10:26:06.414795+0000","last_became_active":"2024-09-16T10:13:43.124429+0000","last_became_peered":"2024-09-16T10:13:43.124429+0000","last_unstale":"2024-09-16T10:26:06.414795+0000","last_undegraded":"2024-09-16T10:26:06.414795+0000","last_fullsized":"2024-09-16T10:26:06.414795+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:21.598754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:21.598754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.440379+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"375'4019","reported_seq":4056,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.532091+0000","last_change":"2024-09-16T10:25:30.948418+0000","last_active":"2024-09-16T10:26:08.532091+0000","last_peered":"2024-09-16T10:26:08.532091+0000","last_clean":"2024-09-16T10:26:08.532091+0000","last_became_active":"2024-09-16T10:25:30.947760+0000","last_became_peered":"2024-09-16T10:25:30.947760+0000","last_unstale":"2024-09-16T10:26:08.532091+0000","last_undegraded":"2024-09-16T10:26:08.532091+0000","last_fullsized":"2024-09-16T10:26:08.532091+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.237682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263389184,"num_objects":4019,"num_object_clones":0,"num_object_copies":12057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4019,"num_write_kb":257216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":141,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.964520+0000","last_change":"2024-09-16T10:25:02.365887+0000","last_active":"2024-09-16T10:25:55.964520+0000","last_peered":"2024-09-16T10:25:55.964520+0000","last_clean":"2024-09-16T10:25:55.964520+0000","last_became_active":"2024-09-16T10:25:00.955466+0000","last_became_peered":"2024-09-16T10:25:00.955466+0000","last_unstale":"2024-09-16T10:25:55.964520+0000","last_undegraded":"2024-09-16T10:25:55.964520+0000","last_fullsized":"2024-09-16T10:25:55.964520+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:32.133909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":694,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393489+0000","last_change":"2024-09-16T10:25:13.649143+0000","last_active":"2024-09-16T10:26:06.393489+0000","last_peered":"2024-09-16T10:26:06.393489+0000","last_clean":"2024-09-16T10:26:06.393489+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:26:06.393489+0000","last_undegraded":"2024-09-16T10:26:06.393489+0000","last_fullsized":"2024-09-16T10:26:06.393489+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:25:13.648981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.648981+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.641189+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":680,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393314+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:26:06.393314+0000","last_peered":"2024-09-16T10:26:06.393314+0000","last_clean":"2024-09-16T10:26:06.393314+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:26:06.393314+0000","last_undegraded":"2024-09-16T10:26:06.393314+0000","last_fullsized":"2024-09-16T10:26:06.393314+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:26.834421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":211,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419651+0000","last_change":"2024-09-16T10:25:10.324756+0000","last_active":"2024-09-16T10:26:06.419651+0000","last_peered":"2024-09-16T10:26:06.419651+0000","last_clean":"2024-09-16T10:26:06.419651+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:26:06.419651+0000","last_undegraded":"2024-09-16T10:26:06.419651+0000","last_fullsized":"2024-09-16T10:26:06.419651+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:25:10.324680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:25:10.324680+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.799924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":207,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419550+0000","last_change":"2024-09-16T10:22:29.905776+0000","last_active":"2024-09-16T10:26:06.419550+0000","last_peered":"2024-09-16T10:26:06.419550+0000","last_clean":"2024-09-16T10:26:06.419550+0000","last_became_active":"2024-09-16T10:22:02.901173+0000","last_became_peered":"2024-09-16T10:22:02.901173+0000","last_unstale":"2024-09-16T10:26:06.419550+0000","last_undegraded":"2024-09-16T10:26:06.419550+0000","last_fullsized":"2024-09-16T10:26:06.419550+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.460922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":68,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402906+0000","last_change":"2024-09-16T10:25:32.463597+0000","last_active":"2024-09-16T10:26:06.402906+0000","last_peered":"2024-09-16T10:26:06.402906+0000","last_clean":"2024-09-16T10:26:06.402906+0000","last_became_active":"2024-09-16T10:25:00.950144+0000","last_became_peered":"2024-09-16T10:25:00.950144+0000","last_unstale":"2024-09-16T10:26:06.402906+0000","last_undegraded":"2024-09-16T10:26:06.402906+0000","last_fullsized":"2024-09-16T10:26:06.402906+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.360271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"374'3507","reported_seq":3542,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:26:04.342850+0000","last_change":"2024-09-16T10:25:30.950233+0000","last_active":"2024-09-16T10:26:04.342850+0000","last_peered":"2024-09-16T10:26:04.342850+0000","last_clean":"2024-09-16T10:26:04.342850+0000","last_became_active":"2024-09-16T10:25:30.948708+0000","last_became_peered":"2024-09-16T10:25:30.948708+0000","last_unstale":"2024-09-16T10:26:04.342850+0000","last_undegraded":"2024-09-16T10:26:04.342850+0000","last_fullsized":"2024-09-16T10:26:04.342850+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:35.650968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229834752,"num_objects":3507,"num_object_clones":0,"num_object_copies":10521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3507,"num_write_kb":224448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":558,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.407751+0000","last_change":"2024-09-16T10:24:27.887030+0000","last_active":"2024-09-16T10:26:06.407751+0000","last_peered":"2024-09-16T10:26:06.407751+0000","last_clean":"2024-09-16T10:26:06.407751+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:26:06.407751+0000","last_undegraded":"2024-09-16T10:26:06.407751+0000","last_fullsized":"2024-09-16T10:26:06.407751+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:27.886961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:27.886961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":768,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963740+0000","last_change":"2024-09-16T10:22:24.213089+0000","last_active":"2024-09-16T10:25:55.963740+0000","last_peered":"2024-09-16T10:25:55.963740+0000","last_clean":"2024-09-16T10:25:55.963740+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:25:55.963740+0000","last_undegraded":"2024-09-16T10:25:55.963740+0000","last_fullsized":"2024-09-16T10:25:55.963740+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:22:24.213008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.213008+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":694,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963322+0000","last_change":"2024-09-16T10:22:26.226703+0000","last_active":"2024-09-16T10:25:55.963322+0000","last_peered":"2024-09-16T10:25:55.963322+0000","last_clean":"2024-09-16T10:25:55.963322+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:25:55.963322+0000","last_undegraded":"2024-09-16T10:25:55.963322+0000","last_fullsized":"2024-09-16T10:25:55.963322+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.226632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.226632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":170,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:07.131333+0000","last_change":"2024-09-16T10:25:00.956734+0000","last_active":"2024-09-16T10:26:07.131333+0000","last_peered":"2024-09-16T10:26:07.131333+0000","last_clean":"2024-09-16T10:26:07.131333+0000","last_became_active":"2024-09-16T10:25:00.956304+0000","last_became_peered":"2024-09-16T10:25:00.956304+0000","last_unstale":"2024-09-16T10:26:07.131333+0000","last_undegraded":"2024-09-16T10:26:07.131333+0000","last_fullsized":"2024-09-16T10:26:07.131333+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:23:41.597682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.597682+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:02.121047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":681,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418607+0000","last_change":"2024-09-16T10:25:35.847648+0000","last_active":"2024-09-16T10:26:06.418607+0000","last_peered":"2024-09-16T10:26:06.418607+0000","last_clean":"2024-09-16T10:26:06.418607+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:26:06.418607+0000","last_undegraded":"2024-09-16T10:26:06.418607+0000","last_fullsized":"2024-09-16T10:26:06.418607+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.847598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.847598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:00.609111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":685,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392855+0000","last_change":"2024-09-16T10:25:35.937305+0000","last_active":"2024-09-16T10:26:06.392855+0000","last_peered":"2024-09-16T10:26:06.392855+0000","last_clean":"2024-09-16T10:26:06.392855+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:26:06.392855+0000","last_undegraded":"2024-09-16T10:26:06.392855+0000","last_fullsized":"2024-09-16T10:26:06.392855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.937257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.937257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.310507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"375'3973","reported_seq":4010,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.343812+0000","last_change":"2024-09-16T10:25:30.622105+0000","last_active":"2024-09-16T10:26:08.343812+0000","last_peered":"2024-09-16T10:26:08.343812+0000","last_clean":"2024-09-16T10:26:08.343812+0000","last_became_active":"2024-09-16T10:25:30.621779+0000","last_became_peered":"2024-09-16T10:25:30.621779+0000","last_unstale":"2024-09-16T10:26:08.343812+0000","last_undegraded":"2024-09-16T10:26:08.343812+0000","last_fullsized":"2024-09-16T10:26:08.343812+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3973,"log_dups_size":0,"ondisk_log_size":3973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.268900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260374528,"num_objects":3973,"num_object_clones":0,"num_object_copies":11919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3973,"num_write_kb":254272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"328'28","reported_seq":851,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.403043+0000","last_change":"2024-09-16T10:20:31.611899+0000","last_active":"2024-09-16T10:26:06.403043+0000","last_peered":"2024-09-16T10:26:06.403043+0000","last_clean":"2024-09-16T10:26:06.403043+0000","last_became_active":"2024-09-16T10:13:39.106095+0000","last_became_peered":"2024-09-16T10:13:39.106095+0000","last_unstale":"2024-09-16T10:26:06.403043+0000","last_undegraded":"2024-09-16T10:26:06.403043+0000","last_fullsized":"2024-09-16T10:26:06.403043+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:26.575852+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":644,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414632+0000","last_change":"2024-09-16T10:21:34.364578+0000","last_active":"2024-09-16T10:26:06.414632+0000","last_peered":"2024-09-16T10:26:06.414632+0000","last_clean":"2024-09-16T10:26:06.414632+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:26:06.414632+0000","last_undegraded":"2024-09-16T10:26:06.414632+0000","last_fullsized":"2024-09-16T10:26:06.414632+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:34.364472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.364472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:46.441650+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":141,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.964327+0000","last_change":"2024-09-16T10:25:01.375800+0000","last_active":"2024-09-16T10:25:55.964327+0000","last_peered":"2024-09-16T10:25:55.964327+0000","last_clean":"2024-09-16T10:25:55.964327+0000","last_became_active":"2024-09-16T10:25:00.955671+0000","last_became_peered":"2024-09-16T10:25:00.955671+0000","last_unstale":"2024-09-16T10:25:55.964327+0000","last_undegraded":"2024-09-16T10:25:55.964327+0000","last_fullsized":"2024-09-16T10:25:55.964327+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.945066+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":540,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418986+0000","last_change":"2024-09-16T10:25:35.591372+0000","last_active":"2024-09-16T10:26:06.418986+0000","last_peered":"2024-09-16T10:26:06.418986+0000","last_clean":"2024-09-16T10:26:06.418986+0000","last_became_active":"2024-09-16T10:25:00.946698+0000","last_became_peered":"2024-09-16T10:25:00.946698+0000","last_unstale":"2024-09-16T10:26:06.418986+0000","last_undegraded":"2024-09-16T10:26:06.418986+0000","last_fullsized":"2024-09-16T10:26:06.418986+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.591323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.194521+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":669,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418436+0000","last_change":"2024-09-16T10:24:33.905860+0000","last_active":"2024-09-16T10:26:06.418436+0000","last_peered":"2024-09-16T10:26:06.418436+0000","last_clean":"2024-09-16T10:26:06.418436+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:26:06.418436+0000","last_undegraded":"2024-09-16T10:26:06.418436+0000","last_fullsized":"2024-09-16T10:26:06.418436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.905785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.905785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"375'3986","reported_seq":4023,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.338639+0000","last_change":"2024-09-16T10:25:30.621715+0000","last_active":"2024-09-16T10:26:08.338639+0000","last_peered":"2024-09-16T10:26:08.338639+0000","last_clean":"2024-09-16T10:26:08.338639+0000","last_became_active":"2024-09-16T10:25:30.621428+0000","last_became_peered":"2024-09-16T10:25:30.621428+0000","last_unstale":"2024-09-16T10:26:08.338639+0000","last_undegraded":"2024-09-16T10:26:08.338639+0000","last_fullsized":"2024-09-16T10:26:08.338639+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:38.872942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261226496,"num_objects":3986,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":255104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":566,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.407506+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:26:06.407506+0000","last_peered":"2024-09-16T10:26:06.407506+0000","last_clean":"2024-09-16T10:26:06.407506+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:26:06.407506+0000","last_undegraded":"2024-09-16T10:26:06.407506+0000","last_fullsized":"2024-09-16T10:26:06.407506+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:00.198762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":726,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.394070+0000","last_change":"2024-09-16T10:24:34.142042+0000","last_active":"2024-09-16T10:26:06.394070+0000","last_peered":"2024-09-16T10:26:06.394070+0000","last_clean":"2024-09-16T10:26:06.394070+0000","last_became_active":"2024-09-16T10:19:09.300186+0000","last_became_peered":"2024-09-16T10:19:09.300186+0000","last_unstale":"2024-09-16T10:26:06.394070+0000","last_undegraded":"2024-09-16T10:26:06.394070+0000","last_fullsized":"2024-09-16T10:26:06.394070+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:24:34.141958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.141958+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":686,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393729+0000","last_change":"2024-09-16T10:22:20.877473+0000","last_active":"2024-09-16T10:26:06.393729+0000","last_peered":"2024-09-16T10:26:06.393729+0000","last_clean":"2024-09-16T10:26:06.393729+0000","last_became_active":"2024-09-16T10:22:02.897952+0000","last_became_peered":"2024-09-16T10:22:02.897952+0000","last_unstale":"2024-09-16T10:26:06.393729+0000","last_undegraded":"2024-09-16T10:26:06.393729+0000","last_fullsized":"2024-09-16T10:26:06.393729+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:44.276665+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":681,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418295+0000","last_change":"2024-09-16T10:25:05.858177+0000","last_active":"2024-09-16T10:26:06.418295+0000","last_peered":"2024-09-16T10:26:06.418295+0000","last_clean":"2024-09-16T10:26:06.418295+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:26:06.418295+0000","last_undegraded":"2024-09-16T10:26:06.418295+0000","last_fullsized":"2024-09-16T10:26:06.418295+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.858129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.858129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:12.657363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":616,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418523+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:26:06.418523+0000","last_peered":"2024-09-16T10:26:06.418523+0000","last_clean":"2024-09-16T10:26:06.418523+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:26:06.418523+0000","last_undegraded":"2024-09-16T10:26:06.418523+0000","last_fullsized":"2024-09-16T10:26:06.418523+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:00.417765+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"375'4108","reported_seq":4145,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.860023+0000","last_change":"2024-09-16T10:25:31.161519+0000","last_active":"2024-09-16T10:26:08.860023+0000","last_peered":"2024-09-16T10:26:08.860023+0000","last_clean":"2024-09-16T10:26:08.860023+0000","last_became_active":"2024-09-16T10:25:31.161181+0000","last_became_peered":"2024-09-16T10:25:31.161181+0000","last_unstale":"2024-09-16T10:26:08.860023+0000","last_undegraded":"2024-09-16T10:26:08.860023+0000","last_fullsized":"2024-09-16T10:26:08.860023+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4108,"log_dups_size":0,"ondisk_log_size":4108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:42.844942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269221888,"num_objects":4108,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":262912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":533,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417587+0000","last_change":"2024-09-16T10:25:04.820847+0000","last_active":"2024-09-16T10:26:06.417587+0000","last_peered":"2024-09-16T10:26:06.417587+0000","last_clean":"2024-09-16T10:26:06.417587+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:26:06.417587+0000","last_undegraded":"2024-09-16T10:26:06.417587+0000","last_fullsized":"2024-09-16T10:26:06.417587+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:04.820775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:04.820775+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.871951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":775,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393591+0000","last_change":"2024-09-16T10:23:09.748114+0000","last_active":"2024-09-16T10:26:06.393591+0000","last_peered":"2024-09-16T10:26:06.393591+0000","last_clean":"2024-09-16T10:26:06.393591+0000","last_became_active":"2024-09-16T10:22:02.898219+0000","last_became_peered":"2024-09-16T10:22:02.898219+0000","last_unstale":"2024-09-16T10:26:06.393591+0000","last_undegraded":"2024-09-16T10:26:06.393591+0000","last_fullsized":"2024-09-16T10:26:06.393591+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-16T10:23:09.748031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:09.748031+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.221819+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":672,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402719+0000","last_change":"2024-09-16T10:21:53.914293+0000","last_active":"2024-09-16T10:26:06.402719+0000","last_peered":"2024-09-16T10:26:06.402719+0000","last_clean":"2024-09-16T10:26:06.402719+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:26:06.402719+0000","last_undegraded":"2024-09-16T10:26:06.402719+0000","last_fullsized":"2024-09-16T10:26:06.402719+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:53.914251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:53.914251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.468841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":665,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392961+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:26:06.392961+0000","last_peered":"2024-09-16T10:26:06.392961+0000","last_clean":"2024-09-16T10:26:06.392961+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:26:06.392961+0000","last_undegraded":"2024-09-16T10:26:06.392961+0000","last_fullsized":"2024-09-16T10:26:06.392961+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.814189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":215,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.420923+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:26:06.420923+0000","last_peered":"2024-09-16T10:26:06.420923+0000","last_clean":"2024-09-16T10:26:06.420923+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:26:06.420923+0000","last_undegraded":"2024-09-16T10:26:06.420923+0000","last_fullsized":"2024-09-16T10:26:06.420923+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.866924+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"375'4021","reported_seq":4058,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:09.005022+0000","last_change":"2024-09-16T10:25:30.613792+0000","last_active":"2024-09-16T10:26:09.005022+0000","last_peered":"2024-09-16T10:26:09.005022+0000","last_clean":"2024-09-16T10:26:09.005022+0000","last_became_active":"2024-09-16T10:25:30.613465+0000","last_became_peered":"2024-09-16T10:25:30.613465+0000","last_unstale":"2024-09-16T10:26:09.005022+0000","last_undegraded":"2024-09-16T10:26:09.005022+0000","last_fullsized":"2024-09-16T10:26:09.005022+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.984379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":257344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":335,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414502+0000","last_change":"2024-09-16T10:25:01.481822+0000","last_active":"2024-09-16T10:26:06.414502+0000","last_peered":"2024-09-16T10:26:06.414502+0000","last_clean":"2024-09-16T10:26:06.414502+0000","last_became_active":"2024-09-16T10:25:00.955829+0000","last_became_peered":"2024-09-16T10:25:00.955829+0000","last_unstale":"2024-09-16T10:26:06.414502+0000","last_undegraded":"2024-09-16T10:26:06.414502+0000","last_fullsized":"2024-09-16T10:26:06.414502+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.481748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.481748+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:13.239898+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":690,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393318+0000","last_change":"2024-09-16T10:21:31.888831+0000","last_active":"2024-09-16T10:26:06.393318+0000","last_peered":"2024-09-16T10:26:06.393318+0000","last_clean":"2024-09-16T10:26:06.393318+0000","last_became_active":"2024-09-16T10:13:39.093563+0000","last_became_peered":"2024-09-16T10:13:39.093563+0000","last_unstale":"2024-09-16T10:26:06.393318+0000","last_undegraded":"2024-09-16T10:26:06.393318+0000","last_fullsized":"2024-09-16T10:26:06.393318+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.213178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":663,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415474+0000","last_change":"2024-09-16T10:24:25.364236+0000","last_active":"2024-09-16T10:26:06.415474+0000","last_peered":"2024-09-16T10:26:06.415474+0000","last_clean":"2024-09-16T10:26:06.415474+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:26:06.415474+0000","last_undegraded":"2024-09-16T10:26:06.415474+0000","last_fullsized":"2024-09-16T10:26:06.415474+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:25.364176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:25.364176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:04.388506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"282'10","reported_seq":261,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419762+0000","last_change":"2024-09-16T10:22:24.799901+0000","last_active":"2024-09-16T10:26:06.419762+0000","last_peered":"2024-09-16T10:26:06.419762+0000","last_clean":"2024-09-16T10:26:06.419762+0000","last_became_active":"2024-09-16T10:22:02.902992+0000","last_became_peered":"2024-09-16T10:22:02.902992+0000","last_unstale":"2024-09-16T10:26:06.419762+0000","last_undegraded":"2024-09-16T10:26:06.419762+0000","last_fullsized":"2024-09-16T10:26:06.419762+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.520234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":690,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393979+0000","last_change":"2024-09-16T10:25:09.348134+0000","last_active":"2024-09-16T10:26:06.393979+0000","last_peered":"2024-09-16T10:26:06.393979+0000","last_clean":"2024-09-16T10:26:06.393979+0000","last_became_active":"2024-09-16T10:22:02.899113+0000","last_became_peered":"2024-09-16T10:22:02.899113+0000","last_unstale":"2024-09-16T10:26:06.393979+0000","last_undegraded":"2024-09-16T10:26:06.393979+0000","last_fullsized":"2024-09-16T10:26:06.393979+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:09.348090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.348090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.427158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"375'3930","reported_seq":3967,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.860884+0000","last_change":"2024-09-16T10:25:31.161545+0000","last_active":"2024-09-16T10:26:08.860884+0000","last_peered":"2024-09-16T10:26:08.860884+0000","last_clean":"2024-09-16T10:26:08.860884+0000","last_became_active":"2024-09-16T10:25:31.161230+0000","last_became_peered":"2024-09-16T10:25:31.161230+0000","last_unstale":"2024-09-16T10:26:08.860884+0000","last_undegraded":"2024-09-16T10:26:08.860884+0000","last_fullsized":"2024-09-16T10:26:08.860884+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.886610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257556480,"num_objects":3930,"num_object_clones":0,"num_object_copies":11790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":251520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":254,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:07.404613+0000","last_change":"2024-09-16T10:26:07.404613+0000","last_active":"2024-09-16T10:26:07.404613+0000","last_peered":"2024-09-16T10:26:07.404613+0000","last_clean":"2024-09-16T10:26:07.404613+0000","last_became_active":"2024-09-16T10:23:31.480326+0000","last_became_peered":"2024-09-16T10:23:31.480326+0000","last_unstale":"2024-09-16T10:26:07.404613+0000","last_undegraded":"2024-09-16T10:26:07.404613+0000","last_fullsized":"2024-09-16T10:26:07.404613+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:07.404565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:07.404565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:32.423254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":691,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.966721+0000","last_change":"2024-09-16T10:25:16.446472+0000","last_active":"2024-09-16T10:25:55.966721+0000","last_peered":"2024-09-16T10:25:55.966721+0000","last_clean":"2024-09-16T10:25:55.966721+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:25:55.966721+0000","last_undegraded":"2024-09-16T10:25:55.966721+0000","last_fullsized":"2024-09-16T10:25:55.966721+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:25:16.446381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.446381+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:31.794942+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":689,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419002+0000","last_change":"2024-09-16T10:25:08.916850+0000","last_active":"2024-09-16T10:26:06.419002+0000","last_peered":"2024-09-16T10:26:06.419002+0000","last_clean":"2024-09-16T10:26:06.419002+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:26:06.419002+0000","last_undegraded":"2024-09-16T10:26:06.419002+0000","last_fullsized":"2024-09-16T10:26:06.419002+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.916778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.916778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:14.226842+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":207,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419005+0000","last_change":"2024-09-16T10:25:09.346506+0000","last_active":"2024-09-16T10:26:06.419005+0000","last_peered":"2024-09-16T10:26:06.419005+0000","last_clean":"2024-09-16T10:26:06.419005+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:26:06.419005+0000","last_undegraded":"2024-09-16T10:26:06.419005+0000","last_fullsized":"2024-09-16T10:26:06.419005+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:09.346437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.346437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.598246+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":695,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963225+0000","last_change":"2024-09-16T10:22:02.911912+0000","last_active":"2024-09-16T10:25:55.963225+0000","last_peered":"2024-09-16T10:25:55.963225+0000","last_clean":"2024-09-16T10:25:55.963225+0000","last_became_active":"2024-09-16T10:22:02.896680+0000","last_became_peered":"2024-09-16T10:22:02.896680+0000","last_unstale":"2024-09-16T10:25:55.963225+0000","last_undegraded":"2024-09-16T10:25:55.963225+0000","last_fullsized":"2024-09-16T10:25:55.963225+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:18:05.103993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:18:05.103993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.267128+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"374'3698","reported_seq":3732,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:26:04.364498+0000","last_change":"2024-09-16T10:25:31.162408+0000","last_active":"2024-09-16T10:26:04.364498+0000","last_peered":"2024-09-16T10:26:04.364498+0000","last_clean":"2024-09-16T10:26:04.364498+0000","last_became_active":"2024-09-16T10:25:31.162120+0000","last_became_peered":"2024-09-16T10:25:31.162120+0000","last_unstale":"2024-09-16T10:26:04.364498+0000","last_undegraded":"2024-09-16T10:26:04.364498+0000","last_fullsized":"2024-09-16T10:26:04.364498+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":3698,"log_dups_size":0,"ondisk_log_size":3698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.047456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242352128,"num_objects":3698,"num_object_clones":0,"num_object_copies":11094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":236672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":645,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.403313+0000","last_change":"2024-09-16T10:25:40.594525+0000","last_active":"2024-09-16T10:26:06.403313+0000","last_peered":"2024-09-16T10:26:06.403313+0000","last_clean":"2024-09-16T10:26:06.403313+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:26:06.403313+0000","last_undegraded":"2024-09-16T10:26:06.403313+0000","last_fullsized":"2024-09-16T10:26:06.403313+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:40.594454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.594454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:45.283077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":747,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402481+0000","last_change":"2024-09-16T10:25:35.488583+0000","last_active":"2024-09-16T10:26:06.402481+0000","last_peered":"2024-09-16T10:26:06.402481+0000","last_clean":"2024-09-16T10:26:06.402481+0000","last_became_active":"2024-09-16T10:13:39.105021+0000","last_became_peered":"2024-09-16T10:13:39.105021+0000","last_unstale":"2024-09-16T10:26:06.402481+0000","last_undegraded":"2024-09-16T10:26:06.402481+0000","last_fullsized":"2024-09-16T10:26:06.402481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:40.463926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":59,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418780+0000","last_change":"2024-09-16T10:25:00.947814+0000","last_active":"2024-09-16T10:26:06.418780+0000","last_peered":"2024-09-16T10:26:06.418780+0000","last_clean":"2024-09-16T10:26:06.418780+0000","last_became_active":"2024-09-16T10:25:00.947502+0000","last_became_peered":"2024-09-16T10:25:00.947502+0000","last_unstale":"2024-09-16T10:26:06.418780+0000","last_undegraded":"2024-09-16T10:26:06.418780+0000","last_fullsized":"2024-09-16T10:26:06.418780+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:33.528600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:33.528600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":207,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419862+0000","last_change":"2024-09-16T10:22:26.818522+0000","last_active":"2024-09-16T10:26:06.419862+0000","last_peered":"2024-09-16T10:26:06.419862+0000","last_clean":"2024-09-16T10:26:06.419862+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:26:06.419862+0000","last_undegraded":"2024-09-16T10:26:06.419862+0000","last_fullsized":"2024-09-16T10:26:06.419862+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:26.818439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:26.818439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.481964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":544,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419949+0000","last_change":"2024-09-16T10:25:34.550982+0000","last_active":"2024-09-16T10:26:06.419949+0000","last_peered":"2024-09-16T10:26:06.419949+0000","last_clean":"2024-09-16T10:26:06.419949+0000","last_became_active":"2024-09-16T10:25:00.954013+0000","last_became_peered":"2024-09-16T10:25:00.954013+0000","last_unstale":"2024-09-16T10:26:06.419949+0000","last_undegraded":"2024-09-16T10:26:06.419949+0000","last_fullsized":"2024-09-16T10:26:06.419949+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:34.550928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:34.550928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:04.254803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"375'4077","reported_seq":4114,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.352338+0000","last_change":"2024-09-16T10:25:31.163086+0000","last_active":"2024-09-16T10:26:08.352338+0000","last_peered":"2024-09-16T10:26:08.352338+0000","last_clean":"2024-09-16T10:26:08.352338+0000","last_became_active":"2024-09-16T10:25:31.162769+0000","last_became_peered":"2024-09-16T10:25:31.162769+0000","last_unstale":"2024-09-16T10:26:08.352338+0000","last_undegraded":"2024-09-16T10:26:08.352338+0000","last_fullsized":"2024-09-16T10:26:08.352338+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:43.363042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267190272,"num_objects":4077,"num_object_clones":0,"num_object_copies":12231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":260928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":588,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417893+0000","last_change":"2024-09-16T10:24:31.490967+0000","last_active":"2024-09-16T10:26:06.417893+0000","last_peered":"2024-09-16T10:26:06.417893+0000","last_clean":"2024-09-16T10:26:06.417893+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:26:06.417893+0000","last_undegraded":"2024-09-16T10:26:06.417893+0000","last_fullsized":"2024-09-16T10:26:06.417893+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.490890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.490890+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":700,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392889+0000","last_change":"2024-09-16T10:23:41.811222+0000","last_active":"2024-09-16T10:26:06.392889+0000","last_peered":"2024-09-16T10:26:06.392889+0000","last_clean":"2024-09-16T10:26:06.392889+0000","last_became_active":"2024-09-16T10:13:39.095254+0000","last_became_peered":"2024-09-16T10:13:39.095254+0000","last_unstale":"2024-09-16T10:26:06.392889+0000","last_undegraded":"2024-09-16T10:26:06.392889+0000","last_fullsized":"2024-09-16T10:26:06.392889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:41.811140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.811140+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.191555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415618+0000","last_change":"2024-09-16T10:25:41.990922+0000","last_active":"2024-09-16T10:26:06.415618+0000","last_peered":"2024-09-16T10:26:06.415618+0000","last_clean":"2024-09-16T10:26:06.415618+0000","last_became_active":"2024-09-16T10:25:41.990483+0000","last_became_peered":"2024-09-16T10:25:41.990483+0000","last_unstale":"2024-09-16T10:26:06.415618+0000","last_undegraded":"2024-09-16T10:26:06.415618+0000","last_fullsized":"2024-09-16T10:26:06.415618+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.405338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.405338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:27.926501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":207,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.420042+0000","last_change":"2024-09-16T10:22:03.705654+0000","last_active":"2024-09-16T10:26:06.420042+0000","last_peered":"2024-09-16T10:26:06.420042+0000","last_clean":"2024-09-16T10:26:06.420042+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:26:06.420042+0000","last_undegraded":"2024-09-16T10:26:06.420042+0000","last_fullsized":"2024-09-16T10:26:06.420042+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:03.705604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:03.705604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.415920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414426+0000","last_change":"2024-09-16T10:25:41.478330+0000","last_active":"2024-09-16T10:26:06.414426+0000","last_peered":"2024-09-16T10:26:06.414426+0000","last_clean":"2024-09-16T10:26:06.414426+0000","last_became_active":"2024-09-16T10:13:43.125512+0000","last_became_peered":"2024-09-16T10:13:43.125512+0000","last_unstale":"2024-09-16T10:26:06.414426+0000","last_undegraded":"2024-09-16T10:26:06.414426+0000","last_fullsized":"2024-09-16T10:26:06.414426+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:41.478267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:41.478267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:06.996051+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"375'4025","reported_seq":4062,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.539937+0000","last_change":"2024-09-16T10:25:30.621276+0000","last_active":"2024-09-16T10:26:08.539937+0000","last_peered":"2024-09-16T10:26:08.539937+0000","last_clean":"2024-09-16T10:26:08.539937+0000","last_became_active":"2024-09-16T10:25:30.620971+0000","last_became_peered":"2024-09-16T10:25:30.620971+0000","last_unstale":"2024-09-16T10:26:08.539937+0000","last_undegraded":"2024-09-16T10:26:08.539937+0000","last_fullsized":"2024-09-16T10:26:08.539937+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4025,"log_dups_size":0,"ondisk_log_size":4025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:55.992526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263782400,"num_objects":4025,"num_object_clones":0,"num_object_copies":12075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":257600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":485,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393453+0000","last_change":"2024-09-16T10:25:01.368064+0000","last_active":"2024-09-16T10:26:06.393453+0000","last_peered":"2024-09-16T10:26:06.393453+0000","last_clean":"2024-09-16T10:26:06.393453+0000","last_became_active":"2024-09-16T10:25:00.960443+0000","last_became_peered":"2024-09-16T10:25:00.960443+0000","last_unstale":"2024-09-16T10:26:06.393453+0000","last_undegraded":"2024-09-16T10:26:06.393453+0000","last_fullsized":"2024-09-16T10:26:06.393453+0000","mapping_epoch":351,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.367979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.367979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:11.835419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":704,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417697+0000","last_change":"2024-09-16T10:24:36.805155+0000","last_active":"2024-09-16T10:26:06.417697+0000","last_peered":"2024-09-16T10:26:06.417697+0000","last_clean":"2024-09-16T10:26:06.417697+0000","last_became_active":"2024-09-16T10:13:39.104400+0000","last_became_peered":"2024-09-16T10:13:39.104400+0000","last_unstale":"2024-09-16T10:26:06.417697+0000","last_undegraded":"2024-09-16T10:26:06.417697+0000","last_fullsized":"2024-09-16T10:26:06.417697+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:24:36.805084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:36.805084+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":69,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393850+0000","last_change":"2024-09-16T10:25:00.952412+0000","last_active":"2024-09-16T10:26:06.393850+0000","last_peered":"2024-09-16T10:26:06.393850+0000","last_clean":"2024-09-16T10:26:06.393850+0000","last_became_active":"2024-09-16T10:25:00.952061+0000","last_became_peered":"2024-09-16T10:25:00.952061+0000","last_unstale":"2024-09-16T10:26:06.393850+0000","last_undegraded":"2024-09-16T10:26:06.393850+0000","last_fullsized":"2024-09-16T10:26:06.393850+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:24:24.486825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:24.486825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"352'9","reported_seq":107,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402324+0000","last_change":"2024-09-16T10:25:37.549535+0000","last_active":"2024-09-16T10:26:06.402324+0000","last_peered":"2024-09-16T10:26:06.402324+0000","last_clean":"2024-09-16T10:26:06.402324+0000","last_became_active":"2024-09-16T10:25:00.946371+0000","last_became_peered":"2024-09-16T10:25:00.946371+0000","last_unstale":"2024-09-16T10:26:06.402324+0000","last_undegraded":"2024-09-16T10:26:06.402324+0000","last_fullsized":"2024-09-16T10:26:06.402324+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'9","last_scrub_stamp":"2024-09-16T10:25:37.549468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.549468+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:39.799942+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":677,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417839+0000","last_change":"2024-09-16T10:24:34.863064+0000","last_active":"2024-09-16T10:26:06.417839+0000","last_peered":"2024-09-16T10:26:06.417839+0000","last_clean":"2024-09-16T10:26:06.417839+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:26:06.417839+0000","last_undegraded":"2024-09-16T10:26:06.417839+0000","last_fullsized":"2024-09-16T10:26:06.417839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:34.862992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:34.862992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:02.994243+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":253,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418089+0000","last_change":"2024-09-16T10:25:00.825815+0000","last_active":"2024-09-16T10:26:06.418089+0000","last_peered":"2024-09-16T10:26:06.418089+0000","last_clean":"2024-09-16T10:26:06.418089+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:26:06.418089+0000","last_undegraded":"2024-09-16T10:26:06.418089+0000","last_fullsized":"2024-09-16T10:26:06.418089+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:00.825743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:00.825743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:11.827743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":664,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.403428+0000","last_change":"2024-09-16T10:19:17.168809+0000","last_active":"2024-09-16T10:26:06.403428+0000","last_peered":"2024-09-16T10:26:06.403428+0000","last_clean":"2024-09-16T10:26:06.403428+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:26:06.403428+0000","last_undegraded":"2024-09-16T10:26:06.403428+0000","last_fullsized":"2024-09-16T10:26:06.403428+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:17.168734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:17.168734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:19.558479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":685,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.416577+0000","last_change":"2024-09-16T10:17:38.202459+0000","last_active":"2024-09-16T10:26:06.416577+0000","last_peered":"2024-09-16T10:26:06.416577+0000","last_clean":"2024-09-16T10:26:06.416577+0000","last_became_active":"2024-09-16T10:13:39.107604+0000","last_became_peered":"2024-09-16T10:13:39.107604+0000","last_unstale":"2024-09-16T10:26:06.416577+0000","last_undegraded":"2024-09-16T10:26:06.416577+0000","last_fullsized":"2024-09-16T10:26:06.416577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:58.436548+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":665,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418305+0000","last_change":"2024-09-16T10:22:01.997512+0000","last_active":"2024-09-16T10:26:06.418305+0000","last_peered":"2024-09-16T10:26:06.418305+0000","last_clean":"2024-09-16T10:26:06.418305+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:26:06.418305+0000","last_undegraded":"2024-09-16T10:26:06.418305+0000","last_fullsized":"2024-09-16T10:26:06.418305+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.997458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.997458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:45.836621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":660,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.412956+0000","last_change":"2024-09-16T10:25:11.324922+0000","last_active":"2024-09-16T10:26:06.412956+0000","last_peered":"2024-09-16T10:26:06.412956+0000","last_clean":"2024-09-16T10:26:06.412956+0000","last_became_active":"2024-09-16T10:13:43.283358+0000","last_became_peered":"2024-09-16T10:13:43.283358+0000","last_unstale":"2024-09-16T10:26:06.412956+0000","last_undegraded":"2024-09-16T10:26:06.412956+0000","last_fullsized":"2024-09-16T10:26:06.412956+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:33.037219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":575,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.407606+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:26:06.407606+0000","last_peered":"2024-09-16T10:26:06.407606+0000","last_clean":"2024-09-16T10:26:06.407606+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:26:06.407606+0000","last_undegraded":"2024-09-16T10:26:06.407606+0000","last_fullsized":"2024-09-16T10:26:06.407606+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:05.174002+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":691,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963851+0000","last_change":"2024-09-16T10:18:17.167770+0000","last_active":"2024-09-16T10:25:55.963851+0000","last_peered":"2024-09-16T10:25:55.963851+0000","last_clean":"2024-09-16T10:25:55.963851+0000","last_became_active":"2024-09-16T10:13:39.107620+0000","last_became_peered":"2024-09-16T10:13:39.107620+0000","last_unstale":"2024-09-16T10:25:55.963851+0000","last_undegraded":"2024-09-16T10:25:55.963851+0000","last_fullsized":"2024-09-16T10:25:55.963851+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:18:17.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:18:17.167121+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.299672+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392774+0000","last_change":"2024-09-16T10:25:00.944900+0000","last_active":"2024-09-16T10:26:06.392774+0000","last_peered":"2024-09-16T10:26:06.392774+0000","last_clean":"2024-09-16T10:26:06.392774+0000","last_became_active":"2024-09-16T10:25:00.944574+0000","last_became_peered":"2024-09-16T10:25:00.944574+0000","last_unstale":"2024-09-16T10:26:06.392774+0000","last_undegraded":"2024-09-16T10:26:06.392774+0000","last_fullsized":"2024-09-16T10:26:06.392774+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:37.608524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.608524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:56.262148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":663,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.416278+0000","last_change":"2024-09-16T10:22:20.500717+0000","last_active":"2024-09-16T10:26:06.416278+0000","last_peered":"2024-09-16T10:26:06.416278+0000","last_clean":"2024-09-16T10:26:06.416278+0000","last_became_active":"2024-09-16T10:13:41.118595+0000","last_became_peered":"2024-09-16T10:13:41.118595+0000","last_unstale":"2024-09-16T10:26:06.416278+0000","last_undegraded":"2024-09-16T10:26:06.416278+0000","last_fullsized":"2024-09-16T10:26:06.416278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.500502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.500502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:20.469447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":207,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.419249+0000","last_change":"2024-09-16T10:25:13.357255+0000","last_active":"2024-09-16T10:26:06.419249+0000","last_peered":"2024-09-16T10:26:06.419249+0000","last_clean":"2024-09-16T10:26:06.419249+0000","last_became_active":"2024-09-16T10:22:02.902845+0000","last_became_peered":"2024-09-16T10:22:02.902845+0000","last_unstale":"2024-09-16T10:26:06.419249+0000","last_undegraded":"2024-09-16T10:26:06.419249+0000","last_fullsized":"2024-09-16T10:26:06.419249+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:37.501033+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":244,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.417571+0000","last_change":"2024-09-16T10:25:09.961140+0000","last_active":"2024-09-16T10:26:06.417571+0000","last_peered":"2024-09-16T10:26:06.417571+0000","last_clean":"2024-09-16T10:26:06.417571+0000","last_became_active":"2024-09-16T10:23:00.902731+0000","last_became_peered":"2024-09-16T10:23:00.902731+0000","last_unstale":"2024-09-16T10:26:06.417571+0000","last_undegraded":"2024-09-16T10:26:06.417571+0000","last_fullsized":"2024-09-16T10:26:06.417571+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:09.961098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.961098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:18.332815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":686,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418869+0000","last_change":"2024-09-16T10:16:33.313089+0000","last_active":"2024-09-16T10:26:06.418869+0000","last_peered":"2024-09-16T10:26:06.418869+0000","last_clean":"2024-09-16T10:26:06.418869+0000","last_became_active":"2024-09-16T10:13:39.105608+0000","last_became_peered":"2024-09-16T10:13:39.105608+0000","last_unstale":"2024-09-16T10:26:06.418869+0000","last_undegraded":"2024-09-16T10:26:06.418869+0000","last_fullsized":"2024-09-16T10:26:06.418869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:57.013506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":678,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.964643+0000","last_change":"2024-09-16T10:21:58.224572+0000","last_active":"2024-09-16T10:25:55.964643+0000","last_peered":"2024-09-16T10:25:55.964643+0000","last_clean":"2024-09-16T10:25:55.964643+0000","last_became_active":"2024-09-16T10:13:37.097207+0000","last_became_peered":"2024-09-16T10:13:37.097207+0000","last_unstale":"2024-09-16T10:25:55.964643+0000","last_undegraded":"2024-09-16T10:25:55.964643+0000","last_fullsized":"2024-09-16T10:25:55.964643+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:52.557932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"282'10","reported_seq":868,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418475+0000","last_change":"2024-09-16T10:25:16.464789+0000","last_active":"2024-09-16T10:26:06.418475+0000","last_peered":"2024-09-16T10:26:06.418475+0000","last_clean":"2024-09-16T10:26:06.418475+0000","last_became_active":"2024-09-16T10:22:02.911369+0000","last_became_peered":"2024-09-16T10:22:02.911369+0000","last_unstale":"2024-09-16T10:26:06.418475+0000","last_undegraded":"2024-09-16T10:26:06.418475+0000","last_fullsized":"2024-09-16T10:26:06.418475+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:25.206981+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":690,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393618+0000","last_change":"2024-09-16T10:23:38.791832+0000","last_active":"2024-09-16T10:26:06.393618+0000","last_peered":"2024-09-16T10:26:06.393618+0000","last_clean":"2024-09-16T10:26:06.393618+0000","last_became_active":"2024-09-16T10:22:02.898028+0000","last_became_peered":"2024-09-16T10:22:02.898028+0000","last_unstale":"2024-09-16T10:26:06.393618+0000","last_undegraded":"2024-09-16T10:26:06.393618+0000","last_fullsized":"2024-09-16T10:26:06.393618+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:38.791750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:38.791750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:23.207250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":248,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963455+0000","last_change":"2024-09-16T10:24:32.496216+0000","last_active":"2024-09-16T10:25:55.963455+0000","last_peered":"2024-09-16T10:25:55.963455+0000","last_clean":"2024-09-16T10:25:55.963455+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:25:55.963455+0000","last_undegraded":"2024-09-16T10:25:55.963455+0000","last_fullsized":"2024-09-16T10:25:55.963455+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.495461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.495461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":713,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418118+0000","last_change":"2024-09-16T10:25:08.470676+0000","last_active":"2024-09-16T10:26:06.418118+0000","last_peered":"2024-09-16T10:26:06.418118+0000","last_clean":"2024-09-16T10:26:06.418118+0000","last_became_active":"2024-09-16T10:13:39.105240+0000","last_became_peered":"2024-09-16T10:13:39.105240+0000","last_unstale":"2024-09-16T10:26:06.418118+0000","last_undegraded":"2024-09-16T10:26:06.418118+0000","last_fullsized":"2024-09-16T10:26:06.418118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:25:08.470600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.470600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:17.849780+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":686,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.412470+0000","last_change":"2024-09-16T10:24:23.469834+0000","last_active":"2024-09-16T10:26:06.412470+0000","last_peered":"2024-09-16T10:26:06.412470+0000","last_clean":"2024-09-16T10:26:06.412470+0000","last_became_active":"2024-09-16T10:22:02.890757+0000","last_became_peered":"2024-09-16T10:22:02.890757+0000","last_unstale":"2024-09-16T10:26:06.412470+0000","last_undegraded":"2024-09-16T10:26:06.412470+0000","last_fullsized":"2024-09-16T10:26:06.412470+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:23.469779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.469779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:53.162820+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":832,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418735+0000","last_change":"2024-09-16T10:24:26.942679+0000","last_active":"2024-09-16T10:26:06.418735+0000","last_peered":"2024-09-16T10:26:06.418735+0000","last_clean":"2024-09-16T10:26:06.418735+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:26:06.418735+0000","last_undegraded":"2024-09-16T10:26:06.418735+0000","last_fullsized":"2024-09-16T10:26:06.418735+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:24:26.942604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.942604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":658,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963287+0000","last_change":"2024-09-16T10:21:20.127762+0000","last_active":"2024-09-16T10:25:55.963287+0000","last_peered":"2024-09-16T10:25:55.963287+0000","last_clean":"2024-09-16T10:25:55.963287+0000","last_became_active":"2024-09-16T10:13:43.120577+0000","last_became_peered":"2024-09-16T10:13:43.120577+0000","last_unstale":"2024-09-16T10:25:55.963287+0000","last_undegraded":"2024-09-16T10:25:55.963287+0000","last_fullsized":"2024-09-16T10:25:55.963287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:47.474897+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":718,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392954+0000","last_change":"2024-09-16T10:25:17.300226+0000","last_active":"2024-09-16T10:26:06.392954+0000","last_peered":"2024-09-16T10:26:06.392954+0000","last_clean":"2024-09-16T10:26:06.392954+0000","last_became_active":"2024-09-16T10:22:10.869891+0000","last_became_peered":"2024-09-16T10:22:10.869891+0000","last_unstale":"2024-09-16T10:26:06.392954+0000","last_undegraded":"2024-09-16T10:26:06.392954+0000","last_fullsized":"2024-09-16T10:26:06.392954+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:17.300150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.300150+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:42.406783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":131,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.392749+0000","last_change":"2024-09-16T10:25:08.657378+0000","last_active":"2024-09-16T10:26:06.392749+0000","last_peered":"2024-09-16T10:26:06.392749+0000","last_clean":"2024-09-16T10:26:06.392749+0000","last_became_active":"2024-09-16T10:23:19.734145+0000","last_became_peered":"2024-09-16T10:23:19.734145+0000","last_unstale":"2024-09-16T10:26:06.392749+0000","last_undegraded":"2024-09-16T10:26:06.392749+0000","last_fullsized":"2024-09-16T10:26:06.392749+0000","mapping_epoch":319,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.565118+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":673,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.412399+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:26:06.412399+0000","last_peered":"2024-09-16T10:26:06.412399+0000","last_clean":"2024-09-16T10:26:06.412399+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:26:06.412399+0000","last_undegraded":"2024-09-16T10:26:06.412399+0000","last_fullsized":"2024-09-16T10:26:06.412399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.630109+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":659,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414291+0000","last_change":"2024-09-16T10:24:57.507342+0000","last_active":"2024-09-16T10:26:06.414291+0000","last_peered":"2024-09-16T10:26:06.414291+0000","last_clean":"2024-09-16T10:26:06.414291+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:26:06.414291+0000","last_undegraded":"2024-09-16T10:26:06.414291+0000","last_fullsized":"2024-09-16T10:26:06.414291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:57.507281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:57.507281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":728,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.412725+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:26:06.412725+0000","last_peered":"2024-09-16T10:26:06.412725+0000","last_clean":"2024-09-16T10:26:06.412725+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:26:06.412725+0000","last_undegraded":"2024-09-16T10:26:06.412725+0000","last_fullsized":"2024-09-16T10:26:06.412725+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:24.824736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":58,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414885+0000","last_change":"2024-09-16T10:25:00.955337+0000","last_active":"2024-09-16T10:26:06.414885+0000","last_peered":"2024-09-16T10:26:06.414885+0000","last_clean":"2024-09-16T10:26:06.414885+0000","last_became_active":"2024-09-16T10:25:00.954747+0000","last_became_peered":"2024-09-16T10:25:00.954747+0000","last_unstale":"2024-09-16T10:26:06.414885+0000","last_undegraded":"2024-09-16T10:26:06.414885+0000","last_fullsized":"2024-09-16T10:26:06.414885+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:25.105215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:25.105215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:21.465171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":636,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414596+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:26:06.414596+0000","last_peered":"2024-09-16T10:26:06.414596+0000","last_clean":"2024-09-16T10:26:06.414596+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:26:06.414596+0000","last_undegraded":"2024-09-16T10:26:06.414596+0000","last_fullsized":"2024-09-16T10:26:06.414596+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.437165+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":650,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963316+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:25:55.963316+0000","last_peered":"2024-09-16T10:25:55.963316+0000","last_clean":"2024-09-16T10:25:55.963316+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:25:55.963316+0000","last_undegraded":"2024-09-16T10:25:55.963316+0000","last_fullsized":"2024-09-16T10:25:55.963316+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:29.416346+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":580,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418750+0000","last_change":"2024-09-16T10:24:32.986890+0000","last_active":"2024-09-16T10:26:06.418750+0000","last_peered":"2024-09-16T10:26:06.418750+0000","last_clean":"2024-09-16T10:26:06.418750+0000","last_became_active":"2024-09-16T10:23:31.473535+0000","last_became_peered":"2024-09-16T10:23:31.473535+0000","last_unstale":"2024-09-16T10:26:06.418750+0000","last_undegraded":"2024-09-16T10:26:06.418750+0000","last_fullsized":"2024-09-16T10:26:06.418750+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.984796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.984796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:52.445768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":682,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.414816+0000","last_change":"2024-09-16T10:19:12.855438+0000","last_active":"2024-09-16T10:26:06.414816+0000","last_peered":"2024-09-16T10:26:06.414816+0000","last_clean":"2024-09-16T10:26:06.414816+0000","last_became_active":"2024-09-16T10:13:39.104949+0000","last_became_peered":"2024-09-16T10:13:39.104949+0000","last_unstale":"2024-09-16T10:26:06.414816+0000","last_undegraded":"2024-09-16T10:26:06.414816+0000","last_fullsized":"2024-09-16T10:26:06.414816+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:59.444416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":681,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418353+0000","last_change":"2024-09-16T10:23:13.803633+0000","last_active":"2024-09-16T10:26:06.418353+0000","last_peered":"2024-09-16T10:26:06.418353+0000","last_clean":"2024-09-16T10:26:06.418353+0000","last_became_active":"2024-09-16T10:13:37.088142+0000","last_became_peered":"2024-09-16T10:13:37.088142+0000","last_unstale":"2024-09-16T10:26:06.418353+0000","last_undegraded":"2024-09-16T10:26:06.418353+0000","last_fullsized":"2024-09-16T10:26:06.418353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:13.803568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.803568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:54.949386+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":672,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.411700+0000","last_change":"2024-09-16T10:20:58.242437+0000","last_active":"2024-09-16T10:26:06.411700+0000","last_peered":"2024-09-16T10:26:06.411700+0000","last_clean":"2024-09-16T10:26:06.411700+0000","last_became_active":"2024-09-16T10:13:41.112547+0000","last_became_peered":"2024-09-16T10:13:41.112547+0000","last_unstale":"2024-09-16T10:26:06.411700+0000","last_undegraded":"2024-09-16T10:26:06.411700+0000","last_fullsized":"2024-09-16T10:26:06.411700+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:54.173293+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415373+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:26:06.415373+0000","last_peered":"2024-09-16T10:26:06.415373+0000","last_clean":"2024-09-16T10:26:06.415373+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:26:06.415373+0000","last_undegraded":"2024-09-16T10:26:06.415373+0000","last_fullsized":"2024-09-16T10:26:06.415373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:30.383093+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":721,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415319+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:26:06.415319+0000","last_peered":"2024-09-16T10:26:06.415319+0000","last_clean":"2024-09-16T10:26:06.415319+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:26:06.415319+0000","last_undegraded":"2024-09-16T10:26:06.415319+0000","last_fullsized":"2024-09-16T10:26:06.415319+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:05.043608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":668,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963864+0000","last_change":"2024-09-16T10:18:15.161069+0000","last_active":"2024-09-16T10:25:55.963864+0000","last_peered":"2024-09-16T10:25:55.963864+0000","last_clean":"2024-09-16T10:25:55.963864+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:25:55.963864+0000","last_undegraded":"2024-09-16T10:25:55.963864+0000","last_fullsized":"2024-09-16T10:25:55.963864+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:18:15.160223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:18:15.160223+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:23.300677+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":672,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.402389+0000","last_change":"2024-09-16T10:25:12.539167+0000","last_active":"2024-09-16T10:26:06.402389+0000","last_peered":"2024-09-16T10:26:06.402389+0000","last_clean":"2024-09-16T10:26:06.402389+0000","last_became_active":"2024-09-16T10:13:41.114159+0000","last_became_peered":"2024-09-16T10:13:41.114159+0000","last_unstale":"2024-09-16T10:26:06.402389+0000","last_undegraded":"2024-09-16T10:26:06.402389+0000","last_fullsized":"2024-09-16T10:26:06.402389+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:12.539106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.539106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:36.533748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":291,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.418840+0000","last_change":"2024-09-16T10:25:00.946850+0000","last_active":"2024-09-16T10:26:06.418840+0000","last_peered":"2024-09-16T10:26:06.418840+0000","last_clean":"2024-09-16T10:26:06.418840+0000","last_became_active":"2024-09-16T10:25:00.946511+0000","last_became_peered":"2024-09-16T10:25:00.946511+0000","last_unstale":"2024-09-16T10:26:06.418840+0000","last_undegraded":"2024-09-16T10:26:06.418840+0000","last_fullsized":"2024-09-16T10:26:06.418840+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:26.432557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.432557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:49.457202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":660,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.393183+0000","last_change":"2024-09-16T10:19:20.641523+0000","last_active":"2024-09-16T10:26:06.393183+0000","last_peered":"2024-09-16T10:26:06.393183+0000","last_clean":"2024-09-16T10:26:06.393183+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:26:06.393183+0000","last_undegraded":"2024-09-16T10:26:06.393183+0000","last_fullsized":"2024-09-16T10:26:06.393183+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:20.641432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:20.641432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:22.811471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":679,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.415461+0000","last_change":"2024-09-16T10:25:02.446887+0000","last_active":"2024-09-16T10:26:06.415461+0000","last_peered":"2024-09-16T10:26:06.415461+0000","last_clean":"2024-09-16T10:26:06.415461+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:26:06.415461+0000","last_undegraded":"2024-09-16T10:26:06.415461+0000","last_fullsized":"2024-09-16T10:26:06.415461+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:02.446742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.446742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":716,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-16T10:26:06.403384+0000","last_change":"2024-09-16T10:25:18.564978+0000","last_active":"2024-09-16T10:26:06.403384+0000","last_peered":"2024-09-16T10:26:06.403384+0000","last_clean":"2024-09-16T10:26:06.403384+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:26:06.403384+0000","last_undegraded":"2024-09-16T10:26:06.403384+0000","last_fullsized":"2024-09-16T10:26:06.403384+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:18.564894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.564894+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:34.312714+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":677,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.963388+0000","last_change":"2024-09-16T10:22:02.901480+0000","last_active":"2024-09-16T10:25:55.963388+0000","last_peered":"2024-09-16T10:25:55.963388+0000","last_clean":"2024-09-16T10:25:55.963388+0000","last_became_active":"2024-09-16T10:22:02.901129+0000","last_became_peered":"2024-09-16T10:22:02.901129+0000","last_unstale":"2024-09-16T10:25:55.963388+0000","last_undegraded":"2024-09-16T10:25:55.963388+0000","last_fullsized":"2024-09-16T10:25:55.963388+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:26:28.261724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4140630016,"num_objects":63181,"num_object_clones":0,"num_object_copies":189543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63181,"num_write_kb":4043584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12338397184,"data_stored":12338397184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63181,"ondisk_log_size":63181,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":681,"ondisk_log_size":681,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234138,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489152,"kb_used_data":1006768,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92256000,"statfs":{"total":95995035648,"available":94470144000,"internally_reserved":0,"allocated":1030930432,"data_stored":1030345454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528478,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2195384,"kb_used_data":1758824,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91549768,"statfs":{"total":95995035648,"available":93746962432,"internally_reserved":0,"allocated":1801035776,"data_stored":1800458332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":447019462},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790116,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1867912,"kb_used_data":1403576,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91877240,"statfs":{"total":95995035648,"available":94082293760,"internally_reserved":0,"allocated":1437261824,"data_stored":1436670002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986345,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2383832,"kb_used_data":2001416,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91361320,"statfs":{"total":95995035648,"available":93553991680,"internally_reserved":0,"allocated":2049449984,"data_stored":2048840813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215278,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1830616,"kb_used_data":1276040,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91914536,"statfs":{"total":95995035648,"available":94120484864,"internally_reserved":0,"allocated":1306664960,"data_stored":1306057172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":567867848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411508,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1828288,"kb_used_data":1522224,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91916864,"statfs":{"total":95995035648,"available":94122868736,"internally_reserved":0,"allocated":1558757376,"data_stored":1558172555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1320552,"kb_used_data":761812,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92424600,"statfs":{"total":95995035648,"available":94642790400,"internally_reserved":0,"allocated":780095488,"data_stored":779474549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738556,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2740540,"kb_used_data":2249836,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91004612,"statfs":{"total":95995035648,"available":93188722688,"internally_reserved":0,"allocated":2303832064,"data_stored":2303244834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2362966016,"data_stored":2362966016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":793182208,"data_stored":793182208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1569128448,"data_stored":1569128448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1328742400,"data_stored":1328742400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2090860544,"data_stored":2090860544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1328021504,"data_stored":1328021504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1831010304,"data_stored":1831010304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1034485760,"data_stored":1034485760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:12.991 INFO:tasks.thrashosds.thrasher:Setting 1.5 to [6, 3, 5] 2024-09-16T10:26:12.991 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.5', '6', '3', '5'] 2024-09-16T10:26:12.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 1.5 6 3 5 2024-09-16T10:26:13.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:13.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:13.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:13.150 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:13.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:13.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:13.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:13.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:13.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:13.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:13.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:13.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:13.537 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:26:13.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:13.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:13.649 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:13.653 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:13.657 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:13.677 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:13.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 37 16 71042 71026 119.956 81.875 0.0154983 0.00832986 2024-09-16T10:26:13.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:13 smithi063 ceph-mon[29993]: 4.19 scrub starts 2024-09-16T10:26:13.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:13 smithi063 ceph-mon[29993]: 4.19 scrub ok 2024-09-16T10:26:13.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:13 smithi063 ceph-mon[39470]: 4.19 scrub starts 2024-09-16T10:26:13.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:13 smithi063 ceph-mon[39470]: 4.19 scrub ok 2024-09-16T10:26:13.732 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:13.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:13.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:13 smithi165 ceph-mon[32696]: 4.19 scrub starts 2024-09-16T10:26:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:13 smithi165 ceph-mon[32696]: 4.19 scrub ok 2024-09-16T10:26:13.837 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:13.883 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:13.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:13.985 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:14.033 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:14.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:14.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:14.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:14.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:14.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:14.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:14.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:14.403 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:14.447 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:14.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:14.548 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:14.594 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:14.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:14.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:14.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 38 16 72307 72291 118.88 79.0625 0.00674704 0.00840676 2024-09-16T10:26:14.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[29993]: 1.11 scrub ok 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[29993]: pgmap v669: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[29993]: 3.18 scrub starts 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[29993]: 3.18 scrub ok 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[39470]: 1.11 scrub ok 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[39470]: pgmap v669: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[39470]: 3.18 scrub starts 2024-09-16T10:26:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[39470]: 3.18 scrub ok 2024-09-16T10:26:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:26:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:14 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:26:14.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:14.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:14 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:26:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:14 smithi165 ceph-mon[32696]: 1.11 scrub ok 2024-09-16T10:26:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:14 smithi165 ceph-mon[32696]: pgmap v669: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-16T10:26:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:14 smithi165 ceph-mon[32696]: 3.18 scrub starts 2024-09-16T10:26:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:14 smithi165 ceph-mon[32696]: 3.18 scrub ok 2024-09-16T10:26:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:14 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:26:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:14 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:26:14.828 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:14.875 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:14.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:14.976 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:15.031 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:15.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:15.132 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:15.173 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:15.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:15.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:15.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:15.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:15.316 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:15.380 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:15.380 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:15.409 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:15.420 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:15.420 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:15.450 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:15.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:15.456 INFO:teuthology.orchestra.run.smithi063.stderr:set 1.5 pg_upmap mapping to [6,3,5] 2024-09-16T10:26:15.456 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:26:15.550 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:15.550 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:15.550 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[29993]: 6.11 scrub starts 2024-09-16T10:26:15.550 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[29993]: 6.11 scrub ok 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1126066453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[29993]: from='client.26140 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/344459292' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]} : dispatch 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[29993]: from='client.26143 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]} : dispatch 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1430839629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[39470]: 6.11 scrub starts 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[39470]: 6.11 scrub ok 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1126066453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[39470]: from='client.26140 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:15.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/344459292' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]} : dispatch 2024-09-16T10:26:15.552 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[39470]: from='client.26143 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]} : dispatch 2024-09-16T10:26:15.552 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1430839629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:15.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:15.591 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:15.591 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:15.596 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:15.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:15.625 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:15.625 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:15.660 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:15.660 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:15.693 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:15.694 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:15.695 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 39 15 73405 73390 117.593 68.6875 0.0111129 0.00849921 2024-09-16T10:26:15.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:15.729 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:26:15.729 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:26:15.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:15.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:15 smithi165 ceph-mon[32696]: 6.11 scrub starts 2024-09-16T10:26:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:15 smithi165 ceph-mon[32696]: 6.11 scrub ok 2024-09-16T10:26:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1126066453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:15 smithi165 ceph-mon[32696]: from='client.26140 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/344459292' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]} : dispatch 2024-09-16T10:26:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:15 smithi165 ceph-mon[32696]: from='client.26143 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]} : dispatch 2024-09-16T10:26:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1430839629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:15.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:15.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:15.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:15.979 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:16.022 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:16.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:16.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:16.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:16.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:16.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:16.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:16.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:16.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:16.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:16.696 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:26:16.695081+0000 min lat: 0.00115798 max lat: 0.155052 avg lat: 0.0085582 2024-09-16T10:26:16.696 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:26:16.696 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 40 16 74765 74749 116.776 84.9375 0.0130562 0.0085582 2024-09-16T10:26:16.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:16.728 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:16.728 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: 2.2 scrub starts 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: 2.2 scrub ok 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: 5.e scrub starts 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: 5.e scrub ok 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: pgmap v670: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: from='client.26143 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]}]': finished 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: osdmap e377: 8 total, 8 up, 8 in 2024-09-16T10:26:16.729 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[29993]: osdmap e378: 8 total, 8 up, 8 in 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: 2.2 scrub starts 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: 2.2 scrub ok 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: 5.e scrub starts 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: 5.e scrub ok 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: pgmap v670: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:16.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: from='client.26143 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]}]': finished 2024-09-16T10:26:16.731 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: osdmap e377: 8 total, 8 up, 8 in 2024-09-16T10:26:16.731 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:16 smithi063 ceph-mon[39470]: osdmap e378: 8 total, 8 up, 8 in 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: 2.2 scrub starts 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: 2.2 scrub ok 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: 5.e scrub starts 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: 5.e scrub ok 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: pgmap v670: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:26:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: from='client.26143 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.5", "id": [6, 3, 5]}]': finished 2024-09-16T10:26:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: osdmap e377: 8 total, 8 up, 8 in 2024-09-16T10:26:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:16 smithi165 ceph-mon[32696]: osdmap e378: 8 total, 8 up, 8 in 2024-09-16T10:26:16.807 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.855 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:16.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:16.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:16.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:16.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:17.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:17.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:17.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:17.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:17.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:17.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:17.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:17.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:17.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:26:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:26:17.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:17.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:17.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:17.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:17.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:17.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:17.696 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 41 16 76176 76160 116.079 88.1875 0.0178172 0.00861016 2024-09-16T10:26:17.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:17.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:17.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:17 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:17 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:17 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:26:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:17 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:26:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:17 smithi165 ceph-mon[32696]: osdmap e379: 8 total, 8 up, 8 in 2024-09-16T10:26:17.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:17.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:17.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:17.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:26:17.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:26:17.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[29993]: osdmap e379: 8 total, 8 up, 8 in 2024-09-16T10:26:17.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:17.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:17.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:26:17.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:26:17.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:17 smithi063 ceph-mon[39470]: osdmap e379: 8 total, 8 up, 8 in 2024-09-16T10:26:17.936 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:17.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:18.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:18.074 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:18.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:18.176 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:18.213 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:18.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:18.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:18.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:18.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:18.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:18.500 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:18.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:18.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:18.639 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:18.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:18.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:18.696 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 42 16 77583 77567 115.409 87.9375 0.00865612 0.00866006 2024-09-16T10:26:18.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:18.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:18.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-mon[32696]: 1.5 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:26:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-mon[32696]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:26:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-mon[32696]: 3.19 scrub starts 2024-09-16T10:26:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-mon[32696]: 3.19 scrub ok 2024-09-16T10:26:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-mon[32696]: pgmap v674: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:26:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-mon[32696]: 4.1c scrub starts 2024-09-16T10:26:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-mon[32696]: 4.1c scrub ok 2024-09-16T10:26:18.838 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[29993]: 1.5 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[29993]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[29993]: 3.19 scrub starts 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[29993]: 3.19 scrub ok 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[29993]: pgmap v674: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[29993]: 4.1c scrub starts 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[29993]: 4.1c scrub ok 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[39470]: 1.5 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[39470]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:26:18.839 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[39470]: 3.19 scrub starts 2024-09-16T10:26:18.840 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[39470]: 3.19 scrub ok 2024-09-16T10:26:18.840 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[39470]: pgmap v674: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:26:18.840 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[39470]: 4.1c scrub starts 2024-09-16T10:26:18.840 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:18 smithi063 ceph-mon[39470]: 4.1c scrub ok 2024-09-16T10:26:18.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:18.924 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:18.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:19.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:19.068 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:19.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:19.171 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:19.218 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:19.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:19.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:26:19.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:26:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:26:19.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:19.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:19.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:19.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:19.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:19.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:19.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:26:19.553 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:19 smithi063 ceph-mon[29993]: 2.1 scrub starts 2024-09-16T10:26:19.553 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:19 smithi063 ceph-mon[29993]: 2.1 scrub ok 2024-09-16T10:26:19.553 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:19 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:19.553 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:19 smithi063 ceph-mon[39470]: 2.1 scrub starts 2024-09-16T10:26:19.553 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:19 smithi063 ceph-mon[39470]: 2.1 scrub ok 2024-09-16T10:26:19.554 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:19 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:19.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:19.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:19.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:19.697 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:19.697 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 43 16 78871 78855 114.593 80.5 0.00706434 0.00872073 2024-09-16T10:26:19.708 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:19.774 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:19.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:19 smithi165 ceph-mon[32696]: 2.1 scrub starts 2024-09-16T10:26:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:19 smithi165 ceph-mon[32696]: 2.1 scrub ok 2024-09-16T10:26:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:19 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:19.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:19.932 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:19.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:19.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:26:20.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:20.083 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:20.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:20.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:20.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:20.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:20.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:20.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:20.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:20.385 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:20.385 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:20.392 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:20.422 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:20.422 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:20.452 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:20.462 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:20.462 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:20.501 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:20.502 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:20.525 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:20.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:20.535 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:20.536 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:20.575 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:20.575 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:20.609 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:20.609 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:20.627 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:20.641 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:26:20.641 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:26:20.677 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:20.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:20.688 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[29993]: 6.7 scrub starts 2024-09-16T10:26:20.688 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[29993]: 6.7 scrub ok 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[29993]: pgmap v675: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.5 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[29993]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/234756335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[39470]: 6.7 scrub starts 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[39470]: 6.7 scrub ok 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[39470]: pgmap v675: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.5 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:26:20.689 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:26:20.690 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[39470]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:26:20.690 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/234756335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:20.699 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 44 16 80052 80036 113.662 73.8125 0.00963564 0.008793 2024-09-16T10:26:20.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:20 smithi165 ceph-mon[32696]: 6.7 scrub starts 2024-09-16T10:26:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:20 smithi165 ceph-mon[32696]: 6.7 scrub ok 2024-09-16T10:26:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:20 smithi165 ceph-mon[32696]: pgmap v675: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.5 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:26:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:20 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:26:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:20 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:26:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:20 smithi165 ceph-mon[32696]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:26:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/234756335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:20.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:20.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:20.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:20.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:20.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:20.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:21.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:21.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:21.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:21.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:21.232 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:21.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:21.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:21.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:21.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:21.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:21.490 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:21.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:21.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:21.630 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:21.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:21.699 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 45 16 81282 81266 112.844 76.875 0.00988745 0.00885698 2024-09-16T10:26:21.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:21.766 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:21.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 4.8 scrub starts 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 4.8 scrub ok 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 4.14 scrub starts 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 4.14 scrub ok 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 3.10 scrub starts 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 3.10 scrub ok 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 6.9 scrub starts 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: 6.9 scrub ok 2024-09-16T10:26:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:21 smithi165 ceph-mon[32696]: osdmap e380: 8 total, 8 up, 8 in 2024-09-16T10:26:21.868 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:21.904 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:21.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:21.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 4.8 scrub starts 2024-09-16T10:26:21.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 4.8 scrub ok 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 4.14 scrub starts 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 4.14 scrub ok 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 3.10 scrub starts 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 3.10 scrub ok 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 6.9 scrub starts 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: 6.9 scrub ok 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[29993]: osdmap e380: 8 total, 8 up, 8 in 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 4.8 scrub starts 2024-09-16T10:26:21.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 4.8 scrub ok 2024-09-16T10:26:21.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 4.14 scrub starts 2024-09-16T10:26:21.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 4.14 scrub ok 2024-09-16T10:26:21.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 3.10 scrub starts 2024-09-16T10:26:21.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 3.10 scrub ok 2024-09-16T10:26:21.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 6.9 scrub starts 2024-09-16T10:26:21.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: 6.9 scrub ok 2024-09-16T10:26:21.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:21 smithi063 ceph-mon[39470]: osdmap e380: 8 total, 8 up, 8 in 2024-09-16T10:26:22.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:22.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:22.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:22.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:22.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:22.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:22.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:22.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:22.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:22.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:22.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:22.699 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 46 16 82591 82575 112.169 81.8125 0.0270777 0.00891004 2024-09-16T10:26:22.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:22 smithi165 ceph-mon[32696]: pgmap v677: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:26:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:22 smithi165 ceph-mon[32696]: 5.17 scrub starts 2024-09-16T10:26:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:22 smithi165 ceph-mon[32696]: 5.17 scrub ok 2024-09-16T10:26:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:22 smithi165 ceph-mon[32696]: 5.8 scrub starts 2024-09-16T10:26:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:22 smithi165 ceph-mon[32696]: 5.8 scrub ok 2024-09-16T10:26:22.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:22.847 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:22.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[29993]: pgmap v677: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[29993]: 5.17 scrub starts 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[29993]: 5.17 scrub ok 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[29993]: 5.8 scrub starts 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[29993]: 5.8 scrub ok 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[39470]: pgmap v677: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[39470]: 5.17 scrub starts 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[39470]: 5.17 scrub ok 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[39470]: 5.8 scrub starts 2024-09-16T10:26:22.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:22 smithi063 ceph-mon[39470]: 5.8 scrub ok 2024-09-16T10:26:22.971 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:22.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:22.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:23.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:23.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:23.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:23.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:23.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:23.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:23.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:23.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:23.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:23.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:23.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:23.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:23.605 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:23.631 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:23.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:23.699 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 47 16 83605 83589 111.131 63.375 0.0181175 0.00899344 2024-09-16T10:26:23.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[29993]: 5.1d scrub starts 2024-09-16T10:26:23.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[29993]: 5.1d scrub ok 2024-09-16T10:26:23.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[29993]: 5.a scrub starts 2024-09-16T10:26:23.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[29993]: 5.a scrub ok 2024-09-16T10:26:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[39470]: 5.1d scrub starts 2024-09-16T10:26:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[39470]: 5.1d scrub ok 2024-09-16T10:26:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[39470]: 5.a scrub starts 2024-09-16T10:26:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:23 smithi063 ceph-mon[39470]: 5.a scrub ok 2024-09-16T10:26:23.733 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:23.777 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:23.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:23.786 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:26:23.786 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:26:23.786 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:26:23.786 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:26:23.786 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:26:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:23 smithi165 ceph-mon[32696]: 5.1d scrub starts 2024-09-16T10:26:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:23 smithi165 ceph-mon[32696]: 5.1d scrub ok 2024-09-16T10:26:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:23 smithi165 ceph-mon[32696]: 5.a scrub starts 2024-09-16T10:26:23.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:23 smithi165 ceph-mon[32696]: 5.a scrub ok 2024-09-16T10:26:23.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:23.893 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:26:23.893 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:26:23.893 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T10:26:23.894 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-16T10:26:23.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 0 2024-09-16T10:26:23.916 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:23.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:24.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:24.059 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:24.073 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:24.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:24.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:24.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:24.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:24.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:24.360 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:24.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:24.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:24.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:24.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:24.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:24.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:24.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:24.699 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 48 16 84919 84903 110.527 82.125 0.0118635 0.00904262 2024-09-16T10:26:24.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:24.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[29993]: 6.1d scrub starts 2024-09-16T10:26:24.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[29993]: 6.1d scrub ok 2024-09-16T10:26:24.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[29993]: pgmap v678: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 1.8 KiB/s, 0 objects/s recovering 2024-09-16T10:26:24.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:24.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:24.779 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[39470]: 6.1d scrub starts 2024-09-16T10:26:24.779 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[39470]: 6.1d scrub ok 2024-09-16T10:26:24.779 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[39470]: pgmap v678: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 1.8 KiB/s, 0 objects/s recovering 2024-09-16T10:26:24.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:24.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:24.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:24.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:24 smithi165 ceph-mon[32696]: 6.1d scrub starts 2024-09-16T10:26:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:24 smithi165 ceph-mon[32696]: 6.1d scrub ok 2024-09-16T10:26:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:24 smithi165 ceph-mon[32696]: pgmap v678: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 1.8 KiB/s, 0 objects/s recovering 2024-09-16T10:26:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:24.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:24.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:24.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:25.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:25.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:25.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:25.190 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:25.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:25.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:25.343 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:25.389 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:25.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:25.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:25.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:25.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:25.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:25.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:25.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:25.700 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 49 16 86149 86133 109.84 76.875 0.0104291 0.00909922 2024-09-16T10:26:25.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:25.762 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:25.764 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:26:25.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:26:25.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:26:25.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[29993]: 4.16 scrub starts 2024-09-16T10:26:25.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[29993]: 4.16 scrub ok 2024-09-16T10:26:25.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[29993]: 5.1 scrub starts 2024-09-16T10:26:25.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[29993]: 5.1 scrub ok 2024-09-16T10:26:25.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:26:25.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:26:25.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[39470]: 4.16 scrub starts 2024-09-16T10:26:25.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[39470]: 4.16 scrub ok 2024-09-16T10:26:25.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[39470]: 5.1 scrub starts 2024-09-16T10:26:25.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:25 smithi063 ceph-mon[39470]: 5.1 scrub ok 2024-09-16T10:26:25.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:25.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:25 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:26:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:25 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:26:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:25 smithi165 ceph-mon[32696]: 4.16 scrub starts 2024-09-16T10:26:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:25 smithi165 ceph-mon[32696]: 4.16 scrub ok 2024-09-16T10:26:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:25 smithi165 ceph-mon[32696]: 5.1 scrub starts 2024-09-16T10:26:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:25 smithi165 ceph-mon[32696]: 5.1 scrub ok 2024-09-16T10:26:25.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:25.935 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:25.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:26.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:26.086 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:26.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:26.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:26.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:26.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:26.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:26.367 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:26.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:26.443 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:26.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:26.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:26.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:26.532 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:26:26.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:26.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:26.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:26.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:26:26.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 50 16 87293 87277 109.073 71.5 0.00659026 0.00916269 2024-09-16T10:26:26.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:26.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:26.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:26.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:26 smithi063 ceph-mon[29993]: pgmap v679: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 1.6 KiB/s, 0 objects/s recovering 2024-09-16T10:26:26.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2175292155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:26.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:26 smithi063 ceph-mon[39470]: pgmap v679: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 1.6 KiB/s, 0 objects/s recovering 2024-09-16T10:26:26.782 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2175292155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:26.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:26 smithi165 ceph-mon[32696]: pgmap v679: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 1.6 KiB/s, 0 objects/s recovering 2024-09-16T10:26:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2175292155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:26.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:26.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:26.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:27.047 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:27.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:27.151 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:27.194 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:27.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:27.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:27.315 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:26:27.315 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:26:27.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:26:27.315 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9.36b0083b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.84667161499999, 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41720589800000002, 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:26:27.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463115+0000", 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7430000000000001e-06 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463140+0000", 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4477e-05 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463151+0000", 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0891e-05 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463161+0000", 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0020000000000001e-05 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464130+0000", 2024-09-16T10:26:27.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00096871499999999998 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464137+0000", 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1749999999999999e-06 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.320 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880306+0000", 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41616966999999999 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880320+0000", 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3207000000000001e-05 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9.669f9f4b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.321 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.805222001, 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37579170000000001, 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.322 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504565+0000", 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9400000000000001e-06 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504589+0000", 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3971000000000001e-05 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504602+0000", 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2539e-05 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504610+0000", 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4460000000000001e-06 2024-09-16T10:26:27.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504674+0000", 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3895999999999999e-05 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504682+0000", 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0260000000000007e-06 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880348+0000", 2024-09-16T10:26:27.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37566548300000002 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880355+0000", 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3989999999999998e-06 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9.deba987b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.79906107299999, 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36966423799999998, 2024-09-16T10:26:27.326 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:26:27.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:26:27.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.329 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.329 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510727+0000", 2024-09-16T10:26:27.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8370000000000001e-06 2024-09-16T10:26:27.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510777+0000", 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9957999999999998e-05 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510794+0000", 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7356000000000001e-05 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510820+0000", 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6071000000000002e-05 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510892+0000", 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1520999999999995e-05 2024-09-16T10:26:27.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510901+0000", 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5920000000000001e-06 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880380+0000", 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36947966100000001 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.332 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880388+0000", 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.242e-06 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9.255df14b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.752250237, 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32288046799999998, 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.333 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:26:27.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557537+0000", 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9080000000000002e-06 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557562+0000", 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5131e-05 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.335 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557575+0000", 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3039e-05 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557582+0000", 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0110000000000003e-06 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557855+0000", 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027304500000000003 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557867+0000", 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1507e-05 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880409+0000", 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32254223399999998 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880416+0000", 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5930000000000002e-06 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9.3f093dfb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:26:27.338 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.70807565800001, 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27873467899999999, 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:26:27.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601712+0000", 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4609999999999998e-06 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601740+0000", 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8008999999999999e-05 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601759+0000", 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8427000000000002e-05 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601767+0000", 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5539999999999998e-06 2024-09-16T10:26:27.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601927+0000", 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015967299999999999 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601935+0000", 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8040000000000002e-06 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880436+0000", 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27850148000000002 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880444+0000", 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2709999999999992e-06 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.343 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9.c91b333b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.682760712, 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25344918100000002, 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.344 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627028+0000", 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.948e-06 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627081+0000", 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3631999999999997e-05 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627100+0000", 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9191999999999999e-05 2024-09-16T10:26:27.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627110+0000", 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6409999999999997e-06 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627207+0000", 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6762999999999995e-05 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627215+0000", 2024-09-16T10:26:27.348 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2260000000000005e-06 2024-09-16T10:26:27.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:27.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880467+0000", 2024-09-16T10:26:27.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25325206 2024-09-16T10:26:27.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880474+0000", 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7190000000000004e-06 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9.7e4a5ddb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:26:27.350 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.67448742400001, 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.245202226, 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:26:27.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635300+0000", 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2620000000000001e-06 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635352+0000", 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2079000000000001e-05 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635365+0000", 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3081000000000001e-05 2024-09-16T10:26:27.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635373+0000", 2024-09-16T10:26:27.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6909999999999997e-06 2024-09-16T10:26:27.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635435+0000", 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2271000000000001e-05 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635445+0000", 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6099999999999995e-06 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880493+0000", 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.24504827800000001 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880500+0000", 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9539999999999998e-06 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9.61c3adbb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.356 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.66772486599999, 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23846684600000001, 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:26:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642062+0000", 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0269999999999998e-06 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642104+0000", 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1473000000000001e-05 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642120+0000", 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6235e-05 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642127+0000", 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2470000000000002e-06 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642184+0000", 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6725999999999999e-05 2024-09-16T10:26:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642192+0000", 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1200000000000002e-06 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880520+0000", 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23832803799999999 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880527+0000", 2024-09-16T10:26:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9800000000000001e-06 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60273 9.b 9.bb30e96b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.61918641, 2024-09-16T10:26:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.189954294, 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60273 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690601+0000", 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2479999999999999e-06 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690656+0000", 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4951000000000003e-05 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690673+0000", 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6543999999999999e-05 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690680+0000", 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8329999999999994e-06 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.366 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690741+0000", 2024-09-16T10:26:27.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0491000000000001e-05 2024-09-16T10:26:27.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.366 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.366 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.366 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:26:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:26:27.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:27.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:27.367 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690754+0000", 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2806000000000001e-05 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880547+0000", 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18979368599999999 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880553+0000", 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7350000000000001e-06 2024-09-16T10:26:27.368 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60317 9.b 9.71cef2b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.58631083, 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.15710459700000001, 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.369 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60317 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723477+0000", 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4119999999999999e-06 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723606+0000", 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012882700000000001 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723622+0000", 2024-09-16T10:26:27.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6436e-05 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723630+0000", 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.089e-06 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723701+0000", 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0729000000000007e-05 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723710+0000", 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8140000000000007e-06 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880572+0000", 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.156862641 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880579+0000", 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6490000000000002e-06 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.373 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60326 9.b 9.d49aff0b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.576369669, 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14719354100000001, 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.374 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60326 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733418+0000", 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.385e-06 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733480+0000", 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2331999999999999e-05 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733498+0000", 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8111e-05 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733506+0000", 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0129999999999993e-06 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733571+0000", 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4437000000000007e-05 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733582+0000", 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1073000000000001e-05 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880601+0000", 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.147019074 2024-09-16T10:26:27.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880609+0000", 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1159999999999999e-06 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60333 9.b 9.f3eeac1b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 231.57196633800001, 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14281871500000001, 2024-09-16T10:26:27.379 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60333 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737822+0000", 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.649e-06 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737888+0000", 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6540999999999999e-05 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737908+0000", 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9419000000000002e-05 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737917+0000", 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0790000000000007e-06 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.738071+0000", 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015447900000000001 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.738079+0000", 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8590000000000005e-06 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880631+0000", 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.142552229 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880638+0000", 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.46e-06 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:41321 10.5 10:afd8c321:::benchmark_data_smithi063_149043_object41320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:25:54.246702+0000", 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 33.063083253999999, 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.103930254, 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 41321 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.385 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.246702+0000", 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.246702+0000", 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.246704+0000", 2024-09-16T10:26:27.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4559999999999999e-06 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.246743+0000", 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9814999999999997e-05 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.246753+0000", 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2590000000000006e-06 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.246759+0000", 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8310000000000003e-06 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.246820+0000", 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0408000000000002e-05 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.326452+0000", 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.079632407000000002 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.326580+0000", 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012814900000000001 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.328253+0000", 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016729469999999999 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.328633+0000", 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037984200000000003 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.350503+0000", 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.021870154999999999 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.350557+0000", 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3445000000000001e-05 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:25:54.350632+0000", 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5539999999999998e-05 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.16152.0:73195 10.e e376/362 10:780b6aaa:::benchmark_data_smithi063_149043_object73194:head v 376'4555, mlcod=376'4553)", 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:15.435844+0000", 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.873941627000001, 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10670737299999999, 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435844+0000", 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435844+0000", 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435845+0000", 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.446e-06 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435882+0000", 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7058000000000003e-05 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435894+0000", 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1817000000000001e-05 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435901+0000", 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3830000000000003e-06 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436104+0000", 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020238 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436124+0000", 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9806999999999999e-05 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.542522+0000", 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.106398338 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.542551+0000", 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9144000000000001e-05 2024-09-16T10:26:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.16152.0:73193 10.f e376/362 10:f7ec0fff:::benchmark_data_smithi063_149043_object73192:head v 376'4466, mlcod=376'4464)", 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:15.435954+0000", 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.873831703, 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10654870600000001, 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435954+0000", 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435954+0000", 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435955+0000", 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5000000000000001e-07 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435986+0000", 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1838000000000002e-05 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.435995+0000", 2024-09-16T10:26:27.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4179999999999997e-06 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436001+0000", 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1929999999999998e-06 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.437157+0000", 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0011562180000000001 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.437175+0000", 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7445e-05 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.542454+0000", 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.105278894 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.542502+0000", 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8749999999999999e-05 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.400 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:73196 10.3 10:c1765f1e:::benchmark_data_smithi063_149043_object73195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:15.436898+0000", 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.872887533, 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.109648416, 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:26:27.401 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 73196 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436898+0000", 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436898+0000", 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436900+0000", 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8819999999999999e-06 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436924+0000", 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4655e-05 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.403 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436936+0000", 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1712999999999999e-05 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.436944+0000", 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6000000000000001e-06 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.437154+0000", 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00021036699999999999 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.437299+0000", 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145341 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.437430+0000", 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013110199999999999 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:27.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.541989+0000", 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.104558714 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.543271+0000", 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001282034 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.543465+0000", 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019343400000000001 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.543516+0000", 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1082999999999999e-05 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.546546+0000", 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030304910000000002 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.16152.0:73196 10.3 e376/362 ondisk, result = 0)", 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:15.440468+0000", 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.869316951, 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.102811572, 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.440468+0000", 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.440468+0000", 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.440470+0000", 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4589999999999999e-06 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.440476+0000", 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0909999999999998e-06 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.440486+0000", 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0541e-05 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.541523+0000", 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.101036347 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.542576+0000", 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001053179 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.543262+0000", 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00068615899999999999 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.543280+0000", 2024-09-16T10:26:27.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7796e-05 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.16152.0:73196 10.3 e376/362 ondisk, result = 0)", 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:15.441512+0000", 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.868273541000001, 2024-09-16T10:26:27.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10504777799999999, 2024-09-16T10:26:27.413 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.413 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.413 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:26:27.413 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.441512+0000", 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.441512+0000", 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.441513+0000", 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.362e-06 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.441519+0000", 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9170000000000003e-06 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.541894+0000", 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.100374633 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.541900+0000", 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2469999999999997e-06 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.543303+0000", 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014027180000000001 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.543459+0000", 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015629899999999999 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:15.546560+0000", 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003100602 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:80881 10.5 10:a37c8b6b:::benchmark_data_smithi063_149043_object80880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:21.315492+0000", 2024-09-16T10:26:27.417 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 5.9942936329999998, 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12199196, 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 80881 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315492+0000", 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315492+0000", 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315494+0000", 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1399999999999998e-06 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315521+0000", 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7375000000000001e-05 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315533+0000", 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2221e-05 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315541+0000", 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.345e-06 2024-09-16T10:26:27.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315598+0000", 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7191000000000002e-05 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315722+0000", 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000124079 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:26:27.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315868+0000", 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000145969 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.320174+0000", 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043059359999999998 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.320417+0000", 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0002429 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.437340+0000", 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116923439 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.437436+0000", 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5841999999999995e-05 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.437484+0000", 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7522999999999999e-05 2024-09-16T10:26:27.423 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:80883 10.5 10:a3020412:::benchmark_data_smithi063_149043_object80882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:21.315859+0000", 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 5.9939262199999996, 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12229240399999999, 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:26:27.424 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 80883 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315859+0000", 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315859+0000", 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315860+0000", 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.136e-06 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315879+0000", 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8961999999999999e-05 2024-09-16T10:26:27.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315887+0000", 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7800000000000001e-06 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.315892+0000", 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6619999999999996e-06 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:26:27.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.318122+0000", 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022299680000000001 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.318212+0000", 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9912000000000002e-05 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.318298+0000", 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6966999999999996e-05 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:26:27.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.320329+0000", 2024-09-16T10:26:27.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002030455 2024-09-16T10:26:27.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.320528+0000", 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019865400000000001 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.437526+0000", 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116998094 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.437995+0000", 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00046901899999999999 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:21.438151+0000", 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015679499999999999 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.431 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.432 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:27.432 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:27.432 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:26:27.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:27.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:27.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-16T10:26:27.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:27.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:27.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:27.700 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 51 16 88330 88314 108.205 64.8125 0.00334367 0.00921351 2024-09-16T10:26:27.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2175292155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[29993]: osdmap e381: 8 total, 8 up, 8 in 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/585995031' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2175292155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:26:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[39470]: osdmap e381: 8 total, 8 up, 8 in 2024-09-16T10:26:27.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/585995031' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-16T10:26:27.733 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:27.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:27.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:27 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:26:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:27 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:26:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:27 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2175292155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:26:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:27 smithi165 ceph-mon[32696]: osdmap e381: 8 total, 8 up, 8 in 2024-09-16T10:26:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/585995031' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-16T10:26:27.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:27.880 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:27.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:27.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.005 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:28.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:28.106 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:28.129 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:28.231 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:28.253 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:28.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:28.377 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:28.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.503 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:28.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:28.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:28.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:28.700 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 52 16 88513 88497 106.344 11.4375 0.00751839 0.00939843 2024-09-16T10:26:28.741 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:28.778 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:28 smithi165 ceph-mon[32696]: pgmap v681: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s 2024-09-16T10:26:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/585995031' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-16T10:26:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:28 smithi165 ceph-mon[32696]: osdmap e382: 8 total, 8 up, 8 in 2024-09-16T10:26:28.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:28.900 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:28 smithi063 ceph-mon[29993]: pgmap v681: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s 2024-09-16T10:26:28.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/585995031' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-16T10:26:28.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:28 smithi063 ceph-mon[29993]: osdmap e382: 8 total, 8 up, 8 in 2024-09-16T10:26:28.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:28 smithi063 ceph-mon[39470]: pgmap v681: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s 2024-09-16T10:26:28.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/585995031' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-16T10:26:28.901 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:28 smithi063 ceph-mon[39470]: osdmap e382: 8 total, 8 up, 8 in 2024-09-16T10:26:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:28.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:28.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:28.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:26:29.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:29.070 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:29.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:29.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:29.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:29.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:29.295 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:26:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:26:29.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:29.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:29.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:29.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:29.497 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:29.517 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:29.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:29.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:29.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:29.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:29.664 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[29993]: osdmap e383: 8 total, 8 up, 8 in 2024-09-16T10:26:29.664 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:29.664 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:29.664 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:29.664 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:29.665 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:26:29.665 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[39470]: osdmap e383: 8 total, 8 up, 8 in 2024-09-16T10:26:29.665 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:29.665 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:29.665 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:29.666 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:29.666 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:29 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:26:29.700 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 53 16 89715 89699 105.755 75.125 0.00603366 0.00945111 2024-09-16T10:26:29.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:29.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:29 smithi165 ceph-mon[32696]: osdmap e383: 8 total, 8 up, 8 in 2024-09-16T10:26:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:29 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:29 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:29 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:29 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:29.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:29 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:26:29.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:29.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:29.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:26:30.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:30.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:30.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:30.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:30.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:30.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.16152.0:90512 10.3 e384/362)", 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:30.438799+0000", 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.002018073, 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020428479999999999, 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:30.438799+0000", 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:30.438799+0000", 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:30.438801+0000", 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0140000000000001e-06 2024-09-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:30.438835+0000", 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4183999999999997e-05 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:30.438855+0000", 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9474e-05 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:30.438864+0000", 2024-09-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7760000000000003e-06 2024-09-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 1 2024-09-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:26:30.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:30.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:30.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:26:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:30.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:30.700 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 54 15 90809 90794 105.064 68.4375 0.00505346 0.00951288 2024-09-16T10:26:30.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:30.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:30.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:30.897 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:30.945 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:30.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:30.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[29993]: pgmap v684: 168 pgs: 10 peering, 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 3 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[29993]: osdmap e384: 8 total, 8 up, 8 in 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/302464234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[39470]: pgmap v684: 168 pgs: 10 peering, 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-16T10:26:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 3 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-16T10:26:30.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[39470]: osdmap e384: 8 total, 8 up, 8 in 2024-09-16T10:26:30.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/302464234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:31.046 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:30 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:26:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:30 smithi165 ceph-mon[32696]: pgmap v684: 168 pgs: 10 peering, 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 4.4 KiB/s, 0 objects/s recovering 2024-09-16T10:26:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:30 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 3 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-16T10:26:31.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:30 smithi165 ceph-mon[32696]: osdmap e384: 8 total, 8 up, 8 in 2024-09-16T10:26:31.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/302464234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:31.092 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:31.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:31.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:31.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:31.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:31.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:31.331 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:26:31.377 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:31.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:31.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:31.524 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:31.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:31.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:31.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:31.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:31.700 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 55 16 92101 92085 104.621 80.6875 0.0120792 0.0095533 2024-09-16T10:26:31.762 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:31.799 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:31.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:31.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:31.941 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:31.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:32.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:32.102 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:32.121 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:32.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:32.192 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:32.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:32.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:32.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:32.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:32.402 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:32.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:32.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:32.539 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:32.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:32.641 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:32.664 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:32.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:32.700 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 56 16 93220 93204 104.001 69.9375 0.00338267 0.0095938 2024-09-16T10:26:32.766 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:32.806 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:32.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:32.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:32.851 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:32 smithi063 ceph-mon[29993]: pgmap v686: 168 pgs: 10 peering, 1 active+remapped+backfill_toofull, 157 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 5.5 KiB/s, 0 objects/s recovering 2024-09-16T10:26:32.852 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:32 smithi063 ceph-mon[39470]: pgmap v686: 168 pgs: 10 peering, 1 active+remapped+backfill_toofull, 157 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 5.5 KiB/s, 0 objects/s recovering 2024-09-16T10:26:32.908 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:32.944 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:32.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:33.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:26:33.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:32 smithi165 ceph-mon[32696]: pgmap v686: 168 pgs: 10 peering, 1 active+remapped+backfill_toofull, 157 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 5.5 KiB/s, 0 objects/s recovering 2024-09-16T10:26:33.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:33.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:26:33.149 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:33.149 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:33.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:33.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:33.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:33.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:33.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:33.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:33.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:33.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:33.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:33.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:33.587 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:26:33.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:33.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:33.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 57 16 94415 94399 103.487 74.6875 0.00683993 0.00965827 2024-09-16T10:26:33.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:33.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:33.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:33.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:33.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:33.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1504897316' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[29993]: from='client.46103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1504897316' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[39470]: from='client.46103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:33 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:33.985 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:34.035 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:34.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:34.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:26:34.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1504897316' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:33 smithi165 ceph-mon[32696]: from='client.46103 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:33 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:33 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:34.094 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:26:34.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:34.170 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:34.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:34.271 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:34.341 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:34.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:34.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:34.454 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:26:34.454 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:26:34.455 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:26:34.455 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-16T10:26:34.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.7 2024-09-16T10:26:34.492 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:34.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:34.594 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:34.630 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:34.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:34.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 58 16 95578 95562 102.956 72.6875 0.010168 0.0097083 2024-09-16T10:26:34.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[29993]: pgmap v687: 168 pgs: 8 peering, 1 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 902 op/s; 26 KiB/s, 0 objects/s recovering 2024-09-16T10:26:34.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:34.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[29993]: from='client.46103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:34.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[29993]: osdmap e385: 8 total, 8 up, 8 in 2024-09-16T10:26:34.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[39470]: pgmap v687: 168 pgs: 8 peering, 1 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 902 op/s; 26 KiB/s, 0 objects/s recovering 2024-09-16T10:26:34.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:34.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[39470]: from='client.46103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:34.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:34 smithi063 ceph-mon[39470]: osdmap e385: 8 total, 8 up, 8 in 2024-09-16T10:26:34.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:34.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:34.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:34.802 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:34 smithi165 systemd[1]: Stopping Ceph osd.7 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:26:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:34 smithi165 ceph-mon[32696]: pgmap v687: 168 pgs: 8 peering, 1 active+remapped+backfill_toofull, 159 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 902 op/s; 26 KiB/s, 0 objects/s recovering 2024-09-16T10:26:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:34 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:34 smithi165 ceph-mon[32696]: from='client.46103 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:34 smithi165 ceph-mon[32696]: osdmap e385: 8 total, 8 up, 8 in 2024-09-16T10:26:34.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:34.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:34.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:35.015 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:35.053 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:35.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:35.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:35.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:35.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:35.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:35.338 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:35.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:35.418 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:35.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:35.446 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:35.493 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:35.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:35.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:35.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:35.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:35.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 59 16 96678 96662 102.376 68.75 0.0101796 0.00976334 2024-09-16T10:26:35.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:35.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:35.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:35.832 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:35 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-09-16T10:26:35.832 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:35 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-09-16T10:26:35.904 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:35.944 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:35.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:36.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:35 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 7 pgs peering) 2024-09-16T10:26:36.046 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:36.091 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:36.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:36.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:36.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:36.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:36.276 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:36.277 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:36.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-16T10:26:36.012+0000 7f38237b1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:26:36.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-16T10:26:36.012+0000 7f38237b1640 -1 osd.7 386 *** Got signal Terminated *** 2024-09-16T10:26:36.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[52770]: 2024-09-16T10:26:36.012+0000 7f38237b1640 -1 osd.7 386 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:26:36.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:36.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:36.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:36.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:36.506 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:36.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:36.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:36.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:36.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:36.651 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:36.652 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:36.652 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:36.652 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:36.660 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:36.698 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:36.698 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:36.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:26:36.700257+0000 min lat: 0.00115798 max lat: 1.03966 avg lat: 0.00981048 2024-09-16T10:26:36.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:26:36.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 60 16 97839 97823 101.879 72.5625 0.0159406 0.00981048 2024-09-16T10:26:36.734 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:36.735 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:36.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:36.766 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:26:36.766 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[29993]: pgmap v689: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 602 B/s rd, 76 MiB/s wr, 1.22k op/s; 19 KiB/s, 0 objects/s recovering 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[29993]: osdmap e386: 8 total, 8 up, 8 in 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[29993]: osd.7 marked itself down and dead 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3974298005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[39470]: pgmap v689: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 602 B/s rd, 76 MiB/s wr, 1.22k op/s; 19 KiB/s, 0 objects/s recovering 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:36.767 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[39470]: osdmap e386: 8 total, 8 up, 8 in 2024-09-16T10:26:36.768 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[39470]: osd.7 marked itself down and dead 2024-09-16T10:26:36.768 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3974298005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:36.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:36.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:36.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:36.925 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:36.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:37.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-mon[32696]: pgmap v689: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 602 B/s rd, 76 MiB/s wr, 1.22k op/s; 19 KiB/s, 0 objects/s recovering 2024-09-16T10:26:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-mon[32696]: osdmap e386: 8 total, 8 up, 8 in 2024-09-16T10:26:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-mon[32696]: osd.7 marked itself down and dead 2024-09-16T10:26:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3974298005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:37.057 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:37.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:37.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:37.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:37.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:26:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.307 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:37.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:37.370 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:26:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:26:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:26:37.409 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:37.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:37.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:37.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:37.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:37.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 61 16 98273 98257 100.654 27.125 0.00338218 0.00982107 2024-09-16T10:26:37.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:37.808 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:37.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:26:37.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:37 smithi165 ceph-mon[32696]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T10:26:37.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:37 smithi165 ceph-mon[32696]: osdmap e387: 8 total, 7 up, 8 in 2024-09-16T10:26:37.869 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:37 smithi165 podman[114041]: 2024-09-16 10:26:37.613434319 +0000 UTC m=+2.864885551 container died 08315d538e103284b1021d363a4bcf201e1c068a1a33e0c841b30ba4b7a72c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T10:26:37.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:37.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:37.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:37.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:37 smithi063 ceph-mon[39470]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T10:26:37.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:37 smithi063 ceph-mon[39470]: osdmap e387: 8 total, 7 up, 8 in 2024-09-16T10:26:37.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:37 smithi063 ceph-mon[29993]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T10:26:37.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:37 smithi063 ceph-mon[29993]: osdmap e387: 8 total, 7 up, 8 in 2024-09-16T10:26:38.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:38.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:38.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:38.124 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:38 smithi165 podman[114041]: 2024-09-16 10:26:38.110738234 +0000 UTC m=+3.362189473 container cleanup 08315d538e103284b1021d363a4bcf201e1c068a1a33e0c841b30ba4b7a72c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-16T10:26:38.124 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:38 smithi165 bash[114041]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7 2024-09-16T10:26:38.181 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:38.231 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:38.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:38.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:38.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:38.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:38.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:38.472 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:38.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:38.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:38.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:38.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:38.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:38.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:38 smithi165 ceph-mon[32696]: pgmap v692: 168 pgs: 6 stale+active+clean, 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 965 op/s; 21 KiB/s, 0 objects/s recovering 2024-09-16T10:26:38.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:38 smithi165 ceph-mon[32696]: osdmap e388: 8 total, 7 up, 8 in 2024-09-16T10:26:38.657 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:38 smithi165 podman[114203]: 2024-09-16 10:26:38.379314579 +0000 UTC m=+0.768380758 container remove 08315d538e103284b1021d363a4bcf201e1c068a1a33e0c841b30ba4b7a72c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-16T10:26:38.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 62 16 99091 99075 99.8549 51.125 0.00777083 0.0100101 2024-09-16T10:26:38.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:38.778 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:38.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:38.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:38.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:38.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:38.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:26:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:26:38.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:38 smithi063 ceph-mon[29993]: pgmap v692: 168 pgs: 6 stale+active+clean, 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 965 op/s; 21 KiB/s, 0 objects/s recovering 2024-09-16T10:26:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:38 smithi063 ceph-mon[29993]: osdmap e388: 8 total, 7 up, 8 in 2024-09-16T10:26:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:38 smithi063 ceph-mon[39470]: pgmap v692: 168 pgs: 6 stale+active+clean, 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 60 MiB/s wr, 965 op/s; 21 KiB/s, 0 objects/s recovering 2024-09-16T10:26:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:38 smithi063 ceph-mon[39470]: osdmap e388: 8 total, 7 up, 8 in 2024-09-16T10:26:39.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:39.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:99562 10.e e388/382 10:7fb4f318:::benchmark_data_smithi063_149043_object99561:head v 388'6171, mlcod=388'6169)", 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:39.091506+0000", 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.003858748, 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003891888, 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:39.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091506+0000", 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091506+0000", 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091507+0000", 2024-09-16T10:26:39.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3370000000000001e-06 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091530+0000", 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2993000000000002e-05 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091551+0000", 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1199000000000002e-05 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091558+0000", 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5540000000000003e-06 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091717+0000", 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015980899999999999 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091735+0000", 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7805999999999999e-05 2024-09-16T10:26:39.101 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:99563 10.d e388/362 10:b508c232:::benchmark_data_smithi063_149043_object99562:head v 388'6233, mlcod=388'6227)", 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:39.091575+0000", 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0037896520000000001, 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038788170000000001, 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091575+0000", 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091575+0000", 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091575+0000", 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7100000000000002e-07 2024-09-16T10:26:39.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091604+0000", 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9261999999999999e-05 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091610+0000", 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.02e-06 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:39.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.091616+0000", 2024-09-16T10:26:39.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1800000000000004e-06 2024-09-16T10:26:39.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:39.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.092177+0000", 2024-09-16T10:26:39.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056116 2024-09-16T10:26:39.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:39.092193+0000", 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6673000000000001e-05 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-09-16T10:26:39.106 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:26:39.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:39.197 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:39.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:39.336 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:39.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:39.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:39.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:26:39.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:39.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:39.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 63 15 100359 100344 99.5287 79.3125 0.00468457 0.0100426 2024-09-16T10:26:39.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:39.732 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:26:39.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:39.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:39 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:39.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:39.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:39 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:39.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:39.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3336353887' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:39.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:39 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:39.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:39.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3336353887' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:39.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:39.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:39.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3336353887' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:26:40.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:40.133 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:40.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:40.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:40.271 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:40.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:40.373 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:40.400 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:40.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:40.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:40.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:40.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:40.633 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:40.673 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:40.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:40.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 64 16 101448 101432 99.036 68 0.0344363 0.010092 2024-09-16T10:26:40.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:40.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:40.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:40.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:40.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:40.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:40.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:40 smithi165 ceph-mon[32696]: pgmap v694: 168 pgs: 34 active+undersized, 1 stale+active+clean, 14 active+undersized+degraded, 1 active+remapped+backfill_toofull, 118 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 6193/294333 objects degraded (2.104%); 11 KiB/s, 0 objects/s recovering 2024-09-16T10:26:40.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:40 smithi165 ceph-mon[32696]: Health check failed: Degraded data redundancy: 6193/294333 objects degraded (2.104%), 14 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:40.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:40 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:40.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3336353887' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:26:40.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:40 smithi165 ceph-mon[32696]: osdmap e389: 8 total, 7 up, 8 in 2024-09-16T10:26:41.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:26:41.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:41.097 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:41.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:41.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[29993]: pgmap v694: 168 pgs: 34 active+undersized, 1 stale+active+clean, 14 active+undersized+degraded, 1 active+remapped+backfill_toofull, 118 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 6193/294333 objects degraded (2.104%); 11 KiB/s, 0 objects/s recovering 2024-09-16T10:26:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[29993]: Health check failed: Degraded data redundancy: 6193/294333 objects degraded (2.104%), 14 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3336353887' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:26:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[29993]: osdmap e389: 8 total, 7 up, 8 in 2024-09-16T10:26:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[39470]: pgmap v694: 168 pgs: 34 active+undersized, 1 stale+active+clean, 14 active+undersized+degraded, 1 active+remapped+backfill_toofull, 118 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 6193/294333 objects degraded (2.104%); 11 KiB/s, 0 objects/s recovering 2024-09-16T10:26:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[39470]: Health check failed: Degraded data redundancy: 6193/294333 objects degraded (2.104%), 14 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3336353887' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:26:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:40 smithi063 ceph-mon[39470]: osdmap e389: 8 total, 7 up, 8 in 2024-09-16T10:26:41.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:41.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:41.275 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:41 smithi165 podman[114609]: 2024-09-16 10:26:41.137366664 +0000 UTC m=+0.030769206 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:26:41.343 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:41.388 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:41.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:41.489 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:41.515 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:41.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:41.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:41 smithi165 podman[114609]: 2024-09-16 10:26:41.273962156 +0000 UTC m=+0.167364693 container create d9f2f854253046de4e135929b97865478e4cb61573e4a99b56588e9b19b316a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-16T10:26:41.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:41.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:41.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:41.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 65 16 102476 102460 98.5008 64.25 0.0104045 0.0101473 2024-09-16T10:26:41.718 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:41.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:41.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:41.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:41.842 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:41 smithi165 podman[114609]: 2024-09-16 10:26:41.552063527 +0000 UTC m=+0.445466057 container init d9f2f854253046de4e135929b97865478e4cb61573e4a99b56588e9b19b316a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-16T10:26:41.843 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:41 smithi165 podman[114609]: 2024-09-16 10:26:41.555893521 +0000 UTC m=+0.449296057 container start d9f2f854253046de4e135929b97865478e4cb61573e4a99b56588e9b19b316a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-16T10:26:41.843 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:41 smithi165 podman[114609]: 2024-09-16 10:26:41.652228294 +0000 UTC m=+0.545630827 container attach d9f2f854253046de4e135929b97865478e4cb61573e4a99b56588e9b19b316a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T10:26:41.902 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:41.949 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:41.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:42.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:42.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:42.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:42.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:42.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:42.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:42.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:26:42 smithi165 podman[114609]: 2024-09-16 10:26:42.021354791 +0000 UTC m=+0.914757319 container died d9f2f854253046de4e135929b97865478e4cb61573e4a99b56588e9b19b316a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-16T10:26:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:41 smithi165 ceph-mon[32696]: osdmap e390: 8 total, 7 up, 8 in 2024-09-16T10:26:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:41 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:41 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:41 smithi165 ceph-mon[32696]: pgmap v697: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 1 active+remapped+backfill_toofull, 104 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 24952/298176 objects degraded (8.368%); 29 KiB/s, 0 objects/s recovering 2024-09-16T10:26:42.316 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:42.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:42.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:42.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[29993]: osdmap e390: 8 total, 7 up, 8 in 2024-09-16T10:26:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[29993]: pgmap v697: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 1 active+remapped+backfill_toofull, 104 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 24952/298176 objects degraded (8.368%); 29 KiB/s, 0 objects/s recovering 2024-09-16T10:26:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[39470]: osdmap e390: 8 total, 7 up, 8 in 2024-09-16T10:26:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:42.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:41 smithi063 ceph-mon[39470]: pgmap v697: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 1 active+remapped+backfill_toofull, 104 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 24952/298176 objects degraded (8.368%); 29 KiB/s, 0 objects/s recovering 2024-09-16T10:26:42.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:42.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:42.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:42.542 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:26:42.542 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:26:42.542 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:26:42.543 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:26:42.543 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:26:42.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:42.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:42.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:42.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 66 15 103633 103618 98.105 72.375 0.0124967 0.0101887 2024-09-16T10:26:42.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:42.763 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:42.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:42.865 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:42.906 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:42.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:42.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.7.service' 2024-09-16T10:26:43.007 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:43.045 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:43.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:43.072 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:26:43.072 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-16T10:26:43.072 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-16T10:26:43.073 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T10:26:43.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -n osd.7 --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-16T10:26:43.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:43.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:43.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:43.276 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:43 smithi165 ceph-mon[32696]: osdmap e391: 8 total, 7 up, 8 in 2024-09-16T10:26:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:43 smithi165 ceph-mon[32696]: 4.9 scrub starts 2024-09-16T10:26:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:43 smithi165 ceph-mon[32696]: 4.9 scrub ok 2024-09-16T10:26:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:43 smithi165 ceph-mon[32696]: 3.a scrub starts 2024-09-16T10:26:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:43 smithi165 ceph-mon[32696]: 3.a scrub ok 2024-09-16T10:26:43.311 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:43.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:43.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:43.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[29993]: osdmap e391: 8 total, 7 up, 8 in 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[29993]: 4.9 scrub starts 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[29993]: 4.9 scrub ok 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[29993]: 3.a scrub starts 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[29993]: 3.a scrub ok 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[39470]: osdmap e391: 8 total, 7 up, 8 in 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[39470]: 4.9 scrub starts 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[39470]: 4.9 scrub ok 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[39470]: 3.a scrub starts 2024-09-16T10:26:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:43 smithi063 ceph-mon[39470]: 3.a scrub ok 2024-09-16T10:26:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:43.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:43.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:43.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:43.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:43.686 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:43.696 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/osd.7/config 2024-09-16T10:26:43.701 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 67 16 104902 104886 97.8235 79.25 0.0145818 0.0102171 2024-09-16T10:26:43.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:26:43.732 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:43.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:43.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:43.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:43.892 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:43.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:43.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:44.032 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:44.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:44.131 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:44 smithi063 ceph-mon[29993]: pgmap v699: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 1 active+remapped+backfill_toofull, 104 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 956 B/s rd, 56 MiB/s wr, 894 op/s; 24952/298176 objects degraded (8.368%); 24 KiB/s, 0 objects/s recovering 2024-09-16T10:26:44.132 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:44 smithi063 ceph-mon[39470]: pgmap v699: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 1 active+remapped+backfill_toofull, 104 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 956 B/s rd, 56 MiB/s wr, 894 op/s; 24952/298176 objects degraded (8.368%); 24 KiB/s, 0 objects/s recovering 2024-09-16T10:26:44.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:44.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:44.275 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:44.297 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:44 smithi165 ceph-mon[32696]: pgmap v699: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 1 active+remapped+backfill_toofull, 104 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 956 B/s rd, 56 MiB/s wr, 894 op/s; 24952/298176 objects degraded (8.368%); 24 KiB/s, 0 objects/s recovering 2024-09-16T10:26:44.318 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:44.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:44.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:44.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:44.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:44.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:44.458 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:44.458 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:44.490 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:44.523 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:44.523 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:44.547 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:44.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:44.561 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:44.561 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:44.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:44.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:44.596 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:44.630 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:44.630 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:44.662 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:44.662 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:44.666 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:26:44.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:44.702 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 68 16 106064 106048 97.4517 72.625 0.00700398 0.0102565 2024-09-16T10:26:44.728 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:44.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:44.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:44.866 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:44.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:44.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:45.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:45.058 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T10:26:45.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:45.148 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:45.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:45.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:45.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:45 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:26:45.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:45 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:26:45.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:45 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:26:45.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:45 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:26:45.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:45 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/352260289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:45.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:45 smithi165 ceph-mon[32696]: 4.0 scrub starts 2024-09-16T10:26:45.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:45 smithi165 ceph-mon[32696]: 4.0 scrub ok 2024-09-16T10:26:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:45.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:45.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:45.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:45.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:45.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:26:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:26:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:26:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:26:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/352260289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[29993]: 4.0 scrub starts 2024-09-16T10:26:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[29993]: 4.0 scrub ok 2024-09-16T10:26:45.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:26:45.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:26:45.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:26:45.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:26:45.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/352260289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:45.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[39470]: 4.0 scrub starts 2024-09-16T10:26:45.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:45 smithi063 ceph-mon[39470]: 4.0 scrub ok 2024-09-16T10:26:45.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:26:45.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:26:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:45.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:45.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:26:45.669 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:45.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 69 16 107325 107309 97.1813 78.8125 0.00595649 0.0102852 2024-09-16T10:26:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:45.755 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:45.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:45.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:45.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:45.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:45.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:46.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:46.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:46.101 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:46.119 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:46 smithi165 ceph-mon[32696]: pgmap v700: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 26378/315588 objects degraded (8.358%); 11 KiB/s, 0 objects/s recovering 2024-09-16T10:26:46.120 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:46 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:26:46.120 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:46 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:26:46.120 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:46 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 24952/298176 objects degraded (8.368%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:46.120 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:46 smithi165 ceph-mon[32696]: osdmap e392: 8 total, 7 up, 8 in 2024-09-16T10:26:46.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:46.146 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:46.146 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:46.186 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:46.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:46.287 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[29993]: pgmap v700: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 26378/315588 objects degraded (8.358%); 11 KiB/s, 0 objects/s recovering 2024-09-16T10:26:46.287 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:26:46.287 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:26:46.287 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 24952/298176 objects degraded (8.368%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:46.287 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[29993]: osdmap e392: 8 total, 7 up, 8 in 2024-09-16T10:26:46.288 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[39470]: pgmap v700: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 26378/315588 objects degraded (8.358%); 11 KiB/s, 0 objects/s recovering 2024-09-16T10:26:46.288 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:26:46.288 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:26:46.288 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 24952/298176 objects degraded (8.368%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:46.288 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:46 smithi063 ceph-mon[39470]: osdmap e392: 8 total, 7 up, 8 in 2024-09-16T10:26:46.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:46.325 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:46.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:46.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:46.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:46.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:46.564 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:46.602 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:46.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:46.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 70 16 108540 108524 96.8778 75.9375 0.00908964 0.010317 2024-09-16T10:26:46.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:46.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:46.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:46.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:46.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:46.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:47.023 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:47.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:47.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:26:47.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:47.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:47.292 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:26:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[29993]: 5.16 scrub starts 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[29993]: 5.16 scrub ok 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:26:47.293 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[39470]: 5.16 scrub starts 2024-09-16T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:47 smithi063 ceph-mon[39470]: 5.16 scrub ok 2024-09-16T10:26:47.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:47 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:26:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:47 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:26:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:47 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:47 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:47 smithi165 ceph-mon[32696]: 5.16 scrub starts 2024-09-16T10:26:47.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:47 smithi165 ceph-mon[32696]: 5.16 scrub ok 2024-09-16T10:26:47.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:47.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:47.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:47.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:47.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:47.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:47.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:47.547 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:26:47.553 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:47.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:47.650 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:47.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:47.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 71 16 109775 109759 96.6004 77.1875 0.0102849 0.0103471 2024-09-16T10:26:47.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:47.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:47.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:47.940 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:47.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:47.989 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:48.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:26:48.103 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:48.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:48.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:48.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:48.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:48.263 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:48.263 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[29993]: 5.3 scrub starts 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[29993]: 5.3 scrub ok 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4029082512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[29993]: from='client.46118 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[29993]: pgmap v702: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 57 MiB/s wr, 912 op/s; 26378/315588 objects degraded (8.358%) 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[39470]: 5.3 scrub starts 2024-09-16T10:26:48.299 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[39470]: 5.3 scrub ok 2024-09-16T10:26:48.300 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4029082512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:48.300 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[39470]: from='client.46118 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:48.300 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[39470]: pgmap v702: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 57 MiB/s wr, 912 op/s; 26378/315588 objects degraded (8.358%) 2024-09-16T10:26:48.300 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:26:48.300 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:48 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:26:48.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:48.369 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:48.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:48.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:48.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:48.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:48.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-mon[32696]: 5.3 scrub starts 2024-09-16T10:26:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-mon[32696]: 5.3 scrub ok 2024-09-16T10:26:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4029082512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-mon[32696]: from='client.46118 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:26:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-mon[32696]: pgmap v702: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 57 MiB/s wr, 912 op/s; 26378/315588 objects degraded (8.358%) 2024-09-16T10:26:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:26:48.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:26:48.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:48.656 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:48.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:48.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 72 16 110933 110917 96.2639 72.375 0.0275578 0.0103831 2024-09-16T10:26:48.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:48.799 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:48.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:48.801 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:48.801 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:48.838 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:48.838 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:48.870 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:48.871 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:48.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:48.905 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:48.905 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:48.936 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:48.937 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:48.947 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:48.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:48.969 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:48.969 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:49.005 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:49.006 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:49.006 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:26:49.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:26:49.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:26:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:26:49.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:49.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:49.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:49.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:49.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:26:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[29993]: from='client.46118 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[29993]: osdmap e393: 8 total, 7 up, 8 in 2024-09-16T10:26:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:26:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:26:49.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3805931446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:26:49.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[39470]: from='client.46118 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:49.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[39470]: osdmap e393: 8 total, 7 up, 8 in 2024-09-16T10:26:49.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:26:49.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:26:49.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:49 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3805931446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:6.b 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:6.9 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:6.6 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:1.7 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:4.11 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:4.5 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:3.f 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:1.2 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:5.6 2024-09-16T10:26:49.208 INFO:teuthology.orchestra.run.smithi165.stdout:6.3 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:3.d 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:6.8 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:5.1a 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:3.16 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:3.13 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:4.17 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:3.12 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:3.11 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:4.14 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:3.1b 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:2.0 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:1.4 2024-09-16T10:26:49.209 INFO:teuthology.orchestra.run.smithi165.stdout:10.c 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:1.10 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:10.9 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:1.14 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:3.7 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:10.8 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:4.16 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:6.1c 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:1.9 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:3.8 2024-09-16T10:26:49.210 INFO:teuthology.orchestra.run.smithi165.stdout:3.9 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:10.1 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:4.3 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:5.1f 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:1.e 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:5.9 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:1.0 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:4.1e 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:6.15 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:4.8 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:4.a 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:5.10 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:5.11 2024-09-16T10:26:49.211 INFO:teuthology.orchestra.run.smithi165.stdout:5.1e 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:5.14 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:3.14 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:5.17 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:5.19 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:5.1 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:4.15 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:1.d 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:5.1b 2024-09-16T10:26:49.212 INFO:teuthology.orchestra.run.smithi165.stdout:5.4 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:5.d 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:6.12 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:6.14 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:6.16 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:6.18 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:6.1b 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:3.15 2024-09-16T10:26:49.213 INFO:teuthology.orchestra.run.smithi165.stdout:6.5 2024-09-16T10:26:49.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:49.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:49.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:49 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:26:49.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:49 smithi165 ceph-mon[32696]: from='client.46118 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:26:49.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:49 smithi165 ceph-mon[32696]: osdmap e393: 8 total, 7 up, 8 in 2024-09-16T10:26:49.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:49 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:26:49.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:49 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:26:49.305 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:49 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3805931446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:49.348 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:49.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:49.422 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:49.485 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:49.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:49.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:49.656 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:49.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:49.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 73 16 112138 112122 95.9768 75.3125 0.0133478 0.0104146 2024-09-16T10:26:49.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:49.799 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:49.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:49.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:49.949 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:49.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:50.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:50.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:50.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:50.193 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[29993]: 1.8 scrub starts 2024-09-16T10:26:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[29993]: 1.8 scrub ok 2024-09-16T10:26:50.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[29993]: pgmap v704: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 26730/329256 objects degraded (8.118%) 2024-09-16T10:26:50.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:50.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[39470]: 1.8 scrub starts 2024-09-16T10:26:50.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[39470]: 1.8 scrub ok 2024-09-16T10:26:50.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:50 smithi063 ceph-mon[39470]: pgmap v704: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 26730/329256 objects degraded (8.118%) 2024-09-16T10:26:50.239 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:50.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:50.243 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:26:50.243 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:26:50.243 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:26:50.243 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:26:50.243 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:26:50.342 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:50.380 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:50.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:50.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:50.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:50.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:26:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:50 smithi165 ceph-mon[32696]: 1.8 scrub starts 2024-09-16T10:26:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:50 smithi165 ceph-mon[32696]: 1.8 scrub ok 2024-09-16T10:26:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:50 smithi165 ceph-mon[32696]: pgmap v704: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 26730/329256 objects degraded (8.118%) 2024-09-16T10:26:50.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:50.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:50.690 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:26:50.690 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:26:50.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 74 16 113446 113430 95.7845 81.75 0.0137173 0.0104356 2024-09-16T10:26:50.781 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:50.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:50.925 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:50.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:51.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:26:51.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:51.108 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:51.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:51.209 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:51.249 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:51.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:51 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 26730/329256 objects degraded (8.118%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:51.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:51 smithi165 ceph-mon[32696]: osdmap e394: 8 total, 7 up, 8 in 2024-09-16T10:26:51.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:51.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:51.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:51.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:51 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 26730/329256 objects degraded (8.118%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:51 smithi063 ceph-mon[29993]: osdmap e394: 8 total, 7 up, 8 in 2024-09-16T10:26:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:51 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 26730/329256 objects degraded (8.118%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:51 smithi063 ceph-mon[39470]: osdmap e394: 8 total, 7 up, 8 in 2024-09-16T10:26:51.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:51.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -n osd.7 --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 4.15 --file /var/log/ceph/exp.4.15.7 2024-09-16T10:26:51.533 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:51.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:51.589 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:51.635 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:51.690 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:51.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:51.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 75 16 114722 114706 95.5705 79.75 0.0130926 0.0104591 2024-09-16T10:26:51.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:51.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:51.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:51.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:51.896 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/osd.7/config 2024-09-16T10:26:51.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:51.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:26:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:51.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:52.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:52.138 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:52.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51212 8.4 8:27b2b5e1:::benchmark_data_smithi063_101290_object51211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:26:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 458.89923452900001, 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98431940500000004, 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51212 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320712+0000", 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320713+0000", 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.559e-06 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320735+0000", 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2010000000000001e-05 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320745+0000", 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6690000000000001e-06 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320750+0000", 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2939999999999996e-06 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320816+0000", 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5090000000000002e-05 2024-09-16T10:26:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.320824+0000", 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8599999999999999e-06 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295578+0000", 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97475403100000002 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295697+0000", 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118301 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.295799+0000", 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102376 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304751+0000", 2024-09-16T10:26:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089521540000000004 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304762+0000", 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1044e-05 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304922+0000", 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015943199999999999 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304966+0000", 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4545000000000001e-05 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305031+0000", 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5040000000000001e-05 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51220 8.4 8:2e2a097c:::benchmark_data_smithi063_101290_object51219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 458.89406659799999, 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97951371499999995, 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51220 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325880+0000", 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325882+0000", 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.827e-06 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325937+0000", 2024-09-16T10:26:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5121000000000003e-05 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325948+0000", 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1066999999999999e-05 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.325954+0000", 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999999e-06 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326011+0000", 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6937000000000001e-05 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326023+0000", 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1344e-05 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298310+0000", 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97228777099999997 2024-09-16T10:26:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298432+0000", 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012194200000000001 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.298661+0000", 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000228497 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.304876+0000", 2024-09-16T10:26:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062151819999999997 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305227+0000", 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035094899999999998 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305288+0000", 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0804000000000003e-05 2024-09-16T10:26:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305361+0000", 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2706999999999994e-05 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305394+0000", 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2972999999999998e-05 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51223 8.4 8:287d1d16:::benchmark_data_smithi063_101290_object51222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 458.89342755600001, 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98451053700000002, 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51223 2024-09-16T10:26:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326519+0000", 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326520+0000", 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.079e-06 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326573+0000", 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2924999999999999e-05 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326581+0000", 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3659999999999992e-06 2024-09-16T10:26:52.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326586+0000", 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8369999999999996e-06 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326636+0000", 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9667999999999999e-05 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.326646+0000", 2024-09-16T10:26:52.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0389e-05 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302523+0000", 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.975876682 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302647+0000", 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123745 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.302774+0000", 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012772799999999999 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305245+0000", 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024711160000000002 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309074+0000", 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038286169999999999 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.310943+0000", 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018688979999999999 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.310987+0000", 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4057999999999997e-05 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311029+0000", 2024-09-16T10:26:52.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2429000000000003e-05 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51230 8.4 8:23e3d41e:::benchmark_data_smithi063_101290_object51229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 458.88178977199999, 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97448984100000002, 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.248 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51230 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338157+0000", 2024-09-16T10:26:52.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338159+0000", 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1610000000000001e-06 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338222+0000", 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3161000000000001e-05 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338235+0000", 2024-09-16T10:26:52.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3129000000000001e-05 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338243+0000", 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8420000000000005e-06 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338299+0000", 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6477999999999997e-05 2024-09-16T10:26:52.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.338313+0000", 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3852000000000001e-05 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305424+0000", 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96711084599999997 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305539+0000", 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115146 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.305678+0000", 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013881700000000001 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311084+0000", 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054059110000000002 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.311113+0000", 2024-09-16T10:26:52.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9405999999999999e-05 2024-09-16T10:26:52.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.255 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:52.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312548+0000", 2024-09-16T10:26:52.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001434533 2024-09-16T10:26:52.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312609+0000", 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1503999999999999e-05 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312646+0000", 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7054999999999997e-05 2024-09-16T10:26:52.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51243 8.4 8:253d325f:::benchmark_data_smithi063_101290_object51242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 458.87751078100001, 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97387738800000001, 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51243 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:26:52.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342436+0000", 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342437+0000", 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.505e-06 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342507+0000", 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9720000000000003e-05 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342520+0000", 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3478999999999999e-05 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342529+0000", 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3380000000000005e-06 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342652+0000", 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012287600000000001 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.342658+0000", 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5889999999999999e-06 2024-09-16T10:26:52.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307308+0000", 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96465014199999999 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307405+0000", 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6510999999999998e-05 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.307533+0000", 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012844199999999999 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.312655+0000", 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051218239999999996 2024-09-16T10:26:52.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.315330+0000", 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026750229999999999 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316199+0000", 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086935199999999999 2024-09-16T10:26:52.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316263+0000", 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3326999999999999e-05 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316313+0000", 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0260000000000003e-05 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51249 8.4 8:208cf65c:::benchmark_data_smithi063_101290_object51248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 458.87110683999998, 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96761772599999996, 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51249 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348840+0000", 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348841+0000", 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5460000000000001e-06 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348879+0000", 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7716999999999998e-05 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348889+0000", 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0327e-05 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348896+0000", 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.742e-06 2024-09-16T10:26:52.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348953+0000", 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7028999999999999e-05 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:13.348960+0000", 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4209999999999998e-06 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309120+0000", 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96015961500000002 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309226+0000", 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106158 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.309348+0000", 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012231699999999999 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.315758+0000", 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064091460000000001 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316153+0000", 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039516799999999999 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316354+0000", 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000201452 2024-09-16T10:26:52.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316402+0000", 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7577999999999997e-05 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.316457+0000", 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5510000000000002e-05 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.276 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51270 8.d 8:b5d1ccff:::benchmark_data_smithi063_101290_object51269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.91383515899997, 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.001723546, 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.277 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51270 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306111+0000", 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306113+0000", 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.136e-06 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306261+0000", 2024-09-16T10:26:52.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147632 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306274+0000", 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2850000000000001e-05 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306282+0000", 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.388e-06 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306347+0000", 2024-09-16T10:26:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4898000000000001e-05 2024-09-16T10:26:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.306359+0000", 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2149e-05 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.296824+0000", 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99046429999999996 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.296988+0000", 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000164017 2024-09-16T10:26:52.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.297156+0000", 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016855799999999999 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307591+0000", 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010434652000000001 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307602+0000", 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.083e-05 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307749+0000", 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014768800000000001 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307800+0000", 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0800999999999997e-05 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307835+0000", 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4647000000000002e-05 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51294 8.d 8:b3f59715:::benchmark_data_smithi063_101290_object51293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.89894687100002, 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.986972195, 2024-09-16T10:26:52.285 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51294 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:26:52.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321000+0000", 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321002+0000", 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2060000000000001e-06 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321046+0000", 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4345000000000003e-05 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321063+0000", 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6688e-05 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321071+0000", 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321156+0000", 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4980000000000003e-05 2024-09-16T10:26:52.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.321163+0000", 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9949999999999999e-06 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299344+0000", 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97818064299999996 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299480+0000", 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013635700000000001 2024-09-16T10:26:52.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.299684+0000", 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000203857 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307681+0000", 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079967979999999994 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307717+0000", 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6297999999999997e-05 2024-09-16T10:26:52.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307904+0000", 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018723000000000001 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307943+0000", 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8831999999999997e-05 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307972+0000", 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8555999999999999e-05 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51296 8.d 8:b655e86a:::benchmark_data_smithi063_101290_object51295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.89654836800003, 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99276492999999999, 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.293 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51296 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323398+0000", 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323400+0000", 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4789999999999999e-06 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323445+0000", 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5637999999999999e-05 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323455+0000", 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0127e-05 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323461+0000", 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.339e-06 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323504+0000", 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3461999999999998e-05 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.323513+0000", 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2269999999999999e-06 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302178+0000", 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97866425099999999 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302303+0000", 2024-09-16T10:26:52.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125862 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.302439+0000", 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013551200000000001 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307685+0000", 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052464349999999998 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307870+0000", 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184811 2024-09-16T10:26:52.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316064+0000", 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081937899999999994 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316116+0000", 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1690999999999997e-05 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316163+0000", 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7305999999999997e-05 2024-09-16T10:26:52.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51303 8.d 8:b2b2f5d7:::benchmark_data_smithi063_101290_object51302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.89114695199999, 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99301229000000002, 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.300 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51303 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328799+0000", 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328801+0000", 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2470000000000001e-06 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328903+0000", 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102128 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328928+0000", 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4760999999999999e-05 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328936+0000", 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4400000000000005e-06 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.328998+0000", 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1628e-05 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.329009+0000", 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1758e-05 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.304926+0000", 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97591637799999997 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.305040+0000", 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113761 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.305141+0000", 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101047 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312025+0000", 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068839119999999998 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312061+0000", 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6918999999999997e-05 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321718+0000", 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096562799999999997 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321768+0000", 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0266000000000003e-05 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321812+0000", 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3764999999999999e-05 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51322 8.d 8:b9f45f41:::benchmark_data_smithi063_101290_object51321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.87848120799998, 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.980450343, 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.307 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51322 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341465+0000", 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341467+0000", 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1409999999999999e-06 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341501+0000", 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3157000000000001e-05 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341513+0000", 2024-09-16T10:26:52.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2296999999999999e-05 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341520+0000", 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5290000000000003e-06 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341603+0000", 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2180000000000003e-05 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.341614+0000", 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1748e-05 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.307997+0000", 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96638288500000002 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.308109+0000", 2024-09-16T10:26:52.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112235 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.308233+0000", 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012397800000000001 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312071+0000", 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003837619 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.315989+0000", 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039176020000000001 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321846+0000", 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058578290000000002 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321886+0000", 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9400999999999998e-05 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321916+0000", 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9742000000000001e-05 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51332 8.d 8:baf4d594:::benchmark_data_smithi063_101290_object51331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.869740939, 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97184618199999995, 2024-09-16T10:26:52.314 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51332 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:26:52.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350205+0000", 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350208+0000", 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.176e-06 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350266+0000", 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7840999999999997e-05 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350279+0000", 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3718e-05 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350287+0000", 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5240000000000003e-06 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350342+0000", 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4755e-05 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.350350+0000", 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1259999999999998e-06 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.309962+0000", 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95961197799999998 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.310060+0000", 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8022000000000004e-05 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.310179+0000", 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119713 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.315951+0000", 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057717239999999998 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316034+0000", 2024-09-16T10:26:52.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2670000000000006e-05 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321980+0000", 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059458560000000002 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322021+0000", 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1328999999999997e-05 2024-09-16T10:26:52.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322052+0000", 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0750000000000002e-05 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51340 8.d 8:be7e8c60:::benchmark_data_smithi063_101290_object51339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.86326867299999, 2024-09-16T10:26:52.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96553988000000002, 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51340 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:26:52.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356678+0000", 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356679+0000", 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356716+0000", 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6591000000000003e-05 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356726+0000", 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6560000000000004e-06 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356733+0000", 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9789999999999996e-06 2024-09-16T10:26:52.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356809+0000", 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6790000000000002e-05 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.356819+0000", 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8269999999999993e-06 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312094+0000", 2024-09-16T10:26:52.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.955275333 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312207+0000", 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112341 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.312340+0000", 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013320399999999999 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316004+0000", 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036634889999999998 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316206+0000", 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020242200000000001 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322131+0000", 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059253600000000002 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322183+0000", 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1372999999999998e-05 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322218+0000", 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4950000000000002e-05 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51359 8.d 8:be832237:::benchmark_data_smithi063_101290_object51358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.330 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.78339128300001, 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88882225199999998, 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51359 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436555+0000", 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436556+0000", 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.328e-06 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436612+0000", 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5040999999999997e-05 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436622+0000", 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0491e-05 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436629+0000", 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9929999999999998e-06 2024-09-16T10:26:52.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436696+0000", 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6544000000000005e-05 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.436710+0000", 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4058e-05 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314089+0000", 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87737920700000005 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314198+0000", 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109348 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.314324+0000", 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012599 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321682+0000", 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073583529999999998 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322263+0000", 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058049700000000002 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325295+0000", 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030324670000000001 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325342+0000", 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6499999999999999e-05 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.325377+0000", 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5435000000000002e-05 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.337 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:51370 8.d 8:b1e0f19d:::benchmark_data_smithi063_101290_object51369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.775183078, 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88139332999999997, 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 51370 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444763+0000", 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444765+0000", 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.412e-06 2024-09-16T10:26:52.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444815+0000", 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0201999999999998e-05 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444825+0000", 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0175e-05 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444830+0000", 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1429999999999997e-06 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444910+0000", 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9300000000000003e-05 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:14.444919+0000", 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2499999999999995e-06 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316237+0000", 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87131822199999998 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316359+0000", 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012239799999999999 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.316474+0000", 2024-09-16T10:26:52.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011482800000000001 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.321948+0000", 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054734090000000003 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.322268+0000", 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032003100000000003 2024-09-16T10:26:52.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326071+0000", 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038029539999999999 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326120+0000", 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9326999999999997e-05 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:19:15.326157+0000", 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6678999999999999e-05 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:2951 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 93976561478912 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.683357175000001, 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0211994769999999, 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.345 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2951 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536592+0000", 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2979999999999999e-06 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536598+0000", 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-09-16T10:26:52.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536615+0000", 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7496999999999999e-05 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536624+0000", 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9099999999999994e-06 2024-09-16T10:26:52.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:52 smithi165 ceph-mon[32696]: pgmap v706: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 27829/333660 objects degraded (8.341%) 2024-09-16T10:26:52.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:52 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:26:52.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:52 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:26:52.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:52 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:26:52.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:52 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:26:52.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:52 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:26:52.350 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:52.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:52.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536869+0000", 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024484799999999998 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536876+0000", 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1910000000000003e-06 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.547076+0000", 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010199542000000001 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.547092+0000", 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5818e-05 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.552328+0000", 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0052359230000001 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.552503+0000", 2024-09-16T10:26:52.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017550000000000001 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.557789+0000", 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052855139999999998 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98204 10.9 10:9d9b818e:::benchmark_data_smithi063_149043_object98203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:26:52.353 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.238357537000001, 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.034086345, 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98204 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981591+0000", 2024-09-16T10:26:52.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981670+0000", 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9151000000000002e-05 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981680+0000", 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8109999999999997e-06 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981687+0000", 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9959999999999996e-06 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981740+0000", 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3491e-05 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981886+0000", 2024-09-16T10:26:52.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014617800000000001 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.982038+0000", 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151927 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.990399+0000", 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083606689999999994 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999930+0000", 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095309130000000002 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999946+0000", 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6342999999999999e-05 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015600+0000", 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0156542 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015675+0000", 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.483e-05 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98205 10.9 10:919b6e78:::benchmark_data_smithi063_149043_object98204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.238238163, 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.046984186, 2024-09-16T10:26:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98205 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981709+0000", 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4499999999999995e-07 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981760+0000", 2024-09-16T10:26:52.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0654e-05 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981766+0000", 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6089999999999997e-06 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981772+0000", 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7620000000000001e-06 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.985245+0000", 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034727030000000002 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.986079+0000", 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083413300000000003 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.986209+0000", 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012954199999999999 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999975+0000", 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013766733 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999982+0000", 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8759999999999999e-06 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015907+0000", 2024-09-16T10:26:52.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0159249210000001 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016460+0000", 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055315400000000004 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028602+0000", 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012142065 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028692+0000", 2024-09-16T10:26:52.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0088999999999998e-05 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98225 10.9 10:9f9b788e:::benchmark_data_smithi063_149043_object98224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.198161539000001, 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0071949410000001, 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.367 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98225 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:26:52.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021788+0000", 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2959999999999999e-06 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021880+0000", 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1638999999999995e-05 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021898+0000", 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873e-05 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021907+0000", 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1789999999999997e-06 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021961+0000", 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4026000000000001e-05 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021969+0000", 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9840000000000001e-06 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016509+0000", 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99453982200000002 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016634+0000", 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012573099999999999 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016773+0000", 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013850399999999999 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.021953+0000", 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051800379999999997 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028816+0000", 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068627699999999998 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028873+0000", 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7655999999999998e-05 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028980+0000", 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010642299999999999 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98247 10.9 10:95b7428a:::benchmark_data_smithi063_149043_object98246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.183295428999999, 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99248770799999997, 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98247 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036653+0000", 2024-09-16T10:26:52.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9139999999999998e-06 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036702+0000", 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9490000000000002e-05 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036716+0000", 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3521e-05 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036723+0000", 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4259999999999997e-06 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036784+0000", 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1013e-05 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036790+0000", 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4739999999999996e-06 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.018768+0000", 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98197859600000004 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.018953+0000", 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184358 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.019066+0000", 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011326 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.021961+0000", 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028944869999999998 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029022+0000", 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070617960000000004 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029056+0000", 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4079999999999999e-05 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:26:52.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029139+0000", 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2293000000000006e-05 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:26:52.380 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:26:52.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:52.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[29993]: pgmap v706: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 27829/333660 objects degraded (8.341%) 2024-09-16T10:26:52.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:26:52.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[39470]: pgmap v706: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 27829/333660 objects degraded (8.341%) 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:26:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:52 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:26:52.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:52.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:52.499 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:52.578 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:52.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:52.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:26:52.703 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 76 16 115787 115771 95.1887 66.5625 0.0139938 0.0105003 2024-09-16T10:26:52.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:52.794 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:52.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:52.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:52.948 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:52.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:53.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:53.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:53.198 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:53.247 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:53.257 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:53.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:53.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:53 smithi165 ceph-mon[32696]: osdmap e395: 8 total, 7 up, 8 in 2024-09-16T10:26:53.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:26:53.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:26:53.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:53 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:53.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:26:53.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:53.388 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[29993]: osdmap e395: 8 total, 7 up, 8 in 2024-09-16T10:26:53.389 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:26:53.389 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:26:53.389 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:53.390 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[39470]: osdmap e395: 8 total, 7 up, 8 in 2024-09-16T10:26:53.390 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:26:53.390 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:26:53.390 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:53 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:53.421 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:53.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:53.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:53.565 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:53.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:53.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:53.704 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 77 16 116838 116822 94.8043 65.6875 0.0148083 0.0105432 2024-09-16T10:26:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:53.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:53.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:53.861 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:53.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:53.878 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:26:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:26:53.914 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:26:53.950 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:26:53.951 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:26:53.963 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:53.988 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:26:53.988 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:26:54.006 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:54.022 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:26:54.022 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:26:54.055 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:26:54.088 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:26:54.088 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:26:54.094 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:26:54.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:54.111 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:26:54.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:54.160 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:54.244 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:54.283 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:54.325 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: 6.0 scrub starts 2024-09-16T10:26:54.325 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: 6.0 scrub ok 2024-09-16T10:26:54.325 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: pgmap v708: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 27829/333660 objects degraded (8.341%) 2024-09-16T10:26:54.325 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:26:54.325 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:26:54.325 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2943490993' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:54.325 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: from='client.26197 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/529119418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: 6.0 scrub starts 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: 6.0 scrub ok 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: pgmap v708: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 27829/333660 objects degraded (8.341%) 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:26:54.326 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:26:54.327 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2943490993' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:54.327 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: from='client.26197 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:54.327 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:54.327 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:54.327 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/529119418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:54.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:54.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:54.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:54.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: 6.0 scrub starts 2024-09-16T10:26:54.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: 6.0 scrub ok 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: pgmap v708: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s; 27829/333660 objects degraded (8.341%) 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2943490993' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: from='client.26197 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:26:54.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/529119418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:26:54.554 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:54.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:54.682 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T10:26:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:54.704 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 78 16 118092 118076 94.5936 78.375 0.00645569 0.0105673 2024-09-16T10:26:54.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:54.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:54.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:54.958 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:54.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:55.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:55.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:55.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:55.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:55 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:55.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:55 smithi165 ceph-mon[32696]: from='client.26197 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:26:55.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:55 smithi165 ceph-mon[32696]: osdmap e396: 8 total, 7 up, 8 in 2024-09-16T10:26:55.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:55.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:55.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:55.337 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:55.372 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:55.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:55 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:55 smithi063 ceph-mon[29993]: from='client.26197 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:26:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:55 smithi063 ceph-mon[29993]: osdmap e396: 8 total, 7 up, 8 in 2024-09-16T10:26:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:55 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:26:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:55 smithi063 ceph-mon[39470]: from='client.26197 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:26:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:55 smithi063 ceph-mon[39470]: osdmap e396: 8 total, 7 up, 8 in 2024-09-16T10:26:55.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:55.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:55.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:55.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:55.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:55.704 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 79 16 119317 119301 94.3653 76.5625 0.0165128 0.0105923 2024-09-16T10:26:55.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:55.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:55.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:55.837 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:55.838 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:55.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:55.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:55.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:56.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:56.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:56.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:56.159 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:26:56.191 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:26:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:56.265 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:56.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:56.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:56.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:56.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:56.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:56 smithi063 ceph-mon[29993]: pgmap v710: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s; 29349/351591 objects degraded (8.347%) 2024-09-16T10:26:56.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:56 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 27829/333660 objects degraded (8.341%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:56 smithi063 ceph-mon[39470]: pgmap v710: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s; 29349/351591 objects degraded (8.347%) 2024-09-16T10:26:56.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:56 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 27829/333660 objects degraded (8.341%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:56.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:56.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:56 smithi165 ceph-mon[32696]: pgmap v710: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s; 29349/351591 objects degraded (8.347%) 2024-09-16T10:26:56.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:56 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 27829/333660 objects degraded (8.341%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:26:56.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:56.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:56.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:56.687 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:56.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:56.704 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:26:56.704083+0000 min lat: 0.00115798 max lat: 1.05572 avg lat: 0.0106084 2024-09-16T10:26:56.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:26:56.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 80 16 120649 120633 94.2263 83.25 0.0300079 0.0106084 2024-09-16T10:26:56.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:26:56.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:56.831 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:56.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:56.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:56.972 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:56.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:57.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:57.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:57.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:57.189 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:26:57.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:57.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:57.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:57.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:57.408 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:57.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:57.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:26:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:26:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:26:57.509 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:57.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:57.654 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:26:57.680 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:26:57.690 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:57.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:26:57.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 81 16 121933 121917 94.0537 80.25 0.00881818 0.0106279 2024-09-16T10:26:57.792 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:57.832 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:57.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:57.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:57.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:57.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:58.077 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:58.126 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:58.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:58.132 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:26:58.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:26:58.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:26:58.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:58.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:26:58.379 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:58.411 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:26:58.411 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:26:58.422 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:58.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:58.475 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:26:58 smithi165 ceph-mon[32696]: pgmap v711: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 804 B/s rd, 59 MiB/s wr, 940 op/s; 29349/351591 objects degraded (8.347%) 2024-09-16T10:26:58.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:26:58.560 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:58.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:26:58.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:58.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:58.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:58.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:26:58 smithi063 ceph-mon[29993]: pgmap v711: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 804 B/s rd, 59 MiB/s wr, 940 op/s; 29349/351591 objects degraded (8.347%) 2024-09-16T10:26:58.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:26:58 smithi063 ceph-mon[39470]: pgmap v711: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 804 B/s rd, 59 MiB/s wr, 940 op/s; 29349/351591 objects degraded (8.347%) 2024-09-16T10:26:58.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 82 16 123044 123028 93.7535 69.4375 0.0104228 0.0106614 2024-09-16T10:26:58.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:26:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:58.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:58.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:26:58.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:58.960 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:58.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:59.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:26:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:26:59.063 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:59.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:59.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:59.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:59.201 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:59.230 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:59.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:59.249 INFO:teuthology.orchestra.run.smithi165.stderr:Exporting 4.15 info 4.15( v 328'15 (0'0,328'15] local-lis/les=60/61 n=4 ec=60/60 lis/c=60/60 les/c/f=61/61/0 sis=60) 2024-09-16T10:26:59.252 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:ab569f81:gc::gc.12:head# 2024-09-16T10:26:59.252 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:ad3f17aa:::obj_delete_at_hint.0000000111:head# 2024-09-16T10:26:59.252 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:ad7a3a02:::obj_delete_at_hint.0000000048:head# 2024-09-16T10:26:59.252 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:ae95b544:::obj_delete_at_hint.0000000020:head# 2024-09-16T10:26:59.252 INFO:teuthology.orchestra.run.smithi165.stderr:Export successful 2024-09-16T10:26:59.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:26:59.332 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:26:59.375 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:59.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:26:59.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:59.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:59.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:59.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:26:59.655 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:26:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:59.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:26:59.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 83 16 124170 124154 93.4718 70.375 0.0579613 0.0106904 2024-09-16T10:26:59.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:59.794 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:59.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:26:59.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:26:59.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:26:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:26:59.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:00.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:00.068 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:00.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:00.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:00.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:00.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:00.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:00.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:00.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:00.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -n osd.7 --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 4.15 2024-09-16T10:27:00.414 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:27:00.424 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:27:00.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:00.433 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:00 smithi165 ceph-mon[32696]: pgmap v712: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 29740/365565 objects degraded (8.135%) 2024-09-16T10:27:00.433 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4062497964' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:00.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:00.491 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:00.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:00.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:00.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:00.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:00.678 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:00 smithi063 ceph-mon[29993]: pgmap v712: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 29740/365565 objects degraded (8.135%) 2024-09-16T10:27:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4062497964' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:00 smithi063 ceph-mon[39470]: pgmap v712: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 29740/365565 objects degraded (8.135%) 2024-09-16T10:27:00.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4062497964' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:00.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 84 16 125351 125335 93.2376 73.8125 0.0077629 0.0107207 2024-09-16T10:27:00.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:00.731 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/osd.7/config 2024-09-16T10:27:00.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:00.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:00.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:00.917 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:00.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:01.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:01.022 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:27:01.022 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:27:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:01.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:01.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:01.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:01.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:01.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:01.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:01.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:01.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:01.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4062497964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[29993]: osdmap e397: 8 total, 7 up, 8 in 2024-09-16T10:27:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 29740/365565 objects degraded (8.135%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:01.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[29993]: osdmap e398: 8 total, 7 up, 8 in 2024-09-16T10:27:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4062497964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:01.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[39470]: osdmap e397: 8 total, 7 up, 8 in 2024-09-16T10:27:01.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 29740/365565 objects degraded (8.135%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:01.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:01 smithi063 ceph-mon[39470]: osdmap e398: 8 total, 7 up, 8 in 2024-09-16T10:27:01.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:01.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:01 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4062497964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:01 smithi165 ceph-mon[32696]: osdmap e397: 8 total, 7 up, 8 in 2024-09-16T10:27:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:01 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 29740/365565 objects degraded (8.135%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:01 smithi165 ceph-mon[32696]: osdmap e398: 8 total, 7 up, 8 in 2024-09-16T10:27:01.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:01.607 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:01.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:01.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 85 16 126587 126571 93.0495 77.25 0.00993034 0.0107426 2024-09-16T10:27:01.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:01.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:01.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:01.838 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:01.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:01.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:01.900 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:01.900 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:01.939 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:01.939 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:01.972 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:01.972 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:01.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:02.010 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:02.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:02.046 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:02.047 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:02.086 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:02.088 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:02.118 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:02.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:02.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:02.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:02.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:02.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:02.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:02.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:02.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:02.658 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:02.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:02 smithi063 ceph-mon[29993]: pgmap v715: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 707 B/s rd, 67 MiB/s wr, 1.07k op/s; 30892/370167 objects degraded (8.345%) 2024-09-16T10:27:02.681 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4002363233' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:02.681 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:02 smithi063 ceph-mon[39470]: pgmap v715: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 707 B/s rd, 67 MiB/s wr, 1.07k op/s; 30892/370167 objects degraded (8.345%) 2024-09-16T10:27:02.682 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4002363233' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:02.696 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:02.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 86 16 127879 127863 92.9064 80.75 0.00689952 0.010759 2024-09-16T10:27:02.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:02.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:02 smithi165 ceph-mon[32696]: pgmap v715: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 707 B/s rd, 67 MiB/s wr, 1.07k op/s; 30892/370167 objects degraded (8.345%) 2024-09-16T10:27:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4002363233' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:02.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:02.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:27:02.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:27:02.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:02.983 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:02.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:03.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:27:03.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:03.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:03.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:03.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:03.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:03.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:03.252 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:27:03.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:03.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:03.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:03.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:03.497 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:03.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:03.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:03.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:03.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:03.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:03.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 87 16 128986 128970 92.6337 69.1875 0.0103781 0.0107904 2024-09-16T10:27:03.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:03.777 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:03.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:03.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:03.923 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:03.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:04.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:04.074 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:04.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:04.172 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:27:04.172 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:27:04.172 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:129480 10.8 10:1d3b9866:::benchmark_data_smithi063_149043_object129479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:04.161969+0000", 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0067034249999999998, 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067364160000000003, 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 129480 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.173 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.161969+0000", 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.161969+0000", 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.174 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.161971+0000", 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9700000000000002e-06 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.161997+0000", 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5731000000000001e-05 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162009+0000", 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2685e-05 2024-09-16T10:27:04.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162017+0000", 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9000000000000006e-06 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162852+0000", 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00083505000000000005 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.163060+0000", 2024-09-16T10:27:04.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000207467 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2", 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.163210+0000", 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000150586 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:129482 10.c 10:379630d5:::benchmark_data_smithi063_149043_object129481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:04.162353+0000", 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0063196260000000001, 2024-09-16T10:27:04.177 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0064257819999999997, 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 129482 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162353+0000", 2024-09-16T10:27:04.178 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162353+0000", 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162354+0000", 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.471e-06 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.179 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162377+0000", 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.243e-05 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162387+0000", 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0447e-05 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162395+0000", 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5989999999999996e-06 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162450+0000", 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5198999999999998e-05 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162606+0000", 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015636799999999999 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4", 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162768+0000", 2024-09-16T10:27:04.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016204199999999999 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.167770+0000", 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0050016879999999998 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:129483 10.4 10:21dfe2a0:::benchmark_data_smithi063_149043_object129482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:04.162406+0000", 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0062667799999999996, 2024-09-16T10:27:04.182 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0064532130000000002, 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 129483 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162406+0000", 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162406+0000", 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162406+0000", 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6899999999999997e-07 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162427+0000", 2024-09-16T10:27:04.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0503e-05 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162433+0000", 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.618e-06 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162439+0000", 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5400000000000003e-06 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:04.185 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165574+0000", 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031355369999999999 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165733+0000", 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015851799999999999 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,4", 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165935+0000", 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000202129 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.186 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:129485 10.c 10.4b1e432c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:04.162950+0000", 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0057221140000000004, 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00597257, 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:04.187 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 129485 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162950+0000", 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162950+0000", 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.188 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162951+0000", 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3799999999999996e-07 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162977+0000", 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6080000000000001e-05 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162986+0000", 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9700000000000005e-06 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162992+0000", 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9789999999999998e-06 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.16152.0:129482 10.c e398/387 ondisk, result = 0)", 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:04.165686+0000", 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0029865529999999999, 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032803419999999999, 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:04.190 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165686+0000", 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165686+0000", 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.191 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165687+0000", 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5600000000000001e-06 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165696+0000", 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4039999999999995e-06 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165706+0000", 2024-09-16T10:27:04.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0694e-05 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.165713+0000", 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9269999999999999e-06 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.167755+0000", 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020414840000000001 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:04.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.167763+0000", 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4849999999999992e-06 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.16152.0:129483 10.4 e398/362)", 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:04.168394+0000", 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.00027829999999999999, 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00061540499999999997, 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:27:04.194 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.168394+0000", 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.168394+0000", 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:04.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.168396+0000", 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.513e-06 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.168404+0000", 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2800000000000003e-06 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.168413+0000", 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4259999999999992e-06 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.168420+0000", 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1250000000000004e-06 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:129467 10.4 10:26b5dca4:::benchmark_data_smithi063_149043_object129466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:04.143212+0000", 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.025460244, 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.025841070000000001, 2024-09-16T10:27:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 129467 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.143212+0000", 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.143212+0000", 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.143213+0000", 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8299999999999995e-07 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:04.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.143279+0000", 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5563999999999998e-05 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.143300+0000", 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1053e-05 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.143306+0000", 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6839999999999999e-06 2024-09-16T10:27:04.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.155715+0000", 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012408697 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.155872+0000", 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000157016 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,4", 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.156021+0000", 2024-09-16T10:27:04.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014850200000000001 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162385+0000", 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0063643650000000003 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:04.162419+0000", 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4254000000000001e-05 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:04.203 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:27:04.203 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 7 2024-09-16T10:27:04.203 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:27:04.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:04.245 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:04.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:04.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:04.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:04.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:04.484 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:04.523 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:04.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:04 smithi165 ceph-mon[32696]: pgmap v716: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 774 op/s; 30892/370167 objects degraded (8.345%) 2024-09-16T10:27:04.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:04.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:04.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:04.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:04 smithi063 ceph-mon[29993]: pgmap v716: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 774 op/s; 30892/370167 objects degraded (8.345%) 2024-09-16T10:27:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:04 smithi063 ceph-mon[39470]: pgmap v716: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 774 op/s; 30892/370167 objects degraded (8.345%) 2024-09-16T10:27:04.705 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 88 16 130122 130106 92.3878 71 0.00755288 0.0108199 2024-09-16T10:27:04.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:04.779 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:04.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:04.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:04.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:04.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:04.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:27:05.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:05.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:05.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:05.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:05.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.310 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:05.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:05.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:05.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:05.466 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:05.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:05.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:05.673 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:05.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 89 15 131221 131206 92.1221 68.75 0.0145214 0.0108505 2024-09-16T10:27:05.716 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:05.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:05.849 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:05.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:05.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:05.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:06.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:06.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:06.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:06.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:06.267 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:06.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:06.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:06.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:06.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:06.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:06.537 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:06.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:06.639 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:06.678 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:06.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:06 smithi063 ceph-mon[29993]: pgmap v717: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 32314/387960 objects degraded (8.329%) 2024-09-16T10:27:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:06 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 32314/387960 objects degraded (8.329%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:06 smithi063 ceph-mon[39470]: pgmap v717: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 32314/387960 objects degraded (8.329%) 2024-09-16T10:27:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:06 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 32314/387960 objects degraded (8.329%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total time run: 90.0085 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total writes made: 132434 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Write size: 65536 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object size: 65536 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Bandwidth (MB/sec): 91.9594 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Bandwidth: 33.9875 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max bandwidth (MB/sec): 212 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min bandwidth (MB/sec): 11.4375 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average IOPS: 1471 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev IOPS: 543.799 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max IOPS: 3392 2024-09-16T10:27:06.706 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min IOPS: 183 2024-09-16T10:27:06.707 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average Latency(s): 0.0108702 2024-09-16T10:27:06.707 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Latency(s): 0.0175079 2024-09-16T10:27:06.707 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max latency(s): 1.05572 2024-09-16T10:27:06.707 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min latency(s): 0.00115798 2024-09-16T10:27:06.712 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Cleaning up (deleting benchmark objects) 2024-09-16T10:27:06.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:06 smithi165 ceph-mon[32696]: pgmap v717: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 32314/387960 objects degraded (8.329%) 2024-09-16T10:27:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:06 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 32314/387960 objects degraded (8.329%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:06.820 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:06.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:06.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:06.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:06.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:07.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:07.072 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:27:07.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:07.119 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:07.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:07.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:07.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:07.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:07.337 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:27:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:27:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:27:07.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:07.402 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:07.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:07.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:07.547 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:07.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:07.623 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:07.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:07.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:07.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:07.778 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:07.827 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:07.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:07.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:07.985 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:07.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:08.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:08.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:08.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:08.148 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T10:27:08.173 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:08.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:08.249 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:08.250 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:08.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:08.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:08.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:08.284 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:08.285 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:08.318 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:08.318 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:08.352 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:08.352 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:08.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:08.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:08.381 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:08.381 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:08.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:08.397 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:08.397 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:08.429 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:08.429 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:08.436 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:27:08.441 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:08.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:08.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:08.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:08.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:08.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[29993]: pgmap v718: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 933 op/s; 32314/387960 objects degraded (8.329%) 2024-09-16T10:27:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1916889843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[29993]: from='client.26218 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3788698181' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:08.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[39470]: pgmap v718: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 933 op/s; 32314/387960 objects degraded (8.329%) 2024-09-16T10:27:08.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1916889843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:08.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[39470]: from='client.26218 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:08.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3788698181' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:08.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:08.773 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:08.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:08.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:08.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:08 smithi165 ceph-mon[32696]: pgmap v718: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 933 op/s; 32314/387960 objects degraded (8.329%) 2024-09-16T10:27:08.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1916889843' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:08.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:08 smithi165 ceph-mon[32696]: from='client.26218 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:08.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3788698181' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:08.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:08.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:08.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:09.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:09.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:27:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:27:09.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:09.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:09.103 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:27:09.104 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:27:09.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:09.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:09.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:09.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:09.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:09.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:09.427 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:09.465 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:09.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:09.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:09.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:09.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:09.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[29993]: from='client.26218 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[29993]: osdmap e399: 8 total, 7 up, 8 in 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[39470]: from='client.26218 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[39470]: osdmap e399: 8 total, 7 up, 8 in 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:09.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:09.703 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:09.744 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:09.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:09 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:09 smithi165 ceph-mon[32696]: from='client.26218 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:27:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:09 smithi165 ceph-mon[32696]: osdmap e399: 8 total, 7 up, 8 in 2024-09-16T10:27:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:09.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:09.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:09.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:09.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.025 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:10.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:10.126 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:10.150 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:27:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:10.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:10.304 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:10.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:10.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:10.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:10.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:27:10.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:27:10.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:10.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[29993]: pgmap v720: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.21k op/s; 32422/391362 objects degraded (8.284%) 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[29993]: 10.2 scrub starts 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[39470]: pgmap v720: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.21k op/s; 32422/391362 objects degraded (8.284%) 2024-09-16T10:27:10.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:10 smithi063 ceph-mon[39470]: 10.2 scrub starts 2024-09-16T10:27:10.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:10.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:10.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:10 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:27:10.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:10 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:27:10.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:10 smithi165 ceph-mon[32696]: pgmap v720: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.21k op/s; 32422/391362 objects degraded (8.284%) 2024-09-16T10:27:10.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:10 smithi165 ceph-mon[32696]: 10.2 scrub starts 2024-09-16T10:27:10.807 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:10.847 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:10.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:10.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:10.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:11.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:11.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:11.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:11.142 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:11.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:11.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:11.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:11.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:11.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:11.395 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:11.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:11.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:11.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:11.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:11.634 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:27:11.634 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[29993]: 10.5 scrub starts 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[29993]: 10.d scrub starts 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 32422/391362 objects degraded (8.284%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[29993]: osdmap e400: 8 total, 7 up, 8 in 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[39470]: 10.5 scrub starts 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[39470]: 10.d scrub starts 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 32422/391362 objects degraded (8.284%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:11.635 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:11 smithi063 ceph-mon[39470]: osdmap e400: 8 total, 7 up, 8 in 2024-09-16T10:27:11.675 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:11.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:11.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:11 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:27:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:11 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:27:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:11 smithi165 ceph-mon[32696]: 10.5 scrub starts 2024-09-16T10:27:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:11 smithi165 ceph-mon[32696]: 10.d scrub starts 2024-09-16T10:27:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:11 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 32422/391362 objects degraded (8.284%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:11 smithi165 ceph-mon[32696]: osdmap e400: 8 total, 7 up, 8 in 2024-09-16T10:27:11.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:11.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:11.809 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:27:11.809 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:27:11.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:11.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:11.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:12.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:12.086 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:12.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:12.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:12.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:12.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:12.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:12.367 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:12.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:12.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:12.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:12.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:12.618 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:27:12.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:12.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:12.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:27:12.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:12 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:27:12.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:12 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:27:12.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:12 smithi063 ceph-mon[29993]: pgmap v722: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.45k op/s; 32295/390861 objects degraded (8.263%) 2024-09-16T10:27:12.740 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:12 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:27:12.740 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:12 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:27:12.740 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:12 smithi063 ceph-mon[39470]: pgmap v722: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.45k op/s; 32295/390861 objects degraded (8.263%) 2024-09-16T10:27:12.741 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:12.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:12.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:12 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:27:12.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:12 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:27:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:12 smithi165 ceph-mon[32696]: pgmap v722: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.45k op/s; 32295/390861 objects degraded (8.263%) 2024-09-16T10:27:12.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:12.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:12.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:13.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:13.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:27:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:13.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:13.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:13.133 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:13.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:13.206 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:13.252 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:13.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:13.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:13.402 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:13.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:13.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:13.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:13.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:13.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:13.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:13.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:13.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[29993]: 10.2 scrub ok 2024-09-16T10:27:13.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[29993]: 3.b scrub starts 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[29993]: 3.b scrub ok 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[29993]: 4.f scrub starts 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[29993]: 4.f scrub ok 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[29993]: 10.d scrub ok 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[39470]: 10.2 scrub ok 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[39470]: 3.b scrub starts 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[39470]: 3.b scrub ok 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[39470]: 4.f scrub starts 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[39470]: 4.f scrub ok 2024-09-16T10:27:13.763 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:13 smithi063 ceph-mon[39470]: 10.d scrub ok 2024-09-16T10:27:13.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:13.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:13 smithi165 ceph-mon[32696]: 10.2 scrub ok 2024-09-16T10:27:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:13 smithi165 ceph-mon[32696]: 3.b scrub starts 2024-09-16T10:27:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:13 smithi165 ceph-mon[32696]: 3.b scrub ok 2024-09-16T10:27:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:13 smithi165 ceph-mon[32696]: 4.f scrub starts 2024-09-16T10:27:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:13 smithi165 ceph-mon[32696]: 4.f scrub ok 2024-09-16T10:27:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:13 smithi165 ceph-mon[32696]: 10.d scrub ok 2024-09-16T10:27:13.820 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:13.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:13.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:13.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:13.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:14.030 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:14.031 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:14.034 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:14.034 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:14.055 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:14.072 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:14.073 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:14.074 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:14.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:14.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:14.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:14.233 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:14.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:14.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:14.383 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:14.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.16152.0:141109 10.b e400/362 10:daada646:::benchmark_data_smithi063_149043_object8672:head v 400'8754, mlcod=400'8752)", 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:14.298479+0000", 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.00069443900000000004, 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00072184899999999999, 2024-09-16T10:27:14.468 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298479+0000", 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298479+0000", 2024-09-16T10:27:14.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298481+0000", 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7859999999999999e-06 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298495+0000", 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3721e-05 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298512+0000", 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7278999999999998e-05 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298521+0000", 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6999999999999997e-06 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298640+0000", 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119413 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298658+0000", 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7946999999999999e-05 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:141112 10.2 10:45873757:::benchmark_data_smithi063_149043_object8675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:14.298509+0000", 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.00066483999999999998, 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00075399099999999997, 2024-09-16T10:27:14.472 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 141112 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298509+0000", 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:14.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298509+0000", 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298511+0000", 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.097e-06 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298517+0000", 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6000000000000003e-06 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298529+0000", 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1474000000000001e-05 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298536+0000", 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3780000000000003e-06 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298790+0000", 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025341099999999999 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.298927+0000", 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013759899999999999 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:14.299034+0000", 2024-09-16T10:27:14.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000107129 2024-09-16T10:27:14.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:14.477 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:14.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:14.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:14.477 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:27:14.477 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 2 2024-09-16T10:27:14.477 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:27:14.484 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:14.518 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:14.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:14.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:14.655 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:14.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: 10.5 scrub ok 2024-09-16T10:27:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: 3.e scrub starts 2024-09-16T10:27:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: 3.e scrub ok 2024-09-16T10:27:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: 10.0 scrub starts 2024-09-16T10:27:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: pgmap v723: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 26 MiB/s wr, 707 op/s; 32295/390861 objects degraded (8.263%) 2024-09-16T10:27:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:27:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1002856824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: 10.5 scrub ok 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: 3.e scrub starts 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: 3.e scrub ok 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: 10.0 scrub starts 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: pgmap v723: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 26 MiB/s wr, 707 op/s; 32295/390861 objects degraded (8.263%) 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:27:14.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1002856824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:14.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:14.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:14.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: 10.5 scrub ok 2024-09-16T10:27:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: 3.e scrub starts 2024-09-16T10:27:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: 3.e scrub ok 2024-09-16T10:27:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: 10.0 scrub starts 2024-09-16T10:27:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: pgmap v723: 168 pgs: 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 26 MiB/s wr, 707 op/s; 32295/390861 objects degraded (8.263%) 2024-09-16T10:27:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:27:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:27:14.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1002856824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:14.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:14.928 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:14.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:15.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:15.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:15.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:15.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:15.166 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:15.178 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:27:15.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:15.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:15.302 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:15.348 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:15.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:15.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:15.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:15.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:15.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:15.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:15.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:15.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:15.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:15 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:27:15.747 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:15 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:27:15.747 INFO:teuthology.orchestra.run.smithi165.stdout: marking collection for removal 2024-09-16T10:27:15.747 INFO:teuthology.orchestra.run.smithi165.stdout:setting '_remove' omap key 2024-09-16T10:27:15.747 INFO:teuthology.orchestra.run.smithi165.stdout:finish_remove_pgs 4.15_head removing 4.15 2024-09-16T10:27:15.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:15.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:15.753 INFO:teuthology.orchestra.run.smithi165.stdout:Remove successful 2024-09-16T10:27:15.819 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:15 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:27:15.819 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:15 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:27:15.820 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:15 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:27:15.820 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:15 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:27:15.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:27:15.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:27:15.898 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:15.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:16.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:16.046 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:16.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:16.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:16.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:16.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:16.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:16.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:16.534 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:16 smithi165 ceph-mon[32696]: pgmap v724: 168 pgs: 1 active+clean+scrubbing, 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 1.43k op/s; 30881/373557 objects degraded (8.267%) 2024-09-16T10:27:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:16 smithi165 ceph-mon[32696]: 10.0 scrub ok 2024-09-16T10:27:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:16 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 30881/373557 objects degraded (8.267%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:16.568 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:16.670 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:16.706 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:16.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:16.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:16.950 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:16 smithi063 ceph-mon[29993]: pgmap v724: 168 pgs: 1 active+clean+scrubbing, 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 1.43k op/s; 30881/373557 objects degraded (8.267%) 2024-09-16T10:27:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:16 smithi063 ceph-mon[29993]: 10.0 scrub ok 2024-09-16T10:27:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:16 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 30881/373557 objects degraded (8.267%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:16 smithi063 ceph-mon[39470]: pgmap v724: 168 pgs: 1 active+clean+scrubbing, 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 26 MiB/s wr, 1.43k op/s; 30881/373557 objects degraded (8.267%) 2024-09-16T10:27:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:16 smithi063 ceph-mon[39470]: 10.0 scrub ok 2024-09-16T10:27:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:16 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 30881/373557 objects degraded (8.267%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:16.989 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:16.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:16.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:17.092 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:17.127 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:17.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:17.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:17.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell -n osd.7 --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.15.7 2024-09-16T10:27:17.266 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:17.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:17.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:17.391 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:17.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:27:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:27:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:27:17.463 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/osd.7/config 2024-09-16T10:27:17.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:17.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:17.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:17.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:17.673 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:17.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:17.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:17.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:17.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:27:17.910 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:17.971 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:17.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:18.072 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:18.109 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:18.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:18.211 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:18.234 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:18.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:18.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:18.373 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:18.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:18.475 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:18.526 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:18.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:18.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:18.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:18.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:18.752 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[29993]: pgmap v725: 168 pgs: 1 active+clean+scrubbing, 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 1.28k op/s; 30881/373557 objects degraded (8.267%) 2024-09-16T10:27:18.752 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:27:18.752 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[29993]: 1.5 scrub ok 2024-09-16T10:27:18.752 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[29993]: osdmap e401: 8 total, 7 up, 8 in 2024-09-16T10:27:18.753 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[39470]: pgmap v725: 168 pgs: 1 active+clean+scrubbing, 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 1.28k op/s; 30881/373557 objects degraded (8.267%) 2024-09-16T10:27:18.753 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:27:18.753 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[39470]: 1.5 scrub ok 2024-09-16T10:27:18.753 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:18 smithi063 ceph-mon[39470]: osdmap e401: 8 total, 7 up, 8 in 2024-09-16T10:27:18.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:18.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:18 smithi165 ceph-mon[32696]: pgmap v725: 168 pgs: 1 active+clean+scrubbing, 41 active+undersized, 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 1.28k op/s; 30881/373557 objects degraded (8.267%) 2024-09-16T10:27:18.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:18 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:27:18.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:18 smithi165 ceph-mon[32696]: 1.5 scrub ok 2024-09-16T10:27:18.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:18 smithi165 ceph-mon[32696]: osdmap e401: 8 total, 7 up, 8 in 2024-09-16T10:27:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:18.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:18.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:18.803 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:18.827 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T10:27:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:18.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:18.933 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:18.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:18.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:19.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:19.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:27:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:27:19.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:19.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:19.168 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:19.206 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:19.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:19.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:19.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:19.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:19.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:19.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:19.481 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:19.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:19.557 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:27:19.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:19.621 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:19.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:19.626 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:27:19.626 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:27:19.627 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:27:19.627 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:27:19.627 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:27:19.713 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[29993]: 6.1f scrub starts 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[29993]: 6.1f scrub ok 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[29993]: osdmap e402: 8 total, 7 up, 8 in 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1595103797' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[29993]: from='client.26236 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[39470]: 6.1f scrub starts 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[39470]: 6.1f scrub ok 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[39470]: osdmap e402: 8 total, 7 up, 8 in 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1595103797' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:19.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:19 smithi063 ceph-mon[39470]: from='client.26236 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:19.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:19.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:19.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:19 smithi165 ceph-mon[32696]: 6.1f scrub starts 2024-09-16T10:27:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:19 smithi165 ceph-mon[32696]: 6.1f scrub ok 2024-09-16T10:27:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:19 smithi165 ceph-mon[32696]: osdmap e402: 8 total, 7 up, 8 in 2024-09-16T10:27:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1595103797' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:19 smithi165 ceph-mon[32696]: from='client.26236 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:19.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:19.886 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:19.886 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:19.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:19.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:19.916 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:19.916 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:19.949 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:19.949 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:19.986 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:19.986 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:20.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:20.018 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:20.018 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:20.046 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:20.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:20.052 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:20.053 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:20.088 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:20.089 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:20.089 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:20.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:20.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:20.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:20.285 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:20.321 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:20.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:20.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:20.461 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:20.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:20.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:20.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:20.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:20.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:20.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:20.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:20.775 INFO:teuthology.orchestra.run.smithi165.stdout:get_pg_num_history pg_num_history pg_num_history(e386 pg_nums {1={275=11,306=21},2={57=3},3={58=32},4={60=32},5={62=32},6={64=32},9={276=16},10={362=16}} deleted_pools 360,9) 2024-09-16T10:27:20.775 INFO:teuthology.orchestra.run.smithi165.stdout:Importing pgid 4.15 2024-09-16T10:27:20.776 INFO:teuthology.orchestra.run.smithi165.stdout:Write #4:ab569f81:gc::gc.12:head# 2024-09-16T10:27:20.776 INFO:teuthology.orchestra.run.smithi165.stdout:snapset 0=[]:{} 2024-09-16T10:27:20.776 INFO:teuthology.orchestra.run.smithi165.stdout:Write #4:ad3f17aa:::obj_delete_at_hint.0000000111:head# 2024-09-16T10:27:20.776 INFO:teuthology.orchestra.run.smithi165.stdout:snapset 0=[]:{} 2024-09-16T10:27:20.776 INFO:teuthology.orchestra.run.smithi165.stdout:Write #4:ad7a3a02:::obj_delete_at_hint.0000000048:head# 2024-09-16T10:27:20.777 INFO:teuthology.orchestra.run.smithi165.stdout:snapset 0=[]:{} 2024-09-16T10:27:20.777 INFO:teuthology.orchestra.run.smithi165.stdout:Write #4:ae95b544:::obj_delete_at_hint.0000000020:head# 2024-09-16T10:27:20.777 INFO:teuthology.orchestra.run.smithi165.stdout:snapset 0=[]:{} 2024-09-16T10:27:20.777 INFO:teuthology.orchestra.run.smithi165.stdout:write_pg epoch 369 info 4.15( v 328'15 (0'0,328'15] local-lis/les=60/61 n=4 ec=60/60 lis/c=60/60 les/c/f=61/61/0 sis=60) 2024-09-16T10:27:20.778 INFO:teuthology.orchestra.run.smithi165.stdout:Import successful 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: 10.6 scrub starts 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: pgmap v728: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 30489/359805 objects degraded (8.474%) 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: from='client.26236 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: osdmap e403: 8 total, 7 up, 8 in 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2966903365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:27:20.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:20 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:27:20.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:20.863 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:20.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:20.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: 10.6 scrub starts 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: pgmap v728: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 30489/359805 objects degraded (8.474%) 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: from='client.26236 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: osdmap e403: 8 total, 7 up, 8 in 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2966903365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:27:20.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: 10.6 scrub starts 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: pgmap v728: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 30489/359805 objects degraded (8.474%) 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: from='client.26236 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: osdmap e403: 8 total, 7 up, 8 in 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2966903365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:27:20.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:20 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:27:20.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.013 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:21.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:21.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:27:21.114 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:21.167 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:21.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:21.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:21.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:21.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:21.528 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:21.572 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:21.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:21.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:21.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[29993]: 4.b scrub starts 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[29993]: 4.b scrub ok 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 30489/359805 objects degraded (8.474%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[29993]: osdmap e404: 8 total, 7 up, 8 in 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[29993]: 10.6 scrub ok 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[39470]: 4.b scrub starts 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[39470]: 4.b scrub ok 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 30489/359805 objects degraded (8.474%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[39470]: osdmap e404: 8 total, 7 up, 8 in 2024-09-16T10:27:21.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:21 smithi063 ceph-mon[39470]: 10.6 scrub ok 2024-09-16T10:27:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:21 smithi165 ceph-mon[32696]: 4.b scrub starts 2024-09-16T10:27:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:21 smithi165 ceph-mon[32696]: 4.b scrub ok 2024-09-16T10:27:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:21 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 30489/359805 objects degraded (8.474%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:21 smithi165 ceph-mon[32696]: osdmap e404: 8 total, 7 up, 8 in 2024-09-16T10:27:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:21 smithi165 ceph-mon[32696]: 10.6 scrub ok 2024-09-16T10:27:21.803 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:21.838 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:21.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:21.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:21.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:22.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:22.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:22.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:22.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/exp.4.15.7 2024-09-16T10:27:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:27:22.152 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:27:22.161 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:22.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:22.243 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:22.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:22.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:22.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:22.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:22.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:22.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:22.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:22.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:22.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:22.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:22 smithi063 ceph-mon[29993]: pgmap v731: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.52k op/s; 29390/355278 objects degraded (8.272%) 2024-09-16T10:27:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:22 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:27:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:22 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:27:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:22 smithi063 ceph-mon[39470]: pgmap v731: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.52k op/s; 29390/355278 objects degraded (8.272%) 2024-09-16T10:27:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:22 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:27:22.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:22 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:27:22.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:22.770 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:27:22.770 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:27:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:22.783 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:22.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:22 smithi165 ceph-mon[32696]: pgmap v731: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.52k op/s; 29390/355278 objects degraded (8.272%) 2024-09-16T10:27:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:22 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:27:22.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:22 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:27:22.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:22.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:22.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:23.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:23.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:23.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:23.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:23.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:23.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:23.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:23.296 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:23.325 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:23.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:23.427 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:23.465 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:23.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:23.467 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:27:23.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:23.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:23.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:23.718 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:23.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:23.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:23.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:23.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:23.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:23.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:23 smithi063 ceph-mon[29993]: 2.2 scrub starts 2024-09-16T10:27:23.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:23 smithi063 ceph-mon[29993]: 2.2 scrub ok 2024-09-16T10:27:23.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:23 smithi063 ceph-mon[39470]: 2.2 scrub starts 2024-09-16T10:27:23.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:23 smithi063 ceph-mon[39470]: 2.2 scrub ok 2024-09-16T10:27:24.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.041 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:24.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:23 smithi165 ceph-mon[32696]: 2.2 scrub starts 2024-09-16T10:27:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:23 smithi165 ceph-mon[32696]: 2.2 scrub ok 2024-09-16T10:27:24.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:24.153 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:27:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:24.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:24.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:24.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:24.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.448 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:24.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:24.549 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.576 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:24.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:24.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:24.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:24.812 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:24.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:24.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:24.978 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:24.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:25.080 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:25.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:25.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:25.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[29993]: pgmap v732: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.04k op/s; 29390/355278 objects degraded (8.272%) 2024-09-16T10:27:25.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:25.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:25.167 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[29993]: 5.0 scrub starts 2024-09-16T10:27:25.168 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[29993]: 5.0 scrub ok 2024-09-16T10:27:25.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[39470]: pgmap v732: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.04k op/s; 29390/355278 objects degraded (8.272%) 2024-09-16T10:27:25.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:25.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:25.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[39470]: 5.0 scrub starts 2024-09-16T10:27:25.168 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:24 smithi063 ceph-mon[39470]: 5.0 scrub ok 2024-09-16T10:27:25.221 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:27:25.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:25.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:25.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:24 smithi165 ceph-mon[32696]: pgmap v732: 168 pgs: 1 peering, 41 active+undersized, 22 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.04k op/s; 29390/355278 objects degraded (8.272%) 2024-09-16T10:27:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:24 smithi165 ceph-mon[32696]: 5.0 scrub starts 2024-09-16T10:27:25.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:24 smithi165 ceph-mon[32696]: 5.0 scrub ok 2024-09-16T10:27:25.356 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:25.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:25.373 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:25.422 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:25.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:25.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:25.579 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:25.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:25.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:25.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:25.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:25.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:25.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:25.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:25.943 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:25.983 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:25.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:26.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:26.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:26.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:26.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:26.263 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:26.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:26.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:26.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:26.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:26.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:26.563 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:26.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:26.666 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:26.710 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:26.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:26.811 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:26.853 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:26.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:26.907 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[29993]: pgmap v733: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 27796/336018 objects degraded (8.272%) 2024-09-16T10:27:26.907 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:27:26.907 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[29993]: 10.4 scrub starts 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[29993]: osdmap e405: 8 total, 7 up, 8 in 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 27796/336018 objects degraded (8.272%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[29993]: osdmap e406: 8 total, 7 up, 8 in 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[39470]: pgmap v733: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 27796/336018 objects degraded (8.272%) 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[39470]: 10.4 scrub starts 2024-09-16T10:27:26.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[39470]: osdmap e405: 8 total, 7 up, 8 in 2024-09-16T10:27:26.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 27796/336018 objects degraded (8.272%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:26.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:26 smithi063 ceph-mon[39470]: osdmap e406: 8 total, 7 up, 8 in 2024-09-16T10:27:26.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:26.997 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:26.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:27.035 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:27.100 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:27.101 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:27.102 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:27.135 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:27.135 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:27.144 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:27.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:27.171 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:27.171 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:27.171 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:27:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:27:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:27:27.182 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-16T10:27:27.183 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:27:27.183 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-09-16T10:27:27.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:27:27.205 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:27.206 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:27.241 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:27.241 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:27.246 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:27.281 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:27.285 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:27.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:26 smithi165 ceph-mon[32696]: pgmap v733: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 27796/336018 objects degraded (8.272%) 2024-09-16T10:27:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:26 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:27:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:26 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:27:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:26 smithi165 ceph-mon[32696]: 10.4 scrub starts 2024-09-16T10:27:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:26 smithi165 ceph-mon[32696]: osdmap e405: 8 total, 7 up, 8 in 2024-09-16T10:27:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:26 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 27796/336018 objects degraded (8.272%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:27.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:26 smithi165 ceph-mon[32696]: osdmap e406: 8 total, 7 up, 8 in 2024-09-16T10:27:27.312 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:27.312 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:27.313 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:27.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:27.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:27.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:27.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:27.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:27.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:27.640 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:27.682 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:27.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:27.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:157549 10.3 10:c86ce2c9:::benchmark_data_smithi063_149043_object25112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:27.788041+0000", 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0067360859999999996, 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067709270000000004, 2024-09-16T10:27:27.797 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 157549 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788041+0000", 2024-09-16T10:27:27.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788041+0000", 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788043+0000", 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4499999999999998e-06 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:27.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788049+0000", 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0900000000000001e-06 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788065+0000", 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5675000000000002e-05 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788073+0000", 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7069999999999998e-06 2024-09-16T10:27:27.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788198+0000", 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012485900000000001 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788331+0000", 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013270400000000001 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:27:27.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.788449+0000", 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011753599999999999 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.793309+0000", 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0048601989999999999 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:157530 10.5 10:a22ed26b:::benchmark_data_smithi063_149043_object25093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-16T10:27:27.802 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:27.772839+0000", 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.021938000999999999, 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.022048829999999998, 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 157530 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:27:27.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.772839+0000", 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.772839+0000", 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.772840+0000", 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.474e-06 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.772845+0000", 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0359999999999997e-06 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.772859+0000", 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.415e-05 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.772866+0000", 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8390000000000001e-06 2024-09-16T10:27:27.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.773680+0000", 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00081373299999999997 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.773791+0000", 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111421 2024-09-16T10:27:27.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.773902+0000", 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011107699999999999 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.778158+0000", 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042555270000000003 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:27:27.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:27.778553+0000", 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00039550000000000002 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 2 2024-09-16T10:27:27.808 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:27:27.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:27.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:27.835 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:27:27.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1002665257' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:27:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:27 smithi063 ceph-mon[29993]: from='client.26251 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:27:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2675716322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1002665257' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:27:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:27 smithi063 ceph-mon[39470]: from='client.26251 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:27:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2675716322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:27.957 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:27.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:28.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:28.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:28.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:28.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:28.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1002665257' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:27:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:27 smithi165 ceph-mon[32696]: from='client.26251 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:27:28.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2675716322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:28.233 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:28.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:28.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:28.374 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:28.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:28.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:28.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:28.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:28.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:28.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:28.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:28.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:28.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:28.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:28.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:28.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:28.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:28.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:29.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:29.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:29.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:29.165 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:29.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:28 smithi165 ceph-mon[32696]: pgmap v736: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.01k op/s; 27796/336018 objects degraded (8.272%) 2024-09-16T10:27:29.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:28 smithi165 ceph-mon[32696]: 3.17 scrub starts 2024-09-16T10:27:29.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:28 smithi165 ceph-mon[32696]: 3.17 scrub ok 2024-09-16T10:27:29.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:28 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:29.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:28 smithi165 ceph-mon[32696]: from='client.26251 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:27:29.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:28 smithi165 ceph-mon[32696]: osdmap e407: 8 total, 7 up, 8 in 2024-09-16T10:27:29.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:27:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:27:29.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[29993]: pgmap v736: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.01k op/s; 27796/336018 objects degraded (8.272%) 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[29993]: 3.17 scrub starts 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[29993]: 3.17 scrub ok 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[29993]: from='client.26251 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[29993]: osdmap e407: 8 total, 7 up, 8 in 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[39470]: pgmap v736: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.01k op/s; 27796/336018 objects degraded (8.272%) 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[39470]: 3.17 scrub starts 2024-09-16T10:27:29.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[39470]: 3.17 scrub ok 2024-09-16T10:27:29.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:29.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[39470]: from='client.26251 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:27:29.204 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:28 smithi063 ceph-mon[39470]: osdmap e407: 8 total, 7 up, 8 in 2024-09-16T10:27:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:29.205 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:29.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:29.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:29.319 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:27:29.358 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:27:29.372 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:29.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:29.474 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:29.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:29.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:29.649 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:29.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:29.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:29.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:29.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:29.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:29.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:29.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:30.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:30.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:27:30.062 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:30.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:30.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:30.216 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:30.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:30.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:30.387 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:30.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:30.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:30.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:30.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:30.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:30.666 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:30.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:30.757 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:30.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:30.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:30.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:30.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:27:30.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:30.927 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:30.927 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:30 smithi063 ceph-mon[29993]: pgmap v738: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.80k op/s; 27445/322761 objects degraded (8.503%) 2024-09-16T10:27:30.927 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:30.927 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:30 smithi063 ceph-mon[39470]: pgmap v738: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.80k op/s; 27445/322761 objects degraded (8.503%) 2024-09-16T10:27:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:30.984 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:30.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:31.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:31.092 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:27:31.095 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:27:31.124 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:31.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:31.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:31.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:31.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:31.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:31.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:30 smithi165 ceph-mon[32696]: pgmap v738: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.80k op/s; 27445/322761 objects degraded (8.503%) 2024-09-16T10:27:31.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:31.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:27:31.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:31.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:31.390 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:27:31.390 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:27:31.390 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:27:31.390 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:27:31.390 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:27:31.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:31.527 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:31.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:31.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:31.676 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:31.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:31.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:31.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:31.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:31.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:31.939 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:31.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:31 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 27445/322761 objects degraded (8.503%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:31.940 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:31 smithi063 ceph-mon[29993]: osdmap e408: 8 total, 7 up, 8 in 2024-09-16T10:27:31.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:31 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 27445/322761 objects degraded (8.503%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:31.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:31 smithi063 ceph-mon[39470]: osdmap e408: 8 total, 7 up, 8 in 2024-09-16T10:27:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:31.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:32.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:32.078 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:32.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:32.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:32.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:32.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:31 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 27445/322761 objects degraded (8.503%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:32.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:31 smithi165 ceph-mon[32696]: osdmap e408: 8 total, 7 up, 8 in 2024-09-16T10:27:32.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:32.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:32.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:32.398 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2024-09-16T10:27:31.006356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6935773958,"num_objects":106022,"num_object_clones":0,"num_object_copies":318066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106022,"num_whiteouts":0,"num_read":1288,"num_read_kb":1045,"num_write":159774,"num_write_kb":8475896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":816773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81828,"ondisk_log_size":81828,"up":441,"acting":441,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24380724,"kb_used_data":20239560,"kb_used_omap":12,"kb_used_meta":4140723,"kb_avail":725580492,"statfs":{"total":767960285184,"available":742994423808,"internally_reserved":0,"allocated":20725309440,"data_stored":20719340243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4240100913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,6,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-392167424,"num_objects":-5984,"num_object_clones":0,"num_object_copies":-17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5984,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5984,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.643760"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":759,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.846809+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:27:27.846809+0000","last_peered":"2024-09-16T10:27:27.846809+0000","last_clean":"2024-09-16T10:27:27.846809+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:27:27.846809+0000","last_undegraded":"2024-09-16T10:27:27.846809+0000","last_fullsized":"2024-09-16T10:27:27.846809+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:58.647110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":705,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.840858+0000","last_change":"2024-09-16T10:26:53.392698+0000","last_active":"2024-09-16T10:27:27.840858+0000","last_peered":"2024-09-16T10:27:27.840858+0000","last_clean":"2024-09-16T10:27:27.840858+0000","last_became_active":"2024-09-16T10:26:28.557096+0000","last_became_peered":"2024-09-16T10:26:28.557096+0000","last_unstale":"2024-09-16T10:27:27.840858+0000","last_undegraded":"2024-09-16T10:27:27.840858+0000","last_fullsized":"2024-09-16T10:27:27.840858+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:53.392646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:53.392646+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:10.475144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":749,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.837687+0000","last_change":"2024-09-16T10:26:38.001941+0000","last_active":"2024-09-16T10:27:27.837687+0000","last_peered":"2024-09-16T10:27:27.837687+0000","last_clean":"2024-09-16T10:26:35.984553+0000","last_became_active":"2024-09-16T10:26:38.001941+0000","last_became_peered":"2024-09-16T10:26:38.001941+0000","last_unstale":"2024-09-16T10:27:27.837687+0000","last_undegraded":"2024-09-16T10:27:27.837687+0000","last_fullsized":"2024-09-16T10:26:37.987716+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.940459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.940459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:15.402666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":61,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828024+0000","last_change":"2024-09-16T10:26:28.552367+0000","last_active":"2024-09-16T10:27:27.828024+0000","last_peered":"2024-09-16T10:27:27.828024+0000","last_clean":"2024-09-16T10:27:27.828024+0000","last_became_active":"2024-09-16T10:26:28.551847+0000","last_became_peered":"2024-09-16T10:26:28.551847+0000","last_unstale":"2024-09-16T10:27:27.828024+0000","last_undegraded":"2024-09-16T10:27:27.828024+0000","last_fullsized":"2024-09-16T10:27:27.828024+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.378376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.378376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:31.380398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":727,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.840897+0000","last_change":"2024-09-16T10:26:38.012843+0000","last_active":"2024-09-16T10:27:27.840897+0000","last_peered":"2024-09-16T10:27:27.840897+0000","last_clean":"2024-09-16T10:26:35.993698+0000","last_became_active":"2024-09-16T10:26:38.012843+0000","last_became_peered":"2024-09-16T10:26:38.012843+0000","last_unstale":"2024-09-16T10:27:27.840897+0000","last_undegraded":"2024-09-16T10:27:27.840897+0000","last_fullsized":"2024-09-16T10:26:37.997407+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.535138+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":766,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.843438+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:27:27.843438+0000","last_peered":"2024-09-16T10:27:27.843438+0000","last_clean":"2024-09-16T10:27:27.843438+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:27:27.843438+0000","last_undegraded":"2024-09-16T10:27:27.843438+0000","last_fullsized":"2024-09-16T10:27:27.843438+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":836,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.842756+0000","last_change":"2024-09-16T10:26:12.334813+0000","last_active":"2024-09-16T10:27:27.842756+0000","last_peered":"2024-09-16T10:27:27.842756+0000","last_clean":"2024-09-16T10:27:27.842756+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:27:27.842756+0000","last_undegraded":"2024-09-16T10:27:27.842756+0000","last_fullsized":"2024-09-16T10:27:27.842756+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:12.334720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.334720+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.177726+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":69,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.840862+0000","last_change":"2024-09-16T10:26:45.470165+0000","last_active":"2024-09-16T10:27:27.840862+0000","last_peered":"2024-09-16T10:27:27.840862+0000","last_clean":"2024-09-16T10:27:27.840862+0000","last_became_active":"2024-09-16T10:26:28.556509+0000","last_became_peered":"2024-09-16T10:26:28.556509+0000","last_unstale":"2024-09-16T10:27:27.840862+0000","last_undegraded":"2024-09-16T10:27:27.840862+0000","last_fullsized":"2024-09-16T10:27:27.840862+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.470014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.470014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.195883+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":769,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.829252+0000","last_change":"2024-09-16T10:26:48.078096+0000","last_active":"2024-09-16T10:27:27.829252+0000","last_peered":"2024-09-16T10:27:27.829252+0000","last_clean":"2024-09-16T10:27:27.829252+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:27:27.829252+0000","last_undegraded":"2024-09-16T10:27:27.829252+0000","last_fullsized":"2024-09-16T10:27:27.829252+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:48.078028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:48.078028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:03.820478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":663,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.843475+0000","last_change":"2024-09-16T10:26:28.551030+0000","last_active":"2024-09-16T10:27:27.843475+0000","last_peered":"2024-09-16T10:27:27.843475+0000","last_clean":"2024-09-16T10:27:27.843475+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:27:27.843475+0000","last_undegraded":"2024-09-16T10:27:27.843475+0000","last_fullsized":"2024-09-16T10:27:27.843475+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:24:23.258886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.258886+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:55.385976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":749,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.828308+0000","last_change":"2024-09-16T10:26:38.007550+0000","last_active":"2024-09-16T10:27:27.828308+0000","last_peered":"2024-09-16T10:27:27.828308+0000","last_clean":"2024-09-16T10:26:35.982994+0000","last_became_active":"2024-09-16T10:26:38.007550+0000","last_became_peered":"2024-09-16T10:26:38.007550+0000","last_unstale":"2024-09-16T10:27:27.828308+0000","last_undegraded":"2024-09-16T10:27:27.828308+0000","last_fullsized":"2024-09-16T10:26:37.992243+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.834823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.834823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:12.147875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":744,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.846060+0000","last_change":"2024-09-16T10:26:37.997637+0000","last_active":"2024-09-16T10:27:27.846060+0000","last_peered":"2024-09-16T10:27:27.846060+0000","last_clean":"2024-09-16T10:26:35.992361+0000","last_became_active":"2024-09-16T10:26:37.997637+0000","last_became_peered":"2024-09-16T10:26:37.997637+0000","last_unstale":"2024-09-16T10:27:27.846060+0000","last_undegraded":"2024-09-16T10:27:27.846060+0000","last_fullsized":"2024-09-16T10:26:37.989396+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:08.436243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.436243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:35.375307+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":713,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828905+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:27:27.828905+0000","last_peered":"2024-09-16T10:27:27.828905+0000","last_clean":"2024-09-16T10:27:27.828905+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:27:27.828905+0000","last_undegraded":"2024-09-16T10:27:27.828905+0000","last_fullsized":"2024-09-16T10:27:27.828905+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:54.979150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":651,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841488+0000","last_change":"2024-09-16T10:26:28.557316+0000","last_active":"2024-09-16T10:27:27.841488+0000","last_peered":"2024-09-16T10:27:27.841488+0000","last_clean":"2024-09-16T10:27:27.841488+0000","last_became_active":"2024-09-16T10:26:28.556933+0000","last_became_peered":"2024-09-16T10:26:28.556933+0000","last_unstale":"2024-09-16T10:27:27.841488+0000","last_undegraded":"2024-09-16T10:27:27.841488+0000","last_fullsized":"2024-09-16T10:27:27.841488+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:57.426521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":48,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.838782+0000","last_change":"2024-09-16T10:26:38.003607+0000","last_active":"2024-09-16T10:27:27.838782+0000","last_peered":"2024-09-16T10:27:27.838782+0000","last_clean":"2024-09-16T10:13:40.081041+0000","last_became_active":"2024-09-16T10:26:38.003607+0000","last_became_peered":"2024-09-16T10:26:38.003607+0000","last_unstale":"2024-09-16T10:27:27.838782+0000","last_undegraded":"2024-09-16T10:27:27.838782+0000","last_fullsized":"2024-09-16T10:26:37.988017+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.369005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.369005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:45.104920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":748,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828148+0000","last_change":"2024-09-16T10:23:42.856401+0000","last_active":"2024-09-16T10:27:27.828148+0000","last_peered":"2024-09-16T10:27:27.828148+0000","last_clean":"2024-09-16T10:27:27.828148+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:27:27.828148+0000","last_undegraded":"2024-09-16T10:27:27.828148+0000","last_fullsized":"2024-09-16T10:27:27.828148+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.856353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.856353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:31.006170+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":61,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841367+0000","last_change":"2024-09-16T10:26:28.556518+0000","last_active":"2024-09-16T10:27:27.841367+0000","last_peered":"2024-09-16T10:27:27.841367+0000","last_clean":"2024-09-16T10:27:27.841367+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:27:27.841367+0000","last_undegraded":"2024-09-16T10:27:27.841367+0000","last_fullsized":"2024-09-16T10:27:27.841367+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:31.436663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":740,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.846837+0000","last_change":"2024-09-16T10:26:37.996618+0000","last_active":"2024-09-16T10:27:27.846837+0000","last_peered":"2024-09-16T10:27:27.846837+0000","last_clean":"2024-09-16T10:26:35.990280+0000","last_became_active":"2024-09-16T10:26:37.996618+0000","last_became_peered":"2024-09-16T10:26:37.996618+0000","last_unstale":"2024-09-16T10:27:27.846837+0000","last_undegraded":"2024-09-16T10:27:27.846837+0000","last_fullsized":"2024-09-16T10:26:37.991795+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:11.654674+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":314,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.831728+0000","last_change":"2024-09-16T10:26:17.566065+0000","last_active":"2024-09-16T10:27:27.831728+0000","last_peered":"2024-09-16T10:27:27.831728+0000","last_clean":"2024-09-16T10:27:27.831728+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:27:27.831728+0000","last_undegraded":"2024-09-16T10:27:27.831728+0000","last_fullsized":"2024-09-16T10:27:27.831728+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:17.565970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.565970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:34.494158+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":737,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.843228+0000","last_change":"2024-09-16T10:26:22.230127+0000","last_active":"2024-09-16T10:27:27.843228+0000","last_peered":"2024-09-16T10:27:27.843228+0000","last_clean":"2024-09-16T10:27:27.843228+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:27:27.843228+0000","last_undegraded":"2024-09-16T10:27:27.843228+0000","last_fullsized":"2024-09-16T10:27:27.843228+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.230054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.230054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":697,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.842150+0000","last_change":"2024-09-16T10:27:18.318288+0000","last_active":"2024-09-16T10:27:27.842150+0000","last_peered":"2024-09-16T10:27:27.842150+0000","last_clean":"2024-09-16T10:27:27.842150+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:27:27.842150+0000","last_undegraded":"2024-09-16T10:27:27.842150+0000","last_fullsized":"2024-09-16T10:27:27.842150+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:18.318214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:18.318214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:28.213657+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":533,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.840042+0000","last_change":"2024-09-16T10:26:28.551407+0000","last_active":"2024-09-16T10:27:27.840042+0000","last_peered":"2024-09-16T10:27:27.840042+0000","last_clean":"2024-09-16T10:27:27.840042+0000","last_became_active":"2024-09-16T10:26:28.551058+0000","last_became_peered":"2024-09-16T10:26:28.551058+0000","last_unstale":"2024-09-16T10:27:27.840042+0000","last_undegraded":"2024-09-16T10:27:27.840042+0000","last_fullsized":"2024-09-16T10:27:27.840042+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:54.217157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":798,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.847425+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:27:27.847425+0000","last_peered":"2024-09-16T10:27:27.847425+0000","last_clean":"2024-09-16T10:27:27.847425+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:27:27.847425+0000","last_undegraded":"2024-09-16T10:27:27.847425+0000","last_fullsized":"2024-09-16T10:27:27.847425+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:33.432273+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":541,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.848204+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:27:27.848204+0000","last_peered":"2024-09-16T10:27:27.848204+0000","last_clean":"2024-09-16T10:27:27.848204+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:27:27.848204+0000","last_undegraded":"2024-09-16T10:27:27.848204+0000","last_fullsized":"2024-09-16T10:27:27.848204+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":148,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.828098+0000","last_change":"2024-09-16T10:26:38.004428+0000","last_active":"2024-09-16T10:27:27.828098+0000","last_peered":"2024-09-16T10:27:27.828098+0000","last_clean":"2024-09-16T10:26:35.980045+0000","last_became_active":"2024-09-16T10:26:38.004428+0000","last_became_peered":"2024-09-16T10:26:38.004428+0000","last_unstale":"2024-09-16T10:27:27.828098+0000","last_undegraded":"2024-09-16T10:26:37.992071+0000","last_fullsized":"2024-09-16T10:26:37.991949+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:15.092834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":752,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.837971+0000","last_change":"2024-09-16T10:26:17.204523+0000","last_active":"2024-09-16T10:27:27.837971+0000","last_peered":"2024-09-16T10:27:27.837971+0000","last_clean":"2024-09-16T10:27:27.837971+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:27:27.837971+0000","last_undegraded":"2024-09-16T10:27:27.837971+0000","last_fullsized":"2024-09-16T10:27:27.837971+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:17.204441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.204441+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":775,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.853016+0000","last_change":"2024-09-16T10:26:38.010506+0000","last_active":"2024-09-16T10:27:27.853016+0000","last_peered":"2024-09-16T10:27:27.853016+0000","last_clean":"2024-09-16T10:26:35.992165+0000","last_became_active":"2024-09-16T10:26:38.010506+0000","last_became_peered":"2024-09-16T10:26:38.010506+0000","last_unstale":"2024-09-16T10:27:27.853016+0000","last_undegraded":"2024-09-16T10:26:37.992894+0000","last_fullsized":"2024-09-16T10:26:37.992782+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:55.749722+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"388'9","reported_seq":919,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.841635+0000","last_change":"2024-09-16T10:26:38.004232+0000","last_active":"2024-09-16T10:27:27.841635+0000","last_peered":"2024-09-16T10:27:27.841635+0000","last_clean":"2024-09-16T10:26:35.985776+0000","last_became_active":"2024-09-16T10:26:38.004232+0000","last_became_peered":"2024-09-16T10:26:38.004232+0000","last_unstale":"2024-09-16T10:27:27.841635+0000","last_undegraded":"2024-09-16T10:26:37.998183+0000","last_fullsized":"2024-09-16T10:26:37.998085+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:16.538706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.538706+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"407'9888","reported_seq":10003,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:29.652761+0000","last_change":"2024-09-16T10:26:38.001295+0000","last_active":"2024-09-16T10:27:29.652761+0000","last_peered":"2024-09-16T10:27:29.652761+0000","last_clean":"2024-09-16T10:26:36.970644+0000","last_became_active":"2024-09-16T10:26:38.001295+0000","last_became_peered":"2024-09-16T10:26:38.001295+0000","last_unstale":"2024-09-16T10:27:29.652761+0000","last_undegraded":"2024-09-16T10:26:37.989401+0000","last_fullsized":"2024-09-16T10:26:37.989298+0000","mapping_epoch":387,"log_start":"380'5100","ondisk_log_start":"380'5100","created":362,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4788,"log_dups_size":0,"ondisk_log_size":4788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:18.594189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424280064,"num_objects":6474,"num_object_clones":0,"num_object_copies":19422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":6474}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":591,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.827925+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:27:27.827925+0000","last_peered":"2024-09-16T10:27:27.827925+0000","last_clean":"2024-09-16T10:27:27.827925+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:27:27.827925+0000","last_undegraded":"2024-09-16T10:27:27.827925+0000","last_fullsized":"2024-09-16T10:27:27.827925+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:29.097602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":785,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.842299+0000","last_change":"2024-09-16T10:27:12.407528+0000","last_active":"2024-09-16T10:27:27.842299+0000","last_peered":"2024-09-16T10:27:27.842299+0000","last_clean":"2024-09-16T10:27:27.842299+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:27:27.842299+0000","last_undegraded":"2024-09-16T10:27:27.842299+0000","last_fullsized":"2024-09-16T10:27:27.842299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:27:12.407441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.407441+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:41.269329+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":756,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841288+0000","last_change":"2024-09-16T10:26:38.002212+0000","last_active":"2024-09-16T10:27:27.841288+0000","last_peered":"2024-09-16T10:27:27.841288+0000","last_clean":"2024-09-16T10:26:35.988090+0000","last_became_active":"2024-09-16T10:26:38.002212+0000","last_became_peered":"2024-09-16T10:26:38.002212+0000","last_unstale":"2024-09-16T10:27:27.841288+0000","last_undegraded":"2024-09-16T10:27:27.841288+0000","last_fullsized":"2024-09-16T10:26:37.994787+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":99,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828322+0000","last_change":"2024-09-16T10:26:28.553272+0000","last_active":"2024-09-16T10:27:27.828322+0000","last_peered":"2024-09-16T10:27:27.828322+0000","last_clean":"2024-09-16T10:27:27.828322+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:27:27.828322+0000","last_undegraded":"2024-09-16T10:27:27.828322+0000","last_fullsized":"2024-09-16T10:27:27.828322+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:15.308775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:15.308775+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:29.488676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":734,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828513+0000","last_change":"2024-09-16T10:24:15.859366+0000","last_active":"2024-09-16T10:27:27.828513+0000","last_peered":"2024-09-16T10:27:27.828513+0000","last_clean":"2024-09-16T10:27:27.828513+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:27:27.828513+0000","last_undegraded":"2024-09-16T10:27:27.828513+0000","last_fullsized":"2024-09-16T10:27:27.828513+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:15.859299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.859299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.117548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"407'9793","reported_seq":9901,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.418621+0000","last_change":"2024-09-16T10:26:28.565690+0000","last_active":"2024-09-16T10:27:28.418621+0000","last_peered":"2024-09-16T10:27:28.418621+0000","last_clean":"2024-09-16T10:27:28.418621+0000","last_became_active":"2024-09-16T10:26:28.555135+0000","last_became_peered":"2024-09-16T10:26:28.555135+0000","last_unstale":"2024-09-16T10:27:28.418621+0000","last_undegraded":"2024-09-16T10:27:28.418621+0000","last_fullsized":"2024-09-16T10:27:28.418621+0000","mapping_epoch":382,"log_start":"379'4800","ondisk_log_start":"379'4800","created":362,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4993,"log_dups_size":0,"ondisk_log_size":4993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435748864,"num_objects":6649,"num_object_clones":0,"num_object_copies":19947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9793,"num_write_kb":526144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":173,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841557+0000","last_change":"2024-09-16T10:27:18.552950+0000","last_active":"2024-09-16T10:27:27.841557+0000","last_peered":"2024-09-16T10:27:27.841557+0000","last_clean":"2024-09-16T10:27:27.841557+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:27:27.841557+0000","last_undegraded":"2024-09-16T10:27:27.841557+0000","last_fullsized":"2024-09-16T10:27:27.841557+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:27:17.482427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:27:17.482427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:42.753548+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":134,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.840544+0000","last_change":"2024-09-16T10:26:44.445756+0000","last_active":"2024-09-16T10:27:27.840544+0000","last_peered":"2024-09-16T10:27:27.840544+0000","last_clean":"2024-09-16T10:27:27.840544+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:27:27.840544+0000","last_undegraded":"2024-09-16T10:27:27.840544+0000","last_fullsized":"2024-09-16T10:27:27.840544+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:26:44.445661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.445661+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.574050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":721,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.842567+0000","last_change":"2024-09-16T10:26:38.008838+0000","last_active":"2024-09-16T10:27:27.842567+0000","last_peered":"2024-09-16T10:27:27.842567+0000","last_clean":"2024-09-16T10:26:35.994682+0000","last_became_active":"2024-09-16T10:26:38.008838+0000","last_became_peered":"2024-09-16T10:26:38.008838+0000","last_unstale":"2024-09-16T10:27:27.842567+0000","last_undegraded":"2024-09-16T10:27:27.842567+0000","last_fullsized":"2024-09-16T10:26:37.995364+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.404383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.404383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":487,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.840638+0000","last_change":"2024-09-16T10:26:37.998714+0000","last_active":"2024-09-16T10:27:27.840638+0000","last_peered":"2024-09-16T10:27:27.840638+0000","last_clean":"2024-09-16T10:26:35.984906+0000","last_became_active":"2024-09-16T10:26:37.998714+0000","last_became_peered":"2024-09-16T10:26:37.998714+0000","last_unstale":"2024-09-16T10:27:27.840638+0000","last_undegraded":"2024-09-16T10:27:27.840638+0000","last_fullsized":"2024-09-16T10:26:37.992483+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.521586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.521586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.643455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":545,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.843451+0000","last_change":"2024-09-16T10:26:28.551067+0000","last_active":"2024-09-16T10:27:27.843451+0000","last_peered":"2024-09-16T10:27:27.843451+0000","last_clean":"2024-09-16T10:27:27.843451+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:27:27.843451+0000","last_undegraded":"2024-09-16T10:27:27.843451+0000","last_fullsized":"2024-09-16T10:27:27.843451+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:22.206414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:22.206414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:37.462061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"407'9889","reported_seq":9990,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.434545+0000","last_change":"2024-09-16T10:25:31.163385+0000","last_active":"2024-09-16T10:27:28.434545+0000","last_peered":"2024-09-16T10:27:28.434545+0000","last_clean":"2024-09-16T10:27:28.434545+0000","last_became_active":"2024-09-16T10:25:31.162906+0000","last_became_peered":"2024-09-16T10:25:31.162906+0000","last_unstale":"2024-09-16T10:27:28.434545+0000","last_undegraded":"2024-09-16T10:27:28.434545+0000","last_fullsized":"2024-09-16T10:27:28.434545+0000","mapping_epoch":362,"log_start":"379'4900","ondisk_log_start":"379'4900","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4989,"log_dups_size":0,"ondisk_log_size":4989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437321728,"num_objects":6673,"num_object_clones":0,"num_object_copies":20019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9889,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":638,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.848406+0000","last_change":"2024-09-16T10:26:38.000943+0000","last_active":"2024-09-16T10:27:27.848406+0000","last_peered":"2024-09-16T10:27:27.848406+0000","last_clean":"2024-09-16T10:26:35.994238+0000","last_became_active":"2024-09-16T10:26:38.000943+0000","last_became_peered":"2024-09-16T10:26:38.000943+0000","last_unstale":"2024-09-16T10:27:27.848406+0000","last_undegraded":"2024-09-16T10:27:27.848406+0000","last_fullsized":"2024-09-16T10:26:37.996963+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.343066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.343066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:48.503120+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":682,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828316+0000","last_change":"2024-09-16T10:26:28.552507+0000","last_active":"2024-09-16T10:27:27.828316+0000","last_peered":"2024-09-16T10:27:27.828316+0000","last_clean":"2024-09-16T10:27:27.828316+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:27:27.828316+0000","last_undegraded":"2024-09-16T10:27:27.828316+0000","last_fullsized":"2024-09-16T10:27:27.828316+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:26:11.381501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:11.381501+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":738,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.853467+0000","last_change":"2024-09-16T10:19:45.133742+0000","last_active":"2024-09-16T10:27:27.853467+0000","last_peered":"2024-09-16T10:27:27.853467+0000","last_clean":"2024-09-16T10:27:27.853467+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:27:27.853467+0000","last_undegraded":"2024-09-16T10:27:27.853467+0000","last_fullsized":"2024-09-16T10:27:27.853467+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:19:45.133678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:19:45.133678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:34.482928+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":922,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.840920+0000","last_change":"2024-09-16T10:27:24.281038+0000","last_active":"2024-09-16T10:27:27.840920+0000","last_peered":"2024-09-16T10:27:27.840920+0000","last_clean":"2024-09-16T10:27:27.840920+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:27:27.840920+0000","last_undegraded":"2024-09-16T10:27:27.840920+0000","last_fullsized":"2024-09-16T10:27:27.840920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:27:24.280960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:24.280960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:36.038661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":589,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841108+0000","last_change":"2024-09-16T10:26:38.006141+0000","last_active":"2024-09-16T10:27:27.841108+0000","last_peered":"2024-09-16T10:27:27.841108+0000","last_clean":"2024-09-16T10:26:35.985257+0000","last_became_active":"2024-09-16T10:26:38.006141+0000","last_became_peered":"2024-09-16T10:26:38.006141+0000","last_unstale":"2024-09-16T10:27:27.841108+0000","last_undegraded":"2024-09-16T10:27:27.841108+0000","last_fullsized":"2024-09-16T10:26:37.998986+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:43.479873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"407'10151","reported_seq":10265,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:29.653051+0000","last_change":"2024-09-16T10:26:38.031331+0000","last_active":"2024-09-16T10:27:29.653051+0000","last_peered":"2024-09-16T10:27:29.653051+0000","last_clean":"2024-09-16T10:26:36.986703+0000","last_became_active":"2024-09-16T10:26:38.015586+0000","last_became_peered":"2024-09-16T10:26:38.015586+0000","last_unstale":"2024-09-16T10:27:29.653051+0000","last_undegraded":"2024-09-16T10:26:38.003109+0000","last_fullsized":"2024-09-16T10:26:37.990926+0000","mapping_epoch":387,"log_start":"380'5300","ondisk_log_start":"380'5300","created":362,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:24.620408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430112768,"num_objects":6563,"num_object_clones":0,"num_object_copies":19689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":6563}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":688,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841389+0000","last_change":"2024-09-16T10:26:38.011823+0000","last_active":"2024-09-16T10:27:27.841389+0000","last_peered":"2024-09-16T10:27:27.841389+0000","last_clean":"2024-09-16T10:26:35.985526+0000","last_became_active":"2024-09-16T10:26:38.011823+0000","last_became_peered":"2024-09-16T10:26:38.011823+0000","last_unstale":"2024-09-16T10:27:27.841389+0000","last_undegraded":"2024-09-16T10:27:27.841389+0000","last_fullsized":"2024-09-16T10:26:38.001181+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.617190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.617190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:46.435042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":780,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.844299+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:27:27.844299+0000","last_peered":"2024-09-16T10:27:27.844299+0000","last_clean":"2024-09-16T10:27:27.844299+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:27:27.844299+0000","last_undegraded":"2024-09-16T10:27:27.844299+0000","last_fullsized":"2024-09-16T10:27:27.844299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:00.233082+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":760,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.829292+0000","last_change":"2024-09-16T10:24:59.820337+0000","last_active":"2024-09-16T10:27:27.829292+0000","last_peered":"2024-09-16T10:27:27.829292+0000","last_clean":"2024-09-16T10:27:27.829292+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:27:27.829292+0000","last_undegraded":"2024-09-16T10:27:27.829292+0000","last_fullsized":"2024-09-16T10:27:27.829292+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:59.820273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.820273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:33.022019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":904,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.846046+0000","last_change":"2024-09-16T10:26:46.635924+0000","last_active":"2024-09-16T10:27:27.846046+0000","last_peered":"2024-09-16T10:27:27.846046+0000","last_clean":"2024-09-16T10:27:27.846046+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:27:27.846046+0000","last_undegraded":"2024-09-16T10:27:27.846046+0000","last_fullsized":"2024-09-16T10:27:27.846046+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:46.635855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.635855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:50.833434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":755,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.851171+0000","last_change":"2024-09-16T10:26:52.648107+0000","last_active":"2024-09-16T10:27:27.851171+0000","last_peered":"2024-09-16T10:27:27.851171+0000","last_clean":"2024-09-16T10:27:27.851171+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:27:27.851171+0000","last_undegraded":"2024-09-16T10:27:27.851171+0000","last_fullsized":"2024-09-16T10:27:27.851171+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:52.648041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:52.648041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.033578+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"407'10013","reported_seq":10202,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.458992+0000","last_change":"2024-09-16T10:27:12.593342+0000","last_active":"2024-09-16T10:27:28.458992+0000","last_peered":"2024-09-16T10:27:28.458992+0000","last_clean":"2024-09-16T10:27:28.458992+0000","last_became_active":"2024-09-16T10:25:30.620945+0000","last_became_peered":"2024-09-16T10:25:30.620945+0000","last_unstale":"2024-09-16T10:27:28.458992+0000","last_undegraded":"2024-09-16T10:27:28.458992+0000","last_fullsized":"2024-09-16T10:27:28.458992+0000","mapping_epoch":362,"log_start":"379'5000","ondisk_log_start":"379'5000","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8755","last_scrub_stamp":"2024-09-16T10:27:12.593235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.593235+0000","objects_scrubbed":7966,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:40.603030+0000","scrub_duration":2292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433324056,"num_objects":6613,"num_object_clones":0,"num_object_copies":19839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10013,"num_write_kb":531969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":697,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841261+0000","last_change":"2024-09-16T10:26:51.422615+0000","last_active":"2024-09-16T10:27:27.841261+0000","last_peered":"2024-09-16T10:27:27.841261+0000","last_clean":"2024-09-16T10:27:27.841261+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:27:27.841261+0000","last_undegraded":"2024-09-16T10:27:27.841261+0000","last_fullsized":"2024-09-16T10:27:27.841261+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:51.422550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:51.422550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.793681+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":796,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.846452+0000","last_change":"2024-09-16T10:26:38.007576+0000","last_active":"2024-09-16T10:27:27.846452+0000","last_peered":"2024-09-16T10:27:27.846452+0000","last_clean":"2024-09-16T10:26:35.990050+0000","last_became_active":"2024-09-16T10:26:38.007576+0000","last_became_peered":"2024-09-16T10:26:38.007576+0000","last_unstale":"2024-09-16T10:27:27.846452+0000","last_undegraded":"2024-09-16T10:26:37.992428+0000","last_fullsized":"2024-09-16T10:26:37.991190+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.655534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":760,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.838655+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:27:27.838655+0000","last_peered":"2024-09-16T10:27:27.838655+0000","last_clean":"2024-09-16T10:27:27.838655+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:27:27.838655+0000","last_undegraded":"2024-09-16T10:27:27.838655+0000","last_fullsized":"2024-09-16T10:27:27.838655+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.276933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":753,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841019+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:27:27.841019+0000","last_peered":"2024-09-16T10:27:27.841019+0000","last_clean":"2024-09-16T10:27:27.841019+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:27:27.841019+0000","last_undegraded":"2024-09-16T10:27:27.841019+0000","last_fullsized":"2024-09-16T10:27:27.841019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":740,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.837597+0000","last_change":"2024-09-16T10:22:02.891444+0000","last_active":"2024-09-16T10:27:27.837597+0000","last_peered":"2024-09-16T10:27:27.837597+0000","last_clean":"2024-09-16T10:27:27.837597+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:27:27.837597+0000","last_undegraded":"2024-09-16T10:27:27.837597+0000","last_fullsized":"2024-09-16T10:27:27.837597+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:35.589332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:35.589332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:35.203311+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"407'9992","reported_seq":10093,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.451402+0000","last_change":"2024-09-16T10:25:30.622174+0000","last_active":"2024-09-16T10:27:28.451402+0000","last_peered":"2024-09-16T10:27:28.451402+0000","last_clean":"2024-09-16T10:27:28.451402+0000","last_became_active":"2024-09-16T10:25:30.621742+0000","last_became_peered":"2024-09-16T10:25:30.621742+0000","last_unstale":"2024-09-16T10:27:28.451402+0000","last_undegraded":"2024-09-16T10:27:28.451402+0000","last_fullsized":"2024-09-16T10:27:28.451402+0000","mapping_epoch":362,"log_start":"379'5000","ondisk_log_start":"379'5000","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4992,"log_dups_size":0,"ondisk_log_size":4992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439091200,"num_objects":6700,"num_object_clones":0,"num_object_copies":20100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":643,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.827933+0000","last_change":"2024-09-16T10:26:09.992070+0000","last_active":"2024-09-16T10:27:27.827933+0000","last_peered":"2024-09-16T10:27:27.827933+0000","last_clean":"2024-09-16T10:27:27.827933+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:27:27.827933+0000","last_undegraded":"2024-09-16T10:27:27.827933+0000","last_fullsized":"2024-09-16T10:27:27.827933+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.992000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.992000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:32.736640+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":1057,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.838514+0000","last_change":"2024-09-16T10:27:14.409646+0000","last_active":"2024-09-16T10:27:27.838514+0000","last_peered":"2024-09-16T10:27:27.838514+0000","last_clean":"2024-09-16T10:27:27.838514+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:27:27.838514+0000","last_undegraded":"2024-09-16T10:27:27.838514+0000","last_fullsized":"2024-09-16T10:27:27.838514+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:27:14.409552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:14.409552+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:38.459925+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":61,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.846747+0000","last_change":"2024-09-16T10:26:28.554041+0000","last_active":"2024-09-16T10:27:27.846747+0000","last_peered":"2024-09-16T10:27:27.846747+0000","last_clean":"2024-09-16T10:27:27.846747+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:27:27.846747+0000","last_undegraded":"2024-09-16T10:27:27.846747+0000","last_fullsized":"2024-09-16T10:27:27.846747+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:13.097133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:13.097133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:35.494697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":764,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.829486+0000","last_change":"2024-09-16T10:27:23.145340+0000","last_active":"2024-09-16T10:27:27.829486+0000","last_peered":"2024-09-16T10:27:27.829486+0000","last_clean":"2024-09-16T10:27:27.829486+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:27:27.829486+0000","last_undegraded":"2024-09-16T10:27:27.829486+0000","last_fullsized":"2024-09-16T10:27:27.829486+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:27:23.145278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:27:23.145278+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:36.993455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":729,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.849947+0000","last_change":"2024-09-16T10:25:37.359150+0000","last_active":"2024-09-16T10:27:27.849947+0000","last_peered":"2024-09-16T10:27:27.849947+0000","last_clean":"2024-09-16T10:27:27.849947+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:27:27.849947+0000","last_undegraded":"2024-09-16T10:27:27.849947+0000","last_fullsized":"2024-09-16T10:27:27.849947+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:37.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":752,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.843189+0000","last_change":"2024-09-16T10:26:38.012326+0000","last_active":"2024-09-16T10:27:27.843189+0000","last_peered":"2024-09-16T10:27:27.843189+0000","last_clean":"2024-09-16T10:26:35.989607+0000","last_became_active":"2024-09-16T10:26:38.012326+0000","last_became_peered":"2024-09-16T10:26:38.012326+0000","last_unstale":"2024-09-16T10:27:27.843189+0000","last_undegraded":"2024-09-16T10:27:27.843189+0000","last_fullsized":"2024-09-16T10:26:38.005246+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:26.349415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:26.349415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:36.622386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"407'9908","reported_seq":10016,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:28.598076+0000","last_change":"2024-09-16T10:26:38.029552+0000","last_active":"2024-09-16T10:27:28.598076+0000","last_peered":"2024-09-16T10:27:28.598076+0000","last_clean":"2024-09-16T10:26:36.981417+0000","last_became_active":"2024-09-16T10:26:38.012353+0000","last_became_peered":"2024-09-16T10:26:38.012353+0000","last_unstale":"2024-09-16T10:27:28.598076+0000","last_undegraded":"2024-09-16T10:26:37.998360+0000","last_fullsized":"2024-09-16T10:26:37.991486+0000","mapping_epoch":387,"log_start":"380'5219","ondisk_log_start":"380'5219","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:16.139506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432275456,"num_objects":6596,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":528128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6596}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":212,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.846084+0000","last_change":"2024-09-16T10:26:38.007174+0000","last_active":"2024-09-16T10:27:27.846084+0000","last_peered":"2024-09-16T10:27:27.846084+0000","last_clean":"2024-09-16T10:26:35.990809+0000","last_became_active":"2024-09-16T10:26:38.007174+0000","last_became_peered":"2024-09-16T10:26:38.007174+0000","last_unstale":"2024-09-16T10:27:27.846084+0000","last_undegraded":"2024-09-16T10:27:27.846084+0000","last_fullsized":"2024-09-16T10:26:37.993534+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:17.574207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":759,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.842406+0000","last_change":"2024-09-16T10:25:13.649143+0000","last_active":"2024-09-16T10:27:27.842406+0000","last_peered":"2024-09-16T10:27:27.842406+0000","last_clean":"2024-09-16T10:27:27.842406+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:27:27.842406+0000","last_undegraded":"2024-09-16T10:27:27.842406+0000","last_fullsized":"2024-09-16T10:27:27.842406+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:25:13.648981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.648981+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.327665+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":744,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.840677+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:27:27.840677+0000","last_peered":"2024-09-16T10:27:27.840677+0000","last_clean":"2024-09-16T10:27:27.840677+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:27:27.840677+0000","last_undegraded":"2024-09-16T10:27:27.840677+0000","last_fullsized":"2024-09-16T10:27:27.840677+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:32.209565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":284,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.831733+0000","last_change":"2024-09-16T10:26:18.549126+0000","last_active":"2024-09-16T10:27:27.831733+0000","last_peered":"2024-09-16T10:27:27.831733+0000","last_clean":"2024-09-16T10:27:27.831733+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:27:27.831733+0000","last_undegraded":"2024-09-16T10:27:27.831733+0000","last_fullsized":"2024-09-16T10:27:27.831733+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:26:18.549005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:26:18.549005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:36.374740+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":276,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.833388+0000","last_change":"2024-09-16T10:26:38.004937+0000","last_active":"2024-09-16T10:27:27.833388+0000","last_peered":"2024-09-16T10:27:27.833388+0000","last_clean":"2024-09-16T10:26:35.983335+0000","last_became_active":"2024-09-16T10:26:38.004937+0000","last_became_peered":"2024-09-16T10:26:38.004937+0000","last_unstale":"2024-09-16T10:27:27.833388+0000","last_undegraded":"2024-09-16T10:27:27.833388+0000","last_fullsized":"2024-09-16T10:26:37.992855+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.224879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":137,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841140+0000","last_change":"2024-09-16T10:26:38.011765+0000","last_active":"2024-09-16T10:27:27.841140+0000","last_peered":"2024-09-16T10:27:27.841140+0000","last_clean":"2024-09-16T10:26:35.985415+0000","last_became_active":"2024-09-16T10:26:38.011765+0000","last_became_peered":"2024-09-16T10:26:38.011765+0000","last_unstale":"2024-09-16T10:27:27.841140+0000","last_undegraded":"2024-09-16T10:27:27.841140+0000","last_fullsized":"2024-09-16T10:26:38.000155+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"407'9814","reported_seq":9918,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:29.667778+0000","last_change":"2024-09-16T10:26:37.996077+0000","last_active":"2024-09-16T10:27:29.667778+0000","last_peered":"2024-09-16T10:27:29.667778+0000","last_clean":"2024-09-16T10:26:36.966822+0000","last_became_active":"2024-09-16T10:26:37.996077+0000","last_became_peered":"2024-09-16T10:26:37.996077+0000","last_unstale":"2024-09-16T10:27:29.667778+0000","last_undegraded":"2024-09-16T10:26:37.990629+0000","last_fullsized":"2024-09-16T10:26:37.990510+0000","mapping_epoch":387,"log_start":"380'5000","ondisk_log_start":"380'5000","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:22.579785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428605440,"num_objects":6540,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9814,"num_write_kb":523328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":6540}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":631,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.839138+0000","last_change":"2024-09-16T10:26:09.201694+0000","last_active":"2024-09-16T10:27:27.839138+0000","last_peered":"2024-09-16T10:27:27.839138+0000","last_clean":"2024-09-16T10:27:27.839138+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:27:27.839138+0000","last_undegraded":"2024-09-16T10:27:27.839138+0000","last_fullsized":"2024-09-16T10:27:27.839138+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.201559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.201559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:34.278969+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":843,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.853146+0000","last_change":"2024-09-16T10:27:09.622319+0000","last_active":"2024-09-16T10:27:27.853146+0000","last_peered":"2024-09-16T10:27:27.853146+0000","last_clean":"2024-09-16T10:27:27.853146+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:27:27.853146+0000","last_undegraded":"2024-09-16T10:27:27.853146+0000","last_fullsized":"2024-09-16T10:27:27.853146+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:27:09.622239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.622239+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:28.341911+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":769,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.846097+0000","last_change":"2024-09-16T10:27:10.642822+0000","last_active":"2024-09-16T10:27:27.846097+0000","last_peered":"2024-09-16T10:27:27.846097+0000","last_clean":"2024-09-16T10:27:27.846097+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:27:27.846097+0000","last_undegraded":"2024-09-16T10:27:27.846097+0000","last_fullsized":"2024-09-16T10:27:27.846097+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:10.642739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:10.642739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:12.787144+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":318,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.840538+0000","last_change":"2024-09-16T10:26:37.994028+0000","last_active":"2024-09-16T10:27:27.840538+0000","last_peered":"2024-09-16T10:27:27.840538+0000","last_clean":"2024-09-16T10:26:36.191294+0000","last_became_active":"2024-09-16T10:26:37.994028+0000","last_became_peered":"2024-09-16T10:26:37.994028+0000","last_unstale":"2024-09-16T10:27:27.840538+0000","last_undegraded":"2024-09-16T10:26:37.988066+0000","last_fullsized":"2024-09-16T10:26:37.987957+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:23:41.597682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.597682+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:43.104021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":153,"num_read_kb":148,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":754,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828640+0000","last_change":"2024-09-16T10:27:20.130780+0000","last_active":"2024-09-16T10:27:27.828640+0000","last_peered":"2024-09-16T10:27:27.828640+0000","last_clean":"2024-09-16T10:27:27.828640+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:27:27.828640+0000","last_undegraded":"2024-09-16T10:27:27.828640+0000","last_fullsized":"2024-09-16T10:27:27.828640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:20.130715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.130715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:33.131596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":758,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.838254+0000","last_change":"2024-09-16T10:27:19.374485+0000","last_active":"2024-09-16T10:27:27.838254+0000","last_peered":"2024-09-16T10:27:27.838254+0000","last_clean":"2024-09-16T10:27:27.838254+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:27:27.838254+0000","last_undegraded":"2024-09-16T10:27:27.838254+0000","last_fullsized":"2024-09-16T10:27:27.838254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:42.304216+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"407'9830","reported_seq":9931,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.457125+0000","last_change":"2024-09-16T10:25:30.622105+0000","last_active":"2024-09-16T10:27:28.457125+0000","last_peered":"2024-09-16T10:27:28.457125+0000","last_clean":"2024-09-16T10:27:28.457125+0000","last_became_active":"2024-09-16T10:25:30.621779+0000","last_became_peered":"2024-09-16T10:25:30.621779+0000","last_unstale":"2024-09-16T10:27:28.457125+0000","last_undegraded":"2024-09-16T10:27:28.457125+0000","last_fullsized":"2024-09-16T10:27:28.457125+0000","mapping_epoch":362,"log_start":"379'4900","ondisk_log_start":"379'4900","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433061888,"num_objects":6608,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"388'29","reported_seq":936,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.841351+0000","last_change":"2024-09-16T10:26:38.004519+0000","last_active":"2024-09-16T10:27:27.841351+0000","last_peered":"2024-09-16T10:27:27.841351+0000","last_clean":"2024-09-16T10:26:35.985510+0000","last_became_active":"2024-09-16T10:26:38.004519+0000","last_became_peered":"2024-09-16T10:26:38.004519+0000","last_unstale":"2024-09-16T10:27:27.841351+0000","last_undegraded":"2024-09-16T10:26:37.998195+0000","last_fullsized":"2024-09-16T10:26:37.998079+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:40.493400+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":717,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.843587+0000","last_change":"2024-09-16T10:26:47.493915+0000","last_active":"2024-09-16T10:27:27.843587+0000","last_peered":"2024-09-16T10:27:27.843587+0000","last_clean":"2024-09-16T10:27:27.843587+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:27:27.843587+0000","last_undegraded":"2024-09-16T10:27:27.843587+0000","last_fullsized":"2024-09-16T10:27:27.843587+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:47.493848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:47.493848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:15.957630+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":212,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.851341+0000","last_change":"2024-09-16T10:26:37.999195+0000","last_active":"2024-09-16T10:27:27.851341+0000","last_peered":"2024-09-16T10:27:27.851341+0000","last_clean":"2024-09-16T10:26:35.992856+0000","last_became_active":"2024-09-16T10:26:37.999195+0000","last_became_peered":"2024-09-16T10:26:37.999195+0000","last_unstale":"2024-09-16T10:27:27.851341+0000","last_undegraded":"2024-09-16T10:27:27.851341+0000","last_fullsized":"2024-09-16T10:26:37.991079+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:14.650988+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":609,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.830720+0000","last_change":"2024-09-16T10:26:38.002654+0000","last_active":"2024-09-16T10:27:27.830720+0000","last_peered":"2024-09-16T10:27:27.830720+0000","last_clean":"2024-09-16T10:26:35.982477+0000","last_became_active":"2024-09-16T10:26:38.002654+0000","last_became_peered":"2024-09-16T10:26:38.002654+0000","last_unstale":"2024-09-16T10:27:27.830720+0000","last_undegraded":"2024-09-16T10:27:27.830720+0000","last_fullsized":"2024-09-16T10:26:37.992362+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.591323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:11.293510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":742,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828515+0000","last_change":"2024-09-16T10:26:19.084467+0000","last_active":"2024-09-16T10:27:27.828515+0000","last_peered":"2024-09-16T10:27:27.828515+0000","last_clean":"2024-09-16T10:27:27.828515+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:27:27.828515+0000","last_undegraded":"2024-09-16T10:27:27.828515+0000","last_fullsized":"2024-09-16T10:27:27.828515+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:19.084409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:19.084409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"407'9902","reported_seq":10089,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.450040+0000","last_change":"2024-09-16T10:27:15.421579+0000","last_active":"2024-09-16T10:27:28.450040+0000","last_peered":"2024-09-16T10:27:28.450040+0000","last_clean":"2024-09-16T10:27:28.450040+0000","last_became_active":"2024-09-16T10:25:30.621428+0000","last_became_peered":"2024-09-16T10:25:30.621428+0000","last_unstale":"2024-09-16T10:27:28.450040+0000","last_undegraded":"2024-09-16T10:27:28.450040+0000","last_fullsized":"2024-09-16T10:27:28.450040+0000","mapping_epoch":362,"log_start":"379'4900","ondisk_log_start":"379'4900","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8912","last_scrub_stamp":"2024-09-16T10:27:15.421284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:15.421284+0000","objects_scrubbed":7743,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:17.636847+0000","scrub_duration":2117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436862976,"num_objects":6666,"num_object_clones":0,"num_object_copies":19998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":530176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":631,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.838932+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:27:27.838932+0000","last_peered":"2024-09-16T10:27:27.838932+0000","last_clean":"2024-09-16T10:27:27.838932+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:27:27.838932+0000","last_undegraded":"2024-09-16T10:27:27.838932+0000","last_fullsized":"2024-09-16T10:27:27.838932+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:06.340246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":682,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841395+0000","last_change":"2024-09-16T10:26:28.556079+0000","last_active":"2024-09-16T10:27:27.841395+0000","last_peered":"2024-09-16T10:27:27.841395+0000","last_clean":"2024-09-16T10:27:27.841395+0000","last_became_active":"2024-09-16T10:26:28.554604+0000","last_became_peered":"2024-09-16T10:26:28.554604+0000","last_unstale":"2024-09-16T10:27:27.841395+0000","last_undegraded":"2024-09-16T10:27:27.841395+0000","last_fullsized":"2024-09-16T10:27:27.841395+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:26:10.386128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.386128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:31.372272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":545,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.831335+0000","last_change":"2024-09-16T10:26:37.998773+0000","last_active":"2024-09-16T10:27:27.831335+0000","last_peered":"2024-09-16T10:27:27.831335+0000","last_clean":"2024-09-16T10:26:35.986576+0000","last_became_active":"2024-09-16T10:26:37.998773+0000","last_became_peered":"2024-09-16T10:26:37.998773+0000","last_unstale":"2024-09-16T10:27:27.831335+0000","last_undegraded":"2024-09-16T10:27:27.831335+0000","last_fullsized":"2024-09-16T10:26:37.991873+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.191187+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":746,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828436+0000","last_change":"2024-09-16T10:25:05.858177+0000","last_active":"2024-09-16T10:27:27.828436+0000","last_peered":"2024-09-16T10:27:27.828436+0000","last_clean":"2024-09-16T10:27:27.828436+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:27:27.828436+0000","last_undegraded":"2024-09-16T10:27:27.828436+0000","last_fullsized":"2024-09-16T10:27:27.828436+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.858129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.858129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:37.033784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":680,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.848306+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:27:27.848306+0000","last_peered":"2024-09-16T10:27:27.848306+0000","last_clean":"2024-09-16T10:27:27.848306+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:27:27.848306+0000","last_undegraded":"2024-09-16T10:27:27.848306+0000","last_fullsized":"2024-09-16T10:27:27.848306+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.375991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"407'10007","reported_seq":10107,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.952687+0000","last_change":"2024-09-16T10:25:31.161519+0000","last_active":"2024-09-16T10:27:28.952687+0000","last_peered":"2024-09-16T10:27:28.952687+0000","last_clean":"2024-09-16T10:27:28.952687+0000","last_became_active":"2024-09-16T10:25:31.161181+0000","last_became_peered":"2024-09-16T10:25:31.161181+0000","last_unstale":"2024-09-16T10:27:28.952687+0000","last_undegraded":"2024-09-16T10:27:28.952687+0000","last_fullsized":"2024-09-16T10:27:28.952687+0000","mapping_epoch":362,"log_start":"374'3800","ondisk_log_start":"374'3800","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:07.354225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439025664,"num_objects":6699,"num_object_clones":0,"num_object_copies":20097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10007,"num_write_kb":534592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":606,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.827605+0000","last_change":"2024-09-16T10:26:49.093325+0000","last_active":"2024-09-16T10:27:27.827605+0000","last_peered":"2024-09-16T10:27:27.827605+0000","last_clean":"2024-09-16T10:27:27.827605+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:27:27.827605+0000","last_undegraded":"2024-09-16T10:27:27.827605+0000","last_fullsized":"2024-09-16T10:27:27.827605+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:49.093245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:49.093245+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:09.427767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":747,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.832994+0000","last_change":"2024-09-16T10:27:21.459062+0000","last_active":"2024-09-16T10:27:27.832994+0000","last_peered":"2024-09-16T10:27:27.832994+0000","last_clean":"2024-09-16T10:27:27.832994+0000","last_became_active":"2024-09-16T10:26:28.553171+0000","last_became_peered":"2024-09-16T10:26:28.553171+0000","last_unstale":"2024-09-16T10:27:27.832994+0000","last_undegraded":"2024-09-16T10:27:27.832994+0000","last_fullsized":"2024-09-16T10:27:27.832994+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:27:21.458969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:21.458969+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:21.673624+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":745,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841690+0000","last_change":"2024-09-16T10:26:42.490866+0000","last_active":"2024-09-16T10:27:27.841690+0000","last_peered":"2024-09-16T10:27:27.841690+0000","last_clean":"2024-09-16T10:27:27.841690+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:27:27.841690+0000","last_undegraded":"2024-09-16T10:27:27.841690+0000","last_fullsized":"2024-09-16T10:27:27.841690+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:42.490731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.490731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:51.400478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":730,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.838334+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:27:27.838334+0000","last_peered":"2024-09-16T10:27:27.838334+0000","last_clean":"2024-09-16T10:27:27.838334+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:27:27.838334+0000","last_undegraded":"2024-09-16T10:27:27.838334+0000","last_fullsized":"2024-09-16T10:27:27.838334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:39.188525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":280,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.830931+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:27:27.830931+0000","last_peered":"2024-09-16T10:27:27.830931+0000","last_clean":"2024-09-16T10:27:27.830931+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:27:27.830931+0000","last_undegraded":"2024-09-16T10:27:27.830931+0000","last_fullsized":"2024-09-16T10:27:27.830931+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:06.309247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"407'9989","reported_seq":10178,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:29.060329+0000","last_change":"2024-09-16T10:27:12.147878+0000","last_active":"2024-09-16T10:27:29.060329+0000","last_peered":"2024-09-16T10:27:29.060329+0000","last_clean":"2024-09-16T10:27:29.060329+0000","last_became_active":"2024-09-16T10:25:30.613465+0000","last_became_peered":"2024-09-16T10:25:30.613465+0000","last_unstale":"2024-09-16T10:27:29.060329+0000","last_undegraded":"2024-09-16T10:27:29.060329+0000","last_fullsized":"2024-09-16T10:27:29.060329+0000","mapping_epoch":362,"log_start":"380'5100","ondisk_log_start":"380'5100","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8705","last_scrub_stamp":"2024-09-16T10:27:12.147648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.147648+0000","objects_scrubbed":7965,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:21.702757+0000","scrub_duration":2770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431423488,"num_objects":6583,"num_object_clones":0,"num_object_copies":19749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9989,"num_write_kb":530304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":412,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841452+0000","last_change":"2024-09-16T10:26:38.005912+0000","last_active":"2024-09-16T10:27:27.841452+0000","last_peered":"2024-09-16T10:27:27.841452+0000","last_clean":"2024-09-16T10:26:35.988312+0000","last_became_active":"2024-09-16T10:26:38.005912+0000","last_became_peered":"2024-09-16T10:26:38.005912+0000","last_unstale":"2024-09-16T10:27:27.841452+0000","last_undegraded":"2024-09-16T10:27:27.841452+0000","last_fullsized":"2024-09-16T10:26:37.994305+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:16.433199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:16.433199+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":654,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.848053+0000","last_change":"2024-09-16T10:26:28.553833+0000","last_active":"2024-09-16T10:27:27.848053+0000","last_peered":"2024-09-16T10:27:27.848053+0000","last_clean":"2024-09-16T10:27:27.848053+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:27:27.848053+0000","last_undegraded":"2024-09-16T10:27:27.848053+0000","last_fullsized":"2024-09-16T10:27:27.848053+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:29.375323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":736,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.847172+0000","last_change":"2024-09-16T10:27:12.155297+0000","last_active":"2024-09-16T10:27:27.847172+0000","last_peered":"2024-09-16T10:27:27.847172+0000","last_clean":"2024-09-16T10:27:27.847172+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:27:27.847172+0000","last_undegraded":"2024-09-16T10:27:27.847172+0000","last_fullsized":"2024-09-16T10:27:27.847172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:12.155186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.155186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:19.522860+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'11","reported_seq":346,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.833131+0000","last_change":"2024-09-16T10:26:38.003009+0000","last_active":"2024-09-16T10:27:27.833131+0000","last_peered":"2024-09-16T10:27:27.833131+0000","last_clean":"2024-09-16T10:26:35.991853+0000","last_became_active":"2024-09-16T10:26:38.003009+0000","last_became_peered":"2024-09-16T10:26:38.003009+0000","last_unstale":"2024-09-16T10:27:27.833131+0000","last_undegraded":"2024-09-16T10:26:37.992209+0000","last_fullsized":"2024-09-16T10:26:37.992083+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:41.235528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":545,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.842668+0000","last_change":"2024-09-16T10:27:13.405307+0000","last_active":"2024-09-16T10:27:27.842668+0000","last_peered":"2024-09-16T10:27:27.842668+0000","last_clean":"2024-09-16T10:27:27.842668+0000","last_became_active":"2024-09-16T10:26:28.550959+0000","last_became_peered":"2024-09-16T10:26:28.550959+0000","last_unstale":"2024-09-16T10:27:27.842668+0000","last_undegraded":"2024-09-16T10:27:27.842668+0000","last_fullsized":"2024-09-16T10:27:27.842668+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.405239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.405239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:30.309220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"407'9720","reported_seq":9906,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.971639+0000","last_change":"2024-09-16T10:27:13.214531+0000","last_active":"2024-09-16T10:27:28.971639+0000","last_peered":"2024-09-16T10:27:28.971639+0000","last_clean":"2024-09-16T10:27:28.971639+0000","last_became_active":"2024-09-16T10:25:31.161230+0000","last_became_peered":"2024-09-16T10:25:31.161230+0000","last_unstale":"2024-09-16T10:27:28.971639+0000","last_undegraded":"2024-09-16T10:27:28.971639+0000","last_fullsized":"2024-09-16T10:27:28.971639+0000","mapping_epoch":362,"log_start":"374'3500","ondisk_log_start":"374'3500","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8546","last_scrub_stamp":"2024-09-16T10:27:13.213001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.213001+0000","objects_scrubbed":7773,"log_size":6220,"log_dups_size":0,"ondisk_log_size":6220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:15.065529+0000","scrub_duration":2932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426770432,"num_objects":6512,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9720,"num_write_kb":519424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":125,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.840920+0000","last_change":"2024-09-16T10:26:38.011429+0000","last_active":"2024-09-16T10:27:27.840920+0000","last_peered":"2024-09-16T10:27:27.840920+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:26:38.011429+0000","last_became_peered":"2024-09-16T10:26:38.011429+0000","last_unstale":"2024-09-16T10:27:27.840920+0000","last_undegraded":"2024-09-16T10:27:27.840920+0000","last_fullsized":"2024-09-16T10:26:37.996799+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:07.404565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:07.404565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:27:38.510875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":766,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.846354+0000","last_change":"2024-09-16T10:27:20.653839+0000","last_active":"2024-09-16T10:27:27.846354+0000","last_peered":"2024-09-16T10:27:27.846354+0000","last_clean":"2024-09-16T10:27:27.846354+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:27:27.846354+0000","last_undegraded":"2024-09-16T10:27:27.846354+0000","last_fullsized":"2024-09-16T10:27:27.846354+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:27:20.653762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.653762+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:27.487791+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":762,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828989+0000","last_change":"2024-09-16T10:26:44.070285+0000","last_active":"2024-09-16T10:27:27.828989+0000","last_peered":"2024-09-16T10:27:27.828989+0000","last_clean":"2024-09-16T10:27:27.828989+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:27:27.828989+0000","last_undegraded":"2024-09-16T10:27:27.828989+0000","last_fullsized":"2024-09-16T10:27:27.828989+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:44.070218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.070218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:03.598523+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":280,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.830546+0000","last_change":"2024-09-16T10:26:22.501028+0000","last_active":"2024-09-16T10:27:27.830546+0000","last_peered":"2024-09-16T10:27:27.830546+0000","last_clean":"2024-09-16T10:27:27.830546+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:27:27.830546+0000","last_undegraded":"2024-09-16T10:27:27.830546+0000","last_fullsized":"2024-09-16T10:27:27.830546+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.500963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.500963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:40.312070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":774,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.846040+0000","last_change":"2024-09-16T10:26:37.998943+0000","last_active":"2024-09-16T10:27:27.846040+0000","last_peered":"2024-09-16T10:27:27.846040+0000","last_clean":"2024-09-16T10:26:35.991432+0000","last_became_active":"2024-09-16T10:26:37.998943+0000","last_became_peered":"2024-09-16T10:26:37.998943+0000","last_unstale":"2024-09-16T10:27:27.846040+0000","last_undegraded":"2024-09-16T10:26:37.990081+0000","last_fullsized":"2024-09-16T10:26:37.989850+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:26:20.555476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.555476+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:33.810438+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"407'10096","reported_seq":10250,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:29.658261+0000","last_change":"2024-09-16T10:27:27.863209+0000","last_active":"2024-09-16T10:27:29.658261+0000","last_peered":"2024-09-16T10:27:29.658261+0000","last_clean":"2024-09-16T10:27:29.658261+0000","last_became_active":"2024-09-16T10:25:31.162120+0000","last_became_peered":"2024-09-16T10:25:31.162120+0000","last_unstale":"2024-09-16T10:27:29.658261+0000","last_undegraded":"2024-09-16T10:27:29.658261+0000","last_fullsized":"2024-09-16T10:27:29.658261+0000","mapping_epoch":362,"log_start":"380'5300","ondisk_log_start":"380'5300","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":4849,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:57.862196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432144384,"num_objects":6594,"num_object_clones":0,"num_object_copies":19782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":534080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":710,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841758+0000","last_change":"2024-09-16T10:25:40.594525+0000","last_active":"2024-09-16T10:27:27.841758+0000","last_peered":"2024-09-16T10:27:27.841758+0000","last_clean":"2024-09-16T10:27:27.841758+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:27:27.841758+0000","last_undegraded":"2024-09-16T10:27:27.841758+0000","last_fullsized":"2024-09-16T10:27:27.841758+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:40.594454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.594454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":816,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.841114+0000","last_change":"2024-09-16T10:26:37.999534+0000","last_active":"2024-09-16T10:27:27.841114+0000","last_peered":"2024-09-16T10:27:27.841114+0000","last_clean":"2024-09-16T10:26:35.985800+0000","last_became_active":"2024-09-16T10:26:37.999534+0000","last_became_peered":"2024-09-16T10:26:37.999534+0000","last_unstale":"2024-09-16T10:27:27.841114+0000","last_undegraded":"2024-09-16T10:26:37.994151+0000","last_fullsized":"2024-09-16T10:26:37.994038+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":136,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.828862+0000","last_change":"2024-09-16T10:26:38.002861+0000","last_active":"2024-09-16T10:27:27.828862+0000","last_peered":"2024-09-16T10:27:27.828862+0000","last_clean":"2024-09-16T10:26:35.979755+0000","last_became_active":"2024-09-16T10:26:38.002861+0000","last_became_peered":"2024-09-16T10:26:38.002861+0000","last_unstale":"2024-09-16T10:27:27.828862+0000","last_undegraded":"2024-09-16T10:27:27.828862+0000","last_fullsized":"2024-09-16T10:26:37.992825+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.031940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.031940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:31.578544+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":280,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.835788+0000","last_change":"2024-09-16T10:26:43.199824+0000","last_active":"2024-09-16T10:27:27.835788+0000","last_peered":"2024-09-16T10:27:27.835788+0000","last_clean":"2024-09-16T10:27:27.835788+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:27:27.835788+0000","last_undegraded":"2024-09-16T10:27:27.835788+0000","last_fullsized":"2024-09-16T10:27:27.835788+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:43.199752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:43.199752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.646490+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":613,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.832132+0000","last_change":"2024-09-16T10:26:37.997577+0000","last_active":"2024-09-16T10:27:27.832132+0000","last_peered":"2024-09-16T10:27:27.832132+0000","last_clean":"2024-09-16T10:26:35.985465+0000","last_became_active":"2024-09-16T10:26:37.997577+0000","last_became_peered":"2024-09-16T10:26:37.997577+0000","last_unstale":"2024-09-16T10:27:27.832132+0000","last_undegraded":"2024-09-16T10:27:27.832132+0000","last_fullsized":"2024-09-16T10:26:37.992049+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:34.550928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:34.550928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:12.344839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"407'10066","reported_seq":10167,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.442330+0000","last_change":"2024-09-16T10:25:31.163086+0000","last_active":"2024-09-16T10:27:28.442330+0000","last_peered":"2024-09-16T10:27:28.442330+0000","last_clean":"2024-09-16T10:27:28.442330+0000","last_became_active":"2024-09-16T10:25:31.162769+0000","last_became_peered":"2024-09-16T10:25:31.162769+0000","last_unstale":"2024-09-16T10:27:28.442330+0000","last_undegraded":"2024-09-16T10:27:28.442330+0000","last_fullsized":"2024-09-16T10:27:28.442330+0000","mapping_epoch":362,"log_start":"379'5100","ondisk_log_start":"379'5100","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438960128,"num_objects":6698,"num_object_clones":0,"num_object_copies":20094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10066,"num_write_kb":536448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":661,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.847694+0000","last_change":"2024-09-16T10:26:10.271093+0000","last_active":"2024-09-16T10:27:27.847694+0000","last_peered":"2024-09-16T10:27:27.847694+0000","last_clean":"2024-09-16T10:27:27.847694+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:27:27.847694+0000","last_undegraded":"2024-09-16T10:27:27.847694+0000","last_fullsized":"2024-09-16T10:27:27.847694+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:10.270833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.270833+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":670,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.837869+0000","last_change":"2024-09-16T10:26:42.132944+0000","last_active":"2024-09-16T10:27:27.837869+0000","last_peered":"2024-09-16T10:27:27.837869+0000","last_clean":"2024-09-16T10:27:27.837869+0000","last_became_active":"2024-09-16T10:26:28.553750+0000","last_became_peered":"2024-09-16T10:26:28.553750+0000","last_unstale":"2024-09-16T10:27:27.837869+0000","last_undegraded":"2024-09-16T10:27:27.837869+0000","last_fullsized":"2024-09-16T10:27:27.837869+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:26:42.132853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.132853+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:59.756132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":92,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.843179+0000","last_change":"2024-09-16T10:27:13.285419+0000","last_active":"2024-09-16T10:27:27.843179+0000","last_peered":"2024-09-16T10:27:27.843179+0000","last_clean":"2024-09-16T10:27:27.843179+0000","last_became_active":"2024-09-16T10:25:41.990483+0000","last_became_peered":"2024-09-16T10:25:41.990483+0000","last_unstale":"2024-09-16T10:27:27.843179+0000","last_undegraded":"2024-09-16T10:27:27.843179+0000","last_fullsized":"2024-09-16T10:27:27.843179+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:30.443308+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":280,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.832277+0000","last_change":"2024-09-16T10:26:21.479904+0000","last_active":"2024-09-16T10:27:27.832277+0000","last_peered":"2024-09-16T10:27:27.832277+0000","last_clean":"2024-09-16T10:27:27.832277+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:27:27.832277+0000","last_undegraded":"2024-09-16T10:27:27.832277+0000","last_fullsized":"2024-09-16T10:27:27.832277+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:32.357640+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":744,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841089+0000","last_change":"2024-09-16T10:26:37.999997+0000","last_active":"2024-09-16T10:27:27.841089+0000","last_peered":"2024-09-16T10:27:27.841089+0000","last_clean":"2024-09-16T10:26:35.993796+0000","last_became_active":"2024-09-16T10:26:37.999997+0000","last_became_peered":"2024-09-16T10:26:37.999997+0000","last_unstale":"2024-09-16T10:27:27.841089+0000","last_undegraded":"2024-09-16T10:27:27.841089+0000","last_fullsized":"2024-09-16T10:26:37.994297+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:41.478267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:41.478267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"407'9989","reported_seq":10172,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:28.599121+0000","last_change":"2024-09-16T10:27:21.056386+0000","last_active":"2024-09-16T10:27:28.599121+0000","last_peered":"2024-09-16T10:27:28.599121+0000","last_clean":"2024-09-16T10:27:28.599121+0000","last_became_active":"2024-09-16T10:25:30.620971+0000","last_became_peered":"2024-09-16T10:25:30.620971+0000","last_unstale":"2024-09-16T10:27:28.599121+0000","last_undegraded":"2024-09-16T10:27:28.599121+0000","last_fullsized":"2024-09-16T10:27:28.599121+0000","mapping_epoch":362,"log_start":"380'5309","ondisk_log_start":"380'5309","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9410","last_scrub_stamp":"2024-09-16T10:27:21.056270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:21.056270+0000","objects_scrubbed":7319,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:22.497647+0000","scrub_duration":1957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436142080,"num_objects":6655,"num_object_clones":0,"num_object_copies":19965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9989,"num_write_kb":532608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":449,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.842829+0000","last_change":"2024-09-16T10:26:38.003592+0000","last_active":"2024-09-16T10:27:27.842829+0000","last_peered":"2024-09-16T10:27:27.842829+0000","last_clean":"2024-09-16T10:26:35.991480+0000","last_became_active":"2024-09-16T10:26:38.003592+0000","last_became_peered":"2024-09-16T10:26:38.003592+0000","last_unstale":"2024-09-16T10:27:27.842829+0000","last_undegraded":"2024-09-16T10:27:27.842829+0000","last_fullsized":"2024-09-16T10:26:37.987896+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.367979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.367979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:27:38.184896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":781,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.827802+0000","last_change":"2024-09-16T10:26:38.003098+0000","last_active":"2024-09-16T10:27:27.827802+0000","last_peered":"2024-09-16T10:27:27.827802+0000","last_clean":"2024-09-16T10:26:35.979403+0000","last_became_active":"2024-09-16T10:26:38.003098+0000","last_became_peered":"2024-09-16T10:26:38.003098+0000","last_unstale":"2024-09-16T10:27:27.827802+0000","last_undegraded":"2024-09-16T10:26:37.994683+0000","last_fullsized":"2024-09-16T10:26:37.991648+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:20.104708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.104708+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":75,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.846503+0000","last_change":"2024-09-16T10:26:38.001345+0000","last_active":"2024-09-16T10:27:27.846503+0000","last_peered":"2024-09-16T10:27:27.846503+0000","last_clean":"2024-09-16T10:26:35.989298+0000","last_became_active":"2024-09-16T10:26:38.001345+0000","last_became_peered":"2024-09-16T10:26:38.001345+0000","last_unstale":"2024-09-16T10:27:27.846503+0000","last_undegraded":"2024-09-16T10:26:37.995998+0000","last_fullsized":"2024-09-16T10:26:37.995884+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:08.381010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.381010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"388'10","reported_seq":192,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.840648+0000","last_change":"2024-09-16T10:26:37.998709+0000","last_active":"2024-09-16T10:27:27.840648+0000","last_peered":"2024-09-16T10:27:27.840648+0000","last_clean":"2024-09-16T10:26:35.984994+0000","last_became_active":"2024-09-16T10:26:37.998709+0000","last_became_peered":"2024-09-16T10:26:37.998709+0000","last_unstale":"2024-09-16T10:27:27.840648+0000","last_undegraded":"2024-09-16T10:26:37.992930+0000","last_fullsized":"2024-09-16T10:26:37.992802+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'9","last_scrub_stamp":"2024-09-16T10:25:37.549468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.549468+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":750,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828114+0000","last_change":"2024-09-16T10:27:09.163203+0000","last_active":"2024-09-16T10:27:27.828114+0000","last_peered":"2024-09-16T10:27:27.828114+0000","last_clean":"2024-09-16T10:27:27.828114+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:27:27.828114+0000","last_undegraded":"2024-09-16T10:27:27.828114+0000","last_fullsized":"2024-09-16T10:27:27.828114+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:09.163139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.163139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:19.821510+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":318,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.829391+0000","last_change":"2024-09-16T10:25:00.825815+0000","last_active":"2024-09-16T10:27:27.829391+0000","last_peered":"2024-09-16T10:27:27.829391+0000","last_clean":"2024-09-16T10:27:27.829391+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:27:27.829391+0000","last_undegraded":"2024-09-16T10:27:27.829391+0000","last_fullsized":"2024-09-16T10:27:27.829391+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:00.825743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:00.825743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:34.973010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":736,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841034+0000","last_change":"2024-09-16T10:26:20.469286+0000","last_active":"2024-09-16T10:27:27.841034+0000","last_peered":"2024-09-16T10:27:27.841034+0000","last_clean":"2024-09-16T10:27:27.841034+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:27:27.841034+0000","last_undegraded":"2024-09-16T10:27:27.841034+0000","last_fullsized":"2024-09-16T10:27:27.841034+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:20.469240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.469240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":753,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.842374+0000","last_change":"2024-09-16T10:26:37.999313+0000","last_active":"2024-09-16T10:27:27.842374+0000","last_peered":"2024-09-16T10:27:27.842374+0000","last_clean":"2024-09-16T10:26:35.994509+0000","last_became_active":"2024-09-16T10:26:37.999313+0000","last_became_peered":"2024-09-16T10:26:37.999313+0000","last_unstale":"2024-09-16T10:27:27.842374+0000","last_undegraded":"2024-09-16T10:26:37.995519+0000","last_fullsized":"2024-09-16T10:26:37.995412+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:40.517672+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":737,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.828468+0000","last_change":"2024-09-16T10:26:46.061085+0000","last_active":"2024-09-16T10:27:27.828468+0000","last_peered":"2024-09-16T10:27:27.828468+0000","last_clean":"2024-09-16T10:27:27.828468+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:27:27.828468+0000","last_undegraded":"2024-09-16T10:27:27.828468+0000","last_fullsized":"2024-09-16T10:27:27.828468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:53.687250+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":49,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841111+0000","last_change":"2024-09-16T10:26:38.003751+0000","last_active":"2024-09-16T10:27:27.841111+0000","last_peered":"2024-09-16T10:27:27.841111+0000","last_clean":"2024-09-16T10:13:42.098882+0000","last_became_active":"2024-09-16T10:26:38.003751+0000","last_became_peered":"2024-09-16T10:26:38.003751+0000","last_unstale":"2024-09-16T10:27:27.841111+0000","last_undegraded":"2024-09-16T10:27:27.841111+0000","last_fullsized":"2024-09-16T10:26:37.996606+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.301809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":639,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.838982+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:27:27.838982+0000","last_peered":"2024-09-16T10:27:27.838982+0000","last_clean":"2024-09-16T10:27:27.838982+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:27:27.838982+0000","last_undegraded":"2024-09-16T10:27:27.838982+0000","last_fullsized":"2024-09-16T10:27:27.838982+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:07.337271+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":769,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.853267+0000","last_change":"2024-09-16T10:26:38.014278+0000","last_active":"2024-09-16T10:27:27.853267+0000","last_peered":"2024-09-16T10:27:27.853267+0000","last_clean":"2024-09-16T10:26:35.992350+0000","last_became_active":"2024-09-16T10:26:38.014278+0000","last_became_peered":"2024-09-16T10:26:38.014278+0000","last_unstale":"2024-09-16T10:27:27.853267+0000","last_undegraded":"2024-09-16T10:26:37.993362+0000","last_fullsized":"2024-09-16T10:26:37.993237+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:24.517639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.517639+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:54.289736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":67,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.841102+0000","last_change":"2024-09-16T10:26:37.999023+0000","last_active":"2024-09-16T10:27:27.841102+0000","last_peered":"2024-09-16T10:27:27.841102+0000","last_clean":"2024-09-16T10:26:35.985349+0000","last_became_active":"2024-09-16T10:26:37.999023+0000","last_became_peered":"2024-09-16T10:26:37.999023+0000","last_unstale":"2024-09-16T10:27:27.841102+0000","last_undegraded":"2024-09-16T10:27:27.841102+0000","last_fullsized":"2024-09-16T10:26:37.994403+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:37.608524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.608524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:49.455774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":739,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.842114+0000","last_change":"2024-09-16T10:26:37.999668+0000","last_active":"2024-09-16T10:27:27.842114+0000","last_peered":"2024-09-16T10:27:27.842114+0000","last_clean":"2024-09-16T10:26:35.993488+0000","last_became_active":"2024-09-16T10:26:37.999668+0000","last_became_peered":"2024-09-16T10:26:37.999668+0000","last_unstale":"2024-09-16T10:27:27.842114+0000","last_undegraded":"2024-09-16T10:27:27.842114+0000","last_fullsized":"2024-09-16T10:26:37.995739+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.419085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.419085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:39.094608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":276,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.831853+0000","last_change":"2024-09-16T10:26:38.003699+0000","last_active":"2024-09-16T10:27:27.831853+0000","last_peered":"2024-09-16T10:27:27.831853+0000","last_clean":"2024-09-16T10:26:35.982840+0000","last_became_active":"2024-09-16T10:26:38.003699+0000","last_became_peered":"2024-09-16T10:26:38.003699+0000","last_unstale":"2024-09-16T10:27:27.831853+0000","last_undegraded":"2024-09-16T10:27:27.831853+0000","last_fullsized":"2024-09-16T10:26:37.993072+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:05.327493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":312,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.827589+0000","last_change":"2024-09-16T10:26:38.002460+0000","last_active":"2024-09-16T10:27:27.827589+0000","last_peered":"2024-09-16T10:27:27.827589+0000","last_clean":"2024-09-16T10:26:35.979255+0000","last_became_active":"2024-09-16T10:26:38.002460+0000","last_became_peered":"2024-09-16T10:26:38.002460+0000","last_unstale":"2024-09-16T10:27:27.827589+0000","last_undegraded":"2024-09-16T10:27:27.827589+0000","last_fullsized":"2024-09-16T10:26:37.991442+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:09.961098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.961098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:57.051524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":754,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.829081+0000","last_change":"2024-09-16T10:26:38.002505+0000","last_active":"2024-09-16T10:27:27.829081+0000","last_peered":"2024-09-16T10:27:27.829081+0000","last_clean":"2024-09-16T10:26:35.979851+0000","last_became_active":"2024-09-16T10:26:38.002505+0000","last_became_peered":"2024-09-16T10:26:38.002505+0000","last_unstale":"2024-09-16T10:27:27.829081+0000","last_undegraded":"2024-09-16T10:26:37.993329+0000","last_fullsized":"2024-09-16T10:26:37.993189+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:40.073686+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":748,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.846770+0000","last_change":"2024-09-16T10:26:38.012076+0000","last_active":"2024-09-16T10:27:27.846770+0000","last_peered":"2024-09-16T10:27:27.846770+0000","last_clean":"2024-09-16T10:26:35.991055+0000","last_became_active":"2024-09-16T10:26:38.012076+0000","last_became_peered":"2024-09-16T10:26:38.012076+0000","last_unstale":"2024-09-16T10:27:27.846770+0000","last_undegraded":"2024-09-16T10:27:27.846770+0000","last_fullsized":"2024-09-16T10:26:37.995935+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.514447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"388'11","reported_seq":952,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.848212+0000","last_change":"2024-09-16T10:26:38.005821+0000","last_active":"2024-09-16T10:27:27.848212+0000","last_peered":"2024-09-16T10:27:27.848212+0000","last_clean":"2024-09-16T10:26:35.994080+0000","last_became_active":"2024-09-16T10:26:38.005821+0000","last_became_peered":"2024-09-16T10:26:38.005821+0000","last_unstale":"2024-09-16T10:27:27.848212+0000","last_undegraded":"2024-09-16T10:26:37.994426+0000","last_fullsized":"2024-09-16T10:26:37.994289+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:58.423570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":545,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.830783+0000","last_change":"2024-09-16T10:26:45.194577+0000","last_active":"2024-09-16T10:27:27.830783+0000","last_peered":"2024-09-16T10:27:27.830783+0000","last_clean":"2024-09-16T10:27:27.830783+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:27:27.830783+0000","last_undegraded":"2024-09-16T10:27:27.830783+0000","last_fullsized":"2024-09-16T10:27:27.830783+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.194454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.194454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:50.881675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":322,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.846263+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:27:27.846263+0000","last_peered":"2024-09-16T10:27:27.846263+0000","last_clean":"2024-09-16T10:27:27.846263+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:27:27.846263+0000","last_undegraded":"2024-09-16T10:27:27.846263+0000","last_fullsized":"2024-09-16T10:27:27.846263+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:37.893896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":789,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.847899+0000","last_change":"2024-09-16T10:26:38.000788+0000","last_active":"2024-09-16T10:27:27.847899+0000","last_peered":"2024-09-16T10:27:27.847899+0000","last_clean":"2024-09-16T10:26:35.994364+0000","last_became_active":"2024-09-16T10:26:38.000788+0000","last_became_peered":"2024-09-16T10:26:38.000788+0000","last_unstale":"2024-09-16T10:27:27.847899+0000","last_undegraded":"2024-09-16T10:26:37.994818+0000","last_fullsized":"2024-09-16T10:26:37.994698+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:26:20.405507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.405507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:27.931577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":529,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.835629+0000","last_change":"2024-09-16T10:26:37.998073+0000","last_active":"2024-09-16T10:27:27.835629+0000","last_peered":"2024-09-16T10:27:27.835629+0000","last_clean":"2024-09-16T10:22:01.240719+0000","last_became_active":"2024-09-16T10:26:37.998073+0000","last_became_peered":"2024-09-16T10:26:37.998073+0000","last_unstale":"2024-09-16T10:27:27.835629+0000","last_undegraded":"2024-09-16T10:27:27.835629+0000","last_fullsized":"2024-09-16T10:26:37.992631+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:23.469779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.469779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:02.271706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":920,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.829229+0000","last_change":"2024-09-16T10:26:11.007271+0000","last_active":"2024-09-16T10:27:27.829229+0000","last_peered":"2024-09-16T10:27:27.829229+0000","last_clean":"2024-09-16T10:27:27.829229+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:27:27.829229+0000","last_undegraded":"2024-09-16T10:27:27.829229+0000","last_fullsized":"2024-09-16T10:27:27.829229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:11.007191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:11.007191+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:40.954797+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":728,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.852333+0000","last_change":"2024-09-16T10:26:38.010154+0000","last_active":"2024-09-16T10:27:27.852333+0000","last_peered":"2024-09-16T10:27:27.852333+0000","last_clean":"2024-09-16T10:26:35.991918+0000","last_became_active":"2024-09-16T10:26:38.010154+0000","last_became_peered":"2024-09-16T10:26:38.010154+0000","last_unstale":"2024-09-16T10:27:27.852333+0000","last_undegraded":"2024-09-16T10:27:27.852333+0000","last_fullsized":"2024-09-16T10:26:37.991189+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:40.616458+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":674,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841314+0000","last_change":"2024-09-16T10:26:28.557427+0000","last_active":"2024-09-16T10:27:27.841314+0000","last_peered":"2024-09-16T10:27:27.841314+0000","last_clean":"2024-09-16T10:27:27.841314+0000","last_became_active":"2024-09-16T10:26:28.557003+0000","last_became_peered":"2024-09-16T10:26:28.557003+0000","last_unstale":"2024-09-16T10:27:27.841314+0000","last_undegraded":"2024-09-16T10:27:27.841314+0000","last_fullsized":"2024-09-16T10:27:27.841314+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:17.300150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.300150+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.264407+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":199,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.838099+0000","last_change":"2024-09-16T10:26:38.003739+0000","last_active":"2024-09-16T10:27:27.838099+0000","last_peered":"2024-09-16T10:27:27.838099+0000","last_clean":"2024-09-16T10:26:35.989456+0000","last_became_active":"2024-09-16T10:26:38.003739+0000","last_became_peered":"2024-09-16T10:26:38.003739+0000","last_unstale":"2024-09-16T10:27:27.838099+0000","last_undegraded":"2024-09-16T10:27:27.838099+0000","last_fullsized":"2024-09-16T10:26:37.988544+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:46.070747+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":737,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.838949+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:27:27.838949+0000","last_peered":"2024-09-16T10:27:27.838949+0000","last_clean":"2024-09-16T10:27:27.838949+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:27:27.838949+0000","last_undegraded":"2024-09-16T10:27:27.838949+0000","last_fullsized":"2024-09-16T10:27:27.838949+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:56.217450+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":731,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841288+0000","last_change":"2024-09-16T10:26:14.413104+0000","last_active":"2024-09-16T10:27:27.841288+0000","last_peered":"2024-09-16T10:27:27.841288+0000","last_clean":"2024-09-16T10:27:27.841288+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:27:27.841288+0000","last_undegraded":"2024-09-16T10:27:27.841288+0000","last_fullsized":"2024-09-16T10:27:27.841288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:14.413019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:14.413019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":792,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.839900+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:27:27.839900+0000","last_peered":"2024-09-16T10:27:27.839900+0000","last_clean":"2024-09-16T10:27:27.839900+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:27:27.839900+0000","last_undegraded":"2024-09-16T10:27:27.839900+0000","last_fullsized":"2024-09-16T10:27:27.839900+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":135,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.847655+0000","last_change":"2024-09-16T10:26:38.011535+0000","last_active":"2024-09-16T10:27:27.847655+0000","last_peered":"2024-09-16T10:27:27.847655+0000","last_clean":"2024-09-16T10:26:35.996853+0000","last_became_active":"2024-09-16T10:26:38.011535+0000","last_became_peered":"2024-09-16T10:26:38.011535+0000","last_unstale":"2024-09-16T10:27:27.847655+0000","last_undegraded":"2024-09-16T10:27:27.847655+0000","last_fullsized":"2024-09-16T10:26:37.998608+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.368081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.368081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:36.525233+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":700,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841334+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:27:27.841334+0000","last_peered":"2024-09-16T10:27:27.841334+0000","last_clean":"2024-09-16T10:27:27.841334+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:27:27.841334+0000","last_undegraded":"2024-09-16T10:27:27.841334+0000","last_fullsized":"2024-09-16T10:27:27.841334+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":716,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.845905+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:27:27.845905+0000","last_peered":"2024-09-16T10:27:27.845905+0000","last_clean":"2024-09-16T10:27:27.845905+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:27:27.845905+0000","last_undegraded":"2024-09-16T10:27:27.845905+0000","last_fullsized":"2024-09-16T10:27:27.845905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:37.535050+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":648,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.830196+0000","last_change":"2024-09-16T10:26:38.004064+0000","last_active":"2024-09-16T10:27:27.830196+0000","last_peered":"2024-09-16T10:27:27.830196+0000","last_clean":"2024-09-16T10:26:35.982353+0000","last_became_active":"2024-09-16T10:26:38.004064+0000","last_became_peered":"2024-09-16T10:26:38.004064+0000","last_unstale":"2024-09-16T10:27:27.830196+0000","last_undegraded":"2024-09-16T10:27:27.830196+0000","last_fullsized":"2024-09-16T10:26:37.992187+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.984796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.984796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:51.123220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":750,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:27.841541+0000","last_change":"2024-09-16T10:26:38.009136+0000","last_active":"2024-09-16T10:27:27.841541+0000","last_peered":"2024-09-16T10:27:27.841541+0000","last_clean":"2024-09-16T10:26:35.988399+0000","last_became_active":"2024-09-16T10:26:38.009136+0000","last_became_peered":"2024-09-16T10:26:38.009136+0000","last_unstale":"2024-09-16T10:27:27.841541+0000","last_undegraded":"2024-09-16T10:26:37.995927+0000","last_fullsized":"2024-09-16T10:26:37.995821+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:43.557288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":749,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.828550+0000","last_change":"2024-09-16T10:26:37.994843+0000","last_active":"2024-09-16T10:27:27.828550+0000","last_peered":"2024-09-16T10:27:27.828550+0000","last_clean":"2024-09-16T10:26:35.980698+0000","last_became_active":"2024-09-16T10:26:37.994843+0000","last_became_peered":"2024-09-16T10:26:37.994843+0000","last_unstale":"2024-09-16T10:27:27.828550+0000","last_undegraded":"2024-09-16T10:27:27.828550+0000","last_fullsized":"2024-09-16T10:26:37.991376+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:13.803568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.803568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.038659+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":48,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.840641+0000","last_change":"2024-09-16T10:26:37.998775+0000","last_active":"2024-09-16T10:27:27.840641+0000","last_peered":"2024-09-16T10:27:27.840641+0000","last_clean":"2024-09-16T10:13:40.081041+0000","last_became_active":"2024-09-16T10:26:37.998775+0000","last_became_peered":"2024-09-16T10:26:37.998775+0000","last_unstale":"2024-09-16T10:27:27.840641+0000","last_undegraded":"2024-09-16T10:27:27.840641+0000","last_fullsized":"2024-09-16T10:26:37.993198+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:41.478139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":739,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841590+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:27:27.841590+0000","last_peered":"2024-09-16T10:27:27.841590+0000","last_clean":"2024-09-16T10:27:27.841590+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:27:27.841590+0000","last_undegraded":"2024-09-16T10:27:27.841590+0000","last_fullsized":"2024-09-16T10:27:27.841590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:36.428840+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":785,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841028+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:27:27.841028+0000","last_peered":"2024-09-16T10:27:27.841028+0000","last_clean":"2024-09-16T10:27:27.841028+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:27:27.841028+0000","last_undegraded":"2024-09-16T10:27:27.841028+0000","last_fullsized":"2024-09-16T10:27:27.841028+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":742,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.846916+0000","last_change":"2024-09-16T10:27:27.598364+0000","last_active":"2024-09-16T10:27:27.846916+0000","last_peered":"2024-09-16T10:27:27.846916+0000","last_clean":"2024-09-16T10:27:27.846916+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:27:27.846916+0000","last_undegraded":"2024-09-16T10:27:27.846916+0000","last_fullsized":"2024-09-16T10:27:27.846916+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:27.598299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:27.598299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:38.278185+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":740,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.840913+0000","last_change":"2024-09-16T10:26:37.999481+0000","last_active":"2024-09-16T10:27:27.840913+0000","last_peered":"2024-09-16T10:27:27.840913+0000","last_clean":"2024-09-16T10:26:35.985145+0000","last_became_active":"2024-09-16T10:26:37.999481+0000","last_became_peered":"2024-09-16T10:26:37.999481+0000","last_unstale":"2024-09-16T10:27:27.840913+0000","last_undegraded":"2024-09-16T10:27:27.840913+0000","last_fullsized":"2024-09-16T10:26:37.993233+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:12.539106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.539106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:07.328826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":360,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.830322+0000","last_change":"2024-09-16T10:26:38.003729+0000","last_active":"2024-09-16T10:27:27.830322+0000","last_peered":"2024-09-16T10:27:27.830322+0000","last_clean":"2024-09-16T10:26:35.982121+0000","last_became_active":"2024-09-16T10:26:38.003729+0000","last_became_peered":"2024-09-16T10:26:38.003729+0000","last_unstale":"2024-09-16T10:27:27.830322+0000","last_undegraded":"2024-09-16T10:27:27.830322+0000","last_fullsized":"2024-09-16T10:26:37.991848+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:26.432557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.432557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:46.234537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":733,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.843009+0000","last_change":"2024-09-16T10:26:23.240494+0000","last_active":"2024-09-16T10:27:27.843009+0000","last_peered":"2024-09-16T10:27:27.843009+0000","last_clean":"2024-09-16T10:27:27.843009+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:27:27.843009+0000","last_undegraded":"2024-09-16T10:27:27.843009+0000","last_fullsized":"2024-09-16T10:27:27.843009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:23.240435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:23.240435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:32.297301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":751,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841188+0000","last_change":"2024-09-16T10:26:13.362416+0000","last_active":"2024-09-16T10:27:27.841188+0000","last_peered":"2024-09-16T10:27:27.841188+0000","last_clean":"2024-09-16T10:27:27.841188+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:27:27.841188+0000","last_undegraded":"2024-09-16T10:27:27.841188+0000","last_fullsized":"2024-09-16T10:27:27.841188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.362347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.362347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":780,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-16T10:27:27.841796+0000","last_change":"2024-09-16T10:25:18.564978+0000","last_active":"2024-09-16T10:27:27.841796+0000","last_peered":"2024-09-16T10:27:27.841796+0000","last_clean":"2024-09-16T10:27:27.841796+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:27:27.841796+0000","last_undegraded":"2024-09-16T10:27:27.841796+0000","last_fullsized":"2024-09-16T10:27:27.841796+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:18.564894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.564894+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:06.370154+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":747,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-09-16T10:27:27.852800+0000","last_change":"2024-09-16T10:26:37.998528+0000","last_active":"2024-09-16T10:27:27.852800+0000","last_peered":"2024-09-16T10:27:27.852800+0000","last_clean":"2024-09-16T10:26:35.992035+0000","last_became_active":"2024-09-16T10:26:37.998528+0000","last_became_peered":"2024-09-16T10:26:37.998528+0000","last_unstale":"2024-09-16T10:27:27.852800+0000","last_undegraded":"2024-09-16T10:27:27.852800+0000","last_fullsized":"2024-09-16T10:26:37.991371+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:39.572571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6935150616,"num_objects":105823,"num_object_clones":0,"num_object_copies":317469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159047,"num_write_kb":8475777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20605382656,"data_stored":20605370440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80819,"ondisk_log_size":80819,"up":44,"acting":44,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":682,"ondisk_log_size":682,"up":86,"acting":86,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":84,"acting":84,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":153,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":552211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":54,"acting":54,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234143,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995628,"kb_used_data":1495196,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91749524,"statfs":{"total":95995035648,"available":93951512576,"internally_reserved":0,"allocated":1531080704,"data_stored":1530449375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":512424394},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528494,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3513276,"kb_used_data":2986476,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":90231876,"statfs":{"total":95995035648,"available":92397441024,"internally_reserved":0,"allocated":3058151424,"data_stored":3057406136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790132,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2669268,"kb_used_data":2150776,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91075884,"statfs":{"total":95995035648,"available":93261705216,"internally_reserved":0,"allocated":2202394624,"data_stored":2201639030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986361,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3929496,"kb_used_data":3442880,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":89815656,"statfs":{"total":95995035648,"available":91971231744,"internally_reserved":0,"allocated":3525509120,"data_stored":3524737310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215294,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2742876,"kb_used_data":2143688,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91002276,"statfs":{"total":95995035648,"available":93186330624,"internally_reserved":0,"allocated":2195136512,"data_stored":2194364952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":613349832},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411524,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2889292,"kb_used_data":2555440,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":90855860,"statfs":{"total":95995035648,"available":93036400640,"internally_reserved":0,"allocated":2616770560,"data_stored":2616021967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1856424,"kb_used_data":1275476,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91888728,"statfs":{"total":95995035648,"available":94094057472,"internally_reserved":0,"allocated":1306087424,"data_stored":1305302713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738573,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4784464,"kb_used_data":4189628,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":88960688,"statfs":{"total":95995035648,"available":91095744512,"internally_reserved":0,"allocated":4290179072,"data_stored":4289418760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4282191872,"data_stored":4282187800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1293287424,"data_stored":1293287424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2596405248,"data_stored":2596405248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2168324096,"data_stored":2168324096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3490844672,"data_stored":3490840600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2186215424,"data_stored":2186215424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3053981696,"data_stored":3053977624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1534132224,"data_stored":1534132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:32.406 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T10:27:32.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 26 2024-09-16T10:27:32.444 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:32.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:32.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:32.606 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:32.654 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:32.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:32.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:32.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:32.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:32.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:32.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:27:32.936 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:32.944 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:32.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:33.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:33.062 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:32 smithi063 ceph-mon[39470]: from='client.16341 v1:172.21.15.63:0/3046154358' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:27:33.062 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:32 smithi063 ceph-mon[39470]: pgmap v740: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.08k op/s; 26240/318066 objects degraded (8.250%) 2024-09-16T10:27:33.063 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:32 smithi063 ceph-mon[29993]: from='client.16341 v1:172.21.15.63:0/3046154358' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:27:33.063 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:32 smithi063 ceph-mon[29993]: pgmap v740: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.08k op/s; 26240/318066 objects degraded (8.250%) 2024-09-16T10:27:33.119 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:33.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:33.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:33.279 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:33.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:33.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:32 smithi165 ceph-mon[32696]: from='client.16341 v1:172.21.15.63:0/3046154358' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:27:33.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:32 smithi165 ceph-mon[32696]: pgmap v740: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.08k op/s; 26240/318066 objects degraded (8.250%) 2024-09-16T10:27:33.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:33.437 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:33.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:33.512 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:33.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:33.584 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:33.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:33.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:33.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:33.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:33.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:33.863 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:33.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:33.934 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:33.935 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:33.965 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:33.977 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:33.977 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:34.016 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:34.016 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:34.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:34.055 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:34.055 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:34.092 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:34.126 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:34.127 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:34.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:34.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:34.166 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:34.166 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:34.166 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:34.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:34.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:34.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:34.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:34.516 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:34.555 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:34.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.697 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:34.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:34.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:34.836 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:34.849 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:27:34.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:34.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:34.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:35.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:35.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:35.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:35.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[29993]: pgmap v741: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 694 B/s rd, 0 B/s wr, 812 op/s; 26240/318066 objects degraded (8.250%) 2024-09-16T10:27:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3848183426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4133204588' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[29993]: from='client.26266 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:27:35.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:27:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[39470]: pgmap v741: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 694 B/s rd, 0 B/s wr, 812 op/s; 26240/318066 objects degraded (8.250%) 2024-09-16T10:27:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3848183426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4133204588' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[39470]: from='client.26266 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:27:35.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:34 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:27:35.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:35.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:34 smithi165 ceph-mon[32696]: pgmap v741: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 694 B/s rd, 0 B/s wr, 812 op/s; 26240/318066 objects degraded (8.250%) 2024-09-16T10:27:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3848183426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4133204588' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:34 smithi165 ceph-mon[32696]: from='client.26266 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:34 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:27:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:34 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:27:35.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:35.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:35.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:35.461 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:35.500 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:35.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:35.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:35.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:35.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:35.743 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:35.780 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:35.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:35.875 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 10 pg_num to 26 2024-09-16T10:27:35.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:35.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:35.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:36.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:36.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:36.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:36.139 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:36.174 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:36.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:36.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[29993]: from='client.26266 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[29993]: osdmap e409: 8 total, 7 up, 8 in 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/275673171' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[29993]: from='client.26278 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[29993]: pgmap v743: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 24697/299217 objects degraded (8.254%) 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[39470]: from='client.26266 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[39470]: osdmap e409: 8 total, 7 up, 8 in 2024-09-16T10:27:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/275673171' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-16T10:27:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[39470]: from='client.26278 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-16T10:27:36.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:35 smithi063 ceph-mon[39470]: pgmap v743: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 24697/299217 objects degraded (8.254%) 2024-09-16T10:27:36.276 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:35 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:35 smithi165 ceph-mon[32696]: from='client.26266 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:35 smithi165 ceph-mon[32696]: osdmap e409: 8 total, 7 up, 8 in 2024-09-16T10:27:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/275673171' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-16T10:27:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:35 smithi165 ceph-mon[32696]: from='client.26278 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-16T10:27:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:35 smithi165 ceph-mon[32696]: pgmap v743: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 24697/299217 objects degraded (8.254%) 2024-09-16T10:27:36.317 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:36.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:36.419 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:36.457 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:36.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:27:36.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:27:36.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:36.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:36.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:36.707 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:36.723 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:27:36.723 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:27:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:36.747 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:36.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:36.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:36.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:36.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:36.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:37.011 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:36 smithi063 ceph-mon[39470]: from='client.26278 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-16T10:27:37.012 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:36 smithi063 ceph-mon[39470]: osdmap e410: 8 total, 7 up, 8 in 2024-09-16T10:27:37.012 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:36 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 24697/299217 objects degraded (8.254%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:37.012 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:36 smithi063 ceph-mon[29993]: from='client.26278 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-16T10:27:37.012 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:36 smithi063 ceph-mon[29993]: osdmap e410: 8 total, 7 up, 8 in 2024-09-16T10:27:37.012 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:36 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 24697/299217 objects degraded (8.254%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:37.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:37.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:37.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:37.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:37.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:37.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:36 smithi165 ceph-mon[32696]: from='client.26278 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-16T10:27:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:36 smithi165 ceph-mon[32696]: osdmap e410: 8 total, 7 up, 8 in 2024-09-16T10:27:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:36 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 24697/299217 objects degraded (8.254%), 22 pgs degraded (PG_DEGRADED) 2024-09-16T10:27:37.344 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:27:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:27:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:27:37.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:37.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:37.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.555 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:37.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:37.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:37.696 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:37.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:37.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:37.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:27:37.983 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:37.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:38.026 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:37 smithi063 ceph-mon[29993]: pgmap v745: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 987 op/s; 24697/299217 objects degraded (8.254%) 2024-09-16T10:27:38.026 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:37 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:38.026 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:37 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:38.026 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:37 smithi063 ceph-mon[39470]: pgmap v745: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 987 op/s; 24697/299217 objects degraded (8.254%) 2024-09-16T10:27:38.026 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:37 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:38.027 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:37 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:38.086 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:38.135 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:38.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:38.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:27:38.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:38.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:37 smithi165 ceph-mon[32696]: pgmap v745: 168 pgs: 41 active+undersized, 22 active+undersized+degraded, 105 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 804 B/s rd, 0 B/s wr, 987 op/s; 24697/299217 objects degraded (8.254%) 2024-09-16T10:27:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:37 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:37 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:38.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:38.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:38.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:38.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:38.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:38.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:27:38.531 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:38.561 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:38.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:38.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:38.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:38.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:38.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:38.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:38.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:38.855 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:38.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:38.975 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:38.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:38.998 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-16T10:27:38.998 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[29993]: osdmap e411: 8 total, 7 up, 8 in 2024-09-16T10:27:38.998 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:38.998 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:38.999 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-16T10:27:38.999 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[39470]: osdmap e411: 8 total, 7 up, 8 in 2024-09-16T10:27:38.999 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:38.999 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:38 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:39.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:39.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:39.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:39.120 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:39.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:39.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:39.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-16T10:27:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:38 smithi165 ceph-mon[32696]: osdmap e411: 8 total, 7 up, 8 in 2024-09-16T10:27:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:38 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:38 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:39.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:27:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:27:39.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:39.383 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:39.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:39.484 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:39.533 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:39.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:39.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:39.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:39.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:39.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:39.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:39.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:39.914 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:27:39.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:39.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:39.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:40.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:27:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:27:40.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:40.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:40.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:40.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: osdmap e412: 8 total, 7 up, 8 in 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: 10.6 scrub starts 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3082531723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: pgmap v748: 178 pgs: 4 peering, 7 unknown, 41 active+undersized, 21 active+undersized+degraded, 105 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 932 op/s; 18526/286563 objects degraded (6.465%) 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: 10.7 scrub starts 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[29993]: 10.2 scrub starts 2024-09-16T10:27:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: osdmap e412: 8 total, 7 up, 8 in 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: 10.6 scrub starts 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3082531723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: pgmap v748: 178 pgs: 4 peering, 7 unknown, 41 active+undersized, 21 active+undersized+degraded, 105 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 932 op/s; 18526/286563 objects degraded (6.465%) 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: 10.7 scrub starts 2024-09-16T10:27:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:39 smithi063 ceph-mon[39470]: 10.2 scrub starts 2024-09-16T10:27:40.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: osdmap e412: 8 total, 7 up, 8 in 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: 10.6 scrub starts 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3082531723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: pgmap v748: 178 pgs: 4 peering, 7 unknown, 41 active+undersized, 21 active+undersized+degraded, 105 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 932 op/s; 18526/286563 objects degraded (6.465%) 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: 10.7 scrub starts 2024-09-16T10:27:40.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:39 smithi165 ceph-mon[32696]: 10.2 scrub starts 2024-09-16T10:27:40.304 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:40.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:40.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:40.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:40.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:40.653 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:40.686 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:40.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:40.821 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:40.923 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:40.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:41.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:41.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:41.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:41.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: 10.5 scrub starts 2024-09-16T10:27:41.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3082531723' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: osdmap e413: 8 total, 7 up, 8 in 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: 6.2 scrub ok 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: 10.0 scrub starts 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[29993]: 10.2 scrub starts 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: 10.5 scrub starts 2024-09-16T10:27:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3082531723' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:27:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-16T10:27:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: osdmap e413: 8 total, 7 up, 8 in 2024-09-16T10:27:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:27:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: 6.2 scrub ok 2024-09-16T10:27:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: 10.0 scrub starts 2024-09-16T10:27:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:40 smithi063 ceph-mon[39470]: 10.2 scrub starts 2024-09-16T10:27:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.204 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:41.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: 10.5 scrub starts 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3082531723' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: osdmap e413: 8 total, 7 up, 8 in 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: 6.2 scrub ok 2024-09-16T10:27:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: 10.0 scrub starts 2024-09-16T10:27:41.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:40 smithi165 ceph-mon[32696]: 10.2 scrub starts 2024-09-16T10:27:41.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:41.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:41.428 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.452 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:41.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:41.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:41.576 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:41.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:41.701 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:41.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:41.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:41.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:41.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:41.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:42.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:42.096 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:42.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:42.197 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.5 scrub starts 2024-09-16T10:27:42.197 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.4 scrub starts 2024-09-16T10:27:42.197 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.5 scrub ok 2024-09-16T10:27:42.197 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.0 scrub ok 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: osdmap e414: 8 total, 7 up, 8 in 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.4 scrub ok 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 18526/286563 objects degraded (6.465%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.2 scrub ok 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.6 scrub starts 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.6 scrub ok 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: pgmap v751: 178 pgs: 4 peering, 4 unknown, 41 active+undersized, 23 active+undersized+degraded, 106 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 17460/282261 objects degraded (6.186%); 0 B/s, 0 objects/s recovering 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.7 scrub starts 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 6.1 scrub starts 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 6.1 scrub ok 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[29993]: 10.7 scrub ok 2024-09-16T10:27:42.198 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.5 scrub starts 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.4 scrub starts 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.5 scrub ok 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.0 scrub ok 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: osdmap e414: 8 total, 7 up, 8 in 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.4 scrub ok 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 18526/286563 objects degraded (6.465%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.2 scrub ok 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.6 scrub starts 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.6 scrub ok 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: pgmap v751: 178 pgs: 4 peering, 4 unknown, 41 active+undersized, 23 active+undersized+degraded, 106 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 17460/282261 objects degraded (6.186%); 0 B/s, 0 objects/s recovering 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.7 scrub starts 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 6.1 scrub starts 2024-09-16T10:27:42.199 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 6.1 scrub ok 2024-09-16T10:27:42.200 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:41 smithi063 ceph-mon[39470]: 10.7 scrub ok 2024-09-16T10:27:42.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:42.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:42.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:42.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.5 scrub starts 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.4 scrub starts 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.5 scrub ok 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.0 scrub ok 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: osdmap e414: 8 total, 7 up, 8 in 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.4 scrub ok 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 18526/286563 objects degraded (6.465%), 21 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.2 scrub ok 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.6 scrub starts 2024-09-16T10:27:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.6 scrub ok 2024-09-16T10:27:42.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: pgmap v751: 178 pgs: 4 peering, 4 unknown, 41 active+undersized, 23 active+undersized+degraded, 106 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 17460/282261 objects degraded (6.186%); 0 B/s, 0 objects/s recovering 2024-09-16T10:27:42.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.7 scrub starts 2024-09-16T10:27:42.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 6.1 scrub starts 2024-09-16T10:27:42.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 6.1 scrub ok 2024-09-16T10:27:42.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:41 smithi165 ceph-mon[32696]: 10.7 scrub ok 2024-09-16T10:27:42.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:42.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:42.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:42.394 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-16T10:27:42.395 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:27:42.395 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-09-16T10:27:42.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:27:42.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:42.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:42.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:42.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:42.630 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:42.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:42.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:42.809 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:42.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:42.908 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:42.909 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:42.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:42.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:42.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:42.942 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.3 scrub starts 2024-09-16T10:27:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.3 scrub ok 2024-09-16T10:27:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10814 2024-09-16T10:27:42.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10506 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 412'10908 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10822 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10442 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10785 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: osdmap e415: 8 total, 7 up, 8 in 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10580 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 412'10506 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10442 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 412'10822 2024-09-16T10:27:42.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 412'10785 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: 10.10 scrub starts 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1931190113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.3 scrub starts 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.3 scrub ok 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10814 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:42.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10506 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 412'10908 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10822 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10442 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10785 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: osdmap e415: 8 total, 7 up, 8 in 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10580 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 412'10506 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10442 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 412'10822 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:42.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 412'10785 2024-09-16T10:27:42.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:42.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: 10.10 scrub starts 2024-09-16T10:27:42.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1931190113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:42.960 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:42.960 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:42.997 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:42.998 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:43.034 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:43.035 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:43.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:43.066 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:43.066 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:43.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:43.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:43.101 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:43.101 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:43.134 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:43.134 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:43.135 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:43.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:43.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:43.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:43.296 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.3 scrub starts 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.3 scrub ok 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10814 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10506 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 412'10908 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10822 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10442 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 412'10785 2024-09-16T10:27:43.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: osdmap e415: 8 total, 7 up, 8 in 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10580 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 412'10506 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10442 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 412'10822 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 412'10729 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 412'10785 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 412'10720 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: 10.10 scrub starts 2024-09-16T10:27:43.304 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1931190113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:43.337 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:43.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:43.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:43.481 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:43.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:43.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:43.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:43.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:43.703 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:43.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:43.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:43.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:43.845 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:43.888 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:43.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:43.989 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:44.032 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:44.079 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:43 smithi063 ceph-mon[29993]: 10.10 scrub ok 2024-09-16T10:27:44.079 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:43 smithi063 ceph-mon[29993]: pgmap v753: 178 pgs: 4 peering, 4 unknown, 41 active+undersized, 23 active+undersized+degraded, 106 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 544 op/s; 17460/282261 objects degraded (6.186%); 0 B/s, 0 objects/s recovering 2024-09-16T10:27:44.079 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:43 smithi063 ceph-mon[39470]: 10.10 scrub ok 2024-09-16T10:27:44.080 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:43 smithi063 ceph-mon[39470]: pgmap v753: 178 pgs: 4 peering, 4 unknown, 41 active+undersized, 23 active+undersized+degraded, 106 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 544 op/s; 17460/282261 objects degraded (6.186%); 0 B/s, 0 objects/s recovering 2024-09-16T10:27:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:27:44.134 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:27:44.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:44.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:44.262 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:44.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:43 smithi165 ceph-mon[32696]: 10.10 scrub ok 2024-09-16T10:27:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:43 smithi165 ceph-mon[32696]: pgmap v753: 178 pgs: 4 peering, 4 unknown, 41 active+undersized, 23 active+undersized+degraded, 106 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 544 op/s; 17460/282261 objects degraded (6.186%); 0 B/s, 0 objects/s recovering 2024-09-16T10:27:44.307 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:44.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:44.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:44.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.581 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:44.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:44.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:44.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:44.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:44.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:44.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:44.952 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:27:44.952 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:27:44.953 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:27:44.953 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:27:44.953 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:27:44.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:44.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:45.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:45.102 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:45.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:45.204 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:45.242 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:45.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:45.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:45.391 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:45.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:45.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:27:45.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:45.536 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:45.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:45.602 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:27:45.609 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:27:45.638 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:45.692 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:45.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:45.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:45.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:45 smithi165 ceph-mon[32696]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:27:45.803 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:45 smithi063 ceph-mon[29993]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:27:45.803 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:45 smithi063 ceph-mon[39470]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:27:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:45.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:45.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:45.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:45.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:45.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:46.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:46.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:46.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:46.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:27:46.160 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:27:46.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:46.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:46.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:46.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:46.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:46.376 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:46.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:46.477 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:46.542 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:46.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:46.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:46.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:46.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:46.776 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:46 smithi165 ceph-mon[32696]: 3.6 scrub starts 2024-09-16T10:27:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:46 smithi165 ceph-mon[32696]: 3.6 scrub ok 2024-09-16T10:27:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:46 smithi165 ceph-mon[32696]: pgmap v754: 178 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 22047/271383 objects degraded (8.124%); 46969/271383 objects misplaced (17.307%); 5.3 MiB/s, 85 objects/s recovering 2024-09-16T10:27:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:46 smithi165 ceph-mon[32696]: from='client.26293 v1:172.21.15.63:0/2783679382' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:27:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:46 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 22047/271383 objects degraded (8.124%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:46.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:46 smithi165 ceph-mon[32696]: osdmap e416: 8 total, 7 up, 8 in 2024-09-16T10:27:46.811 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:46.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:46.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[29993]: 3.6 scrub starts 2024-09-16T10:27:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[29993]: 3.6 scrub ok 2024-09-16T10:27:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[29993]: pgmap v754: 178 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 22047/271383 objects degraded (8.124%); 46969/271383 objects misplaced (17.307%); 5.3 MiB/s, 85 objects/s recovering 2024-09-16T10:27:46.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[29993]: from='client.26293 v1:172.21.15.63:0/2783679382' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:27:46.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 22047/271383 objects degraded (8.124%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:46.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[29993]: osdmap e416: 8 total, 7 up, 8 in 2024-09-16T10:27:46.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[39470]: 3.6 scrub starts 2024-09-16T10:27:46.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[39470]: 3.6 scrub ok 2024-09-16T10:27:46.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[39470]: pgmap v754: 178 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 22047/271383 objects degraded (8.124%); 46969/271383 objects misplaced (17.307%); 5.3 MiB/s, 85 objects/s recovering 2024-09-16T10:27:46.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[39470]: from='client.26293 v1:172.21.15.63:0/2783679382' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:27:46.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 22047/271383 objects degraded (8.124%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:46.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:46 smithi063 ceph-mon[39470]: osdmap e416: 8 total, 7 up, 8 in 2024-09-16T10:27:46.955 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:46.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:47.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:47.080 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:47.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:47.181 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2024-09-16T10:27:45.377499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5915804422,"num_objects":90461,"num_object_clones":0,"num_object_copies":271383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22047,"num_objects_misplaced":46969,"num_objects_unfound":0,"num_objects_dirty":90461,"num_whiteouts":0,"num_read":1303,"num_read_kb":1060,"num_write":132206,"num_write_kb":6351992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":33912453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133309,"ondisk_log_size":133309,"up":468,"acting":468,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":542,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21837332,"kb_used_data":17550644,"kb_used_omap":12,"kb_used_meta":4286515,"kb_avail":728123884,"statfs":{"total":767960285184,"available":745598857216,"internally_reserved":0,"allocated":17971859456,"data_stored":17965608582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4389391921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-331776000,"num_objects":-5060,"num_object_clones":0,"num_object_copies":-15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3521,"num_objects_misplaced":46969,"num_objects_unfound":0,"num_objects_dirty":-5060,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-38069,"num_write_kb":-2123904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":33095680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004062"},"pg_stats":[{"pgid":"10.16","version":"415'10893","reported_seq":11106,"reported_epoch":415,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:27:44.707906+0000","last_change":"2024-09-16T10:27:42.062503+0000","last_active":"2024-09-16T10:27:44.707906+0000","last_peered":"2024-09-16T10:27:44.707906+0000","last_clean":"2024-09-16T10:27:39.914173+0000","last_became_active":"2024-09-16T10:27:42.035626+0000","last_became_peered":"2024-09-16T10:27:42.035626+0000","last_unstale":"2024-09-16T10:27:44.707906+0000","last_undegraded":"2024-09-16T10:27:44.707906+0000","last_fullsized":"2024-09-16T10:27:44.707906+0000","mapping_epoch":414,"log_start":"385'6109","ondisk_log_start":"385'6109","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"404'9410","last_scrub_stamp":"2024-09-16T10:27:21.056270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:21.056270+0000","objects_scrubbed":7319,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183042048,"num_objects":2793,"num_object_clones":0,"num_object_copies":8379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5586,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5529,"num_write_kb":266304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":2793}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":775,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.926452+0000","last_change":"2024-09-16T10:22:04.122045+0000","last_active":"2024-09-16T10:27:41.926452+0000","last_peered":"2024-09-16T10:27:41.926452+0000","last_clean":"2024-09-16T10:27:41.926452+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:27:41.926452+0000","last_undegraded":"2024-09-16T10:27:41.926452+0000","last_fullsized":"2024-09-16T10:27:41.926452+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:04.121966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:04.121966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:58.647110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":721,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922737+0000","last_change":"2024-09-16T10:26:53.392698+0000","last_active":"2024-09-16T10:27:41.922737+0000","last_peered":"2024-09-16T10:27:41.922737+0000","last_clean":"2024-09-16T10:27:41.922737+0000","last_became_active":"2024-09-16T10:26:28.557096+0000","last_became_peered":"2024-09-16T10:26:28.557096+0000","last_unstale":"2024-09-16T10:27:41.922737+0000","last_undegraded":"2024-09-16T10:27:41.922737+0000","last_fullsized":"2024-09-16T10:27:41.922737+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:53.392646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:53.392646+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:10.475144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":765,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.922599+0000","last_change":"2024-09-16T10:26:38.001941+0000","last_active":"2024-09-16T10:27:41.922599+0000","last_peered":"2024-09-16T10:27:41.922599+0000","last_clean":"2024-09-16T10:26:35.984553+0000","last_became_active":"2024-09-16T10:26:38.001941+0000","last_became_peered":"2024-09-16T10:26:38.001941+0000","last_unstale":"2024-09-16T10:27:41.922599+0000","last_undegraded":"2024-09-16T10:27:41.922599+0000","last_fullsized":"2024-09-16T10:26:37.987716+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.940459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.940459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:15.402666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":77,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.914700+0000","last_change":"2024-09-16T10:26:28.552367+0000","last_active":"2024-09-16T10:27:41.914700+0000","last_peered":"2024-09-16T10:27:41.914700+0000","last_clean":"2024-09-16T10:27:41.914700+0000","last_became_active":"2024-09-16T10:26:28.551847+0000","last_became_peered":"2024-09-16T10:26:28.551847+0000","last_unstale":"2024-09-16T10:27:41.914700+0000","last_undegraded":"2024-09-16T10:27:41.914700+0000","last_fullsized":"2024-09-16T10:27:41.914700+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.378376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.378376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"415'10998","reported_seq":11129,"reported_epoch":415,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:27:44.114109+0000","last_change":"2024-09-16T10:27:42.027827+0000","last_active":"2024-09-16T10:27:44.114109+0000","last_peered":"2024-09-16T10:27:44.114109+0000","last_clean":"2024-09-16T10:27:39.879076+0000","last_became_active":"2024-09-16T10:27:41.990190+0000","last_became_peered":"2024-09-16T10:27:41.990190+0000","last_unstale":"2024-09-16T10:27:44.114109+0000","last_undegraded":"2024-09-16T10:27:44.114109+0000","last_fullsized":"2024-09-16T10:27:44.114109+0000","mapping_epoch":414,"log_start":"384'5900","ondisk_log_start":"384'5900","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":5098,"log_dups_size":0,"ondisk_log_size":5098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184320000,"num_objects":2813,"num_object_clones":0,"num_object_copies":8439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2813,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":268224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":2813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":743,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.919781+0000","last_change":"2024-09-16T10:26:38.012843+0000","last_active":"2024-09-16T10:27:41.919781+0000","last_peered":"2024-09-16T10:27:41.919781+0000","last_clean":"2024-09-16T10:26:35.993698+0000","last_became_active":"2024-09-16T10:26:38.012843+0000","last_became_peered":"2024-09-16T10:26:38.012843+0000","last_unstale":"2024-09-16T10:27:41.919781+0000","last_undegraded":"2024-09-16T10:27:41.919781+0000","last_fullsized":"2024-09-16T10:26:37.997407+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:24.255421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:20:24.255421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.535138+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":783,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922285+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:27:41.922285+0000","last_peered":"2024-09-16T10:27:41.922285+0000","last_clean":"2024-09-16T10:27:41.922285+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:27:41.922285+0000","last_undegraded":"2024-09-16T10:27:41.922285+0000","last_fullsized":"2024-09-16T10:27:41.922285+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":852,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923749+0000","last_change":"2024-09-16T10:26:12.334813+0000","last_active":"2024-09-16T10:27:41.923749+0000","last_peered":"2024-09-16T10:27:41.923749+0000","last_clean":"2024-09-16T10:27:41.923749+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:27:41.923749+0000","last_undegraded":"2024-09-16T10:27:41.923749+0000","last_fullsized":"2024-09-16T10:27:41.923749+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:12.334720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.334720+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.177726+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":85,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922840+0000","last_change":"2024-09-16T10:26:45.470165+0000","last_active":"2024-09-16T10:27:41.922840+0000","last_peered":"2024-09-16T10:27:41.922840+0000","last_clean":"2024-09-16T10:27:41.922840+0000","last_became_active":"2024-09-16T10:26:28.556509+0000","last_became_peered":"2024-09-16T10:26:28.556509+0000","last_unstale":"2024-09-16T10:27:41.922840+0000","last_undegraded":"2024-09-16T10:27:41.922840+0000","last_fullsized":"2024-09-16T10:27:41.922840+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.470014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.470014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.195883+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"415'10941","reported_seq":11129,"reported_epoch":415,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:27:44.639511+0000","last_change":"2024-09-16T10:27:42.095124+0000","last_active":"2024-09-16T10:27:44.639511+0000","last_peered":"2024-09-16T10:27:44.639511+0000","last_clean":"2024-09-16T10:27:39.858635+0000","last_became_active":"2024-09-16T10:27:42.027594+0000","last_became_peered":"2024-09-16T10:27:42.027594+0000","last_unstale":"2024-09-16T10:27:44.639511+0000","last_undegraded":"2024-09-16T10:27:44.639511+0000","last_fullsized":"2024-09-16T10:27:44.639511+0000","mapping_epoch":414,"log_start":"385'6000","ondisk_log_start":"385'6000","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":4849,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183074816,"num_objects":2794,"num_object_clones":0,"num_object_copies":8382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5588,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":267040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":2794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":785,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915414+0000","last_change":"2024-09-16T10:26:48.078096+0000","last_active":"2024-09-16T10:27:41.915414+0000","last_peered":"2024-09-16T10:27:41.915414+0000","last_clean":"2024-09-16T10:27:41.915414+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:27:41.915414+0000","last_undegraded":"2024-09-16T10:27:41.915414+0000","last_fullsized":"2024-09-16T10:27:41.915414+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:48.078028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:48.078028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:03.820478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":679,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.920697+0000","last_change":"2024-09-16T10:26:28.551030+0000","last_active":"2024-09-16T10:27:41.920697+0000","last_peered":"2024-09-16T10:27:41.920697+0000","last_clean":"2024-09-16T10:27:41.920697+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:27:41.920697+0000","last_undegraded":"2024-09-16T10:27:41.920697+0000","last_fullsized":"2024-09-16T10:27:41.920697+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:24:23.258886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.258886+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:55.385976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":765,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.915016+0000","last_change":"2024-09-16T10:26:38.007550+0000","last_active":"2024-09-16T10:27:41.915016+0000","last_peered":"2024-09-16T10:27:41.915016+0000","last_clean":"2024-09-16T10:26:35.982994+0000","last_became_active":"2024-09-16T10:26:38.007550+0000","last_became_peered":"2024-09-16T10:26:38.007550+0000","last_unstale":"2024-09-16T10:27:41.915016+0000","last_undegraded":"2024-09-16T10:27:41.915016+0000","last_fullsized":"2024-09-16T10:26:37.992243+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.834823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.834823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:12.147875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":760,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.917630+0000","last_change":"2024-09-16T10:26:37.997637+0000","last_active":"2024-09-16T10:27:41.917630+0000","last_peered":"2024-09-16T10:27:41.917630+0000","last_clean":"2024-09-16T10:26:35.992361+0000","last_became_active":"2024-09-16T10:26:37.997637+0000","last_became_peered":"2024-09-16T10:26:37.997637+0000","last_unstale":"2024-09-16T10:27:41.917630+0000","last_undegraded":"2024-09-16T10:27:41.917630+0000","last_fullsized":"2024-09-16T10:26:37.989396+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:08.436243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.436243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.15","version":"415'10535","reported_seq":10752,"reported_epoch":415,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:27:44.101222+0000","last_change":"2024-09-16T10:27:42.041961+0000","last_active":"2024-09-16T10:27:44.101222+0000","last_peered":"2024-09-16T10:27:44.101222+0000","last_clean":"2024-09-16T10:27:39.890520+0000","last_became_active":"2024-09-16T10:27:42.011408+0000","last_became_peered":"2024-09-16T10:27:42.011408+0000","last_unstale":"2024-09-16T10:27:44.101222+0000","last_undegraded":"2024-09-16T10:27:44.101222+0000","last_fullsized":"2024-09-16T10:27:44.101222+0000","mapping_epoch":414,"log_start":"376'4200","ondisk_log_start":"376'4200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"400'8546","last_scrub_stamp":"2024-09-16T10:27:13.213001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.213001+0000","objects_scrubbed":7773,"log_size":6335,"log_dups_size":0,"ondisk_log_size":6335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182157312,"num_objects":2780,"num_object_clones":0,"num_object_copies":8340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5560,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5337,"num_write_kb":259712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,0,2],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":2780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":729,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915361+0000","last_change":"2024-09-16T10:13:37.093219+0000","last_active":"2024-09-16T10:27:41.915361+0000","last_peered":"2024-09-16T10:27:41.915361+0000","last_clean":"2024-09-16T10:27:41.915361+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:27:41.915361+0000","last_undegraded":"2024-09-16T10:27:41.915361+0000","last_fullsized":"2024-09-16T10:27:41.915361+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:13:36.070782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:54.979150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":667,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.921284+0000","last_change":"2024-09-16T10:26:28.557316+0000","last_active":"2024-09-16T10:27:41.921284+0000","last_peered":"2024-09-16T10:27:41.921284+0000","last_clean":"2024-09-16T10:27:41.921284+0000","last_became_active":"2024-09-16T10:26:28.556933+0000","last_became_peered":"2024-09-16T10:26:28.556933+0000","last_unstale":"2024-09-16T10:27:41.921284+0000","last_undegraded":"2024-09-16T10:27:41.921284+0000","last_fullsized":"2024-09-16T10:27:41.921284+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:57.426521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":64,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.924076+0000","last_change":"2024-09-16T10:26:38.003607+0000","last_active":"2024-09-16T10:27:41.924076+0000","last_peered":"2024-09-16T10:27:41.924076+0000","last_clean":"2024-09-16T10:13:40.081041+0000","last_became_active":"2024-09-16T10:26:38.003607+0000","last_became_peered":"2024-09-16T10:26:38.003607+0000","last_unstale":"2024-09-16T10:27:41.924076+0000","last_undegraded":"2024-09-16T10:27:41.924076+0000","last_fullsized":"2024-09-16T10:26:37.988017+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:15.369005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:15.369005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:45.104920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":764,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.914899+0000","last_change":"2024-09-16T10:23:42.856401+0000","last_active":"2024-09-16T10:27:41.914899+0000","last_peered":"2024-09-16T10:27:41.914899+0000","last_clean":"2024-09-16T10:27:41.914899+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:27:41.914899+0000","last_undegraded":"2024-09-16T10:27:41.914899+0000","last_fullsized":"2024-09-16T10:27:41.914899+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:42.856353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:42.856353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:33.058807+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"415'10823","reported_seq":11051,"reported_epoch":415,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:27:44.706149+0000","last_change":"2024-09-16T10:27:42.063494+0000","last_active":"2024-09-16T10:27:44.706149+0000","last_peered":"2024-09-16T10:27:44.706149+0000","last_clean":"2024-09-16T10:27:39.905149+0000","last_became_active":"2024-09-16T10:27:42.023314+0000","last_became_peered":"2024-09-16T10:27:42.023314+0000","last_unstale":"2024-09-16T10:27:44.706149+0000","last_undegraded":"2024-09-16T10:27:44.706149+0000","last_fullsized":"2024-09-16T10:27:44.706149+0000","mapping_epoch":414,"log_start":"384'5800","ondisk_log_start":"384'5800","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"400'8705","last_scrub_stamp":"2024-09-16T10:27:12.147648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.147648+0000","objects_scrubbed":7965,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183664640,"num_objects":2803,"num_object_clones":0,"num_object_copies":8409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8409,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":265152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":2803}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":77,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923188+0000","last_change":"2024-09-16T10:26:28.556518+0000","last_active":"2024-09-16T10:27:41.923188+0000","last_peered":"2024-09-16T10:27:41.923188+0000","last_clean":"2024-09-16T10:27:41.923188+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:27:41.923188+0000","last_undegraded":"2024-09-16T10:27:41.923188+0000","last_fullsized":"2024-09-16T10:27:41.923188+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:47.697978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:47.697978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:34.541962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":756,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.918767+0000","last_change":"2024-09-16T10:26:37.996618+0000","last_active":"2024-09-16T10:27:41.918767+0000","last_peered":"2024-09-16T10:27:41.918767+0000","last_clean":"2024-09-16T10:26:35.990280+0000","last_became_active":"2024-09-16T10:26:37.996618+0000","last_became_peered":"2024-09-16T10:26:37.996618+0000","last_unstale":"2024-09-16T10:27:41.918767+0000","last_undegraded":"2024-09-16T10:27:41.918767+0000","last_fullsized":"2024-09-16T10:26:37.991795+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:11.654674+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":330,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.916242+0000","last_change":"2024-09-16T10:26:17.566065+0000","last_active":"2024-09-16T10:27:41.916242+0000","last_peered":"2024-09-16T10:27:41.916242+0000","last_clean":"2024-09-16T10:27:41.916242+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:27:41.916242+0000","last_undegraded":"2024-09-16T10:27:41.916242+0000","last_fullsized":"2024-09-16T10:27:41.916242+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:17.565970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.565970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":753,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924464+0000","last_change":"2024-09-16T10:26:22.230127+0000","last_active":"2024-09-16T10:27:41.924464+0000","last_peered":"2024-09-16T10:27:41.924464+0000","last_clean":"2024-09-16T10:27:41.924464+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:27:41.924464+0000","last_undegraded":"2024-09-16T10:27:41.924464+0000","last_fullsized":"2024-09-16T10:27:41.924464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.230054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.230054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:31.445236+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.13","version":"415'10891","reported_seq":11377,"reported_epoch":415,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T10:27:43.996709+0000","last_change":"2024-09-16T10:27:42.102348+0000","last_active":"2024-09-16T10:27:43.996709+0000","last_peered":"2024-09-16T10:27:43.996709+0000","last_clean":"2024-09-16T10:27:39.872359+0000","last_became_active":"2024-09-16T10:27:42.050830+0000","last_became_peered":"2024-09-16T10:27:42.050830+0000","last_unstale":"2024-09-16T10:27:43.996709+0000","last_undegraded":"2024-09-16T10:27:43.996709+0000","last_fullsized":"2024-09-16T10:27:43.996709+0000","mapping_epoch":414,"log_start":"376'4600","ondisk_log_start":"376'4600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186613760,"num_objects":2848,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5523,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5506,"num_write_kb":267296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":11534336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,3,5],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":2848}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":713,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.935978+0000","last_change":"2024-09-16T10:27:18.318288+0000","last_active":"2024-09-16T10:27:41.935978+0000","last_peered":"2024-09-16T10:27:41.935978+0000","last_clean":"2024-09-16T10:27:41.935978+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:27:41.935978+0000","last_undegraded":"2024-09-16T10:27:41.935978+0000","last_fullsized":"2024-09-16T10:27:41.935978+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:18.318214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:18.318214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:28.213657+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":549,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924680+0000","last_change":"2024-09-16T10:26:28.551407+0000","last_active":"2024-09-16T10:27:41.924680+0000","last_peered":"2024-09-16T10:27:41.924680+0000","last_clean":"2024-09-16T10:27:41.924680+0000","last_became_active":"2024-09-16T10:26:28.551058+0000","last_became_peered":"2024-09-16T10:26:28.551058+0000","last_unstale":"2024-09-16T10:27:41.924680+0000","last_undegraded":"2024-09-16T10:27:41.924680+0000","last_fullsized":"2024-09-16T10:27:41.924680+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:54.217157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":814,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.943998+0000","last_change":"2024-09-16T10:23:01.850206+0000","last_active":"2024-09-16T10:27:41.943998+0000","last_peered":"2024-09-16T10:27:41.943998+0000","last_clean":"2024-09-16T10:27:41.943998+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:27:41.943998+0000","last_undegraded":"2024-09-16T10:27:41.943998+0000","last_fullsized":"2024-09-16T10:27:41.943998+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-16T10:23:01.850113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.850113+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:37.225561+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":557,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923062+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:27:41.923062+0000","last_peered":"2024-09-16T10:27:41.923062+0000","last_clean":"2024-09-16T10:27:41.923062+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:27:41.923062+0000","last_undegraded":"2024-09-16T10:27:41.923062+0000","last_fullsized":"2024-09-16T10:27:41.923062+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"415'10778","reported_seq":11030,"reported_epoch":415,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:27:43.458290+0000","last_change":"2024-09-16T10:27:43.143154+0000","last_active":"2024-09-16T10:27:43.458290+0000","last_peered":"2024-09-16T10:27:43.458290+0000","last_clean":"2024-09-16T10:27:43.458290+0000","last_became_active":"2024-09-16T10:27:41.925807+0000","last_became_peered":"2024-09-16T10:27:41.925807+0000","last_unstale":"2024-09-16T10:27:43.458290+0000","last_undegraded":"2024-09-16T10:27:43.458290+0000","last_fullsized":"2024-09-16T10:27:43.458290+0000","mapping_epoch":414,"log_start":"384'5700","ondisk_log_start":"384'5700","created":411,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"415'10766","last_scrub_stamp":"2024-09-16T10:27:43.142849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:43.142849+0000","objects_scrubbed":2897,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:56.201066+0000","scrub_duration":597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188219392,"num_objects":2872,"num_object_clones":0,"num_object_copies":8616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2872,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,1],"avail_no_missing":["6","1","4"],"object_location_counts":[{"shards":"1,4,6","objects":2872}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":165,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.915159+0000","last_change":"2024-09-16T10:26:38.004428+0000","last_active":"2024-09-16T10:27:41.915159+0000","last_peered":"2024-09-16T10:27:41.915159+0000","last_clean":"2024-09-16T10:26:35.980045+0000","last_became_active":"2024-09-16T10:26:38.004428+0000","last_became_peered":"2024-09-16T10:26:38.004428+0000","last_unstale":"2024-09-16T10:27:41.915159+0000","last_undegraded":"2024-09-16T10:26:37.992071+0000","last_fullsized":"2024-09-16T10:26:37.991949+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:15.092834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":768,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924124+0000","last_change":"2024-09-16T10:26:17.204523+0000","last_active":"2024-09-16T10:27:41.924124+0000","last_peered":"2024-09-16T10:27:41.924124+0000","last_clean":"2024-09-16T10:27:41.924124+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:27:41.924124+0000","last_undegraded":"2024-09-16T10:27:41.924124+0000","last_fullsized":"2024-09-16T10:27:41.924124+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:17.204441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.204441+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":792,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.929875+0000","last_change":"2024-09-16T10:26:38.010506+0000","last_active":"2024-09-16T10:27:41.929875+0000","last_peered":"2024-09-16T10:27:41.929875+0000","last_clean":"2024-09-16T10:26:35.992165+0000","last_became_active":"2024-09-16T10:26:38.010506+0000","last_became_peered":"2024-09-16T10:26:38.010506+0000","last_unstale":"2024-09-16T10:27:41.929875+0000","last_undegraded":"2024-09-16T10:26:37.992894+0000","last_fullsized":"2024-09-16T10:26:37.992782+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:34.114142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:34.114142+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:55.749722+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"388'9","reported_seq":938,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.923766+0000","last_change":"2024-09-16T10:26:38.004232+0000","last_active":"2024-09-16T10:27:41.923766+0000","last_peered":"2024-09-16T10:27:41.923766+0000","last_clean":"2024-09-16T10:26:35.985776+0000","last_became_active":"2024-09-16T10:26:38.004232+0000","last_became_peered":"2024-09-16T10:26:38.004232+0000","last_unstale":"2024-09-16T10:27:41.923766+0000","last_undegraded":"2024-09-16T10:26:37.998183+0000","last_fullsized":"2024-09-16T10:26:37.998085+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:25:16.538706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.538706+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"415'10678","reported_seq":10818,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:44.665728+0000","last_change":"2024-09-16T10:27:38.986165+0000","last_active":"2024-09-16T10:27:44.665728+0000","last_peered":"2024-09-16T10:27:44.665728+0000","last_clean":"2024-09-16T10:26:36.970644+0000","last_became_active":"2024-09-16T10:27:38.937148+0000","last_became_peered":"2024-09-16T10:27:38.937148+0000","last_unstale":"2024-09-16T10:27:44.665728+0000","last_undegraded":"2024-09-16T10:27:38.911144+0000","last_fullsized":"2024-09-16T10:27:38.905116+0000","mapping_epoch":387,"log_start":"384'5700","ondisk_log_start":"384'5700","created":362,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4978,"log_dups_size":0,"ondisk_log_size":4978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:28:42.597553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181239808,"num_objects":2765,"num_object_clones":0,"num_object_copies":8295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2765,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":261792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2765}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":607,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.914537+0000","last_change":"2024-09-16T10:23:02.807413+0000","last_active":"2024-09-16T10:27:41.914537+0000","last_peered":"2024-09-16T10:27:41.914537+0000","last_clean":"2024-09-16T10:27:41.914537+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:27:41.914537+0000","last_undegraded":"2024-09-16T10:27:41.914537+0000","last_fullsized":"2024-09-16T10:27:41.914537+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:02.807363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:02.807363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:29.140116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":801,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923734+0000","last_change":"2024-09-16T10:27:12.407528+0000","last_active":"2024-09-16T10:27:41.923734+0000","last_peered":"2024-09-16T10:27:41.923734+0000","last_clean":"2024-09-16T10:27:41.923734+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:27:41.923734+0000","last_undegraded":"2024-09-16T10:27:41.923734+0000","last_fullsized":"2024-09-16T10:27:41.923734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:27:12.407441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.407441+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:41.269329+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":772,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.918935+0000","last_change":"2024-09-16T10:26:38.002212+0000","last_active":"2024-09-16T10:27:41.918935+0000","last_peered":"2024-09-16T10:27:41.918935+0000","last_clean":"2024-09-16T10:26:35.988090+0000","last_became_active":"2024-09-16T10:26:38.002212+0000","last_became_peered":"2024-09-16T10:26:38.002212+0000","last_unstale":"2024-09-16T10:27:41.918935+0000","last_undegraded":"2024-09-16T10:27:41.918935+0000","last_fullsized":"2024-09-16T10:26:37.994787+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":118,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915638+0000","last_change":"2024-09-16T10:26:28.553272+0000","last_active":"2024-09-16T10:27:41.915638+0000","last_peered":"2024-09-16T10:27:41.915638+0000","last_clean":"2024-09-16T10:27:41.915638+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:27:41.915638+0000","last_undegraded":"2024-09-16T10:27:41.915638+0000","last_fullsized":"2024-09-16T10:27:41.915638+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:15.308775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:15.308775+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":750,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915078+0000","last_change":"2024-09-16T10:24:15.859366+0000","last_active":"2024-09-16T10:27:41.915078+0000","last_peered":"2024-09-16T10:27:41.915078+0000","last_clean":"2024-09-16T10:27:41.915078+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:27:41.915078+0000","last_undegraded":"2024-09-16T10:27:41.915078+0000","last_fullsized":"2024-09-16T10:27:41.915078+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:15.859299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:15.859299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.117548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"415'10760","reported_seq":10887,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.435509+0000","last_change":"2024-09-16T10:26:28.565690+0000","last_active":"2024-09-16T10:27:43.435509+0000","last_peered":"2024-09-16T10:27:43.435509+0000","last_clean":"2024-09-16T10:27:43.435509+0000","last_became_active":"2024-09-16T10:26:28.555135+0000","last_became_peered":"2024-09-16T10:26:28.555135+0000","last_unstale":"2024-09-16T10:27:43.435509+0000","last_undegraded":"2024-09-16T10:27:43.435509+0000","last_fullsized":"2024-09-16T10:27:43.435509+0000","mapping_epoch":382,"log_start":"386'6075","ondisk_log_start":"386'6075","created":362,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372375552,"num_objects":5682,"num_object_clones":0,"num_object_copies":17046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":526144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":189,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923314+0000","last_change":"2024-09-16T10:27:18.552950+0000","last_active":"2024-09-16T10:27:41.923314+0000","last_peered":"2024-09-16T10:27:41.923314+0000","last_clean":"2024-09-16T10:27:41.923314+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:27:41.923314+0000","last_undegraded":"2024-09-16T10:27:41.923314+0000","last_fullsized":"2024-09-16T10:27:41.923314+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:27:17.482427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:27:17.482427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:42.753548+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":150,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922013+0000","last_change":"2024-09-16T10:26:44.445756+0000","last_active":"2024-09-16T10:27:41.922013+0000","last_peered":"2024-09-16T10:27:41.922013+0000","last_clean":"2024-09-16T10:27:41.922013+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:27:41.922013+0000","last_undegraded":"2024-09-16T10:27:41.922013+0000","last_fullsized":"2024-09-16T10:27:41.922013+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:26:44.445661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.445661+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.574050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":737,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.936710+0000","last_change":"2024-09-16T10:26:38.008838+0000","last_active":"2024-09-16T10:27:41.936710+0000","last_peered":"2024-09-16T10:27:41.936710+0000","last_clean":"2024-09-16T10:26:35.994682+0000","last_became_active":"2024-09-16T10:26:38.008838+0000","last_became_peered":"2024-09-16T10:26:38.008838+0000","last_unstale":"2024-09-16T10:27:41.936710+0000","last_undegraded":"2024-09-16T10:27:41.936710+0000","last_fullsized":"2024-09-16T10:26:37.995364+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.404383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.404383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":504,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.921970+0000","last_change":"2024-09-16T10:26:37.998714+0000","last_active":"2024-09-16T10:27:41.921970+0000","last_peered":"2024-09-16T10:27:41.921970+0000","last_clean":"2024-09-16T10:26:35.984906+0000","last_became_active":"2024-09-16T10:26:37.998714+0000","last_became_peered":"2024-09-16T10:26:37.998714+0000","last_unstale":"2024-09-16T10:27:41.921970+0000","last_undegraded":"2024-09-16T10:27:41.921970+0000","last_fullsized":"2024-09-16T10:26:37.992483+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.521586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.521586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.643455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":569,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.920599+0000","last_change":"2024-09-16T10:27:40.298188+0000","last_active":"2024-09-16T10:27:41.920599+0000","last_peered":"2024-09-16T10:27:41.920599+0000","last_clean":"2024-09-16T10:27:41.920599+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:27:41.920599+0000","last_undegraded":"2024-09-16T10:27:41.920599+0000","last_fullsized":"2024-09-16T10:27:41.920599+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:29:08.799570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"415'10835","reported_seq":10955,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.462975+0000","last_change":"2024-09-16T10:25:31.163385+0000","last_active":"2024-09-16T10:27:43.462975+0000","last_peered":"2024-09-16T10:27:43.462975+0000","last_clean":"2024-09-16T10:27:43.462975+0000","last_became_active":"2024-09-16T10:25:31.162906+0000","last_became_peered":"2024-09-16T10:25:31.162906+0000","last_unstale":"2024-09-16T10:27:43.462975+0000","last_undegraded":"2024-09-16T10:27:43.462975+0000","last_fullsized":"2024-09-16T10:27:43.462975+0000","mapping_epoch":362,"log_start":"388'6128","ondisk_log_start":"388'6128","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375324672,"num_objects":5727,"num_object_clones":0,"num_object_copies":17181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10835,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":654,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.924152+0000","last_change":"2024-09-16T10:26:38.000943+0000","last_active":"2024-09-16T10:27:41.924152+0000","last_peered":"2024-09-16T10:27:41.924152+0000","last_clean":"2024-09-16T10:26:35.994238+0000","last_became_active":"2024-09-16T10:26:38.000943+0000","last_became_peered":"2024-09-16T10:26:38.000943+0000","last_unstale":"2024-09-16T10:27:41.924152+0000","last_undegraded":"2024-09-16T10:27:41.924152+0000","last_fullsized":"2024-09-16T10:26:37.996963+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:28.343066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:28.343066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:48.503120+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":698,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915001+0000","last_change":"2024-09-16T10:26:28.552507+0000","last_active":"2024-09-16T10:27:41.915001+0000","last_peered":"2024-09-16T10:27:41.915001+0000","last_clean":"2024-09-16T10:27:41.915001+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:27:41.915001+0000","last_undegraded":"2024-09-16T10:27:41.915001+0000","last_fullsized":"2024-09-16T10:27:41.915001+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:26:11.381501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:11.381501+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":762,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:44.624946+0000","last_change":"2024-09-16T10:27:44.624946+0000","last_active":"2024-09-16T10:27:44.624946+0000","last_peered":"2024-09-16T10:27:44.624946+0000","last_clean":"2024-09-16T10:27:44.624946+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:27:44.624946+0000","last_undegraded":"2024-09-16T10:27:44.624946+0000","last_fullsized":"2024-09-16T10:27:44.624946+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:57.569715+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":942,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:42.336556+0000","last_change":"2024-09-16T10:27:24.281038+0000","last_active":"2024-09-16T10:27:42.336556+0000","last_peered":"2024-09-16T10:27:42.336556+0000","last_clean":"2024-09-16T10:27:42.336556+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:27:42.336556+0000","last_undegraded":"2024-09-16T10:27:42.336556+0000","last_fullsized":"2024-09-16T10:27:42.336556+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:27:24.280960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:24.280960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:36.038661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":605,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.922315+0000","last_change":"2024-09-16T10:26:38.006141+0000","last_active":"2024-09-16T10:27:41.922315+0000","last_peered":"2024-09-16T10:27:41.922315+0000","last_clean":"2024-09-16T10:26:35.985257+0000","last_became_active":"2024-09-16T10:26:38.006141+0000","last_became_peered":"2024-09-16T10:26:38.006141+0000","last_unstale":"2024-09-16T10:27:41.922315+0000","last_undegraded":"2024-09-16T10:27:41.922315+0000","last_fullsized":"2024-09-16T10:26:37.998986+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.051380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.051380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:43.479873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"415'11105","reported_seq":11238,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:44.713158+0000","last_change":"2024-09-16T10:26:38.031331+0000","last_active":"2024-09-16T10:27:44.713158+0000","last_peered":"2024-09-16T10:27:44.713158+0000","last_clean":"2024-09-16T10:26:36.986703+0000","last_became_active":"2024-09-16T10:26:38.015586+0000","last_became_peered":"2024-09-16T10:26:38.015586+0000","last_unstale":"2024-09-16T10:27:44.713158+0000","last_undegraded":"2024-09-16T10:26:38.003109+0000","last_fullsized":"2024-09-16T10:26:37.990926+0000","mapping_epoch":387,"log_start":"391'6658","ondisk_log_start":"391'6658","created":362,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367591424,"num_objects":5609,"num_object_clones":0,"num_object_copies":16827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11105,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":5609}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":704,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.923372+0000","last_change":"2024-09-16T10:26:38.011823+0000","last_active":"2024-09-16T10:27:41.923372+0000","last_peered":"2024-09-16T10:27:41.923372+0000","last_clean":"2024-09-16T10:26:35.985526+0000","last_became_active":"2024-09-16T10:26:38.011823+0000","last_became_peered":"2024-09-16T10:26:38.011823+0000","last_unstale":"2024-09-16T10:27:41.923372+0000","last_undegraded":"2024-09-16T10:27:41.923372+0000","last_fullsized":"2024-09-16T10:26:38.001181+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:31.617190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:31.617190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:46.435042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":796,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924453+0000","last_change":"2024-09-16T10:22:20.642298+0000","last_active":"2024-09-16T10:27:41.924453+0000","last_peered":"2024-09-16T10:27:41.924453+0000","last_clean":"2024-09-16T10:27:41.924453+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:27:41.924453+0000","last_undegraded":"2024-09-16T10:27:41.924453+0000","last_fullsized":"2024-09-16T10:27:41.924453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:22:20.642212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.642212+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:00.233082+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":776,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915506+0000","last_change":"2024-09-16T10:24:59.820337+0000","last_active":"2024-09-16T10:27:41.915506+0000","last_peered":"2024-09-16T10:27:41.915506+0000","last_clean":"2024-09-16T10:27:41.915506+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:27:41.915506+0000","last_undegraded":"2024-09-16T10:27:41.915506+0000","last_fullsized":"2024-09-16T10:27:41.915506+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:59.820273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:59.820273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:34.014566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":923,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.918223+0000","last_change":"2024-09-16T10:26:46.635924+0000","last_active":"2024-09-16T10:27:41.918223+0000","last_peered":"2024-09-16T10:27:41.918223+0000","last_clean":"2024-09-16T10:27:41.918223+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:27:41.918223+0000","last_undegraded":"2024-09-16T10:27:41.918223+0000","last_fullsized":"2024-09-16T10:27:41.918223+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:46.635855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.635855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:50.833434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":771,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.917909+0000","last_change":"2024-09-16T10:26:52.648107+0000","last_active":"2024-09-16T10:27:41.917909+0000","last_peered":"2024-09-16T10:27:41.917909+0000","last_clean":"2024-09-16T10:27:41.917909+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:27:41.917909+0000","last_undegraded":"2024-09-16T10:27:41.917909+0000","last_fullsized":"2024-09-16T10:27:41.917909+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:52.648041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:52.648041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.033578+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"415'10964","reported_seq":11172,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.480717+0000","last_change":"2024-09-16T10:27:12.593342+0000","last_active":"2024-09-16T10:27:43.480717+0000","last_peered":"2024-09-16T10:27:43.480717+0000","last_clean":"2024-09-16T10:27:43.480717+0000","last_became_active":"2024-09-16T10:25:30.620945+0000","last_became_peered":"2024-09-16T10:25:30.620945+0000","last_unstale":"2024-09-16T10:27:43.480717+0000","last_undegraded":"2024-09-16T10:27:43.480717+0000","last_fullsized":"2024-09-16T10:27:43.480717+0000","mapping_epoch":362,"log_start":"389'6333","ondisk_log_start":"389'6333","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8755","last_scrub_stamp":"2024-09-16T10:27:12.593235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.593235+0000","objects_scrubbed":7966,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:25.454669+0000","scrub_duration":2292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370999320,"num_objects":5662,"num_object_clones":0,"num_object_copies":16986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10964,"num_write_kb":531969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":713,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923183+0000","last_change":"2024-09-16T10:26:51.422615+0000","last_active":"2024-09-16T10:27:41.923183+0000","last_peered":"2024-09-16T10:27:41.923183+0000","last_clean":"2024-09-16T10:27:41.923183+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:27:41.923183+0000","last_undegraded":"2024-09-16T10:27:41.923183+0000","last_fullsized":"2024-09-16T10:27:41.923183+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:51.422550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:51.422550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.793681+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":812,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.918388+0000","last_change":"2024-09-16T10:26:38.007576+0000","last_active":"2024-09-16T10:27:41.918388+0000","last_peered":"2024-09-16T10:27:41.918388+0000","last_clean":"2024-09-16T10:26:35.990050+0000","last_became_active":"2024-09-16T10:26:38.007576+0000","last_became_peered":"2024-09-16T10:26:38.007576+0000","last_unstale":"2024-09-16T10:27:41.918388+0000","last_undegraded":"2024-09-16T10:26:37.992428+0000","last_fullsized":"2024-09-16T10:26:37.991190+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-16T10:21:27.125775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:27.125775+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.655534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":776,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924106+0000","last_change":"2024-09-16T10:22:05.591908+0000","last_active":"2024-09-16T10:27:41.924106+0000","last_peered":"2024-09-16T10:27:41.924106+0000","last_clean":"2024-09-16T10:27:41.924106+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:27:41.924106+0000","last_undegraded":"2024-09-16T10:27:41.924106+0000","last_fullsized":"2024-09-16T10:27:41.924106+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:05.591836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:05.591836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:04.276933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":769,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922220+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:27:41.922220+0000","last_peered":"2024-09-16T10:27:41.922220+0000","last_clean":"2024-09-16T10:27:41.922220+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:27:41.922220+0000","last_undegraded":"2024-09-16T10:27:41.922220+0000","last_fullsized":"2024-09-16T10:27:41.922220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":764,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922576+0000","last_change":"2024-09-16T10:27:41.587855+0000","last_active":"2024-09-16T10:27:41.922576+0000","last_peered":"2024-09-16T10:27:41.922576+0000","last_clean":"2024-09-16T10:27:41.922576+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:27:41.922576+0000","last_undegraded":"2024-09-16T10:27:41.922576+0000","last_fullsized":"2024-09-16T10:27:41.922576+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:55.373715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"415'10950","reported_seq":11068,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.478887+0000","last_change":"2024-09-16T10:25:30.622174+0000","last_active":"2024-09-16T10:27:43.478887+0000","last_peered":"2024-09-16T10:27:43.478887+0000","last_clean":"2024-09-16T10:27:43.478887+0000","last_became_active":"2024-09-16T10:25:30.621742+0000","last_became_peered":"2024-09-16T10:25:30.621742+0000","last_unstale":"2024-09-16T10:27:43.478887+0000","last_undegraded":"2024-09-16T10:27:43.478887+0000","last_fullsized":"2024-09-16T10:27:43.478887+0000","mapping_epoch":362,"log_start":"389'6333","ondisk_log_start":"389'6333","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376307712,"num_objects":5742,"num_object_clones":0,"num_object_copies":17226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":659,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.914705+0000","last_change":"2024-09-16T10:26:09.992070+0000","last_active":"2024-09-16T10:27:41.914705+0000","last_peered":"2024-09-16T10:27:41.914705+0000","last_clean":"2024-09-16T10:27:41.914705+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:27:41.914705+0000","last_undegraded":"2024-09-16T10:27:41.914705+0000","last_fullsized":"2024-09-16T10:27:41.914705+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.992000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.992000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":1076,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924005+0000","last_change":"2024-09-16T10:27:14.409646+0000","last_active":"2024-09-16T10:27:41.924005+0000","last_peered":"2024-09-16T10:27:41.924005+0000","last_clean":"2024-09-16T10:27:41.924005+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:27:41.924005+0000","last_undegraded":"2024-09-16T10:27:41.924005+0000","last_fullsized":"2024-09-16T10:27:41.924005+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:27:14.409552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:14.409552+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:38.459925+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":77,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.918680+0000","last_change":"2024-09-16T10:26:28.554041+0000","last_active":"2024-09-16T10:27:41.918680+0000","last_peered":"2024-09-16T10:27:41.918680+0000","last_clean":"2024-09-16T10:27:41.918680+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:27:41.918680+0000","last_undegraded":"2024-09-16T10:27:41.918680+0000","last_fullsized":"2024-09-16T10:27:41.918680+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:13.097133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:13.097133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":780,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915261+0000","last_change":"2024-09-16T10:27:23.145340+0000","last_active":"2024-09-16T10:27:41.915261+0000","last_peered":"2024-09-16T10:27:41.915261+0000","last_clean":"2024-09-16T10:27:41.915261+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:27:41.915261+0000","last_undegraded":"2024-09-16T10:27:41.915261+0000","last_fullsized":"2024-09-16T10:27:41.915261+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:27:23.145278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:27:23.145278+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:36.993455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":745,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.921556+0000","last_change":"2024-09-16T10:25:37.359150+0000","last_active":"2024-09-16T10:27:41.921556+0000","last_peered":"2024-09-16T10:27:41.921556+0000","last_clean":"2024-09-16T10:27:41.921556+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:27:41.921556+0000","last_undegraded":"2024-09-16T10:27:41.921556+0000","last_fullsized":"2024-09-16T10:27:41.921556+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:37.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:28.606593+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":768,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.921364+0000","last_change":"2024-09-16T10:26:38.012326+0000","last_active":"2024-09-16T10:27:41.921364+0000","last_peered":"2024-09-16T10:27:41.921364+0000","last_clean":"2024-09-16T10:26:35.989607+0000","last_became_active":"2024-09-16T10:26:38.012326+0000","last_became_peered":"2024-09-16T10:26:38.012326+0000","last_unstale":"2024-09-16T10:27:41.921364+0000","last_undegraded":"2024-09-16T10:27:41.921364+0000","last_fullsized":"2024-09-16T10:26:38.005246+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:26.349415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:26.349415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"415'10790","reported_seq":10919,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:43.581044+0000","last_change":"2024-09-16T10:27:38.928526+0000","last_active":"2024-09-16T10:27:43.581044+0000","last_peered":"2024-09-16T10:27:43.581044+0000","last_clean":"2024-09-16T10:26:36.981417+0000","last_became_active":"2024-09-16T10:27:38.928526+0000","last_became_peered":"2024-09-16T10:27:38.928526+0000","last_unstale":"2024-09-16T10:27:43.581044+0000","last_undegraded":"2024-09-16T10:27:38.906714+0000","last_fullsized":"2024-09-16T10:27:38.902056+0000","mapping_epoch":387,"log_start":"386'6019","ondisk_log_start":"386'6019","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:28:40.033449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183533568,"num_objects":2800,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5451,"num_write_kb":264064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":32768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":2800}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":228,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.919036+0000","last_change":"2024-09-16T10:26:38.007174+0000","last_active":"2024-09-16T10:27:41.919036+0000","last_peered":"2024-09-16T10:27:41.919036+0000","last_clean":"2024-09-16T10:26:35.990809+0000","last_became_active":"2024-09-16T10:26:38.007174+0000","last_became_peered":"2024-09-16T10:26:38.007174+0000","last_unstale":"2024-09-16T10:27:41.919036+0000","last_undegraded":"2024-09-16T10:27:41.919036+0000","last_fullsized":"2024-09-16T10:26:37.993534+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:17.574207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":775,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923850+0000","last_change":"2024-09-16T10:25:13.649143+0000","last_active":"2024-09-16T10:27:41.923850+0000","last_peered":"2024-09-16T10:27:41.923850+0000","last_clean":"2024-09-16T10:27:41.923850+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:27:41.923850+0000","last_undegraded":"2024-09-16T10:27:41.923850+0000","last_fullsized":"2024-09-16T10:27:41.923850+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:25:13.648981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.648981+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.327665+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":761,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.921951+0000","last_change":"2024-09-16T10:22:02.897835+0000","last_active":"2024-09-16T10:27:41.921951+0000","last_peered":"2024-09-16T10:27:41.921951+0000","last_clean":"2024-09-16T10:27:41.921951+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:27:41.921951+0000","last_undegraded":"2024-09-16T10:27:41.921951+0000","last_fullsized":"2024-09-16T10:27:41.921951+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:55.257221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:20:55.257221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:37.572756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":300,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.961067+0000","last_change":"2024-09-16T10:26:18.549126+0000","last_active":"2024-09-16T10:27:41.961067+0000","last_peered":"2024-09-16T10:27:41.961067+0000","last_clean":"2024-09-16T10:27:41.961067+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:27:41.961067+0000","last_undegraded":"2024-09-16T10:27:41.961067+0000","last_fullsized":"2024-09-16T10:27:41.961067+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:26:18.549005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:26:18.549005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":292,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.919945+0000","last_change":"2024-09-16T10:26:38.004937+0000","last_active":"2024-09-16T10:27:41.919945+0000","last_peered":"2024-09-16T10:27:41.919945+0000","last_clean":"2024-09-16T10:26:35.983335+0000","last_became_active":"2024-09-16T10:26:38.004937+0000","last_became_peered":"2024-09-16T10:26:38.004937+0000","last_unstale":"2024-09-16T10:27:41.919945+0000","last_undegraded":"2024-09-16T10:27:41.919945+0000","last_fullsized":"2024-09-16T10:26:37.992855+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:29.905707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:29.905707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:44.224879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":153,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.922999+0000","last_change":"2024-09-16T10:26:38.011765+0000","last_active":"2024-09-16T10:27:41.922999+0000","last_peered":"2024-09-16T10:27:41.922999+0000","last_clean":"2024-09-16T10:26:35.985415+0000","last_became_active":"2024-09-16T10:26:38.011765+0000","last_became_peered":"2024-09-16T10:26:38.011765+0000","last_unstale":"2024-09-16T10:27:41.922999+0000","last_undegraded":"2024-09-16T10:27:41.922999+0000","last_fullsized":"2024-09-16T10:26:38.000155+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"415'10601","reported_seq":10729,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:44.698685+0000","last_change":"2024-09-16T10:27:38.916637+0000","last_active":"2024-09-16T10:27:44.698685+0000","last_peered":"2024-09-16T10:27:44.698685+0000","last_clean":"2024-09-16T10:26:36.966822+0000","last_became_active":"2024-09-16T10:27:38.916637+0000","last_became_peered":"2024-09-16T10:27:38.916637+0000","last_unstale":"2024-09-16T10:27:44.698685+0000","last_undegraded":"2024-09-16T10:27:38.905094+0000","last_fullsized":"2024-09-16T10:27:38.904975+0000","mapping_epoch":387,"log_start":"384'5600","ondisk_log_start":"384'5600","created":362,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:28:41.561673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183566336,"num_objects":2801,"num_object_clones":0,"num_object_copies":8403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":261664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2801}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":647,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923638+0000","last_change":"2024-09-16T10:26:09.201694+0000","last_active":"2024-09-16T10:27:41.923638+0000","last_peered":"2024-09-16T10:27:41.923638+0000","last_clean":"2024-09-16T10:27:41.923638+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:27:41.923638+0000","last_undegraded":"2024-09-16T10:27:41.923638+0000","last_fullsized":"2024-09-16T10:27:41.923638+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.201559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.201559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":859,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:42.007115+0000","last_change":"2024-09-16T10:27:09.622319+0000","last_active":"2024-09-16T10:27:42.007115+0000","last_peered":"2024-09-16T10:27:42.007115+0000","last_clean":"2024-09-16T10:27:42.007115+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:27:42.007115+0000","last_undegraded":"2024-09-16T10:27:42.007115+0000","last_fullsized":"2024-09-16T10:27:42.007115+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:27:09.622239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.622239+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:28.341911+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":785,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.946139+0000","last_change":"2024-09-16T10:27:10.642822+0000","last_active":"2024-09-16T10:27:41.946139+0000","last_peered":"2024-09-16T10:27:41.946139+0000","last_clean":"2024-09-16T10:27:41.946139+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:27:41.946139+0000","last_undegraded":"2024-09-16T10:27:41.946139+0000","last_fullsized":"2024-09-16T10:27:41.946139+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:10.642739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:10.642739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:12.787144+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":349,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:42.149490+0000","last_change":"2024-09-16T10:26:37.994028+0000","last_active":"2024-09-16T10:27:42.149490+0000","last_peered":"2024-09-16T10:27:42.149490+0000","last_clean":"2024-09-16T10:26:36.191294+0000","last_became_active":"2024-09-16T10:26:37.994028+0000","last_became_peered":"2024-09-16T10:26:37.994028+0000","last_unstale":"2024-09-16T10:27:42.149490+0000","last_undegraded":"2024-09-16T10:26:37.988066+0000","last_fullsized":"2024-09-16T10:26:37.987957+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:23:41.597682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.597682+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":168,"num_read_kb":163,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":770,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915838+0000","last_change":"2024-09-16T10:27:20.130780+0000","last_active":"2024-09-16T10:27:41.915838+0000","last_peered":"2024-09-16T10:27:41.915838+0000","last_clean":"2024-09-16T10:27:41.915838+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:27:41.915838+0000","last_undegraded":"2024-09-16T10:27:41.915838+0000","last_fullsized":"2024-09-16T10:27:41.915838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:20.130715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.130715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:33.131596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":774,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923642+0000","last_change":"2024-09-16T10:27:19.374485+0000","last_active":"2024-09-16T10:27:41.923642+0000","last_peered":"2024-09-16T10:27:41.923642+0000","last_clean":"2024-09-16T10:27:41.923642+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:27:41.923642+0000","last_undegraded":"2024-09-16T10:27:41.923642+0000","last_fullsized":"2024-09-16T10:27:41.923642+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:42.304216+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"415'10808","reported_seq":10928,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.476725+0000","last_change":"2024-09-16T10:25:30.622105+0000","last_active":"2024-09-16T10:27:43.476725+0000","last_peered":"2024-09-16T10:27:43.476725+0000","last_clean":"2024-09-16T10:27:43.476725+0000","last_became_active":"2024-09-16T10:25:30.621779+0000","last_became_peered":"2024-09-16T10:25:30.621779+0000","last_unstale":"2024-09-16T10:27:43.476725+0000","last_undegraded":"2024-09-16T10:27:43.476725+0000","last_fullsized":"2024-09-16T10:27:43.476725+0000","mapping_epoch":362,"log_start":"388'6181","ondisk_log_start":"388'6181","created":362,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368967680,"num_objects":5630,"num_object_clones":0,"num_object_copies":16890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"388'29","reported_seq":955,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.923213+0000","last_change":"2024-09-16T10:26:38.004519+0000","last_active":"2024-09-16T10:27:41.923213+0000","last_peered":"2024-09-16T10:27:41.923213+0000","last_clean":"2024-09-16T10:26:35.985510+0000","last_became_active":"2024-09-16T10:26:38.004519+0000","last_became_peered":"2024-09-16T10:26:38.004519+0000","last_unstale":"2024-09-16T10:27:41.923213+0000","last_undegraded":"2024-09-16T10:26:37.998195+0000","last_fullsized":"2024-09-16T10:26:37.998079+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'16","last_scrub_stamp":"2024-09-16T10:20:31.611732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:31.611732+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":733,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.920861+0000","last_change":"2024-09-16T10:26:47.493915+0000","last_active":"2024-09-16T10:27:41.920861+0000","last_peered":"2024-09-16T10:27:41.920861+0000","last_clean":"2024-09-16T10:27:41.920861+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:27:41.920861+0000","last_undegraded":"2024-09-16T10:27:41.920861+0000","last_fullsized":"2024-09-16T10:27:41.920861+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:47.493848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:47.493848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:15.957630+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":228,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.918025+0000","last_change":"2024-09-16T10:26:37.999195+0000","last_active":"2024-09-16T10:27:41.918025+0000","last_peered":"2024-09-16T10:27:41.918025+0000","last_clean":"2024-09-16T10:26:35.992856+0000","last_became_active":"2024-09-16T10:26:37.999195+0000","last_became_peered":"2024-09-16T10:26:37.999195+0000","last_unstale":"2024-09-16T10:27:41.918025+0000","last_undegraded":"2024-09-16T10:27:41.918025+0000","last_fullsized":"2024-09-16T10:26:37.991079+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:14.650988+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":625,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.915882+0000","last_change":"2024-09-16T10:26:38.002654+0000","last_active":"2024-09-16T10:27:41.915882+0000","last_peered":"2024-09-16T10:27:41.915882+0000","last_clean":"2024-09-16T10:26:35.982477+0000","last_became_active":"2024-09-16T10:26:38.002654+0000","last_became_peered":"2024-09-16T10:26:38.002654+0000","last_unstale":"2024-09-16T10:27:41.915882+0000","last_undegraded":"2024-09-16T10:27:41.915882+0000","last_fullsized":"2024-09-16T10:26:37.992362+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.591323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:11.293510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":758,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915734+0000","last_change":"2024-09-16T10:26:19.084467+0000","last_active":"2024-09-16T10:27:41.915734+0000","last_peered":"2024-09-16T10:27:41.915734+0000","last_clean":"2024-09-16T10:27:41.915734+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:27:41.915734+0000","last_undegraded":"2024-09-16T10:27:41.915734+0000","last_fullsized":"2024-09-16T10:27:41.915734+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:19.084409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:19.084409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:28.111371+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"415'10782","reported_seq":11032,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.479619+0000","last_change":"2024-09-16T10:27:40.858675+0000","last_active":"2024-09-16T10:27:43.479619+0000","last_peered":"2024-09-16T10:27:43.479619+0000","last_clean":"2024-09-16T10:27:43.479619+0000","last_became_active":"2024-09-16T10:27:38.936786+0000","last_became_peered":"2024-09-16T10:27:38.936786+0000","last_unstale":"2024-09-16T10:27:43.479619+0000","last_undegraded":"2024-09-16T10:27:43.479619+0000","last_fullsized":"2024-09-16T10:27:43.479619+0000","mapping_epoch":362,"log_start":"384'5700","ondisk_log_start":"384'5700","created":362,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'10731","last_scrub_stamp":"2024-09-16T10:27:40.858589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.858589+0000","objects_scrubbed":2857,"log_size":5082,"log_dups_size":0,"ondisk_log_size":5082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:59.832936+0000","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":8418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":647,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923418+0000","last_change":"2024-09-16T10:23:01.705304+0000","last_active":"2024-09-16T10:27:41.923418+0000","last_peered":"2024-09-16T10:27:41.923418+0000","last_clean":"2024-09-16T10:27:41.923418+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:27:41.923418+0000","last_undegraded":"2024-09-16T10:27:41.923418+0000","last_fullsized":"2024-09-16T10:27:41.923418+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:23:01.705081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:23:01.705081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:06.340246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":698,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923286+0000","last_change":"2024-09-16T10:26:28.556079+0000","last_active":"2024-09-16T10:27:41.923286+0000","last_peered":"2024-09-16T10:27:41.923286+0000","last_clean":"2024-09-16T10:27:41.923286+0000","last_became_active":"2024-09-16T10:26:28.554604+0000","last_became_peered":"2024-09-16T10:26:28.554604+0000","last_unstale":"2024-09-16T10:27:41.923286+0000","last_undegraded":"2024-09-16T10:27:41.923286+0000","last_fullsized":"2024-09-16T10:27:41.923286+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:26:10.386128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.386128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":561,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.960641+0000","last_change":"2024-09-16T10:26:37.998773+0000","last_active":"2024-09-16T10:27:41.960641+0000","last_peered":"2024-09-16T10:27:41.960641+0000","last_clean":"2024-09-16T10:26:35.986576+0000","last_became_active":"2024-09-16T10:26:37.998773+0000","last_became_peered":"2024-09-16T10:26:37.998773+0000","last_unstale":"2024-09-16T10:27:41.960641+0000","last_undegraded":"2024-09-16T10:27:41.960641+0000","last_fullsized":"2024-09-16T10:26:37.991873+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:20.877398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:20.877398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.191187+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":762,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915671+0000","last_change":"2024-09-16T10:25:05.858177+0000","last_active":"2024-09-16T10:27:41.915671+0000","last_peered":"2024-09-16T10:27:41.915671+0000","last_clean":"2024-09-16T10:27:41.915671+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:27:41.915671+0000","last_undegraded":"2024-09-16T10:27:41.915671+0000","last_fullsized":"2024-09-16T10:27:41.915671+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:05.858129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:05.858129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:38.052701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":697,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924059+0000","last_change":"2024-09-16T10:19:57.175078+0000","last_active":"2024-09-16T10:27:41.924059+0000","last_peered":"2024-09-16T10:27:41.924059+0000","last_clean":"2024-09-16T10:27:41.924059+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:27:41.924059+0000","last_undegraded":"2024-09-16T10:27:41.924059+0000","last_fullsized":"2024-09-16T10:27:41.924059+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:57.175013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:19:57.175013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.375991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"415'10905","reported_seq":11068,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.970582+0000","last_change":"2024-09-16T10:27:41.815045+0000","last_active":"2024-09-16T10:27:43.970582+0000","last_peered":"2024-09-16T10:27:43.970582+0000","last_clean":"2024-09-16T10:27:43.970582+0000","last_became_active":"2024-09-16T10:27:38.943280+0000","last_became_peered":"2024-09-16T10:27:38.943280+0000","last_unstale":"2024-09-16T10:27:43.970582+0000","last_undegraded":"2024-09-16T10:27:43.970582+0000","last_fullsized":"2024-09-16T10:27:43.970582+0000","mapping_epoch":362,"log_start":"376'4600","ondisk_log_start":"376'4600","created":362,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'10835","last_scrub_stamp":"2024-09-16T10:27:41.814978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.814978+0000","objects_scrubbed":2841,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:59.027310+0000","scrub_duration":312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181600256,"num_objects":2771,"num_object_clones":0,"num_object_copies":8313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":622,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.914416+0000","last_change":"2024-09-16T10:26:49.093325+0000","last_active":"2024-09-16T10:27:41.914416+0000","last_peered":"2024-09-16T10:27:41.914416+0000","last_clean":"2024-09-16T10:27:41.914416+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:27:41.914416+0000","last_undegraded":"2024-09-16T10:27:41.914416+0000","last_fullsized":"2024-09-16T10:27:41.914416+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:49.093245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:49.093245+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:09.427767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":763,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.916355+0000","last_change":"2024-09-16T10:27:21.459062+0000","last_active":"2024-09-16T10:27:41.916355+0000","last_peered":"2024-09-16T10:27:41.916355+0000","last_clean":"2024-09-16T10:27:41.916355+0000","last_became_active":"2024-09-16T10:26:28.553171+0000","last_became_peered":"2024-09-16T10:26:28.553171+0000","last_unstale":"2024-09-16T10:27:41.916355+0000","last_undegraded":"2024-09-16T10:27:41.916355+0000","last_fullsized":"2024-09-16T10:27:41.916355+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:27:21.458969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:21.458969+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:21.673624+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":761,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922933+0000","last_change":"2024-09-16T10:26:42.490866+0000","last_active":"2024-09-16T10:27:41.922933+0000","last_peered":"2024-09-16T10:27:41.922933+0000","last_clean":"2024-09-16T10:27:41.922933+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:27:41.922933+0000","last_undegraded":"2024-09-16T10:27:41.922933+0000","last_fullsized":"2024-09-16T10:27:41.922933+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:42.490731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.490731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:51.400478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":746,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923748+0000","last_change":"2024-09-16T10:21:57.568725+0000","last_active":"2024-09-16T10:27:41.923748+0000","last_peered":"2024-09-16T10:27:41.923748+0000","last_clean":"2024-09-16T10:27:41.923748+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:27:41.923748+0000","last_undegraded":"2024-09-16T10:27:41.923748+0000","last_fullsized":"2024-09-16T10:27:41.923748+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:57.568649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:21:57.568649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":296,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.938023+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:27:41.938023+0000","last_peered":"2024-09-16T10:27:41.938023+0000","last_clean":"2024-09-16T10:27:41.938023+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:27:41.938023+0000","last_undegraded":"2024-09-16T10:27:41.938023+0000","last_fullsized":"2024-09-16T10:27:41.938023+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:06.309247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"415'10818","reported_seq":11074,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.964656+0000","last_change":"2024-09-16T10:27:41.013049+0000","last_active":"2024-09-16T10:27:43.964656+0000","last_peered":"2024-09-16T10:27:43.964656+0000","last_clean":"2024-09-16T10:27:43.964656+0000","last_became_active":"2024-09-16T10:27:38.932195+0000","last_became_peered":"2024-09-16T10:27:38.932195+0000","last_unstale":"2024-09-16T10:27:43.964656+0000","last_undegraded":"2024-09-16T10:27:43.964656+0000","last_fullsized":"2024-09-16T10:27:43.964656+0000","mapping_epoch":362,"log_start":"384'5800","ondisk_log_start":"384'5800","created":362,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'10735","last_scrub_stamp":"2024-09-16T10:27:41.012948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.012948+0000","objects_scrubbed":2939,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:59.821138+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187170816,"num_objects":2856,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":428,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.920972+0000","last_change":"2024-09-16T10:26:38.005912+0000","last_active":"2024-09-16T10:27:41.920972+0000","last_peered":"2024-09-16T10:27:41.920972+0000","last_clean":"2024-09-16T10:26:35.988312+0000","last_became_active":"2024-09-16T10:26:38.005912+0000","last_became_peered":"2024-09-16T10:26:38.005912+0000","last_unstale":"2024-09-16T10:27:41.920972+0000","last_undegraded":"2024-09-16T10:27:41.920972+0000","last_fullsized":"2024-09-16T10:26:37.994305+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:16.433199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:16.433199+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":670,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922840+0000","last_change":"2024-09-16T10:26:28.553833+0000","last_active":"2024-09-16T10:27:41.922840+0000","last_peered":"2024-09-16T10:27:41.922840+0000","last_clean":"2024-09-16T10:27:41.922840+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:27:41.922840+0000","last_undegraded":"2024-09-16T10:27:41.922840+0000","last_fullsized":"2024-09-16T10:27:41.922840+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-16T10:21:31.888762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:21:31.888762+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:36.187160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":752,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922173+0000","last_change":"2024-09-16T10:27:12.155297+0000","last_active":"2024-09-16T10:27:41.922173+0000","last_peered":"2024-09-16T10:27:41.922173+0000","last_clean":"2024-09-16T10:27:41.922173+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:27:41.922173+0000","last_undegraded":"2024-09-16T10:27:41.922173+0000","last_fullsized":"2024-09-16T10:27:41.922173+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:12.155186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.155186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:19.522860+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'11","reported_seq":365,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.920275+0000","last_change":"2024-09-16T10:26:38.003009+0000","last_active":"2024-09-16T10:27:41.920275+0000","last_peered":"2024-09-16T10:27:41.920275+0000","last_clean":"2024-09-16T10:26:35.991853+0000","last_became_active":"2024-09-16T10:26:38.003009+0000","last_became_peered":"2024-09-16T10:26:38.003009+0000","last_unstale":"2024-09-16T10:27:41.920275+0000","last_undegraded":"2024-09-16T10:26:37.992209+0000","last_fullsized":"2024-09-16T10:26:37.992083+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:22:24.799819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:22:24.799819+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":561,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923395+0000","last_change":"2024-09-16T10:27:13.405307+0000","last_active":"2024-09-16T10:27:41.923395+0000","last_peered":"2024-09-16T10:27:41.923395+0000","last_clean":"2024-09-16T10:27:41.923395+0000","last_became_active":"2024-09-16T10:26:28.550959+0000","last_became_peered":"2024-09-16T10:26:28.550959+0000","last_unstale":"2024-09-16T10:27:41.923395+0000","last_undegraded":"2024-09-16T10:27:41.923395+0000","last_fullsized":"2024-09-16T10:27:41.923395+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.405239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.405239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:30.309220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"415'10542","reported_seq":10792,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.985226+0000","last_change":"2024-09-16T10:27:40.855762+0000","last_active":"2024-09-16T10:27:43.985226+0000","last_peered":"2024-09-16T10:27:43.985226+0000","last_clean":"2024-09-16T10:27:43.985226+0000","last_became_active":"2024-09-16T10:27:38.938068+0000","last_became_peered":"2024-09-16T10:27:38.938068+0000","last_unstale":"2024-09-16T10:27:43.985226+0000","last_undegraded":"2024-09-16T10:27:43.985226+0000","last_fullsized":"2024-09-16T10:27:43.985226+0000","mapping_epoch":362,"log_start":"376'4200","ondisk_log_start":"376'4200","created":362,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'10446","last_scrub_stamp":"2024-09-16T10:27:40.855563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.855563+0000","objects_scrubbed":2891,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:52.992566+0000","scrub_duration":392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183173120,"num_objects":2795,"num_object_clones":0,"num_object_copies":8385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":141,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.921132+0000","last_change":"2024-09-16T10:26:38.011429+0000","last_active":"2024-09-16T10:27:41.921132+0000","last_peered":"2024-09-16T10:27:41.921132+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:26:38.011429+0000","last_became_peered":"2024-09-16T10:26:38.011429+0000","last_unstale":"2024-09-16T10:27:41.921132+0000","last_undegraded":"2024-09-16T10:27:41.921132+0000","last_fullsized":"2024-09-16T10:26:37.996799+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:07.404565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:07.404565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:28:39.305357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":782,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.918284+0000","last_change":"2024-09-16T10:27:20.653839+0000","last_active":"2024-09-16T10:27:41.918284+0000","last_peered":"2024-09-16T10:27:41.918284+0000","last_clean":"2024-09-16T10:27:41.918284+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:27:41.918284+0000","last_undegraded":"2024-09-16T10:27:41.918284+0000","last_fullsized":"2024-09-16T10:27:41.918284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:27:20.653762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.653762+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:27.487791+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":778,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915508+0000","last_change":"2024-09-16T10:26:44.070285+0000","last_active":"2024-09-16T10:27:41.915508+0000","last_peered":"2024-09-16T10:27:41.915508+0000","last_clean":"2024-09-16T10:27:41.915508+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:27:41.915508+0000","last_undegraded":"2024-09-16T10:27:41.915508+0000","last_fullsized":"2024-09-16T10:27:41.915508+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:44.070218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.070218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:03.598523+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":296,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.916015+0000","last_change":"2024-09-16T10:26:22.501028+0000","last_active":"2024-09-16T10:27:41.916015+0000","last_peered":"2024-09-16T10:27:41.916015+0000","last_clean":"2024-09-16T10:27:41.916015+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:27:41.916015+0000","last_undegraded":"2024-09-16T10:27:41.916015+0000","last_fullsized":"2024-09-16T10:27:41.916015+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.500963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.500963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":790,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.944427+0000","last_change":"2024-09-16T10:26:37.998943+0000","last_active":"2024-09-16T10:27:41.944427+0000","last_peered":"2024-09-16T10:27:41.944427+0000","last_clean":"2024-09-16T10:26:35.991432+0000","last_became_active":"2024-09-16T10:26:37.998943+0000","last_became_peered":"2024-09-16T10:26:37.998943+0000","last_unstale":"2024-09-16T10:27:41.944427+0000","last_undegraded":"2024-09-16T10:26:37.990081+0000","last_fullsized":"2024-09-16T10:26:37.989850+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:26:20.555476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.555476+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"415'10952","reported_seq":11173,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:44.666781+0000","last_change":"2024-09-16T10:27:40.984418+0000","last_active":"2024-09-16T10:27:44.666781+0000","last_peered":"2024-09-16T10:27:44.666781+0000","last_clean":"2024-09-16T10:27:44.666781+0000","last_became_active":"2024-09-16T10:27:38.930896+0000","last_became_peered":"2024-09-16T10:27:38.930896+0000","last_unstale":"2024-09-16T10:27:44.666781+0000","last_undegraded":"2024-09-16T10:27:44.666781+0000","last_fullsized":"2024-09-16T10:27:44.666781+0000","mapping_epoch":362,"log_start":"385'6000","ondisk_log_start":"385'6000","created":362,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'10846","last_scrub_stamp":"2024-09-16T10:27:40.984314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.984314+0000","objects_scrubbed":2945,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:29:10.176423+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186056704,"num_objects":2839,"num_object_clones":0,"num_object_copies":8517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":726,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923598+0000","last_change":"2024-09-16T10:25:40.594525+0000","last_active":"2024-09-16T10:27:41.923598+0000","last_peered":"2024-09-16T10:27:41.923598+0000","last_clean":"2024-09-16T10:27:41.923598+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:27:41.923598+0000","last_undegraded":"2024-09-16T10:27:41.923598+0000","last_fullsized":"2024-09-16T10:27:41.923598+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:40.594454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.594454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":832,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.922235+0000","last_change":"2024-09-16T10:26:37.999534+0000","last_active":"2024-09-16T10:27:41.922235+0000","last_peered":"2024-09-16T10:27:41.922235+0000","last_clean":"2024-09-16T10:26:35.985800+0000","last_became_active":"2024-09-16T10:26:37.999534+0000","last_became_peered":"2024-09-16T10:26:37.999534+0000","last_unstale":"2024-09-16T10:27:41.922235+0000","last_undegraded":"2024-09-16T10:26:37.994151+0000","last_fullsized":"2024-09-16T10:26:37.994038+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:30.481556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":152,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.915219+0000","last_change":"2024-09-16T10:26:38.002861+0000","last_active":"2024-09-16T10:27:41.915219+0000","last_peered":"2024-09-16T10:27:41.915219+0000","last_clean":"2024-09-16T10:26:35.979755+0000","last_became_active":"2024-09-16T10:26:38.002861+0000","last_became_peered":"2024-09-16T10:26:38.002861+0000","last_unstale":"2024-09-16T10:27:41.915219+0000","last_undegraded":"2024-09-16T10:27:41.915219+0000","last_fullsized":"2024-09-16T10:26:37.992825+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.031940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.031940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":296,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.978325+0000","last_change":"2024-09-16T10:26:43.199824+0000","last_active":"2024-09-16T10:27:41.978325+0000","last_peered":"2024-09-16T10:27:41.978325+0000","last_clean":"2024-09-16T10:27:41.978325+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:27:41.978325+0000","last_undegraded":"2024-09-16T10:27:41.978325+0000","last_fullsized":"2024-09-16T10:27:41.978325+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:43.199752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:43.199752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.646490+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":629,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.915916+0000","last_change":"2024-09-16T10:26:37.997577+0000","last_active":"2024-09-16T10:27:41.915916+0000","last_peered":"2024-09-16T10:27:41.915916+0000","last_clean":"2024-09-16T10:26:35.985465+0000","last_became_active":"2024-09-16T10:26:37.997577+0000","last_became_peered":"2024-09-16T10:26:37.997577+0000","last_unstale":"2024-09-16T10:27:41.915916+0000","last_undegraded":"2024-09-16T10:27:41.915916+0000","last_fullsized":"2024-09-16T10:26:37.992049+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:34.550928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:34.550928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:12.344839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"415'10968","reported_seq":11134,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.478837+0000","last_change":"2024-09-16T10:27:41.811335+0000","last_active":"2024-09-16T10:27:43.478837+0000","last_peered":"2024-09-16T10:27:43.478837+0000","last_clean":"2024-09-16T10:27:43.478837+0000","last_became_active":"2024-09-16T10:27:38.933847+0000","last_became_peered":"2024-09-16T10:27:38.933847+0000","last_unstale":"2024-09-16T10:27:43.478837+0000","last_undegraded":"2024-09-16T10:27:43.478837+0000","last_fullsized":"2024-09-16T10:27:43.478837+0000","mapping_epoch":362,"log_start":"384'5900","ondisk_log_start":"384'5900","created":362,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'10910","last_scrub_stamp":"2024-09-16T10:27:41.811199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.811199+0000","objects_scrubbed":2967,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:29:07.105493+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190644224,"num_objects":2909,"num_object_clones":0,"num_object_copies":8727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":677,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.921185+0000","last_change":"2024-09-16T10:26:10.271093+0000","last_active":"2024-09-16T10:27:41.921185+0000","last_peered":"2024-09-16T10:27:41.921185+0000","last_clean":"2024-09-16T10:27:41.921185+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:27:41.921185+0000","last_undegraded":"2024-09-16T10:27:41.921185+0000","last_fullsized":"2024-09-16T10:27:41.921185+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:10.270833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.270833+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":686,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922833+0000","last_change":"2024-09-16T10:26:42.132944+0000","last_active":"2024-09-16T10:27:41.922833+0000","last_peered":"2024-09-16T10:27:41.922833+0000","last_clean":"2024-09-16T10:27:41.922833+0000","last_became_active":"2024-09-16T10:26:28.553750+0000","last_became_peered":"2024-09-16T10:26:28.553750+0000","last_unstale":"2024-09-16T10:27:41.922833+0000","last_undegraded":"2024-09-16T10:27:41.922833+0000","last_fullsized":"2024-09-16T10:27:41.922833+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:26:42.132853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.132853+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:59.756132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":108,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.921327+0000","last_change":"2024-09-16T10:27:13.285419+0000","last_active":"2024-09-16T10:27:41.921327+0000","last_peered":"2024-09-16T10:27:41.921327+0000","last_clean":"2024-09-16T10:27:41.921327+0000","last_became_active":"2024-09-16T10:25:41.990483+0000","last_became_peered":"2024-09-16T10:25:41.990483+0000","last_unstale":"2024-09-16T10:27:41.921327+0000","last_undegraded":"2024-09-16T10:27:41.921327+0000","last_fullsized":"2024-09-16T10:27:41.921327+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:30.443308+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":296,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.916252+0000","last_change":"2024-09-16T10:26:21.479904+0000","last_active":"2024-09-16T10:27:41.916252+0000","last_peered":"2024-09-16T10:27:41.916252+0000","last_clean":"2024-09-16T10:27:41.916252+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:27:41.916252+0000","last_undegraded":"2024-09-16T10:27:41.916252+0000","last_fullsized":"2024-09-16T10:27:41.916252+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:37.453644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":760,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.920816+0000","last_change":"2024-09-16T10:26:37.999997+0000","last_active":"2024-09-16T10:27:41.920816+0000","last_peered":"2024-09-16T10:27:41.920816+0000","last_clean":"2024-09-16T10:26:35.993796+0000","last_became_active":"2024-09-16T10:26:37.999997+0000","last_became_peered":"2024-09-16T10:26:37.999997+0000","last_unstale":"2024-09-16T10:27:41.920816+0000","last_undegraded":"2024-09-16T10:27:41.920816+0000","last_fullsized":"2024-09-16T10:26:37.994297+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:41.478267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:41.478267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"415'10840","reported_seq":11088,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:43.613503+0000","last_change":"2024-09-16T10:27:41.321312+0000","last_active":"2024-09-16T10:27:43.613503+0000","last_peered":"2024-09-16T10:27:43.613503+0000","last_clean":"2024-09-16T10:27:43.613503+0000","last_became_active":"2024-09-16T10:27:38.925709+0000","last_became_peered":"2024-09-16T10:27:38.925709+0000","last_unstale":"2024-09-16T10:27:43.613503+0000","last_undegraded":"2024-09-16T10:27:43.613503+0000","last_fullsized":"2024-09-16T10:27:43.613503+0000","mapping_epoch":362,"log_start":"385'6109","ondisk_log_start":"385'6109","created":362,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'10774","last_scrub_stamp":"2024-09-16T10:27:41.321240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.321240+0000","objects_scrubbed":2901,"log_size":4731,"log_dups_size":0,"ondisk_log_size":4731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:46.702422+0000","scrub_duration":292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185794560,"num_objects":2835,"num_object_clones":0,"num_object_copies":8505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":465,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.923204+0000","last_change":"2024-09-16T10:26:38.003592+0000","last_active":"2024-09-16T10:27:41.923204+0000","last_peered":"2024-09-16T10:27:41.923204+0000","last_clean":"2024-09-16T10:26:35.991480+0000","last_became_active":"2024-09-16T10:26:38.003592+0000","last_became_peered":"2024-09-16T10:26:38.003592+0000","last_unstale":"2024-09-16T10:27:41.923204+0000","last_undegraded":"2024-09-16T10:27:41.923204+0000","last_fullsized":"2024-09-16T10:26:37.987896+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.367979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.367979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:28:38.625145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":797,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.914630+0000","last_change":"2024-09-16T10:26:38.003098+0000","last_active":"2024-09-16T10:27:41.914630+0000","last_peered":"2024-09-16T10:27:41.914630+0000","last_clean":"2024-09-16T10:26:35.979403+0000","last_became_active":"2024-09-16T10:26:38.003098+0000","last_became_peered":"2024-09-16T10:26:38.003098+0000","last_unstale":"2024-09-16T10:27:41.914630+0000","last_undegraded":"2024-09-16T10:26:37.994683+0000","last_fullsized":"2024-09-16T10:26:37.991648+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:20.104708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.104708+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:30.141019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":91,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.926198+0000","last_change":"2024-09-16T10:26:38.001345+0000","last_active":"2024-09-16T10:27:41.926198+0000","last_peered":"2024-09-16T10:27:41.926198+0000","last_clean":"2024-09-16T10:26:35.989298+0000","last_became_active":"2024-09-16T10:26:38.001345+0000","last_became_peered":"2024-09-16T10:26:38.001345+0000","last_unstale":"2024-09-16T10:27:41.926198+0000","last_undegraded":"2024-09-16T10:26:37.995998+0000","last_fullsized":"2024-09-16T10:26:37.995884+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:08.381010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.381010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:29.580348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"388'10","reported_seq":211,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.922088+0000","last_change":"2024-09-16T10:26:37.998709+0000","last_active":"2024-09-16T10:27:41.922088+0000","last_peered":"2024-09-16T10:27:41.922088+0000","last_clean":"2024-09-16T10:26:35.984994+0000","last_became_active":"2024-09-16T10:26:37.998709+0000","last_became_peered":"2024-09-16T10:26:37.998709+0000","last_unstale":"2024-09-16T10:27:41.922088+0000","last_undegraded":"2024-09-16T10:26:37.992930+0000","last_fullsized":"2024-09-16T10:26:37.992802+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'9","last_scrub_stamp":"2024-09-16T10:25:37.549468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.549468+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":766,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.914812+0000","last_change":"2024-09-16T10:27:09.163203+0000","last_active":"2024-09-16T10:27:41.914812+0000","last_peered":"2024-09-16T10:27:41.914812+0000","last_clean":"2024-09-16T10:27:41.914812+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:27:41.914812+0000","last_undegraded":"2024-09-16T10:27:41.914812+0000","last_fullsized":"2024-09-16T10:27:41.914812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:09.163139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.163139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:19.821510+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.19","version":"415'10812","reported_seq":11627,"reported_epoch":415,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-16T10:27:43.998073+0000","last_change":"2024-09-16T10:27:42.050208+0000","last_active":"2024-09-16T10:27:43.998073+0000","last_peered":"2024-09-16T10:27:43.998073+0000","last_clean":"2024-09-16T10:26:36.981417+0000","last_became_active":"2024-09-16T10:27:42.019135+0000","last_became_peered":"2024-09-16T10:27:42.019135+0000","last_unstale":"2024-09-16T10:27:43.998073+0000","last_undegraded":"2024-09-16T10:27:41.959921+0000","last_fullsized":"2024-09-16T10:27:41.915468+0000","mapping_epoch":414,"log_start":"386'6019","ondisk_log_start":"386'6019","created":411,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182091776,"num_objects":2779,"num_object_clones":0,"num_object_copies":8337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2613,"num_objects_misplaced":5226,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":264064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":11173888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":334,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.914942+0000","last_change":"2024-09-16T10:25:00.825815+0000","last_active":"2024-09-16T10:27:41.914942+0000","last_peered":"2024-09-16T10:27:41.914942+0000","last_clean":"2024-09-16T10:27:41.914942+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:27:41.914942+0000","last_undegraded":"2024-09-16T10:27:41.914942+0000","last_fullsized":"2024-09-16T10:27:41.914942+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:00.825743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:00.825743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:35.976846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":753,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922920+0000","last_change":"2024-09-16T10:26:20.469286+0000","last_active":"2024-09-16T10:27:41.922920+0000","last_peered":"2024-09-16T10:27:41.922920+0000","last_clean":"2024-09-16T10:27:41.922920+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:27:41.922920+0000","last_undegraded":"2024-09-16T10:27:41.922920+0000","last_fullsized":"2024-09-16T10:27:41.922920+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:20.469240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.469240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":769,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.936457+0000","last_change":"2024-09-16T10:26:37.999313+0000","last_active":"2024-09-16T10:27:41.936457+0000","last_peered":"2024-09-16T10:27:41.936457+0000","last_clean":"2024-09-16T10:26:35.994509+0000","last_became_active":"2024-09-16T10:26:37.999313+0000","last_became_peered":"2024-09-16T10:26:37.999313+0000","last_unstale":"2024-09-16T10:27:41.936457+0000","last_undegraded":"2024-09-16T10:26:37.995519+0000","last_fullsized":"2024-09-16T10:26:37.995412+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-16T10:17:38.202149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:17:38.202149+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":753,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915696+0000","last_change":"2024-09-16T10:26:46.061085+0000","last_active":"2024-09-16T10:27:41.915696+0000","last_peered":"2024-09-16T10:27:41.915696+0000","last_clean":"2024-09-16T10:27:41.915696+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:27:41.915696+0000","last_undegraded":"2024-09-16T10:27:41.915696+0000","last_fullsized":"2024-09-16T10:27:41.915696+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:53.687250+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":65,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.922920+0000","last_change":"2024-09-16T10:26:38.003751+0000","last_active":"2024-09-16T10:27:41.922920+0000","last_peered":"2024-09-16T10:27:41.922920+0000","last_clean":"2024-09-16T10:13:42.098882+0000","last_became_active":"2024-09-16T10:26:38.003751+0000","last_became_peered":"2024-09-16T10:26:38.003751+0000","last_unstale":"2024-09-16T10:27:41.922920+0000","last_undegraded":"2024-09-16T10:27:41.922920+0000","last_fullsized":"2024-09-16T10:26:37.996606+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:08.301809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"415'10590","reported_seq":11199,"reported_epoch":415,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-16T10:27:44.003898+0000","last_change":"2024-09-16T10:27:42.052069+0000","last_active":"2024-09-16T10:27:44.003898+0000","last_peered":"2024-09-16T10:27:44.003898+0000","last_clean":"2024-09-16T10:26:36.966822+0000","last_became_active":"2024-09-16T10:27:42.031971+0000","last_became_peered":"2024-09-16T10:27:42.031971+0000","last_unstale":"2024-09-16T10:27:44.003898+0000","last_undegraded":"2024-09-16T10:27:41.955625+0000","last_fullsized":"2024-09-16T10:27:41.915532+0000","mapping_epoch":414,"log_start":"384'5600","ondisk_log_start":"384'5600","created":411,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184287232,"num_objects":2812,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2653,"num_objects_misplaced":2653,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5365,"num_write_kb":261664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2812}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":656,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923519+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:27:41.923519+0000","last_peered":"2024-09-16T10:27:41.923519+0000","last_clean":"2024-09-16T10:27:41.923519+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:27:41.923519+0000","last_undegraded":"2024-09-16T10:27:41.923519+0000","last_fullsized":"2024-09-16T10:27:41.923519+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:07.337271+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":785,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:42.007231+0000","last_change":"2024-09-16T10:26:38.014278+0000","last_active":"2024-09-16T10:27:42.007231+0000","last_peered":"2024-09-16T10:27:42.007231+0000","last_clean":"2024-09-16T10:26:35.992350+0000","last_became_active":"2024-09-16T10:26:38.014278+0000","last_became_peered":"2024-09-16T10:26:38.014278+0000","last_unstale":"2024-09-16T10:27:42.007231+0000","last_undegraded":"2024-09-16T10:26:37.993362+0000","last_fullsized":"2024-09-16T10:26:37.993237+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:24.517639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.517639+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:54.289736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":83,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.922541+0000","last_change":"2024-09-16T10:26:37.999023+0000","last_active":"2024-09-16T10:27:41.922541+0000","last_peered":"2024-09-16T10:27:41.922541+0000","last_clean":"2024-09-16T10:26:35.985349+0000","last_became_active":"2024-09-16T10:26:37.999023+0000","last_became_peered":"2024-09-16T10:26:37.999023+0000","last_unstale":"2024-09-16T10:27:41.922541+0000","last_undegraded":"2024-09-16T10:27:41.922541+0000","last_fullsized":"2024-09-16T10:26:37.994403+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:37.608524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.608524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:49.455774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":755,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.936143+0000","last_change":"2024-09-16T10:26:37.999668+0000","last_active":"2024-09-16T10:27:41.936143+0000","last_peered":"2024-09-16T10:27:41.936143+0000","last_clean":"2024-09-16T10:26:35.993488+0000","last_became_active":"2024-09-16T10:26:37.999668+0000","last_became_peered":"2024-09-16T10:26:37.999668+0000","last_unstale":"2024-09-16T10:27:41.936143+0000","last_undegraded":"2024-09-16T10:27:41.936143+0000","last_fullsized":"2024-09-16T10:26:37.995739+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.419085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.419085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":292,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.916376+0000","last_change":"2024-09-16T10:26:38.003699+0000","last_active":"2024-09-16T10:27:41.916376+0000","last_peered":"2024-09-16T10:27:41.916376+0000","last_clean":"2024-09-16T10:26:35.982840+0000","last_became_active":"2024-09-16T10:26:38.003699+0000","last_became_peered":"2024-09-16T10:26:38.003699+0000","last_unstale":"2024-09-16T10:27:41.916376+0000","last_undegraded":"2024-09-16T10:27:41.916376+0000","last_fullsized":"2024-09-16T10:26:37.993072+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:13.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:05.327493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":329,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.914541+0000","last_change":"2024-09-16T10:26:38.002460+0000","last_active":"2024-09-16T10:27:41.914541+0000","last_peered":"2024-09-16T10:27:41.914541+0000","last_clean":"2024-09-16T10:26:35.979255+0000","last_became_active":"2024-09-16T10:26:38.002460+0000","last_became_peered":"2024-09-16T10:26:38.002460+0000","last_unstale":"2024-09-16T10:27:41.914541+0000","last_undegraded":"2024-09-16T10:27:41.914541+0000","last_fullsized":"2024-09-16T10:26:37.991442+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:09.961098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:09.961098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:57.051524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":770,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.915381+0000","last_change":"2024-09-16T10:26:38.002505+0000","last_active":"2024-09-16T10:27:41.915381+0000","last_peered":"2024-09-16T10:27:41.915381+0000","last_clean":"2024-09-16T10:26:35.979851+0000","last_became_active":"2024-09-16T10:26:38.002505+0000","last_became_peered":"2024-09-16T10:26:38.002505+0000","last_unstale":"2024-09-16T10:27:41.915381+0000","last_undegraded":"2024-09-16T10:26:37.993329+0000","last_fullsized":"2024-09-16T10:26:37.993189+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:16:33.312992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:16:33.312992+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":765,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.919381+0000","last_change":"2024-09-16T10:26:38.012076+0000","last_active":"2024-09-16T10:27:41.919381+0000","last_peered":"2024-09-16T10:27:41.919381+0000","last_clean":"2024-09-16T10:26:35.991055+0000","last_became_active":"2024-09-16T10:26:38.012076+0000","last_became_peered":"2024-09-16T10:26:38.012076+0000","last_unstale":"2024-09-16T10:27:41.919381+0000","last_undegraded":"2024-09-16T10:27:41.919381+0000","last_fullsized":"2024-09-16T10:26:37.995935+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:58.224525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:21:58.224525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.514447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"388'11","reported_seq":972,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:42.335607+0000","last_change":"2024-09-16T10:26:38.005821+0000","last_active":"2024-09-16T10:27:42.335607+0000","last_peered":"2024-09-16T10:27:42.335607+0000","last_clean":"2024-09-16T10:26:35.994080+0000","last_became_active":"2024-09-16T10:26:38.005821+0000","last_became_peered":"2024-09-16T10:26:38.005821+0000","last_unstale":"2024-09-16T10:27:42.335607+0000","last_undegraded":"2024-09-16T10:26:37.994426+0000","last_fullsized":"2024-09-16T10:26:37.994289+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:58.423570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":561,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.931118+0000","last_change":"2024-09-16T10:26:45.194577+0000","last_active":"2024-09-16T10:27:41.931118+0000","last_peered":"2024-09-16T10:27:41.931118+0000","last_clean":"2024-09-16T10:27:41.931118+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:27:41.931118+0000","last_undegraded":"2024-09-16T10:27:41.931118+0000","last_fullsized":"2024-09-16T10:27:41.931118+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.194454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.194454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:50.881675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":339,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.945988+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:27:41.945988+0000","last_peered":"2024-09-16T10:27:41.945988+0000","last_clean":"2024-09-16T10:27:41.945988+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:27:41.945988+0000","last_undegraded":"2024-09-16T10:27:41.945988+0000","last_fullsized":"2024-09-16T10:27:41.945988+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":806,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.924435+0000","last_change":"2024-09-16T10:26:38.000788+0000","last_active":"2024-09-16T10:27:41.924435+0000","last_peered":"2024-09-16T10:27:41.924435+0000","last_clean":"2024-09-16T10:26:35.994364+0000","last_became_active":"2024-09-16T10:26:38.000788+0000","last_became_peered":"2024-09-16T10:26:38.000788+0000","last_unstale":"2024-09-16T10:27:41.924435+0000","last_undegraded":"2024-09-16T10:26:37.994818+0000","last_fullsized":"2024-09-16T10:26:37.994698+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:26:20.405507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.405507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":545,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.978137+0000","last_change":"2024-09-16T10:26:37.998073+0000","last_active":"2024-09-16T10:27:41.978137+0000","last_peered":"2024-09-16T10:27:41.978137+0000","last_clean":"2024-09-16T10:22:01.240719+0000","last_became_active":"2024-09-16T10:26:37.998073+0000","last_became_peered":"2024-09-16T10:26:37.998073+0000","last_unstale":"2024-09-16T10:27:41.978137+0000","last_undegraded":"2024-09-16T10:27:41.978137+0000","last_fullsized":"2024-09-16T10:26:37.992631+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:23.469779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:24:23.469779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:02.271706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":939,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.915258+0000","last_change":"2024-09-16T10:26:11.007271+0000","last_active":"2024-09-16T10:27:41.915258+0000","last_peered":"2024-09-16T10:27:41.915258+0000","last_clean":"2024-09-16T10:27:41.915258+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:27:41.915258+0000","last_undegraded":"2024-09-16T10:27:41.915258+0000","last_fullsized":"2024-09-16T10:27:41.915258+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:11.007191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:11.007191+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":744,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.929322+0000","last_change":"2024-09-16T10:26:38.010154+0000","last_active":"2024-09-16T10:27:41.929322+0000","last_peered":"2024-09-16T10:27:41.929322+0000","last_clean":"2024-09-16T10:26:35.991918+0000","last_became_active":"2024-09-16T10:26:38.010154+0000","last_became_peered":"2024-09-16T10:26:38.010154+0000","last_unstale":"2024-09-16T10:27:41.929322+0000","last_undegraded":"2024-09-16T10:27:41.929322+0000","last_fullsized":"2024-09-16T10:26:37.991189+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:21:20.127687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:21:20.127687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":690,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922409+0000","last_change":"2024-09-16T10:26:28.557427+0000","last_active":"2024-09-16T10:27:41.922409+0000","last_peered":"2024-09-16T10:27:41.922409+0000","last_clean":"2024-09-16T10:27:41.922409+0000","last_became_active":"2024-09-16T10:26:28.557003+0000","last_became_peered":"2024-09-16T10:26:28.557003+0000","last_unstale":"2024-09-16T10:27:41.922409+0000","last_undegraded":"2024-09-16T10:27:41.922409+0000","last_fullsized":"2024-09-16T10:27:41.922409+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:17.300150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.300150+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:01.264407+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":215,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.923567+0000","last_change":"2024-09-16T10:26:38.003739+0000","last_active":"2024-09-16T10:27:41.923567+0000","last_peered":"2024-09-16T10:27:41.923567+0000","last_clean":"2024-09-16T10:26:35.989456+0000","last_became_active":"2024-09-16T10:26:38.003739+0000","last_became_peered":"2024-09-16T10:26:38.003739+0000","last_unstale":"2024-09-16T10:27:41.923567+0000","last_undegraded":"2024-09-16T10:27:41.923567+0000","last_fullsized":"2024-09-16T10:26:37.988544+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:46.070747+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":754,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924289+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:27:41.924289+0000","last_peered":"2024-09-16T10:27:41.924289+0000","last_clean":"2024-09-16T10:27:41.924289+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:27:41.924289+0000","last_undegraded":"2024-09-16T10:27:41.924289+0000","last_fullsized":"2024-09-16T10:27:41.924289+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:56.217450+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":747,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.919690+0000","last_change":"2024-09-16T10:26:14.413104+0000","last_active":"2024-09-16T10:27:41.919690+0000","last_peered":"2024-09-16T10:27:41.919690+0000","last_clean":"2024-09-16T10:27:41.919690+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:27:41.919690+0000","last_undegraded":"2024-09-16T10:27:41.919690+0000","last_fullsized":"2024-09-16T10:27:41.919690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:14.413019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:14.413019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":809,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.924587+0000","last_change":"2024-09-16T10:22:02.917518+0000","last_active":"2024-09-16T10:27:41.924587+0000","last_peered":"2024-09-16T10:27:41.924587+0000","last_clean":"2024-09-16T10:27:41.924587+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:27:41.924587+0000","last_undegraded":"2024-09-16T10:27:41.924587+0000","last_fullsized":"2024-09-16T10:27:41.924587+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-16T10:20:50.296418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:20:50.296418+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:28.496588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":151,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.944068+0000","last_change":"2024-09-16T10:26:38.011535+0000","last_active":"2024-09-16T10:27:41.944068+0000","last_peered":"2024-09-16T10:27:41.944068+0000","last_clean":"2024-09-16T10:26:35.996853+0000","last_became_active":"2024-09-16T10:26:38.011535+0000","last_became_peered":"2024-09-16T10:26:38.011535+0000","last_unstale":"2024-09-16T10:27:41.944068+0000","last_undegraded":"2024-09-16T10:27:41.944068+0000","last_fullsized":"2024-09-16T10:26:37.998608+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.368081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.368081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":716,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.921091+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:27:41.921091+0000","last_peered":"2024-09-16T10:27:41.921091+0000","last_clean":"2024-09-16T10:27:41.921091+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:27:41.921091+0000","last_undegraded":"2024-09-16T10:27:41.921091+0000","last_fullsized":"2024-09-16T10:27:41.921091+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":733,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.917610+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:27:41.917610+0000","last_peered":"2024-09-16T10:27:41.917610+0000","last_clean":"2024-09-16T10:27:41.917610+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:27:41.917610+0000","last_undegraded":"2024-09-16T10:27:41.917610+0000","last_fullsized":"2024-09-16T10:27:41.917610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":665,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.915787+0000","last_change":"2024-09-16T10:26:38.004064+0000","last_active":"2024-09-16T10:27:41.915787+0000","last_peered":"2024-09-16T10:27:41.915787+0000","last_clean":"2024-09-16T10:26:35.982353+0000","last_became_active":"2024-09-16T10:26:38.004064+0000","last_became_peered":"2024-09-16T10:26:38.004064+0000","last_unstale":"2024-09-16T10:27:41.915787+0000","last_undegraded":"2024-09-16T10:27:41.915787+0000","last_fullsized":"2024-09-16T10:26:37.992187+0000","mapping_epoch":387,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.984796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.984796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:51.123220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":767,"reported_epoch":415,"state":"active+undersized+degraded","last_fresh":"2024-09-16T10:27:41.924046+0000","last_change":"2024-09-16T10:26:38.009136+0000","last_active":"2024-09-16T10:27:41.924046+0000","last_peered":"2024-09-16T10:27:41.924046+0000","last_clean":"2024-09-16T10:26:35.988399+0000","last_became_active":"2024-09-16T10:26:38.009136+0000","last_became_peered":"2024-09-16T10:26:38.009136+0000","last_unstale":"2024-09-16T10:27:41.924046+0000","last_undegraded":"2024-09-16T10:26:37.995927+0000","last_fullsized":"2024-09-16T10:26:37.995821+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-16T10:19:12.855380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:19:12.855380+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":765,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.915696+0000","last_change":"2024-09-16T10:26:37.994843+0000","last_active":"2024-09-16T10:27:41.915696+0000","last_peered":"2024-09-16T10:27:41.915696+0000","last_clean":"2024-09-16T10:26:35.980698+0000","last_became_active":"2024-09-16T10:26:37.994843+0000","last_became_peered":"2024-09-16T10:26:37.994843+0000","last_unstale":"2024-09-16T10:27:41.915696+0000","last_undegraded":"2024-09-16T10:27:41.915696+0000","last_fullsized":"2024-09-16T10:26:37.991376+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:13.803568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:23:13.803568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:42.038659+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":64,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.922445+0000","last_change":"2024-09-16T10:26:37.998775+0000","last_active":"2024-09-16T10:27:41.922445+0000","last_peered":"2024-09-16T10:27:41.922445+0000","last_clean":"2024-09-16T10:13:40.081041+0000","last_became_active":"2024-09-16T10:26:37.998775+0000","last_became_peered":"2024-09-16T10:26:37.998775+0000","last_unstale":"2024-09-16T10:27:41.922445+0000","last_undegraded":"2024-09-16T10:27:41.922445+0000","last_fullsized":"2024-09-16T10:26:37.993198+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:20:58.242051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:20:58.242051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":755,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.922080+0000","last_change":"2024-09-16T10:23:05.996862+0000","last_active":"2024-09-16T10:27:41.922080+0000","last_peered":"2024-09-16T10:27:41.922080+0000","last_clean":"2024-09-16T10:27:41.922080+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:27:41.922080+0000","last_undegraded":"2024-09-16T10:27:41.922080+0000","last_fullsized":"2024-09-16T10:27:41.922080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:23:05.996762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:23:05.996762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:38.272964+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":802,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.918627+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:27:41.918627+0000","last_peered":"2024-09-16T10:27:41.918627+0000","last_clean":"2024-09-16T10:27:41.918627+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:27:41.918627+0000","last_undegraded":"2024-09-16T10:27:41.918627+0000","last_fullsized":"2024-09-16T10:27:41.918627+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":758,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.926478+0000","last_change":"2024-09-16T10:27:27.598364+0000","last_active":"2024-09-16T10:27:41.926478+0000","last_peered":"2024-09-16T10:27:41.926478+0000","last_clean":"2024-09-16T10:27:41.926478+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:27:41.926478+0000","last_undegraded":"2024-09-16T10:27:41.926478+0000","last_fullsized":"2024-09-16T10:27:41.926478+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:27.598299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:27.598299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:38.278185+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":757,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.922753+0000","last_change":"2024-09-16T10:26:37.999481+0000","last_active":"2024-09-16T10:27:41.922753+0000","last_peered":"2024-09-16T10:27:41.922753+0000","last_clean":"2024-09-16T10:26:35.985145+0000","last_became_active":"2024-09-16T10:26:37.999481+0000","last_became_peered":"2024-09-16T10:26:37.999481+0000","last_unstale":"2024-09-16T10:27:41.922753+0000","last_undegraded":"2024-09-16T10:27:41.922753+0000","last_fullsized":"2024-09-16T10:26:37.993233+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:12.539106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:12.539106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:07.328826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":376,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.915730+0000","last_change":"2024-09-16T10:26:38.003729+0000","last_active":"2024-09-16T10:27:41.915730+0000","last_peered":"2024-09-16T10:27:41.915730+0000","last_clean":"2024-09-16T10:26:35.982121+0000","last_became_active":"2024-09-16T10:26:38.003729+0000","last_became_peered":"2024-09-16T10:26:38.003729+0000","last_unstale":"2024-09-16T10:27:41.915730+0000","last_undegraded":"2024-09-16T10:27:41.915730+0000","last_fullsized":"2024-09-16T10:26:37.991848+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:24:26.432557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:24:26.432557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:27:46.234537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"415'10705","reported_seq":10855,"reported_epoch":415,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-16T10:27:44.675731+0000","last_change":"2024-09-16T10:27:42.009947+0000","last_active":"2024-09-16T10:27:44.675731+0000","last_peered":"2024-09-16T10:27:44.675731+0000","last_clean":"2024-09-16T10:26:36.970644+0000","last_became_active":"2024-09-16T10:27:41.982307+0000","last_became_peered":"2024-09-16T10:27:41.982307+0000","last_unstale":"2024-09-16T10:27:44.675731+0000","last_undegraded":"2024-09-16T10:27:41.943573+0000","last_fullsized":"2024-09-16T10:27:41.917106+0000","mapping_epoch":414,"log_start":"384'5700","ondisk_log_start":"384'5700","created":411,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:25:29.598842+0000","last_clean_scrub_stamp":"2024-09-16T10:25:29.598842+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179470336,"num_objects":2739,"num_object_clones":0,"num_object_copies":8217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":2739,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":261792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2739}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":749,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923402+0000","last_change":"2024-09-16T10:26:23.240494+0000","last_active":"2024-09-16T10:27:41.923402+0000","last_peered":"2024-09-16T10:27:41.923402+0000","last_clean":"2024-09-16T10:27:41.923402+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:27:41.923402+0000","last_undegraded":"2024-09-16T10:27:41.923402+0000","last_fullsized":"2024-09-16T10:27:41.923402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:23.240435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:23.240435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":767,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.918831+0000","last_change":"2024-09-16T10:26:13.362416+0000","last_active":"2024-09-16T10:27:41.918831+0000","last_peered":"2024-09-16T10:27:41.918831+0000","last_clean":"2024-09-16T10:27:41.918831+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:27:41.918831+0000","last_undegraded":"2024-09-16T10:27:41.918831+0000","last_fullsized":"2024-09-16T10:27:41.918831+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.362347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.362347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":796,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.923875+0000","last_change":"2024-09-16T10:25:18.564978+0000","last_active":"2024-09-16T10:27:41.923875+0000","last_peered":"2024-09-16T10:27:41.923875+0000","last_clean":"2024-09-16T10:27:41.923875+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:27:41.923875+0000","last_undegraded":"2024-09-16T10:27:41.923875+0000","last_fullsized":"2024-09-16T10:27:41.923875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:25:18.564894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.564894+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:28:06.370154+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":763,"reported_epoch":415,"state":"active+undersized","last_fresh":"2024-09-16T10:27:41.929579+0000","last_change":"2024-09-16T10:26:37.998528+0000","last_active":"2024-09-16T10:27:41.929579+0000","last_peered":"2024-09-16T10:27:41.929579+0000","last_clean":"2024-09-16T10:26:35.992035+0000","last_became_active":"2024-09-16T10:26:37.998528+0000","last_became_peered":"2024-09-16T10:26:37.998528+0000","last_unstale":"2024-09-16T10:27:41.929579+0000","last_undegraded":"2024-09-16T10:27:41.929579+0000","last_fullsized":"2024-09-16T10:26:37.991371+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:19:16.217720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:19:16.217720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":5915181080,"num_objects":90262,"num_object_clones":0,"num_object_copies":270786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21980,"num_objects_misplaced":46969,"num_objects_unfound":0,"num_objects_dirty":90262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131479,"num_write_kb":6351873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":33357824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17859227648,"data_stored":17859215432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132300,"ondisk_log_size":132300,"up":71,"acting":71,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":682,"ondisk_log_size":682,"up":86,"acting":86,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":84,"acting":84,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":168,"num_read_kb":163,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":552211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":54,"acting":54,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234143,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995628,"kb_used_data":1495196,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91749524,"statfs":{"total":95995035648,"available":93951512576,"internally_reserved":0,"allocated":1531080704,"data_stored":1530449375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":512424394},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528497,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3134052,"kb_used_data":2579468,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":90611100,"statfs":{"total":95995035648,"available":92785766400,"internally_reserved":0,"allocated":2641375232,"data_stored":2640590448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790135,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2376304,"kb_used_data":1835608,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91368848,"statfs":{"total":95995035648,"available":93561700352,"internally_reserved":0,"allocated":1879662592,"data_stored":1878867502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986364,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3444356,"kb_used_data":2939756,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":90300796,"statfs":{"total":95995035648,"available":92468015104,"internally_reserved":0,"allocated":3010310144,"data_stored":3009493633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215297,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2444024,"kb_used_data":1839464,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91301128,"statfs":{"total":95995035648,"available":93492355072,"internally_reserved":0,"allocated":1883611136,"data_stored":1882800080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":619051464},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411527,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2554856,"kb_used_data":2180752,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91190296,"statfs":{"total":95995035648,"available":93378863104,"internally_reserved":0,"allocated":2233090048,"data_stored":2232301959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706564,"kb_used_data":1111732,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92038588,"statfs":{"total":95995035648,"available":94247514112,"internally_reserved":0,"allocated":1138413568,"data_stored":1137589361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738576,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4181548,"kb_used_data":3568668,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":89563604,"statfs":{"total":95995035648,"available":91713130496,"internally_reserved":0,"allocated":3654316032,"data_stored":3653516224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3648065536,"data_stored":3648061464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1137115136,"data_stored":1137115136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2212233216,"data_stored":2212233216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1868693504,"data_stored":1868693504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2989297664,"data_stored":2989293592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1861484544,"data_stored":1861484544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2608205824,"data_stored":2608201752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1534132224,"data_stored":1534132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:47.191 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T10:27:47.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 36 2024-09-16T10:27:47.192 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:47.203 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:47.256 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:47.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:47.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:47.366 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:27:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:27:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:27:47.413 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:47.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:47.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:47.559 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:47.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:47.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:47.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:47.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:47.792 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:47.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:47.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:47.958 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:47.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:48.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:48.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:48.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:48.235 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:48.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:48.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:48.380 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:48.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:48.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:48.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:48.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:48.614 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:48.653 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:48.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:48.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:48.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:48.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:48.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:48 smithi165 ceph-mon[32696]: pgmap v756: 178 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 791 B/s rd, 0 op/s; 22047/271383 objects degraded (8.124%); 46969/271383 objects misplaced (17.307%); 4.9 MiB/s, 78 objects/s recovering 2024-09-16T10:27:48.863 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:48 smithi063 ceph-mon[29993]: pgmap v756: 178 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 791 B/s rd, 0 op/s; 22047/271383 objects degraded (8.124%); 46969/271383 objects misplaced (17.307%); 4.9 MiB/s, 78 objects/s recovering 2024-09-16T10:27:48.863 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:48 smithi063 ceph-mon[39470]: pgmap v756: 178 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 791 B/s rd, 0 op/s; 22047/271383 objects degraded (8.124%); 46969/271383 objects misplaced (17.307%); 4.9 MiB/s, 78 objects/s recovering 2024-09-16T10:27:48.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:48.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:48.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:49.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:49.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:49.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:49.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:49.181 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:49.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:49.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:49.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:27:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:27:49.329 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:49.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:49.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:49.470 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:49.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:49.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:49.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:49.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:49.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:49.753 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:49.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:49.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:49 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:27:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:49 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:27:49.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:49 smithi165 ceph-mon[32696]: osdmap e417: 8 total, 7 up, 8 in 2024-09-16T10:27:49.855 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:49.895 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:49.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:49.915 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:49 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:27:49.915 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:49 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:27:49.915 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:49 smithi063 ceph-mon[29993]: osdmap e417: 8 total, 7 up, 8 in 2024-09-16T10:27:49.916 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:49 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:27:49.916 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:49 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:27:49.916 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:49 smithi063 ceph-mon[39470]: osdmap e417: 8 total, 7 up, 8 in 2024-09-16T10:27:49.919 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:49.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:50.040 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:50.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:50.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:50.185 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:50.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:50.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:50.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:50.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:50.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:50.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:50.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:50.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:50.596 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:50.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:50.697 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:50.741 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:50.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:50.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:50.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:50 smithi165 ceph-mon[32696]: pgmap v758: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 20638/258054 objects degraded (7.998%); 43624/258054 objects misplaced (16.905%); 15 MiB/s, 243 objects/s recovering 2024-09-16T10:27:50.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:50 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:27:50.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:50 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:27:50.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:50.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:50.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[29993]: pgmap v758: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 20638/258054 objects degraded (7.998%); 43624/258054 objects misplaced (16.905%); 15 MiB/s, 243 objects/s recovering 2024-09-16T10:27:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:27:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:27:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:27:50.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[39470]: pgmap v758: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 20638/258054 objects degraded (7.998%); 43624/258054 objects misplaced (16.905%); 15 MiB/s, 243 objects/s recovering 2024-09-16T10:27:50.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:27:50.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:50 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:27:50.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:51.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:51.105 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:51.148 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:51.249 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.274 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:51.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:51.375 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:51.518 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:51.557 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:51.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.701 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:51.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:51.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:51 smithi165 ceph-mon[32696]: osdmap e418: 8 total, 7 up, 8 in 2024-09-16T10:27:51.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:51 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 20638/258054 objects degraded (7.998%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:51.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4176459353' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:51.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:51 smithi165 ceph-mon[32696]: from='client.26299 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:51.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:51.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:51.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:51.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[29993]: osdmap e418: 8 total, 7 up, 8 in 2024-09-16T10:27:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 20638/258054 objects degraded (7.998%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4176459353' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[29993]: from='client.26299 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[39470]: osdmap e418: 8 total, 7 up, 8 in 2024-09-16T10:27:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 20638/258054 objects degraded (7.998%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4176459353' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:51 smithi063 ceph-mon[39470]: from='client.26299 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:27:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:51.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:52.020 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:27:52.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:52.106 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:52.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:52.208 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:52.251 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:52.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:52.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:52.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:52.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:52.474 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:52.512 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:52.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:52.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:52.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:52.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:52.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:52.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:52.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:52.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:52 smithi165 ceph-mon[32696]: pgmap v760: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 19728/253026 objects degraded (7.797%); 42086/253026 objects misplaced (16.633%); 14 MiB/s, 217 objects/s recovering 2024-09-16T10:27:52.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:52 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:27:52.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:52 smithi165 ceph-mon[32696]: from='client.26299 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:52.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:52 smithi165 ceph-mon[32696]: osdmap e419: 8 total, 7 up, 8 in 2024-09-16T10:27:52.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:52.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:52.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[29993]: pgmap v760: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 19728/253026 objects degraded (7.797%); 42086/253026 objects misplaced (16.633%); 14 MiB/s, 217 objects/s recovering 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[29993]: from='client.26299 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[29993]: osdmap e419: 8 total, 7 up, 8 in 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[39470]: pgmap v760: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 19728/253026 objects degraded (7.797%); 42086/253026 objects misplaced (16.633%); 14 MiB/s, 217 objects/s recovering 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[39470]: from='client.26299 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:27:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:52 smithi063 ceph-mon[39470]: osdmap e419: 8 total, 7 up, 8 in 2024-09-16T10:27:53.016 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:53.066 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:53.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:53.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:53.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:53.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:53.295 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:53.338 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:53.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:53.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:53.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:53.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:53.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:53.609 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:53.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:53.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:53.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:53.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:53.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:53 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:27:53.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:53 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:27:53.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:53 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:27:53.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:53.889 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:53.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:53.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:53 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:27:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:53 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:27:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:53 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:27:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:53 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:27:53.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:53 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:27:53.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:53 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:27:53.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:54.034 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:54.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:54.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:54.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:54.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:54.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:54.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:54.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:54.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:54.454 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:54.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:54.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:54.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:54.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:54.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:54.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:54.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:54.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:54.870 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:54.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:54.892 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:27:54.895 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[29993]: pgmap v762: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 19728/253026 objects degraded (7.797%); 42086/253026 objects misplaced (16.633%); 14 MiB/s, 217 objects/s recovering 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[39470]: pgmap v762: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 19728/253026 objects degraded (7.797%); 42086/253026 objects misplaced (16.633%); 14 MiB/s, 217 objects/s recovering 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:27:54.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:54 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:27:54.971 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:55.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:55.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:55.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:55.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:54 smithi165 ceph-mon[32696]: pgmap v762: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 19728/253026 objects degraded (7.797%); 42086/253026 objects misplaced (16.633%); 14 MiB/s, 217 objects/s recovering 2024-09-16T10:27:55.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:54 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:27:55.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:27:55.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:54 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:27:55.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:54 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:27:55.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:55.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:55.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:55.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:55.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:55.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:55.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:55.669 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:55.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:55.806 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:55.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:55.939 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:55.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:56.041 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:56.071 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:56.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:56.174 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:56.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[29993]: 3.3 scrub starts 2024-09-16T10:27:56.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[29993]: 3.3 scrub ok 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[29993]: osdmap e420: 8 total, 7 up, 8 in 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[39470]: 3.3 scrub starts 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[39470]: 3.3 scrub ok 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[39470]: osdmap e420: 8 total, 7 up, 8 in 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:27:56.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:27:56.218 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:56.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:55 smithi165 ceph-mon[32696]: 3.3 scrub starts 2024-09-16T10:27:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:55 smithi165 ceph-mon[32696]: 3.3 scrub ok 2024-09-16T10:27:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:55 smithi165 ceph-mon[32696]: osdmap e420: 8 total, 7 up, 8 in 2024-09-16T10:27:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:27:56.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:27:56.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:56.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:56.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:56.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:56.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:56.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:56.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:56.627 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:56.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:56.694 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:27:56.729 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:56.786 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:56.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:56.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:56.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:57.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:57.016 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[29993]: pgmap v764: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 17671/236214 objects degraded (7.481%); 37670/236214 objects misplaced (15.947%); 13 MiB/s, 209 objects/s recovering 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[29993]: osdmap e421: 8 total, 7 up, 8 in 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 17671/236214 objects degraded (7.481%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[29993]: osdmap e422: 8 total, 7 up, 8 in 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3240158216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[39470]: pgmap v764: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 17671/236214 objects degraded (7.481%); 37670/236214 objects misplaced (15.947%); 13 MiB/s, 209 objects/s recovering 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[39470]: osdmap e421: 8 total, 7 up, 8 in 2024-09-16T10:27:57.017 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 17671/236214 objects degraded (7.481%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:57.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[39470]: osdmap e422: 8 total, 7 up, 8 in 2024-09-16T10:27:57.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:27:57.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:27:57.018 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3240158216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:57.059 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:57.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:57.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:57.197 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:57.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:57.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:57.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:56 smithi165 ceph-mon[32696]: pgmap v764: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 17671/236214 objects degraded (7.481%); 37670/236214 objects misplaced (15.947%); 13 MiB/s, 209 objects/s recovering 2024-09-16T10:27:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:56 smithi165 ceph-mon[32696]: osdmap e421: 8 total, 7 up, 8 in 2024-09-16T10:27:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:56 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 17671/236214 objects degraded (7.481%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:27:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:56 smithi165 ceph-mon[32696]: osdmap e422: 8 total, 7 up, 8 in 2024-09-16T10:27:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:56 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:27:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:56 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:27:57.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3240158216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:27:57.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:57.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:57.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:57.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:27:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:27:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:27:57.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:57.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:57.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:57.589 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:57.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:57.681 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:27:57.735 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:27:57.735 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:27:57.735 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:27:57.736 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:27:57.737 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:27:57.737 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:27:57.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:57.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:57.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:57.867 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:57.904 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:57.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:58.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:58.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:58.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:58.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:58.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:58.182 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:58.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:58.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:58 smithi165 ceph-mon[32696]: osdmap e423: 8 total, 7 up, 8 in 2024-09-16T10:27:58.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:58 smithi165 ceph-mon[32696]: 10.b scrub starts 2024-09-16T10:27:58.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:58 smithi165 ceph-mon[32696]: pgmap v768: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 17671/236214 objects degraded (7.481%); 37670/236214 objects misplaced (15.947%); 20 MiB/s, 313 objects/s recovering 2024-09-16T10:27:58.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:58.323 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:58.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:58.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:58.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:58 smithi063 ceph-mon[29993]: osdmap e423: 8 total, 7 up, 8 in 2024-09-16T10:27:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:58 smithi063 ceph-mon[29993]: 10.b scrub starts 2024-09-16T10:27:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:58 smithi063 ceph-mon[29993]: pgmap v768: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 17671/236214 objects degraded (7.481%); 37670/236214 objects misplaced (15.947%); 20 MiB/s, 313 objects/s recovering 2024-09-16T10:27:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:58 smithi063 ceph-mon[39470]: osdmap e423: 8 total, 7 up, 8 in 2024-09-16T10:27:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:58 smithi063 ceph-mon[39470]: 10.b scrub starts 2024-09-16T10:27:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:58 smithi063 ceph-mon[39470]: pgmap v768: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 17671/236214 objects degraded (7.481%); 37670/236214 objects misplaced (15.947%); 20 MiB/s, 313 objects/s recovering 2024-09-16T10:27:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:58.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:58.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:58.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:27:58.623 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:58.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:27:58.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:58.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:58.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:58.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:58.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:58.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:59.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:27:59.017 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:27:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:27:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:59.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:27:59.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:27:59.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:59.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:27:59.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:27:59 smithi165 ceph-mon[32696]: 10.f scrub starts 2024-09-16T10:27:59.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:59.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:59.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:27:59.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:27:59 smithi063 ceph-mon[29993]: 10.f scrub starts 2024-09-16T10:27:59.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:27:59 smithi063 ceph-mon[39470]: 10.f scrub starts 2024-09-16T10:27:59.474 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:59.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:27:59.576 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:27:59.614 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:27:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:59.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:27:59.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:27:59.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:59.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:27:59.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:27:59.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:27:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:27:59.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:27:59.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:00.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:00.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:00.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:00.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.302 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:00.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:00.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:00.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:00.535 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:00.573 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[29993]: 10.b scrub ok 2024-09-16T10:28:00.573 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:00.573 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[29993]: 3.1c scrub starts 2024-09-16T10:28:00.573 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[29993]: 3.1c scrub ok 2024-09-16T10:28:00.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:00.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:00.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[29993]: pgmap v769: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 881 op/s; 16504/224271 objects degraded (7.359%); 34760/224271 objects misplaced (15.499%); 18 MiB/s, 282 objects/s recovering 2024-09-16T10:28:00.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[39470]: 10.b scrub ok 2024-09-16T10:28:00.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:00.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[39470]: 3.1c scrub starts 2024-09-16T10:28:00.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[39470]: 3.1c scrub ok 2024-09-16T10:28:00.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:00.575 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:00.576 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:00 smithi063 ceph-mon[39470]: pgmap v769: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 881 op/s; 16504/224271 objects degraded (7.359%); 34760/224271 objects misplaced (15.499%); 18 MiB/s, 282 objects/s recovering 2024-09-16T10:28:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:00.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:00.578 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:00 smithi165 ceph-mon[32696]: 10.b scrub ok 2024-09-16T10:28:00.578 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:00.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:00 smithi165 ceph-mon[32696]: 3.1c scrub starts 2024-09-16T10:28:00.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:00 smithi165 ceph-mon[32696]: 3.1c scrub ok 2024-09-16T10:28:00.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:00 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:00.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:00 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:00.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:00 smithi165 ceph-mon[32696]: pgmap v769: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 881 op/s; 16504/224271 objects degraded (7.359%); 34760/224271 objects misplaced (15.499%); 18 MiB/s, 282 objects/s recovering 2024-09-16T10:28:00.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:00.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:00.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:00.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:00.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:00.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:00.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:01.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:01.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:01.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:01.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:01.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:01.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:01.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:01.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:01.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:01.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:01.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:01.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:01.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:01.630 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:01.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:01 smithi063 ceph-mon[29993]: 10.f scrub ok 2024-09-16T10:28:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:01 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 16504/224271 objects degraded (7.359%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:01 smithi063 ceph-mon[29993]: osdmap e424: 8 total, 7 up, 8 in 2024-09-16T10:28:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:01 smithi063 ceph-mon[39470]: 10.f scrub ok 2024-09-16T10:28:01.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:01 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 16504/224271 objects degraded (7.359%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:01.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:01 smithi063 ceph-mon[39470]: osdmap e424: 8 total, 7 up, 8 in 2024-09-16T10:28:01.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:01.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:01.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:01.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:01 smithi165 ceph-mon[32696]: 10.f scrub ok 2024-09-16T10:28:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:01 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 16504/224271 objects degraded (7.359%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:01 smithi165 ceph-mon[32696]: osdmap e424: 8 total, 7 up, 8 in 2024-09-16T10:28:01.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:01.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:01.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:01.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:02.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:02.127 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:02.128 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:02.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:02.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:02.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:02.389 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:02.413 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:02 smithi063 ceph-mon[29993]: pgmap v771: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 929 B/s rd, 0 B/s wr, 1.00k op/s; 15649/219636 objects degraded (7.125%); 33380/219636 objects misplaced (15.198%); 14 MiB/s, 231 objects/s recovering 2024-09-16T10:28:02.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:02 smithi063 ceph-mon[39470]: pgmap v771: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 929 B/s rd, 0 B/s wr, 1.00k op/s; 15649/219636 objects degraded (7.125%); 33380/219636 objects misplaced (15.198%); 14 MiB/s, 231 objects/s recovering 2024-09-16T10:28:02.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:02.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:02.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:02.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:02.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:02.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:02 smithi165 ceph-mon[32696]: pgmap v771: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 929 B/s rd, 0 B/s wr, 1.00k op/s; 15649/219636 objects degraded (7.125%); 33380/219636 objects misplaced (15.198%); 14 MiB/s, 231 objects/s recovering 2024-09-16T10:28:02.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:02.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:02.952 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:02.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:03.054 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:03.114 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:03.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:03.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:03.264 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:03.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:03.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:03.365 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:03.414 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:03.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:03.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:03 smithi063 ceph-mon[29993]: 4.13 scrub starts 2024-09-16T10:28:03.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:03 smithi063 ceph-mon[29993]: 4.13 scrub ok 2024-09-16T10:28:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:03 smithi063 ceph-mon[39470]: 4.13 scrub starts 2024-09-16T10:28:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:03 smithi063 ceph-mon[39470]: 4.13 scrub ok 2024-09-16T10:28:03.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:03.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:03.566 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:03.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:03.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:03.711 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:03.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:03 smithi165 ceph-mon[32696]: 4.13 scrub starts 2024-09-16T10:28:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:03 smithi165 ceph-mon[32696]: 4.13 scrub ok 2024-09-16T10:28:03.811 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:03.853 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:03.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:03.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:03.990 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:03.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:04.091 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:04.127 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:04.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:04.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:04.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:04.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:04.408 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:04.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:04.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:04.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:04.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:04.637 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:04.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:04.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:04.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[29993]: 5.e scrub starts 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[29993]: 5.e scrub ok 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[29993]: pgmap v772: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 749 op/s; 15649/219636 objects degraded (7.125%); 33380/219636 objects misplaced (15.198%); 11 MiB/s, 172 objects/s recovering 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3302976771' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"} : dispatch 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[29993]: from='client.46214 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"} : dispatch 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[39470]: 5.e scrub starts 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[39470]: 5.e scrub ok 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[39470]: pgmap v772: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 749 op/s; 15649/219636 objects degraded (7.125%); 33380/219636 objects misplaced (15.198%); 11 MiB/s, 172 objects/s recovering 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3302976771' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"} : dispatch 2024-09-16T10:28:04.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:04 smithi063 ceph-mon[39470]: from='client.46214 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"} : dispatch 2024-09-16T10:28:04.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:04 smithi165 ceph-mon[32696]: 5.e scrub starts 2024-09-16T10:28:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:04 smithi165 ceph-mon[32696]: 5.e scrub ok 2024-09-16T10:28:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:04 smithi165 ceph-mon[32696]: pgmap v772: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 103 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 749 op/s; 15649/219636 objects degraded (7.125%); 33380/219636 objects misplaced (15.198%); 11 MiB/s, 172 objects/s recovering 2024-09-16T10:28:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:04 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3302976771' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"} : dispatch 2024-09-16T10:28:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:04 smithi165 ceph-mon[32696]: from='client.46214 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"} : dispatch 2024-09-16T10:28:04.825 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:04.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:28:04.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:28:04.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:04.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:04.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:04.960 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 10 pg_num to 36 2024-09-16T10:28:05.061 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:05.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:05.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:05.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:05.242 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:05.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:05.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:05.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:05.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:05.471 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:05.514 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:05.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:05.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:05.655 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:05.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:05.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:05 smithi063 ceph-mon[39470]: from='client.46214 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-09-16T10:28:05.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:05 smithi063 ceph-mon[39470]: osdmap e425: 8 total, 7 up, 8 in 2024-09-16T10:28:05.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:05 smithi063 ceph-mon[29993]: from='client.46214 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-09-16T10:28:05.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:05 smithi063 ceph-mon[29993]: osdmap e425: 8 total, 7 up, 8 in 2024-09-16T10:28:05.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:05.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:28:05.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:05 smithi165 ceph-mon[32696]: from='client.46214 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-09-16T10:28:05.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:05 smithi165 ceph-mon[32696]: osdmap e425: 8 total, 7 up, 8 in 2024-09-16T10:28:05.846 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:05.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:05.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:05.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:05.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:06.086 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:06.139 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:06.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:06.219 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:06.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:06.276 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:06.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:06.377 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:06.408 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:06.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:06.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:06.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:06.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:06.633 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:06.662 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:06.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[29993]: pgmap v774: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 13723/203022 objects degraded (6.759%); 29315/203022 objects misplaced (14.439%); 20 MiB/s, 316 objects/s recovering 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 15649/219636 objects degraded (7.125%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[29993]: osdmap e426: 8 total, 7 up, 8 in 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[39470]: pgmap v774: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 13723/203022 objects degraded (6.759%); 29315/203022 objects misplaced (14.439%); 20 MiB/s, 316 objects/s recovering 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 15649/219636 objects degraded (7.125%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-16T10:28:06.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:06 smithi063 ceph-mon[39470]: osdmap e426: 8 total, 7 up, 8 in 2024-09-16T10:28:06.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:06.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:06.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:06.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:06 smithi165 ceph-mon[32696]: pgmap v774: 178 pgs: 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 13723/203022 objects degraded (6.759%); 29315/203022 objects misplaced (14.439%); 20 MiB/s, 316 objects/s recovering 2024-09-16T10:28:06.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:06 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:06 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:06 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 15649/219636 objects degraded (7.125%), 25 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:06 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-16T10:28:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:06 smithi165 ceph-mon[32696]: osdmap e426: 8 total, 7 up, 8 in 2024-09-16T10:28:06.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:06.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:28:06.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:06.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:07.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:07.051 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:07.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:07.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:07.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:07.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:07.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:28:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:28:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:28:07.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:07.290 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:07.291 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:07.332 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:07.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:07.362 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:07.433 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:07.477 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:07.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:07 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:28:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:07 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:28:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:07 smithi165 ceph-mon[32696]: osdmap e427: 8 total, 7 up, 8 in 2024-09-16T10:28:07.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:07 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:28:07.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:07 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:28:07.567 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:07 smithi063 ceph-mon[29993]: osdmap e427: 8 total, 7 up, 8 in 2024-09-16T10:28:07.567 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:07 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:28:07.567 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:07 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:28:07.567 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:07 smithi063 ceph-mon[39470]: osdmap e427: 8 total, 7 up, 8 in 2024-09-16T10:28:07.579 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:07.610 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:07.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:07.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:07.767 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:07.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:07.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:07.893 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:07.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:07.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:28:07.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:08.020 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:08.030 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:08.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:08.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:08.231 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:08.246 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:08.268 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:08.370 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:08.393 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:08.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: 6.d scrub starts 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: 6.d scrub ok 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: 10.0 scrub starts 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: pgmap v777: 188 pgs: 10 unknown, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 923 op/s; 13723/203022 objects degraded (6.759%); 29315/203022 objects misplaced (14.439%); 13 MiB/s, 208 objects/s recovering 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: 10.3 scrub starts 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: 10.0 scrub ok 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: 10.2 scrub starts 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: 10.3 scrub ok 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-16T10:28:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[29993]: osdmap e428: 8 total, 7 up, 8 in 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: 6.d scrub starts 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: 6.d scrub ok 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: 10.0 scrub starts 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: pgmap v777: 188 pgs: 10 unknown, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 923 op/s; 13723/203022 objects degraded (6.759%); 29315/203022 objects misplaced (14.439%); 13 MiB/s, 208 objects/s recovering 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: 10.3 scrub starts 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: 10.0 scrub ok 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: 10.2 scrub starts 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: 10.3 scrub ok 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-16T10:28:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:08 smithi063 ceph-mon[39470]: osdmap e428: 8 total, 7 up, 8 in 2024-09-16T10:28:08.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:08.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:08.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:08.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: 6.d scrub starts 2024-09-16T10:28:08.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: 6.d scrub ok 2024-09-16T10:28:08.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: 10.0 scrub starts 2024-09-16T10:28:08.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: pgmap v777: 188 pgs: 10 unknown, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 923 op/s; 13723/203022 objects degraded (6.759%); 29315/203022 objects misplaced (14.439%); 13 MiB/s, 208 objects/s recovering 2024-09-16T10:28:08.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:08.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-16T10:28:08.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: 10.3 scrub starts 2024-09-16T10:28:08.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: 10.0 scrub ok 2024-09-16T10:28:08.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: 10.2 scrub starts 2024-09-16T10:28:08.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: 10.3 scrub ok 2024-09-16T10:28:08.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-16T10:28:08.625 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-mon[32696]: osdmap e428: 8 total, 7 up, 8 in 2024-09-16T10:28:08.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:08.724 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:08.724 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:08.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:08.768 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:08.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:08.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:08.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:08.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:09.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:09.019 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:28:09.104 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:28:09.105 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:28:09.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:09.122 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:28:09.122 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:28:09.141 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:28:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:28:09.142 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:09.158 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:28:09.158 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:28:09.174 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:28:09.175 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:28:09.191 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:28:09.191 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:28:09.208 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:28:09.208 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:28:09.208 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:28:09.245 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:09.267 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:09.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:28:08 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:28:09.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:09.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:28:09.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:09.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.517 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:09.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:09.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:09.636 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:09.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:09.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: 10.2 scrub ok 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: 10.b scrub starts 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/256534399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: 10.b scrub ok 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: 10.13 scrub starts 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/256534399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:28:09.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: osdmap e429: 8 total, 7 up, 8 in 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/834162184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: 10.2 scrub ok 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: 10.b scrub starts 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/256534399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: 10.b scrub ok 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: 10.13 scrub starts 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:09.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:09.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/256534399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:28:09.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: osdmap e429: 8 total, 7 up, 8 in 2024-09-16T10:28:09.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/834162184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:09.741 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: 10.2 scrub ok 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: 10.b scrub starts 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/256534399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: 10.b scrub ok 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: 10.13 scrub starts 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:09.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:09.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/256534399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:28:09.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: osdmap e429: 8 total, 7 up, 8 in 2024-09-16T10:28:09.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/834162184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:09.765 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:09.866 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:09.888 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:09.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:09.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:10.012 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:10.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:10.114 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:10.155 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:10.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:231096 8.2 8:4cceeac5:::benchmark_data_smithi063_101290_object103776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 405.408234193, 2024-09-16T10:28:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115767351, 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 231096 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754723+0000", 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754727+0000", 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2300000000000002e-06 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754734+0000", 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7850000000000003e-06 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754739+0000", 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7979999999999996e-06 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754843+0000", 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010442000000000001 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754957+0000", 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113748 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.755066+0000", 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109279 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758427+0000", 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033604519999999999 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758995+0000", 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056835100000000003 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870387+0000", 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11139234100000001 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870444+0000", 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6450999999999999e-05 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870489+0000", 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4863000000000001e-05 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212280 9.e 9:7a91a301:::benchmark_data_smithi063_124717_object96049:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 189.49281002699999, 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30939322400000002, 2024-09-16T10:28:10.174 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212280 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670148+0000", 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.283e-06 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670154+0000", 2024-09-16T10:28:10.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8370000000000002e-06 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670167+0000", 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3243e-05 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670176+0000", 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5969999999999992e-06 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670244+0000", 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8726999999999994e-05 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670254+0000", 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9210000000000004e-06 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.978436+0000", 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30818199899999998 2024-09-16T10:28:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979539+0000", 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001102617 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212292 9.e 9:780c7a09:::benchmark_data_smithi063_124717_object96061:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 189.49275830600001, 2024-09-16T10:28:10.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31685680300000002, 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212292 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:28:10.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670198+0000", 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9599999999999999e-07 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670201+0000", 2024-09-16T10:28:10.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1130000000000001e-06 2024-09-16T10:28:10.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670207+0000", 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2689999999999996e-06 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670213+0000", 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.164e-06 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670284+0000", 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1136000000000005e-05 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670291+0000", 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9210000000000003e-06 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979884+0000", 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30959349200000003 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980142+0000", 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025797 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980215+0000", 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3054999999999998e-05 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985118+0000", 2024-09-16T10:28:10.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049030380000000002 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985153+0000", 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4496999999999999e-05 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986997+0000", 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018447450000000001 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987028+0000", 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.099e-05 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987054+0000", 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5717000000000001e-05 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212294 9.e 9:724d3208:::benchmark_data_smithi063_124717_object96063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:28:10.187 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 189.49272653700001, 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.32240128099999998, 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212294 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670230+0000", 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7999999999999995e-07 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670233+0000", 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9919999999999999e-06 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670239+0000", 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3779999999999998e-06 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670245+0000", 2024-09-16T10:28:10.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670316+0000", 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1147000000000002e-05 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670323+0000", 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7209999999999997e-06 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981661+0000", 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31133887799999999 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981742+0000", 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0136000000000007e-05 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981814+0000", 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2065999999999997e-05 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985187+0000", 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003373048 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985193+0000", 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7170000000000002e-06 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992481+0000", 2024-09-16T10:28:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072876870000000002 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992565+0000", 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3726000000000004e-05 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992630+0000", 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5581999999999997e-05 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212314 9.e 9:73a00d32:::benchmark_data_smithi063_124717_object96083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 189.46767966100001, 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29758563799999999, 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212314 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.195 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695278+0000", 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9800000000000001e-06 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695284+0000", 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9349999999999999e-06 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695295+0000", 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0849000000000001e-05 2024-09-16T10:28:10.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695303+0000", 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2130000000000008e-06 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695359+0000", 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5859999999999997e-05 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:28:10.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695366+0000", 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6580000000000002e-06 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982882+0000", 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.287515939 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982949+0000", 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6370000000000003e-05 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.983024+0000", 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5723000000000007e-05 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985197+0000", 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002172195 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986274+0000", 2024-09-16T10:28:10.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001077733 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992764+0000", 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064893520000000003 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992820+0000", 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5903000000000003e-05 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992862+0000", 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1928000000000002e-05 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212323 9.e 9:7cb576e6:::benchmark_data_smithi063_124717_object96092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 189.46218553, 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29218721399999997, 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.202 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212323 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700773+0000", 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5299999999999999e-06 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700779+0000", 2024-09-16T10:28:10.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1399999999999997e-06 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700801+0000", 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1812999999999999e-05 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700809+0000", 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1929999999999993e-06 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700877+0000", 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8199000000000002e-05 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700889+0000", 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1904e-05 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984122+0000", 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28323306799999998 2024-09-16T10:28:10.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984181+0000", 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8847000000000001e-05 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984241+0000", 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0056000000000002e-05 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986252+0000", 2024-09-16T10:28:10.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020111399999999998 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986461+0000", 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000209273 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992908+0000", 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006446523 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992935+0000", 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7016999999999999e-05 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992957+0000", 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2510999999999998e-05 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212342 9.e 9:700843e6:::benchmark_data_smithi063_124717_object96111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 189.44264533099999, 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27274648600000001, 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212342 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720312+0000", 2024-09-16T10:28:10.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0949999999999998e-06 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720318+0000", 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1059999999999996e-06 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720333+0000", 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4173e-05 2024-09-16T10:28:10.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720340+0000", 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0389999999999999e-06 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720405+0000", 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5079000000000005e-05 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720415+0000", 2024-09-16T10:28:10.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0351e-05 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985207+0000", 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.264792321 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985273+0000", 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5215999999999994e-05 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985329+0000", 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6403000000000002e-05 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986392+0000", 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010631939999999999 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986576+0000", 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018346000000000001 2024-09-16T10:28:10.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992994+0000", 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064186979999999996 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993027+0000", 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2262999999999999e-05 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993057+0000", 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0088000000000001e-05 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212688 9.e 9:7ab865c0:::benchmark_data_smithi063_124717_object96457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 188.94195898800001, 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12657935300000001, 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212688 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220999+0000", 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221004+0000", 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1880000000000001e-06 2024-09-16T10:28:10.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221017+0000", 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2802e-05 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221023+0000", 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8789999999999997e-06 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221090+0000", 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6854999999999996e-05 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221200+0000", 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010959399999999999 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221304+0000", 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104456 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.223708+0000", 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024038449999999999 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.224936+0000", 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012280119999999999 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347460+0000", 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122523584 2024-09-16T10:28:10.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347521+0000", 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1199999999999997e-05 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347576+0000", 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4972000000000002e-05 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:213104 9.e 9:709eca86:::benchmark_data_smithi063_124717_object96873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 188.41830575200001, 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12242840100000001, 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213104 2024-09-16T10:28:10.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744652+0000", 2024-09-16T10:28:10.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0219999999999999e-06 2024-09-16T10:28:10.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744657+0000", 2024-09-16T10:28:10.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3020000000000002e-06 2024-09-16T10:28:10.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744668+0000", 2024-09-16T10:28:10.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.118e-05 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744674+0000", 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6500000000000001e-06 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744728+0000", 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3705000000000002e-05 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744820+0000", 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2052000000000006e-05 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:28:10.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744897+0000", 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7200999999999995e-05 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.746741+0000", 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018441250000000001 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.747036+0000", 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029480100000000003 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.866958+0000", 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11992201700000001 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867027+0000", 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9314999999999996e-05 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867078+0000", 2024-09-16T10:28:10.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1031e-05 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:41313 10.1 10:8e698f2c:::benchmark_data_smithi063_149043_object41312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 135.91903224999999, 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10554345399999999, 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.232 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 41313 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243925+0000", 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0329999999999998e-06 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243997+0000", 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1767000000000006e-05 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244022+0000", 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4737999999999999e-05 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244030+0000", 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5699999999999993e-06 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244235+0000", 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020468299999999999 2024-09-16T10:28:10.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244353+0000", 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011790699999999999 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.246158+0000", 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018052179999999999 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.251015+0000", 2024-09-16T10:28:10.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004856595 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.251463+0000", 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000448369 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349318+0000", 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097854294999999994 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349407+0000", 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9024999999999996e-05 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349467+0000", 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0254000000000003e-05 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73195 10.e 10:780b6aaa:::benchmark_data_smithi063_149043_object73194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:28:10.238 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 114.728689196, 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858509, 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73195 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434267+0000", 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434306+0000", 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8250000000000001e-05 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434314+0000", 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2949999999999994e-06 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434319+0000", 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5300000000000004e-06 2024-09-16T10:28:10.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434368+0000", 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8321e-05 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434486+0000", 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117924 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:28:10.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434622+0000", 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013583099999999999 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.437851+0000", 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032291569999999999 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.438873+0000", 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00102191 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542761+0000", 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10388875 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542818+0000", 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6863000000000001e-05 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542852+0000", 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-16T10:28:10.244 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73199 10.e 10:752bb1d5:::benchmark_data_smithi063_149043_object73198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 114.720895441, 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112679315, 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73199 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:28:10.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442062+0000", 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.367e-06 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442108+0000", 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6814999999999998e-05 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442117+0000", 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8410000000000006e-06 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442124+0000", 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6889999999999998e-06 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442179+0000", 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4756000000000002e-05 2024-09-16T10:28:10.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442288+0000", 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109372 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442378+0000", 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0488000000000005e-05 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.444289+0000", 2024-09-16T10:28:10.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001910293 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.444588+0000", 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000299602 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554621+0000", 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110032625 2024-09-16T10:28:10.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554689+0000", 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8324000000000005e-05 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554739+0000", 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0142999999999997e-05 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73208 10.e 10:776621e8:::benchmark_data_smithi063_149043_object73207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 114.71577161800001, 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10772002999999999, 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73208 2024-09-16T10:28:10.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447185+0000", 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2100000000000001e-06 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447333+0000", 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014735900000000001 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447344+0000", 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0994e-05 2024-09-16T10:28:10.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447351+0000", 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1130000000000003e-06 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447410+0000", 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9219999999999999e-05 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447533+0000", 2024-09-16T10:28:10.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123325 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447677+0000", 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143972 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.449865+0000", 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021879680000000002 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.450597+0000", 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073198000000000002 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554808+0000", 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104210409 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554861+0000", 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3298999999999999e-05 2024-09-16T10:28:10.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554904+0000", 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3180999999999999e-05 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73211 10.e 10:77f82def:::benchmark_data_smithi063_149043_object73210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:28:10.258 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 114.71020978, 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10288251299999999, 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73211 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:28:10.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452747+0000", 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.643e-06 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452838+0000", 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0611000000000004e-05 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452849+0000", 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0519e-05 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452855+0000", 2024-09-16T10:28:10.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454022+0000", 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001166582 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454155+0000", 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133013 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:28:10.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454307+0000", 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015232200000000001 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.459163+0000", 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048564769999999997 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.459272+0000", 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109069 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554980+0000", 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095707424999999999 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.555077+0000", 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6832999999999999e-05 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.555628+0000", 2024-09-16T10:28:10.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055162699999999996 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:80880 10.1 10:8f0a7a66:::benchmark_data_smithi063_149043_object80879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 108.84813207099999, 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109496315, 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.265 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 80880 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314825+0000", 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.689e-06 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314901+0000", 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5549000000000005e-05 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314912+0000", 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270999999999999e-05 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314917+0000", 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.169e-06 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314973+0000", 2024-09-16T10:28:10.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5448999999999997e-05 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315116+0000", 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142986 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315268+0000", 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015202799999999999 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.319831+0000", 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004563302 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.319869+0000", 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7737000000000001e-05 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424204+0000", 2024-09-16T10:28:10.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104335386 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424277+0000", 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2435999999999996e-05 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424320+0000", 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3312999999999997e-05 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.271 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:80882 10.1 10:8d1b387e:::benchmark_data_smithi063_149043_object80881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 108.84800311399999, 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111081265, 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:10.272 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 80882 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314954+0000", 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.319e-06 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315013+0000", 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9150000000000001e-05 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315022+0000", 2024-09-16T10:28:10.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9309999999999997e-06 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315027+0000", 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6360000000000002e-06 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317727+0000", 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027003840000000001 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317832+0000", 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010535999999999999 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317972+0000", 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014001700000000001 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.320350+0000", 2024-09-16T10:28:10.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023780020000000002 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.320544+0000", 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019368200000000001 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.425942+0000", 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105398164 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.425992+0000", 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9570000000000001e-05 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.426034+0000", 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2049999999999999e-05 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.278 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:142785 10.c e400/387 10:347f38ce:::benchmark_data_smithi063_149043_object10348:head v 400'9026, mlcod=400'9024)", 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:27:15.581489+0000", 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 54.581467054999997, 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021881356000000001, 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.581489+0000", 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.581489+0000", 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.581489+0000", 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2100000000000002e-07 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.581493+0000", 2024-09-16T10:28:10.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8800000000000001e-06 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.581498+0000", 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.887e-06 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.581502+0000", 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3279999999999999e-06 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.584181+0000", 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026790249999999998 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.584203+0000", 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1781e-05 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.603316+0000", 2024-09-16T10:28:10.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019112885999999999 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:15.603370+0000", 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4147999999999997e-05 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:188229 10.e e420/382 10:7c90ef2c:::benchmark_data_smithi063_149043_object55792:head v 420'11694, mlcod=420'11690)", 2024-09-16T10:28:10.283 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:27:55.542592+0000", 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.620363315000001, 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01439354, 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.542592+0000", 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.542592+0000", 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.542594+0000", 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5370000000000001e-06 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.542601+0000", 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5560000000000002e-06 2024-09-16T10:28:10.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.542617+0000", 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5828000000000002e-05 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.542625+0000", 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7170000000000008e-06 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.543052+0000", 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042699300000000002 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.543071+0000", 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9123999999999999e-05 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.556925+0000", 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013854066999999999 2024-09-16T10:28:10.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.556986+0000", 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0717999999999997e-05 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:188233 10.e e420/382 10:71a08141:::benchmark_data_smithi063_149043_object55796:head v 420'11695, mlcod=420'11690)", 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:27:55.544126+0000", 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.618829226999999, 2024-09-16T10:28:10.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014270112, 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.544126+0000", 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.544126+0000", 2024-09-16T10:28:10.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.544127+0000", 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0950000000000001e-06 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.544133+0000", 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6180000000000003e-06 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.544144+0000", 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0789e-05 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.544150+0000", 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6159999999999998e-06 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.545520+0000", 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013696159999999999 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.545539+0000", 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9097e-05 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.558366+0000", 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012826694 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.558396+0000", 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0586999999999997e-05 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:188248 10.6 e420/411 10:65f8ba35:::benchmark_data_smithi063_149043_object55811:head v 420'11251, mlcod=420'11249)", 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:27:55.551521+0000", 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.611434625999999, 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01363312, 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:10.293 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.551521+0000", 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.551521+0000", 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.551522+0000", 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.398e-06 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.551529+0000", 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.674e-06 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.551541+0000", 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1841000000000001e-05 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.551548+0000", 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7780000000000002e-06 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.556910+0000", 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053623209999999998 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.556972+0000", 2024-09-16T10:28:10.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1579000000000001e-05 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.565125+0000", 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081529500000000008 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:27:55.565154+0000", 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9578999999999999e-05 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.297 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.298 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:10.298 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:10.298 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:28:10.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:10.340 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:10.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:10.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:10.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: 10.13 scrub ok 2024-09-16T10:28:10.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: 10.d scrub starts 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: pgmap v780: 188 pgs: 2 active+clean+remapped, 1 active+clean+scrubbing, 3 remapped+peering, 6 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12504/187143 objects degraded (6.682%); 26829/187143 objects misplaced (14.336%) 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: 10.18 scrub starts 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: 10.19 scrub starts 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 427'12418 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: osdmap e430: 8 total, 7 up, 8 in 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 427'12450 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[29993]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 427'12450 2024-09-16T10:28:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: 10.13 scrub ok 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: 10.d scrub starts 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: pgmap v780: 188 pgs: 2 active+clean+remapped, 1 active+clean+scrubbing, 3 remapped+peering, 6 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12504/187143 objects degraded (6.682%); 26829/187143 objects misplaced (14.336%) 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: 10.18 scrub starts 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: 10.19 scrub starts 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 427'12418 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: osdmap e430: 8 total, 7 up, 8 in 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 427'12450 2024-09-16T10:28:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:10 smithi063 ceph-mon[39470]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 427'12450 2024-09-16T10:28:10.487 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:10.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:10.589 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:28:10.622 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:28:10.633 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:10.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:10.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:10.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:10.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: 10.13 scrub ok 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: 10.d scrub starts 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: pgmap v780: 188 pgs: 2 active+clean+remapped, 1 active+clean+scrubbing, 3 remapped+peering, 6 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12504/187143 objects degraded (6.682%); 26829/187143 objects misplaced (14.336%) 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: 10.18 scrub starts 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: 10.19 scrub starts 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 427'12418 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: osdmap e430: 8 total, 7 up, 8 in 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 427'12450 2024-09-16T10:28:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:10 smithi165 ceph-mon[32696]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 427'12450 2024-09-16T10:28:10.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:10.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:10.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:11.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:11.073 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:11.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:11.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:11.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:11.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:11.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:28:11.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:11.357 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:11.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:11.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:11.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:11.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 427'11596 2024-09-16T10:28:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 427'12623 2024-09-16T10:28:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 427'11436 2024-09-16T10:28:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 427'12460 2024-09-16T10:28:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 427'11583 2024-09-16T10:28:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 427'11596 2024-09-16T10:28:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.23 starting backfill to osd.4 from (0'0,0'0] MAX to 427'11754 2024-09-16T10:28:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 427'12624 2024-09-16T10:28:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 427'12624 2024-09-16T10:28:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 427'12460 2024-09-16T10:28:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: 10.f scrub starts 2024-09-16T10:28:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:11 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 12504/187143 objects degraded (6.682%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:11.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:11.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:11.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:11.641 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 427'11596 2024-09-16T10:28:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 427'12623 2024-09-16T10:28:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 427'11436 2024-09-16T10:28:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 427'12460 2024-09-16T10:28:11.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 427'11583 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 427'11596 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.23 starting backfill to osd.4 from (0'0,0'0] MAX to 427'11754 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 427'12624 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 427'12624 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 427'12460 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: 10.f scrub starts 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 12504/187143 objects degraded (6.682%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 427'11596 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 427'12623 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 427'11436 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 427'12460 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 427'11583 2024-09-16T10:28:11.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 427'11596 2024-09-16T10:28:11.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.23 starting backfill to osd.4 from (0'0,0'0] MAX to 427'11754 2024-09-16T10:28:11.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 427'12624 2024-09-16T10:28:11.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 427'12624 2024-09-16T10:28:11.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 427'12460 2024-09-16T10:28:11.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: 10.f scrub starts 2024-09-16T10:28:11.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:11 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 12504/187143 objects degraded (6.682%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:11.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:11.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:11.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:11.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:28:11.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:11.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:11.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:11.992 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:12.014 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:12.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:12.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:12.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:12.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:12.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:12.286 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:12.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:12.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-16T10:28:12.363 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-16T10:28:12.363 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-09-16T10:28:12.364 DEBUG:teuthology.orchestra.run.smithi165:osd.7> sudo journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.7.service 2024-09-16T10:28:12.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl start ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.7 2024-09-16T10:28:12.389 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:12.433 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:12.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:12.535 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:12.579 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:12.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:12.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:28:12.658 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:12 smithi063 ceph-mon[29993]: 10.1a scrub starts 2024-09-16T10:28:12.658 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:12 smithi063 ceph-mon[29993]: pgmap v782: 188 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 1 active+clean+scrubbing, 7 remapped+peering, 2 unknown, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 9149/194469 objects degraded (4.705%); 26325/194469 objects misplaced (13.537%) 2024-09-16T10:28:12.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:12 smithi063 ceph-mon[39470]: 10.1a scrub starts 2024-09-16T10:28:12.658 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:12 smithi063 ceph-mon[39470]: pgmap v782: 188 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 1 active+clean+scrubbing, 7 remapped+peering, 2 unknown, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 9149/194469 objects degraded (4.705%); 26325/194469 objects misplaced (13.537%) 2024-09-16T10:28:12.681 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:12.743 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:12.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:12 smithi165 ceph-mon[32696]: 10.1a scrub starts 2024-09-16T10:28:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:12 smithi165 ceph-mon[32696]: pgmap v782: 188 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 1 active+clean+scrubbing, 7 remapped+peering, 2 unknown, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 9149/194469 objects degraded (4.705%); 26325/194469 objects misplaced (13.537%) 2024-09-16T10:28:12.802 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:12 smithi165 systemd[1]: Starting Ceph osd.7 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:28:12.844 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:202942 10.23 10:c7c4d4d0:::benchmark_data_smithi063_149043_object70505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:12.233715+0000", 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0042317259999999999, 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042534130000000002, 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:28:12.926 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 202942 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.233715+0000", 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.233715+0000", 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.233717+0000", 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.556e-06 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.233721+0000", 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6759999999999998e-06 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.233730+0000", 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5180000000000004e-06 2024-09-16T10:28:12.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.233734+0000", 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5839999999999996e-06 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.234057+0000", 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032230799999999999 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:12.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.234158+0000", 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010090399999999999 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3,4", 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.234416+0000", 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000257808 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.235752+0000", 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013364449999999999 2024-09-16T10:28:12.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.237326+0000", 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001574212 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.16152.0:202942 10.23 e430/429)", 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:12.237531+0000", 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00041596299999999999, 2024-09-16T10:28:12.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048212400000000001, 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.237531+0000", 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.237531+0000", 2024-09-16T10:28:12.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.237532+0000", 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3060000000000001e-06 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.237537+0000", 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999996e-06 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.237544+0000", 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6410000000000002e-06 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:12.237551+0000", 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4690000000000002e-06 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:28:12.934 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-09-16T10:28:12.935 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:28:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:12.935 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:12.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:13.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:13.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:13.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:13.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:13.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:13.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:13.331 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:13.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:13.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:13.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:13.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:13.689 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:13.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:13 smithi063 ceph-mon[29993]: 10.a scrub starts 2024-09-16T10:28:13.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:13 smithi063 ceph-mon[39470]: 10.a scrub starts 2024-09-16T10:28:13.730 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:13.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:13 smithi165 ceph-mon[32696]: 10.a scrub starts 2024-09-16T10:28:13.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:13.865 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:13.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:13.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:13.996 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:13.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:14.099 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:14.134 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:14.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:14.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:14.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:14.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:14.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:14.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:14.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:14.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:14.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:14.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:14.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:14.667 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:14.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:14.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[29993]: 10.23 scrub starts 2024-09-16T10:28:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[29993]: 10.e scrub starts 2024-09-16T10:28:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[29993]: pgmap v783: 188 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 1 active+clean+scrubbing, 7 remapped+peering, 2 unknown, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 9149/194469 objects degraded (4.705%); 26325/194469 objects misplaced (13.537%) 2024-09-16T10:28:14.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[29993]: osdmap e431: 8 total, 7 up, 8 in 2024-09-16T10:28:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[39470]: 10.23 scrub starts 2024-09-16T10:28:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[39470]: 10.e scrub starts 2024-09-16T10:28:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[39470]: pgmap v783: 188 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 1 active+clean+scrubbing, 7 remapped+peering, 2 unknown, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 9149/194469 objects degraded (4.705%); 26325/194469 objects misplaced (13.537%) 2024-09-16T10:28:14.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:14 smithi063 ceph-mon[39470]: osdmap e431: 8 total, 7 up, 8 in 2024-09-16T10:28:14.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:14 smithi165 ceph-mon[32696]: 10.23 scrub starts 2024-09-16T10:28:14.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:14 smithi165 ceph-mon[32696]: 10.e scrub starts 2024-09-16T10:28:14.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:14 smithi165 ceph-mon[32696]: pgmap v783: 188 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 1 active+clean+scrubbing, 7 remapped+peering, 2 unknown, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 107 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 9149/194469 objects degraded (4.705%); 26325/194469 objects misplaced (13.537%) 2024-09-16T10:28:14.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:14 smithi165 ceph-mon[32696]: osdmap e431: 8 total, 7 up, 8 in 2024-09-16T10:28:14.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:14.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:14.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:14.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:14.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:14.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:15.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:15.067 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:15.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:15.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:28:15.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:15.210 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:15.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:15.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:15.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:15.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:15.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:15.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:15.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:15.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:15.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:15.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:15.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:15.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:15.691 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:15 smithi165 ceph-mon[32696]: osdmap e432: 8 total, 7 up, 8 in 2024-09-16T10:28:15.691 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 podman[122950]: 2024-09-16 10:28:15.278968309 +0000 UTC m=+0.026293423 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:28:15.691 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 podman[122950]: 2024-09-16 10:28:15.42745841 +0000 UTC m=+0.174783517 container create 298abc2b1d30dc49e922cc835b085a9319b9c60f15ed0c3bd0030edb4ac26216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-16T10:28:15.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:15 smithi063 ceph-mon[39470]: osdmap e432: 8 total, 7 up, 8 in 2024-09-16T10:28:15.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:15 smithi063 ceph-mon[29993]: osdmap e432: 8 total, 7 up, 8 in 2024-09-16T10:28:15.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:15.750 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:15.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:15.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:15.856 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:15.893 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:15.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:15.982 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 podman[122950]: 2024-09-16 10:28:15.688973868 +0000 UTC m=+0.436298986 container init 298abc2b1d30dc49e922cc835b085a9319b9c60f15ed0c3bd0030edb4ac26216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-16T10:28:15.983 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 podman[122950]: 2024-09-16 10:28:15.694952892 +0000 UTC m=+0.442277995 container start 298abc2b1d30dc49e922cc835b085a9319b9c60f15ed0c3bd0030edb4ac26216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T10:28:15.983 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 podman[122950]: 2024-09-16 10:28:15.797671053 +0000 UTC m=+0.544996162 container attach 298abc2b1d30dc49e922cc835b085a9319b9c60f15ed0c3bd0030edb4ac26216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, ceph=True) 2024-09-16T10:28:15.983 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T10:28:15.983 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 bash[122950]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T10:28:15.983 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T10:28:15.983 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:15 smithi165 bash[122950]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T10:28:15.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:16.041 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:16.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:16.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:16.143 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:16.188 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:16.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:16.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:16.335 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:16.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:16.438 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:16.482 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:16.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:16.503 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:28:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:28:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:28:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:28:16.504 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:28:16.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:16.642 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:16.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:16.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[29993]: 10.23 scrub starts 2024-09-16T10:28:16.677 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[29993]: pgmap v786: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 2 active+clean+remapped, 1 remapped+peering, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 853 op/s; 10085/182154 objects degraded (5.537%); 42494/182154 objects misplaced (23.329%); 41 MiB/s, 658 objects/s recovering 2024-09-16T10:28:16.677 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:16.677 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 10085/182154 objects degraded (5.537%), 25 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:16.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[39470]: 10.23 scrub starts 2024-09-16T10:28:16.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[39470]: pgmap v786: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 2 active+clean+remapped, 1 remapped+peering, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 853 op/s; 10085/182154 objects degraded (5.537%); 42494/182154 objects misplaced (23.329%); 41 MiB/s, 658 objects/s recovering 2024-09-16T10:28:16.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:16.677 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:16 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 10085/182154 objects degraded (5.537%), 25 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:16.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:16.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-mon[32696]: 10.23 scrub starts 2024-09-16T10:28:16.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-mon[32696]: pgmap v786: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 2 active+clean+remapped, 1 remapped+peering, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 853 op/s; 10085/182154 objects degraded (5.537%); 42494/182154 objects misplaced (23.329%); 41 MiB/s, 658 objects/s recovering 2024-09-16T10:28:16.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:16.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 10085/182154 objects degraded (5.537%), 25 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:16.753 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T10:28:16.753 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 bash[122950]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T10:28:16.754 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-16T10:28:16.754 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 bash[122950]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-16T10:28:16.820 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:16.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:16.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:16.922 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:16.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:16.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:17.052 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-16T10:28:17.052 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 bash[122950]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-16T10:28:17.052 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-16T10:28:17.052 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 bash[122950]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-16T10:28:17.052 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-16T10:28:17.053 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 bash[122950]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-16T10:28:17.053 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T10:28:17.053 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 bash[122950]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T10:28:17.053 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate[123020]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-16T10:28:17.053 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 bash[122950]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-16T10:28:17.053 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:16 smithi165 podman[122950]: 2024-09-16 10:28:16.831542252 +0000 UTC m=+1.578867351 container died 298abc2b1d30dc49e922cc835b085a9319b9c60f15ed0c3bd0030edb4ac26216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731) 2024-09-16T10:28:17.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:17.116 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:17.122 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:17.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:17.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:17.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:17.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:17.261 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:28:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:28:17.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:17.400 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:17.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:17.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:17.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:17.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:17.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:17.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:17.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:17.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:17.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:17.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:17.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:17.797 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:28:17.798 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:28:17.841 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:28:17.842 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:28:17.842 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:17.842 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:28:17.843 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:17 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:17.880 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:28:17.880 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:28:17.887 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:17.916 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:28:17.916 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:28:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:17.928 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:17.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:17.956 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:28:17.957 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:28:17.989 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:28:17.989 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:28:18.025 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:28:18.026 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:28:18.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:18.030 INFO:teuthology.orchestra.run.smithi063.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T10:28:18.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:17 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:18.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:17 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:18.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:17 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:28:18.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:17 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:28:18.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:17 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:18.073 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:18.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:18.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:18.206 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:18.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:18.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:18.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:18.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:18.438 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:18.484 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:18.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:18.565 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:28:18.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:18.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:18.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:18.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:18.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:18.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:18.848 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:18.857 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:18 smithi165 podman[123224]: 2024-09-16 10:28:18.619355403 +0000 UTC m=+1.782056629 container remove 298abc2b1d30dc49e922cc835b085a9319b9c60f15ed0c3bd0030edb4ac26216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-activate, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-16T10:28:18.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:18 smithi165 ceph-mon[32696]: pgmap v787: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 2 active+clean+remapped, 1 remapped+peering, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 694 B/s rd, 0 op/s; 10085/182154 objects degraded (5.537%); 42494/182154 objects misplaced (23.329%); 33 MiB/s, 535 objects/s recovering 2024-09-16T10:28:18.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2218250335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:18.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:18 smithi165 ceph-mon[32696]: from='client.26329 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:18.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1823833608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:18.870 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:18.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[29993]: pgmap v787: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 2 active+clean+remapped, 1 remapped+peering, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 694 B/s rd, 0 op/s; 10085/182154 objects degraded (5.537%); 42494/182154 objects misplaced (23.329%); 33 MiB/s, 535 objects/s recovering 2024-09-16T10:28:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2218250335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[29993]: from='client.26329 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1823833608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:18.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[39470]: pgmap v787: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 2 active+clean+remapped, 1 remapped+peering, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 694 B/s rd, 0 op/s; 10085/182154 objects degraded (5.537%); 42494/182154 objects misplaced (23.329%); 33 MiB/s, 535 objects/s recovering 2024-09-16T10:28:18.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2218250335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:18.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[39470]: from='client.26329 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1823833608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:18.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:19.017 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:19.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:19.163 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:19.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:28:18 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:28:19.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:28:19.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:19.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:19.389 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:19.434 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:19.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:19.554 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:19 smithi165 podman[123450]: 2024-09-16 10:28:19.305898626 +0000 UTC m=+0.016710364 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:28:19.554 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:19 smithi165 podman[123450]: 2024-09-16 10:28:19.424368935 +0000 UTC m=+0.135180660 container create 76d1aad57994b2a8f80541148854e80d92fc8016749ed4a0bc8e4ee7c7af2e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-09-16T10:28:19.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:19.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:19.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:19.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:28:19.821 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:19.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:19 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:19.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:19 smithi165 ceph-mon[32696]: from='client.26329 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:28:19.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:19 smithi165 ceph-mon[32696]: osdmap e433: 8 total, 7 up, 8 in 2024-09-16T10:28:19.831 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:19 smithi165 podman[123450]: 2024-09-16 10:28:19.686240241 +0000 UTC m=+0.397051976 container init 76d1aad57994b2a8f80541148854e80d92fc8016749ed4a0bc8e4ee7c7af2e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-16T10:28:19.831 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:19 smithi165 podman[123450]: 2024-09-16 10:28:19.689604459 +0000 UTC m=+0.400416184 container start 76d1aad57994b2a8f80541148854e80d92fc8016749ed4a0bc8e4ee7c7af2e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-16T10:28:19.831 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:19 smithi165 bash[123450]: 76d1aad57994b2a8f80541148854e80d92fc8016749ed4a0bc8e4ee7c7af2e00 2024-09-16T10:28:19.831 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:19 smithi165 systemd[1]: Started Ceph osd.7 for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:28:19.854 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:19.858 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:19.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:19 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:19.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:19 smithi063 ceph-mon[29993]: from='client.26329 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:28:19.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:19 smithi063 ceph-mon[29993]: osdmap e433: 8 total, 7 up, 8 in 2024-09-16T10:28:19.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:19 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:19.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:19 smithi063 ceph-mon[39470]: from='client.26329 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:28:19.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:19 smithi063 ceph-mon[39470]: osdmap e433: 8 total, 7 up, 8 in 2024-09-16T10:28:19.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:19.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:20.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:20.127 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:20.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:20.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:20.272 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:20.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:20.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:20.418 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:20.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:20.447 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:20.447 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:20.459 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:20.519 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:20.562 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:20.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:20.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:20.695 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:20.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:20.796 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:20 smithi165 ceph-mon[32696]: pgmap v789: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 154 op/s; 9942/173013 objects degraded (5.746%); 38884/173013 objects misplaced (22.475%); 66 MiB/s, 1.05k objects/s recovering 2024-09-16T10:28:20.838 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:20.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:20.908 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:20.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:20 smithi063 ceph-mon[29993]: pgmap v789: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 154 op/s; 9942/173013 objects degraded (5.746%); 38884/173013 objects misplaced (22.475%); 66 MiB/s, 1.05k objects/s recovering 2024-09-16T10:28:20.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:20.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:20 smithi063 ceph-mon[39470]: pgmap v789: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 154 op/s; 9942/173013 objects degraded (5.746%); 38884/173013 objects misplaced (22.475%); 66 MiB/s, 1.05k objects/s recovering 2024-09-16T10:28:20.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:20.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:20.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:21.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:21.099 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:21.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:21.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:21.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:28:21.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:21.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:21.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:21.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:21.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:21.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:21.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:21.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:21.645 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:21.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:21.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:21 smithi165 ceph-mon[32696]: osdmap e434: 8 total, 7 up, 8 in 2024-09-16T10:28:21.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:21 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 9942/173013 objects degraded (5.746%), 25 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:21.702 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:21 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:28:21.559+0000 7f64985ad640 -1 Falling back to public interface 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:0 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:1 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:2 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:config diff 2024-09-16T10:28:21.713 INFO:teuthology.orchestra.run.smithi165.stderr:config diff get 2024-09-16T10:28:21.714 INFO:teuthology.orchestra.run.smithi165.stderr:config get 2024-09-16T10:28:21.714 INFO:teuthology.orchestra.run.smithi165.stderr:config help [] 2024-09-16T10:28:21.714 INFO:teuthology.orchestra.run.smithi165.stderr:config set ... 2024-09-16T10:28:21.714 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2024-09-16T10:28:21.781 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:21.825 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:21.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:21.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:21 smithi063 ceph-mon[29993]: osdmap e434: 8 total, 7 up, 8 in 2024-09-16T10:28:21.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:21 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 9942/173013 objects degraded (5.746%), 25 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:21.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:21 smithi063 ceph-mon[39470]: osdmap e434: 8 total, 7 up, 8 in 2024-09-16T10:28:21.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:21 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 9942/173013 objects degraded (5.746%), 25 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:21.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:21.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:22.060 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:22.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:22.097 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:22.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:22.198 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:22.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:22.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:22.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:22.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:22.432 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:22.454 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:28:22.467 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-16T10:28:22.485 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:22.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:22.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:22.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:22.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:22.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:22.713 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:22 smithi165 ceph-mon[32696]: pgmap v791: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 873 op/s; 8248/168393 objects degraded (4.898%); 36739/168393 objects misplaced (21.817%); 44 MiB/s, 706 objects/s recovering 2024-09-16T10:28:22.714 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:22 smithi165 ceph-mon[32696]: osdmap e435: 8 total, 7 up, 8 in 2024-09-16T10:28:22.737 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:22.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:22.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:22.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:22.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:22 smithi063 ceph-mon[29993]: pgmap v791: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 873 op/s; 8248/168393 objects degraded (4.898%); 36739/168393 objects misplaced (21.817%); 44 MiB/s, 706 objects/s recovering 2024-09-16T10:28:22.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:22 smithi063 ceph-mon[29993]: osdmap e435: 8 total, 7 up, 8 in 2024-09-16T10:28:22.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:22 smithi063 ceph-mon[39470]: pgmap v791: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 873 op/s; 8248/168393 objects degraded (4.898%); 36739/168393 objects misplaced (21.817%); 44 MiB/s, 706 objects/s recovering 2024-09-16T10:28:22.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:22 smithi063 ceph-mon[39470]: osdmap e435: 8 total, 7 up, 8 in 2024-09-16T10:28:22.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:23.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:23.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:23.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:23.244 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:23.276 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:23.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:23.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:23.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:23.641 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.674 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:23.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:23.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:23.796 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:23.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:23.928 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:23.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:23.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:23.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:23 smithi063 ceph-mon[29993]: 10.20 scrub starts 2024-09-16T10:28:23.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:23 smithi063 ceph-mon[29993]: 10.21 scrub starts 2024-09-16T10:28:23.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:23 smithi063 ceph-mon[29993]: osdmap e436: 8 total, 7 up, 8 in 2024-09-16T10:28:23.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:23 smithi063 ceph-mon[39470]: 10.20 scrub starts 2024-09-16T10:28:23.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:23 smithi063 ceph-mon[39470]: 10.21 scrub starts 2024-09-16T10:28:23.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:23 smithi063 ceph-mon[39470]: osdmap e436: 8 total, 7 up, 8 in 2024-09-16T10:28:24.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:24.047 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:24.048 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:24.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:23 smithi165 ceph-mon[32696]: 10.20 scrub starts 2024-09-16T10:28:24.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:23 smithi165 ceph-mon[32696]: 10.21 scrub starts 2024-09-16T10:28:24.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:23 smithi165 ceph-mon[32696]: osdmap e436: 8 total, 7 up, 8 in 2024-09-16T10:28:24.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:24.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:24.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:24.290 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:24.314 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:24.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:24.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:24.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:24.571 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:24.673 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:24.713 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:24.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:24.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:24.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[29993]: pgmap v794: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 659 op/s; 8248/168393 objects degraded (4.898%); 36739/168393 objects misplaced (21.817%); 44 MiB/s, 710 objects/s recovering 2024-09-16T10:28:24.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[29993]: osdmap e437: 8 total, 7 up, 8 in 2024-09-16T10:28:24.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:24.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:24.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[39470]: pgmap v794: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 659 op/s; 8248/168393 objects degraded (4.898%); 36739/168393 objects misplaced (21.817%); 44 MiB/s, 710 objects/s recovering 2024-09-16T10:28:24.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[39470]: osdmap e437: 8 total, 7 up, 8 in 2024-09-16T10:28:24.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:24.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:24.976 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:24.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:25.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:24 smithi165 ceph-mon[32696]: pgmap v794: 188 pgs: 3 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 41 active+undersized, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 108 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 659 op/s; 8248/168393 objects degraded (4.898%); 36739/168393 objects misplaced (21.817%); 44 MiB/s, 710 objects/s recovering 2024-09-16T10:28:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:24 smithi165 ceph-mon[32696]: osdmap e437: 8 total, 7 up, 8 in 2024-09-16T10:28:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:25.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:25.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:25.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:25.206 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:25.235 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:25.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:25.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:25.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:25.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:25.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:25.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:25.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:25.590 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:25.619 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:25.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:25.722 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:25.748 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:25.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:25.802 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:28:25.476+0000 7f64985ad640 -1 osd.7 386 log_to_monitors true 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: 10.17 scrub starts 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: osdmap e438: 8 total, 7 up, 8 in 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: from='osd.7 v1:172.21.15.165:6812/1921603083' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: from='osd.7 v1:172.21.15.165:6812/1921603083' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: osdmap e439: 8 total, 7 up, 8 in 2024-09-16T10:28:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:25 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:28:25.850 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:25.880 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:25.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:25.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: 10.17 scrub starts 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: osdmap e438: 8 total, 7 up, 8 in 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: from='osd.7 v1:172.21.15.165:6812/1921603083' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: from='osd.7 v1:172.21.15.165:6812/1921603083' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: osdmap e439: 8 total, 7 up, 8 in 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: 10.17 scrub starts 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: osdmap e438: 8 total, 7 up, 8 in 2024-09-16T10:28:25.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: from='osd.7 v1:172.21.15.165:6812/1921603083' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:28:25.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T10:28:25.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T10:28:25.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: from='osd.7 v1:172.21.15.165:6812/1921603083' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:28:25.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: osdmap e439: 8 total, 7 up, 8 in 2024-09-16T10:28:25.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:25 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-09-16T10:28:25.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:26.005 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:26.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:26.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:26.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:26.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:26.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:28:26.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:26.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:26.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:26.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:26.415 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:26.455 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:26.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:26.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:28:26 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:28:26.329+0000 7f648c34f640 -1 osd.7 440 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:28:26.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:26.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:26.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:26.620 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:26.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:26.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:26.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:26.833 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:26.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:26.879 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:26.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:26.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[29993]: 10.1f scrub starts 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[29993]: pgmap v797: 188 pgs: 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 111 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 780 op/s; 7045/159033 objects degraded (4.430%); 29595/159033 objects misplaced (18.609%); 44 MiB/s, 696 objects/s recovering 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[29993]: 10.22 scrub starts 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 7045/159033 objects degraded (4.430%), 24 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[29993]: osdmap e440: 8 total, 7 up, 8 in 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[29993]: from='osd.7 ' entity='osd.7' 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[39470]: 10.1f scrub starts 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[39470]: pgmap v797: 188 pgs: 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 111 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 780 op/s; 7045/159033 objects degraded (4.430%); 29595/159033 objects misplaced (18.609%); 44 MiB/s, 696 objects/s recovering 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[39470]: 10.22 scrub starts 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 7045/159033 objects degraded (4.430%), 24 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[39470]: osdmap e440: 8 total, 7 up, 8 in 2024-09-16T10:28:26.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:26 smithi063 ceph-mon[39470]: from='osd.7 ' entity='osd.7' 2024-09-16T10:28:26.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:26 smithi165 ceph-mon[32696]: 10.1f scrub starts 2024-09-16T10:28:26.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:26 smithi165 ceph-mon[32696]: pgmap v797: 188 pgs: 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 111 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 780 op/s; 7045/159033 objects degraded (4.430%); 29595/159033 objects misplaced (18.609%); 44 MiB/s, 696 objects/s recovering 2024-09-16T10:28:26.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:26 smithi165 ceph-mon[32696]: 10.22 scrub starts 2024-09-16T10:28:26.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:26 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 7045/159033 objects degraded (4.430%), 24 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:26.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:26 smithi165 ceph-mon[32696]: osdmap e440: 8 total, 7 up, 8 in 2024-09-16T10:28:26.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:26 smithi165 ceph-mon[32696]: from='osd.7 ' entity='osd.7' 2024-09-16T10:28:26.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:27.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:27.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:27.137 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:27.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:27.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:27.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:27.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:27.385 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:27.385 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:27.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:28:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:28:27.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:28:27.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:27.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:27.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:27.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:27.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:27.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:27.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:27.908 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:27.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:27.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[29993]: 10.1e scrub starts 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[29993]: 10.11 scrub starts 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[29993]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[29993]: osd.7 v1:172.21.15.165:6812/1921603083 boot 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[29993]: osdmap e441: 8 total, 8 up, 8 in 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[39470]: 10.1e scrub starts 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[39470]: 10.11 scrub starts 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[39470]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[39470]: osd.7 v1:172.21.15.165:6812/1921603083 boot 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[39470]: osdmap e441: 8 total, 8 up, 8 in 2024-09-16T10:28:27.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:27 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:28:27.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:27 smithi165 ceph-mon[32696]: 10.1e scrub starts 2024-09-16T10:28:27.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:27 smithi165 ceph-mon[32696]: 10.11 scrub starts 2024-09-16T10:28:27.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:27 smithi165 ceph-mon[32696]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T10:28:27.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:27 smithi165 ceph-mon[32696]: osd.7 v1:172.21.15.165:6812/1921603083 boot 2024-09-16T10:28:27.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:27 smithi165 ceph-mon[32696]: osdmap e441: 8 total, 8 up, 8 in 2024-09-16T10:28:27.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:27 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T10:28:28.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:28.034 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:28.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:28:28.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:28.167 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:28.240 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:28.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:28.300 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:28.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:28.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:28.533 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:28.561 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:28.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:28.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:28.791 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:28.819 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:28.894 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:28:28.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:28:28.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-mon[32696]: pgmap v801: 188 pgs: 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 111 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 563 op/s; 7045/159033 objects degraded (4.430%); 29595/159033 objects misplaced (18.609%); 46 MiB/s, 737 objects/s recovering 2024-09-16T10:28:28.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-mon[32696]: osdmap e442: 8 total, 8 up, 8 in 2024-09-16T10:28:28.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-mon[32696]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 440'11816 2024-09-16T10:28:28.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-mon[32696]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 440'11971 2024-09-16T10:28:28.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:28:28.901 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:28:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:28:28.901 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:28:28.901 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:28:28.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:28.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:28.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:28.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:28:28.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[29993]: pgmap v801: 188 pgs: 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 111 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 563 op/s; 7045/159033 objects degraded (4.430%); 29595/159033 objects misplaced (18.609%); 46 MiB/s, 737 objects/s recovering 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[29993]: osdmap e442: 8 total, 8 up, 8 in 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[29993]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 440'11816 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[29993]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 440'11971 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[39470]: pgmap v801: 188 pgs: 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 3 active+clean+remapped, 1 active+remapped+backfill_wait, 41 active+undersized, 5 active+remapped+backfill_toofull, 22 active+undersized+degraded, 111 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 563 op/s; 7045/159033 objects degraded (4.430%); 29595/159033 objects misplaced (18.609%); 46 MiB/s, 737 objects/s recovering 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[39470]: osdmap e442: 8 total, 8 up, 8 in 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[39470]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 440'11816 2024-09-16T10:28:28.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[39470]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 440'11971 2024-09-16T10:28:28.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:28 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:28:29.050 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:29.081 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:29.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:29.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:29.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:28:28 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:28:29.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:29.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:29.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:29.350 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:29.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:29.452 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:29.505 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:29.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:29.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:29.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:29.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 440'12059 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 440'12849 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 440'12134 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: 10.1b continuing backfill to osd.3 from (397'7781,440'12896] MIN to 440'12896 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: 10.1b continuing backfill to osd.4 from (397'7781,440'12896] MIN to 440'12896 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 440'12896 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: 10.1c scrub starts 2024-09-16T10:28:29.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[29993]: osdmap e443: 8 total, 8 up, 8 in 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 440'12059 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 440'12849 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 440'12134 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: 10.1b continuing backfill to osd.3 from (397'7781,440'12896] MIN to 440'12896 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: 10.1b continuing backfill to osd.4 from (397'7781,440'12896] MIN to 440'12896 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 440'12896 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: 10.1c scrub starts 2024-09-16T10:28:29.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:29 smithi063 ceph-mon[39470]: osdmap e443: 8 total, 8 up, 8 in 2024-09-16T10:28:29.743 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:29.783 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:29.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:29.885 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:29.921 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:29.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:30.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:30.049 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:30.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 440'12059 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 440'12849 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 440'12134 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: 10.1b continuing backfill to osd.3 from (397'7781,440'12896] MIN to 440'12896 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: 10.1b continuing backfill to osd.4 from (397'7781,440'12896] MIN to 440'12896 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 440'12896 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: 10.1c scrub starts 2024-09-16T10:28:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:29 smithi165 ceph-mon[32696]: osdmap e443: 8 total, 8 up, 8 in 2024-09-16T10:28:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T10:28:30.121 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T10:28:30.151 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:30.175 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:30.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:30.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:30.276 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:30.317 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:30.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:30.419 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:30.458 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:30.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:30.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:30.602 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:30.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:30.704 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:30.744 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:30.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:30.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:30.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:30.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[29993]: 10.1e scrub starts 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[29993]: 10.c starting backfill to osd.7 from (372'1500,386'6282] MAX to 440'13086 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[29993]: pgmap v804: 188 pgs: 2 remapped+peering, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 12 active+undersized, 5 active+remapped+backfill_toofull, 8 active+undersized+degraded, 154 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 496 op/s; 4115/153420 objects degraded (2.682%); 27343/153420 objects misplaced (17.822%); 43 MiB/s, 683 objects/s recovering 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[39470]: 10.1e scrub starts 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[39470]: 10.c starting backfill to osd.7 from (372'1500,386'6282] MAX to 440'13086 2024-09-16T10:28:30.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:30 smithi063 ceph-mon[39470]: pgmap v804: 188 pgs: 2 remapped+peering, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 12 active+undersized, 5 active+remapped+backfill_toofull, 8 active+undersized+degraded, 154 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 496 op/s; 4115/153420 objects degraded (2.682%); 27343/153420 objects misplaced (17.822%); 43 MiB/s, 683 objects/s recovering 2024-09-16T10:28:30.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:31.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:31.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:31.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:30 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:28:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:30 smithi165 ceph-mon[32696]: 10.1e scrub starts 2024-09-16T10:28:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:30 smithi165 ceph-mon[32696]: 10.c starting backfill to osd.7 from (372'1500,386'6282] MAX to 440'13086 2024-09-16T10:28:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:30 smithi165 ceph-mon[32696]: pgmap v804: 188 pgs: 2 remapped+peering, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 12 active+undersized, 5 active+remapped+backfill_toofull, 8 active+undersized+degraded, 154 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 496 op/s; 4115/153420 objects degraded (2.682%); 27343/153420 objects misplaced (17.822%); 43 MiB/s, 683 objects/s recovering 2024-09-16T10:28:31.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:31.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:31.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:31.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:31.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:31.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:31.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:31.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:31.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:31.509 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:28:31.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:31.606 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:31.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:31.707 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:31.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:31.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:31.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:31.882 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:31.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:31.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:31 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 4115/153420 objects degraded (2.682%), 8 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:31.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:31 smithi063 ceph-mon[29993]: osdmap e444: 8 total, 8 up, 8 in 2024-09-16T10:28:31.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:31 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 4115/153420 objects degraded (2.682%), 8 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:31.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:31 smithi063 ceph-mon[39470]: osdmap e444: 8 total, 8 up, 8 in 2024-09-16T10:28:31.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:32.008 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:31 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 4115/153420 objects degraded (2.682%), 8 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T10:28:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:31 smithi165 ceph-mon[32696]: osdmap e444: 8 total, 8 up, 8 in 2024-09-16T10:28:32.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:32.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:28:32.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:32.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:32.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:32.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:32.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:32.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:32.543 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:32.645 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:32.664 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:32 smithi063 ceph-mon[29993]: pgmap v806: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 596 op/s; 3125/151245 objects degraded (2.066%); 23009/151245 objects misplaced (15.213%); 64 MiB/s, 1.02k objects/s recovering 2024-09-16T10:28:32.689 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:32.718 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:32.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:32.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:32.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:32 smithi063 ceph-mon[39470]: pgmap v806: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 596 op/s; 3125/151245 objects degraded (2.066%); 23009/151245 objects misplaced (15.213%); 64 MiB/s, 1.02k objects/s recovering 2024-09-16T10:28:32.952 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:32.986 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:32.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:33.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:32 smithi165 ceph-mon[32696]: pgmap v806: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 939 B/s rd, 0 B/s wr, 596 op/s; 3125/151245 objects degraded (2.066%); 23009/151245 objects misplaced (15.213%); 64 MiB/s, 1.02k objects/s recovering 2024-09-16T10:28:33.087 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:33.127 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:33.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:33.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:33.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:33.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:33.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:33.398 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:33.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:33.421 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:33.461 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T10:28:33.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:33.556 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:33.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:33.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:33.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:33.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:33.692 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:28:33.786 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:33.842 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:33.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:33.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:33.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3167736806' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:28:33.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:33 smithi063 ceph-mon[29993]: from='client.46244 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:28:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3167736806' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:28:33.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:33 smithi063 ceph-mon[39470]: from='client.46244 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:28:33.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:33.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3167736806' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:28:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:33 smithi165 ceph-mon[32696]: from='client.46244 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:28:34.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:34.108 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:34.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:34.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:34.249 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:34.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:34.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:34.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:34.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:34.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:34.535 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:34.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:34.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:34.666 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:34.666 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:34.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:34.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:34.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:34.788 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[29993]: pgmap v807: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 433 op/s; 3125/151245 objects degraded (2.066%); 23009/151245 objects misplaced (15.213%); 46 MiB/s, 743 objects/s recovering 2024-09-16T10:28:34.788 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:34.788 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[29993]: from='client.46244 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:28:34.788 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[29993]: osdmap e445: 8 total, 8 up, 8 in 2024-09-16T10:28:34.788 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[39470]: pgmap v807: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 433 op/s; 3125/151245 objects degraded (2.066%); 23009/151245 objects misplaced (15.213%); 46 MiB/s, 743 objects/s recovering 2024-09-16T10:28:34.789 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:34.789 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[39470]: from='client.46244 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:28:34.789 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:34 smithi063 ceph-mon[39470]: osdmap e445: 8 total, 8 up, 8 in 2024-09-16T10:28:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:34 smithi165 ceph-mon[32696]: pgmap v807: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 3 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 433 op/s; 3125/151245 objects degraded (2.066%); 23009/151245 objects misplaced (15.213%); 46 MiB/s, 743 objects/s recovering 2024-09-16T10:28:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:34 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:34 smithi165 ceph-mon[32696]: from='client.46244 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:28:34.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:34 smithi165 ceph-mon[32696]: osdmap e445: 8 total, 8 up, 8 in 2024-09-16T10:28:34.830 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:34.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:34.875 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:34.877 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:28:34.877 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:28:34.909 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:28:34.910 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:28:34.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:34.943 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:28:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:28:34.981 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:28:34.981 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:28:34.987 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:34.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:35.018 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:28:35.019 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:28:35.059 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:28:35.059 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:28:35.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:35.096 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:28:35.096 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:28:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:35.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:35.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:35.135 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:28:35.135 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:28:35.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:35.286 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:35.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:35.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:35.433 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:35.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:35.534 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:35.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:28:35.578 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:35.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:35.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:35.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:35.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:35.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1552713036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:35.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:35.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:35.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:35.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1552713036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1552713036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:35.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:35.983 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:36.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:36.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:36.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:36.243 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:36.265 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:36.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:36.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:36.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:36.514 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:36.534 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:28:36.617 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:36.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:36.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:36.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:36.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:36.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:36.945 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:36.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[29993]: pgmap v809: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 885 op/s; 5961/138330 objects degraded (4.309%); 21189/138330 objects misplaced (15.318%); 36 MiB/s, 572 objects/s recovering 2024-09-16T10:28:36.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 5961/138330 objects degraded (4.309%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[29993]: osdmap e446: 8 total, 8 up, 8 in 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3126665197' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[39470]: pgmap v809: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 885 op/s; 5961/138330 objects degraded (4.309%); 21189/138330 objects misplaced (15.318%); 36 MiB/s, 572 objects/s recovering 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 5961/138330 objects degraded (4.309%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[39470]: osdmap e446: 8 total, 8 up, 8 in 2024-09-16T10:28:36.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3126665197' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:28:37.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T10:28:37.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:37.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:36 smithi165 ceph-mon[32696]: pgmap v809: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 885 op/s; 5961/138330 objects degraded (4.309%); 21189/138330 objects misplaced (15.318%); 36 MiB/s, 572 objects/s recovering 2024-09-16T10:28:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:36 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:36 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 5961/138330 objects degraded (4.309%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:36 smithi165 ceph-mon[32696]: osdmap e446: 8 total, 8 up, 8 in 2024-09-16T10:28:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3126665197' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:28:37.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:37.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:37.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:37.295 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:28:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:28:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:28:37.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:37.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:37.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:37.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:37.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:37.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:37.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:37.811 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:37.852 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:37.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:37.995 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:37.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:38.075 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:28:38.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:38.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:38.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:38.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:38.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:38.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:38.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3126665197' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:28:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:38 smithi165 ceph-mon[32696]: osdmap e447: 8 total, 8 up, 8 in 2024-09-16T10:28:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:38 smithi165 ceph-mon[32696]: pgmap v812: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 718 op/s; 5961/138330 objects degraded (4.309%); 21189/138330 objects misplaced (15.318%); 15 MiB/s, 234 objects/s recovering 2024-09-16T10:28:38.318 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3126665197' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:28:38.318 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:38 smithi063 ceph-mon[29993]: osdmap e447: 8 total, 8 up, 8 in 2024-09-16T10:28:38.318 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:38 smithi063 ceph-mon[29993]: pgmap v812: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 718 op/s; 5961/138330 objects degraded (4.309%); 21189/138330 objects misplaced (15.318%); 15 MiB/s, 234 objects/s recovering 2024-09-16T10:28:38.318 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3126665197' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:28:38.318 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:38 smithi063 ceph-mon[39470]: osdmap e447: 8 total, 8 up, 8 in 2024-09-16T10:28:38.319 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:38 smithi063 ceph-mon[39470]: pgmap v812: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 718 op/s; 5961/138330 objects degraded (4.309%); 21189/138330 objects misplaced (15.318%); 15 MiB/s, 234 objects/s recovering 2024-09-16T10:28:38.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T10:28:38.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:38.488 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:38.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:38.516 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:38.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:38.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:38.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:38.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:38.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:38.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:38.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:38.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:38.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:38.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:39.015 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:28:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:28:39.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:39.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:39.067 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:224106 10.9 e447/442 10:95e9f3aa:::benchmark_data_smithi063_149043_object91669:head v 447'12297, mlcod=386'6101)", 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:39.066532+0000", 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0034401470000000002, 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034664779999999998, 2024-09-16T10:28:39.072 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066532+0000", 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066532+0000", 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:39.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066534+0000", 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3799999999999999e-06 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066541+0000", 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6380000000000004e-06 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:39.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066556+0000", 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4854e-05 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066564+0000", 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5410000000000003e-06 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066606+0000", 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2551e-05 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:39.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:39.066626+0000", 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9839000000000001e-05 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-16T10:28:39.076 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:28:39.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:39.177 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:39.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:39.279 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:39.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1934744434' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:39 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:39.322 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:39.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:39.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:39.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1934744434' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:39 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:39.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:39.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1934744434' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:28:39.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:39 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:39.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:39.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:39.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:39.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:39.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:39.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:39.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:39.862 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:39.903 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:39.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:39.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:40.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:40.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:28:40.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:40.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:40.151 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:40.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:40.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:40.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:40.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1934744434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[39470]: osdmap e448: 8 total, 8 up, 8 in 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[39470]: pgmap v814: 188 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s; 5961/127833 objects degraded (4.663%); 20031/127833 objects misplaced (15.670%); 9.9 MiB/s, 159 objects/s recovering 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1934744434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[29993]: osdmap e448: 8 total, 8 up, 8 in 2024-09-16T10:28:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:40 smithi063 ceph-mon[29993]: pgmap v814: 188 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s; 5961/127833 objects degraded (4.663%); 20031/127833 objects misplaced (15.670%); 9.9 MiB/s, 159 objects/s recovering 2024-09-16T10:28:40.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:40.306 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:40.306 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:40 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:28:40.306 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1934744434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:28:40.306 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:40 smithi165 ceph-mon[32696]: osdmap e448: 8 total, 8 up, 8 in 2024-09-16T10:28:40.306 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:40 smithi165 ceph-mon[32696]: pgmap v814: 188 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s; 5961/127833 objects degraded (4.663%); 20031/127833 objects misplaced (15.670%); 9.9 MiB/s, 159 objects/s recovering 2024-09-16T10:28:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:40.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:40.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:40.438 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:40.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:40.472 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:40.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:40.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:40.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:40.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:40.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:40.752 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:40.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:40.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:40.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:40.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:40.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:41.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:41.080 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:28:41.081 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:28:41.081 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:28:41.081 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:28:41.081 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:28:41.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:41.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:41.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[29993]: 10.18 scrub starts 2024-09-16T10:28:41.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[29993]: 10.19 scrub starts 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[29993]: 10.18 scrub ok 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[29993]: 10.d scrub starts 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[29993]: 10.19 scrub ok 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[29993]: 10.d scrub ok 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 5961/127833 objects degraded (4.663%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[39470]: 10.18 scrub starts 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[39470]: 10.19 scrub starts 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[39470]: 10.18 scrub ok 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[39470]: 10.d scrub starts 2024-09-16T10:28:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[39470]: 10.19 scrub ok 2024-09-16T10:28:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[39470]: 10.d scrub ok 2024-09-16T10:28:41.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:41 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 5961/127833 objects degraded (4.663%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:41.257 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:41.297 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:41 smithi165 ceph-mon[32696]: 10.18 scrub starts 2024-09-16T10:28:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:41 smithi165 ceph-mon[32696]: 10.19 scrub starts 2024-09-16T10:28:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:41 smithi165 ceph-mon[32696]: 10.18 scrub ok 2024-09-16T10:28:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:41 smithi165 ceph-mon[32696]: 10.d scrub starts 2024-09-16T10:28:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:41 smithi165 ceph-mon[32696]: 10.19 scrub ok 2024-09-16T10:28:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:41 smithi165 ceph-mon[32696]: 10.d scrub ok 2024-09-16T10:28:41.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:41 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 5961/127833 objects degraded (4.663%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:41.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:41.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:41.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:41.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:41.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:41.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:41.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:41.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:41.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:41.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:28:41.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:41.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:42.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:42.106 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:42.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:42.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:42.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:42.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:42 smithi165 ceph-mon[32696]: 10.f scrub starts 2024-09-16T10:28:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:42 smithi165 ceph-mon[32696]: pgmap v815: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 821 op/s; 5428/123552 objects degraded (4.393%); 18739/123552 objects misplaced (15.167%); 6.4 MiB/s, 125 objects/s recovering 2024-09-16T10:28:42.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:42 smithi165 ceph-mon[32696]: 10.f scrub ok 2024-09-16T10:28:42.363 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:42.401 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:42.402 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:28:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:42.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:42.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:42 smithi063 ceph-mon[29993]: 10.f scrub starts 2024-09-16T10:28:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:42 smithi063 ceph-mon[29993]: pgmap v815: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 821 op/s; 5428/123552 objects degraded (4.393%); 18739/123552 objects misplaced (15.167%); 6.4 MiB/s, 125 objects/s recovering 2024-09-16T10:28:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:42 smithi063 ceph-mon[29993]: 10.f scrub ok 2024-09-16T10:28:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:42 smithi063 ceph-mon[39470]: 10.f scrub starts 2024-09-16T10:28:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:42 smithi063 ceph-mon[39470]: pgmap v815: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 821 op/s; 5428/123552 objects degraded (4.393%); 18739/123552 objects misplaced (15.167%); 6.4 MiB/s, 125 objects/s recovering 2024-09-16T10:28:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:42 smithi063 ceph-mon[39470]: 10.f scrub ok 2024-09-16T10:28:42.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:42.531 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:42.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:42.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:42.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:42.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:42.772 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:42.812 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:42.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:42.914 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:42.954 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:42.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:43.055 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:28:43.055 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:215379 10.8 e443/442 10:154e210b:::benchmark_data_smithi063_149043_object82942:head v 443'11819, mlcod=388'6072)", 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:29.820980+0000", 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.231871937999999, 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038601530000000002, 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.820980+0000", 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.820980+0000", 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.820981+0000", 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1769999999999999e-06 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.820988+0000", 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.764e-06 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.821000+0000", 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.222e-05 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.821007+0000", 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6610000000000003e-06 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.824723+0000", 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037163109999999999 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.824841+0000", 2024-09-16T10:28:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117088 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.859539+0000", 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034698139000000003 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:29.859582+0000", 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3170000000000002e-05 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216464 10.11 e443/442 10:8bbe5995:::benchmark_data_smithi063_149043_object84027:head v 443'11994, mlcod=389'6270)", 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.058745468, 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180527202, 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994108+0000", 2024-09-16T10:28:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1179999999999999e-06 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994115+0000", 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7109999999999998e-06 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994124+0000", 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8120000000000002e-06 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994131+0000", 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2929999999999997e-06 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996650+0000", 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002519777 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996675+0000", 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4800999999999999e-05 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174595+0000", 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17792013200000001 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174634+0000", 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8557999999999999e-05 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(10.1 443/442 [PushOp(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head, version: 391'6545, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head@391'6545, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.056098095999999, 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19788486499999999, 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:28:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996756+0000", 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4810000000000001e-06 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996812+0000", 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6508999999999997e-05 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996831+0000", 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9009e-05 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996849+0000", 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7612999999999999e-05 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193170+0000", 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19632173999999999 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193181+0000", 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0138e-05 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.194639+0000", 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001458375 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216466 10.9 e443/442 10:96e6e3aa:::benchmark_data_smithi063_149043_object84029:head v 443'12076, mlcod=386'6101)", 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.055660619999999, 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18400087900000001, 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997193+0000", 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2759999999999999e-06 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997200+0000", 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8950000000000001e-06 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997209+0000", 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2010000000000005e-06 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997226+0000", 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7059e-05 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002158+0000", 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049319699999999999 2024-09-16T10:28:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002173+0000", 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4661000000000001e-05 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181097+0000", 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.178924214 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181192+0000", 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5602999999999999e-05 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216473 10.9 e443/442 10:9420f438:::benchmark_data_smithi063_149043_object84036:head v 443'12077, mlcod=386'6101)", 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.052030651000001, 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18039601599999999, 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000823+0000", 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.221e-06 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000830+0000", 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1319999999999996e-06 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000840+0000", 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0378e-05 2024-09-16T10:28:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000847+0000", 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4710000000000004e-06 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007645+0000", 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067982770000000001 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007677+0000", 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2172999999999998e-05 2024-09-16T10:28:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181195+0000", 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17351757300000001 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181218+0000", 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2790999999999999e-05 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216481 10.11 e443/442 10:8cd97c52:::benchmark_data_smithi063_149043_object84044:head v 443'11995, mlcod=389'6270)", 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.046872649000001, 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18177754400000001, 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005981+0000", 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.618e-06 2024-09-16T10:28:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005991+0000", 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4660000000000005e-06 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006004+0000", 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2964e-05 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006012+0000", 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5939999999999994e-06 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007635+0000", 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001622706 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007678+0000", 2024-09-16T10:28:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3282000000000003e-05 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187680+0000", 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180001892 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187757+0000", 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7021999999999995e-05 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216483 10.9 e443/442 10:91f62e6a:::benchmark_data_smithi063_149043_object84046:head v 443'12078, mlcod=386'6101)", 2024-09-16T10:28:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:28:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.045723991999999, 2024-09-16T10:28:43.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.185520875, 2024-09-16T10:28:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007129+0000", 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.198e-06 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007136+0000", 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7780000000000002e-06 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007158+0000", 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2013000000000001e-05 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007166+0000", 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999999e-06 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014305+0000", 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071398319999999996 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014326+0000", 2024-09-16T10:28:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0767999999999998e-05 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192615+0000", 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17828850800000001 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192649+0000", 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4366999999999998e-05 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216472 10.8 e443/442 10:167e16b6:::benchmark_data_smithi063_149043_object84035:head v 443'11858, mlcod=389'6215)", 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.044067605, 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.167553278, 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008786+0000", 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.296e-06 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008820+0000", 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4329999999999998e-05 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008830+0000", 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3360000000000001e-06 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008836+0000", 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3040000000000001e-06 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010938+0000", 2024-09-16T10:28:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021016780000000001 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010959+0000", 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1454000000000002e-05 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176290+0000", 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16533061800000001 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176338+0000", 2024-09-16T10:28:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8262e-05 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216489 10.c e443/442 10:36cb7a7c:::benchmark_data_smithi063_149043_object84052:head v 443'13145, mlcod=443'13143)", 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.040200993999999, 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16334574800000001, 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012653+0000", 2024-09-16T10:28:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.606e-06 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012661+0000", 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0469999999999994e-06 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012673+0000", 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2490000000000001e-05 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012682+0000", 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3450000000000006e-06 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012753+0000", 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1565999999999999e-05 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012773+0000", 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9942999999999999e-05 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175914+0000", 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16314114399999999 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175997+0000", 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2607000000000004e-05 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216490 10.10 e443/441 10:0ed1dccc:::benchmark_data_smithi063_149043_object84053:head v 443'12109, mlcod=443'12107)", 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.038179058000001, 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16562529300000001, 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014675+0000", 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.593e-06 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014687+0000", 2024-09-16T10:28:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2071e-05 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014697+0000", 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0148e-05 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014704+0000", 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.613e-06 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174625+0000", 2024-09-16T10:28:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15992183700000001 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6707e-05 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180235+0000", 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055831880000000002 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180298+0000", 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3136e-05 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216488 10.11 e443/442 10:8a4eba15:::benchmark_data_smithi063_149043_object84051:head v 443'11996, mlcod=389'6270)", 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.037247983, 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.177066683, 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015606+0000", 2024-09-16T10:28:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.404e-06 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015617+0000", 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1769e-05 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015628+0000", 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0489000000000001e-05 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015634+0000", 2024-09-16T10:28:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2960000000000004e-06 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174659+0000", 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15902495699999999 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174681+0000", 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1778999999999999e-05 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192652+0000", 2024-09-16T10:28:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017971311 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192671+0000", 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8678e-05 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216491 10.10 e443/441 10:0e46f635:::benchmark_data_smithi063_149043_object84054:head v 443'12110, mlcod=443'12107)", 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.035561632, 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.163036928, 2024-09-16T10:28:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017292+0000", 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.443e-06 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017300+0000", 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174635+0000", 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.157335014 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174657+0000", 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1985e-05 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176330+0000", 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016733029999999999 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176359+0000", 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9564e-05 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180301+0000", 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00394174 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180328+0000", 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6319e-05 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216495 10.c e443/442 10:33a8a0ac:::benchmark_data_smithi063_149043_object84058:head v 443'13146, mlcod=443'13143)", 2024-09-16T10:28:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.034266637, 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.160016309, 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018587+0000", 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3820000000000001e-06 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018595+0000", 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4759999999999998e-06 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174544+0000", 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15594872200000001 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174566+0000", 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1979000000000001e-05 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174626+0000", 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9899999999999999e-05 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.561e-05 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178550+0000", 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038981969999999999 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178602+0000", 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2043000000000003e-05 2024-09-16T10:28:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:219406 10.8 e445/442 10:1608708f:::benchmark_data_smithi063_149043_object86969:head v 445'11959, mlcod=391'6527)", 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:34.292357+0000", 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.7604956909999991, 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045351198000000002, 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292357+0000", 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292357+0000", 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292357+0000", 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9800000000000003e-07 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292363+0000", 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4859999999999997e-06 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292370+0000", 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7900000000000002e-06 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292375+0000", 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0259999999999998e-06 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292428+0000", 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3118999999999997e-05 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292449+0000", 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1552999999999998e-05 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.337628+0000", 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045179143999999997 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.337708+0000", 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9282000000000004e-05 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:219417 10.8 e445/442 10:161e4172:::benchmark_data_smithi063_149043_object86980:head v 445'11960, mlcod=391'6527)", 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:34.292739+0000", 2024-09-16T10:28:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.7601132620000008, 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045097152000000001, 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292739+0000", 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292739+0000", 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292740+0000", 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.048e-06 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292746+0000", 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5459999999999999e-06 2024-09-16T10:28:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292757+0000", 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0964e-05 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.292762+0000", 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6810000000000003e-06 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.299709+0000", 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069464619999999996 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.299732+0000", 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3598e-05 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.337762+0000", 2024-09-16T10:28:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038030099999999997 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.337836+0000", 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3752999999999997e-05 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:219421 10.8 e445/442 10:10e3a521:::benchmark_data_smithi063_149043_object86984:head v 445'11961, mlcod=391'6527)", 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:34.298321+0000", 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.7545311169999991, 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039581442000000001, 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.298321+0000", 2024-09-16T10:28:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.298321+0000", 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.298323+0000", 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.818e-06 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.298331+0000", 2024-09-16T10:28:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8820000000000001e-06 2024-09-16T10:28:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.298341+0000", 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0086e-05 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.298348+0000", 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1930000000000004e-06 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.305082+0000", 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067343079999999996 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.305107+0000", 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4712e-05 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.337843+0000", 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032735589000000002 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:34.337903+0000", 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9854e-05 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221476 10.9 e446/442 10:900b709d:::benchmark_data_smithi063_149043_object89039:head v 446'12225, mlcod=386'6101)", 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.9742034400000001, 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50937916299999997, 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078650+0000", 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.091e-06 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078657+0000", 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078665+0000", 2024-09-16T10:28:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6619999999999994e-06 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078671+0000", 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1700000000000002e-06 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078762+0000", 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0346999999999994e-05 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078786+0000", 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4314000000000001e-05 2024-09-16T10:28:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587999+0000", 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.509212514 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.588028+0000", 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9326999999999999e-05 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221483 10.c e446/442 10:32ed840b:::benchmark_data_smithi063_149043_object89046:head v 446'13312, mlcod=445'13310)", 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.9734743540000004, 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50397308900000004, 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079379+0000", 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.395e-06 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079385+0000", 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5639999999999996e-06 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079392+0000", 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.481e-06 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079398+0000", 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8950000000000003e-06 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079450+0000", 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1786999999999997e-05 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079469+0000", 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8841000000000001e-05 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583301+0000", 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50383221600000005 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583351+0000", 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9910000000000002e-05 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221482 10.1f e446/441 10:ffb425b8:::benchmark_data_smithi063_149043_object89045:head v 446'13070, mlcod=445'13068)", 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.9732669009999997, 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50798407899999998, 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079587+0000", 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.187e-06 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079593+0000", 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0929999999999999e-06 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079600+0000", 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5070000000000003e-06 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079606+0000", 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7509999999999997e-06 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583349+0000", 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50374310499999997 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583378+0000", 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8649e-05 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587530+0000", 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041521559999999997 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587569+0000", 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9631000000000001e-05 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(10.1 447/442 [PushOp(10:8159d612:::benchmark_data_smithi063_149043_object129605:head, version: 398'8006, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:8159d612:::benchmark_data_smithi063_149043_object129605:head@398'8006, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:37.159449+0000", 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.8934030240000004, 2024-09-16T10:28:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.042268498000000002, 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.159449+0000", 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.159449+0000", 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:28:43.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.159450+0000", 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6300000000000004e-07 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.159507+0000", 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7426999999999998e-05 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.192677+0000", 2024-09-16T10:28:43.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033169555000000003 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.192728+0000", 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1193000000000003e-05 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.199651+0000", 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069230480000000002 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.199664+0000", 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2439000000000001e-05 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:37.201718+0000", 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002054073 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:28:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:28:43.150 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:28:43.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:43.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:43.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:43.298 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:43.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:43.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:43 smithi165 ceph-mon[32696]: 10.1a scrub starts 2024-09-16T10:28:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:43 smithi165 ceph-mon[32696]: 10.1a scrub ok 2024-09-16T10:28:43.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:43.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:43.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:43 smithi063 ceph-mon[29993]: 10.1a scrub starts 2024-09-16T10:28:43.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:43 smithi063 ceph-mon[29993]: 10.1a scrub ok 2024-09-16T10:28:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:43 smithi063 ceph-mon[39470]: 10.1a scrub starts 2024-09-16T10:28:43.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:43 smithi063 ceph-mon[39470]: 10.1a scrub ok 2024-09-16T10:28:43.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:43.562 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:43.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:43.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:43.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:43.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:43.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:43.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:43.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:43.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:43.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:43.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:44.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:44.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:44.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:44.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:28:44.211 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:44.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:44.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:44 smithi165 ceph-mon[32696]: 10.a scrub starts 2024-09-16T10:28:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:44 smithi165 ceph-mon[32696]: pgmap v816: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 667 op/s; 5428/123552 objects degraded (4.393%); 18739/123552 objects misplaced (15.167%); 5.2 MiB/s, 102 objects/s recovering 2024-09-16T10:28:44.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:44 smithi165 ceph-mon[32696]: 10.a scrub ok 2024-09-16T10:28:44.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:44.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:44.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:44 smithi063 ceph-mon[29993]: 10.a scrub starts 2024-09-16T10:28:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:44 smithi063 ceph-mon[29993]: pgmap v816: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 667 op/s; 5428/123552 objects degraded (4.393%); 18739/123552 objects misplaced (15.167%); 5.2 MiB/s, 102 objects/s recovering 2024-09-16T10:28:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:44 smithi063 ceph-mon[29993]: 10.a scrub ok 2024-09-16T10:28:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:44 smithi063 ceph-mon[39470]: 10.a scrub starts 2024-09-16T10:28:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:44 smithi063 ceph-mon[39470]: pgmap v816: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 6 active+remapped+backfill_toofull, 172 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 667 op/s; 5428/123552 objects degraded (4.393%); 18739/123552 objects misplaced (15.167%); 5.2 MiB/s, 102 objects/s recovering 2024-09-16T10:28:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:44 smithi063 ceph-mon[39470]: 10.a scrub ok 2024-09-16T10:28:44.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:44.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:44.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:44.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:44.666 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:44.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:44.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:44.806 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:44.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:44.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:44.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:44.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:45.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:45.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:45.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:45.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:45.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:45.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:45.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:45.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:45.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:45.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:45 smithi063 ceph-mon[29993]: 10.e scrub starts 2024-09-16T10:28:45.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:45 smithi063 ceph-mon[29993]: 10.e scrub ok 2024-09-16T10:28:45.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:45 smithi063 ceph-mon[39470]: 10.e scrub starts 2024-09-16T10:28:45.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:45 smithi063 ceph-mon[39470]: 10.e scrub ok 2024-09-16T10:28:45.486 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:45.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:45.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:45 smithi165 ceph-mon[32696]: 10.e scrub starts 2024-09-16T10:28:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:45 smithi165 ceph-mon[32696]: 10.e scrub ok 2024-09-16T10:28:45.589 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:45.618 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:45.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:45.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:45.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:45.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:45.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:45.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:45.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:45.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:46.116 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.156 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:46.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:46.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:46.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:46.382 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:46.425 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:46.445 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:28:46.446 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:28:46.447 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:28:46.447 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:28:46.447 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:28:46.447 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:28:46.447 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:46 smithi063 ceph-mon[29993]: pgmap v817: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 171 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 5093/107496 objects degraded (4.738%); 15632/107496 objects misplaced (14.542%) 2024-09-16T10:28:46.447 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1396934105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:46.447 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:46 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 5428/123552 objects degraded (4.393%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:46.448 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:46 smithi063 ceph-mon[39470]: pgmap v817: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 171 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 5093/107496 objects degraded (4.738%); 15632/107496 objects misplaced (14.542%) 2024-09-16T10:28:46.448 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1396934105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:46.448 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:46 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 5428/123552 objects degraded (4.393%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:46.527 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:46.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:46 smithi165 ceph-mon[32696]: pgmap v817: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 171 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 5093/107496 objects degraded (4.738%); 15632/107496 objects misplaced (14.542%) 2024-09-16T10:28:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1396934105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:28:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:46 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 5428/123552 objects degraded (4.393%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:46.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:46.654 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:46.695 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:46.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:46.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:46.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:46.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:46.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:47.063 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:47.114 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:47.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:47.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:47.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:47.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:47.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:47.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:47.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:47.389 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:47 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:47.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:47 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:47.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:28:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:28:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:28:47.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:47 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:47.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:47.500 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:47.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:47.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:47.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:47.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:47.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:47.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:47.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:47.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:47.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:48.007 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:48.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:48.144 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:48.167 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:48.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:48.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:48.313 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:48.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:48.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:48.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:48.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:48.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:48 smithi063 ceph-mon[29993]: pgmap v818: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 171 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 920 B/s rd, 0 op/s; 5093/107496 objects degraded (4.738%); 15632/107496 objects misplaced (14.542%) 2024-09-16T10:28:48.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:48 smithi063 ceph-mon[39470]: pgmap v818: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 171 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 920 B/s rd, 0 op/s; 5093/107496 objects degraded (4.738%); 15632/107496 objects misplaced (14.542%) 2024-09-16T10:28:48.551 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:48 smithi165 ceph-mon[32696]: pgmap v818: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 171 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 920 B/s rd, 0 op/s; 5093/107496 objects degraded (4.738%); 15632/107496 objects misplaced (14.542%) 2024-09-16T10:28:48.592 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:48.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:48.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:48.717 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:48.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:48.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:48.858 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:48.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:48.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:49.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:49.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:49.180 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:49.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:49.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:28:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:28:49.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:49.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:49.439 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:49.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:49.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:49.707 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:49.809 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:49.831 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:49.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:49.958 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:49.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:50.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:50.082 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:50.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:50.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:50.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:50.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:50.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:50.357 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:50.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:50.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:50.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:50 smithi063 ceph-mon[29993]: osdmap e449: 8 total, 8 up, 8 in 2024-09-16T10:28:50.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:50 smithi063 ceph-mon[29993]: pgmap v820: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 5093/96684 objects degraded (5.268%); 13953/96684 objects misplaced (14.432%) 2024-09-16T10:28:50.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:50.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:50 smithi063 ceph-mon[39470]: osdmap e449: 8 total, 8 up, 8 in 2024-09-16T10:28:50.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:50 smithi063 ceph-mon[39470]: pgmap v820: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 5093/96684 objects degraded (5.268%); 13953/96684 objects misplaced (14.432%) 2024-09-16T10:28:50.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:50.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:50.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:50.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:28:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:50 smithi165 ceph-mon[32696]: osdmap e449: 8 total, 8 up, 8 in 2024-09-16T10:28:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:50 smithi165 ceph-mon[32696]: pgmap v820: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 5093/96684 objects degraded (5.268%); 13953/96684 objects misplaced (14.432%) 2024-09-16T10:28:50.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:50.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:50.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:50.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:50.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:50.774 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:50.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:50.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:50.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:50.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:51.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:51.040 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:51.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:51.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:51.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:51.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:51.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:51.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:51.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:51.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:51.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[29993]: osdmap e450: 8 total, 8 up, 8 in 2024-09-16T10:28:51.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 5093/96684 objects degraded (5.268%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[39470]: osdmap e450: 8 total, 8 up, 8 in 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:28:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:51 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 5093/96684 objects degraded (5.268%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:51.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:51.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:51 smithi165 ceph-mon[32696]: osdmap e450: 8 total, 8 up, 8 in 2024-09-16T10:28:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:51 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:28:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:51 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:28:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:51 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:28:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:51 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:28:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:51 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 5093/96684 objects degraded (5.268%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:51.567 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:51.606 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:51.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:51.708 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:51.746 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:51.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:51.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:51.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:51.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:51.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:52.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:52.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:52.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:52.155 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:52.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:52.257 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:52.299 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:52.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:52.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:52.427 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[29993]: 10.1d scrub starts 2024-09-16T10:28:52.427 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[29993]: osdmap e451: 8 total, 8 up, 8 in 2024-09-16T10:28:52.427 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[29993]: 10.1d scrub ok 2024-09-16T10:28:52.427 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[29993]: pgmap v823: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 4739/92634 objects degraded (5.116%); 12763/92634 objects misplaced (13.778%); 11 MiB/s, 301 objects/s recovering 2024-09-16T10:28:52.427 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[39470]: 10.1d scrub starts 2024-09-16T10:28:52.427 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[39470]: osdmap e451: 8 total, 8 up, 8 in 2024-09-16T10:28:52.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[39470]: 10.1d scrub ok 2024-09-16T10:28:52.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:52 smithi063 ceph-mon[39470]: pgmap v823: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 4739/92634 objects degraded (5.116%); 12763/92634 objects misplaced (13.778%); 11 MiB/s, 301 objects/s recovering 2024-09-16T10:28:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:52.441 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:52.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:52.528 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:52.542 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:52.542 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:52.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:52 smithi165 ceph-mon[32696]: 10.1d scrub starts 2024-09-16T10:28:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:52 smithi165 ceph-mon[32696]: osdmap e451: 8 total, 8 up, 8 in 2024-09-16T10:28:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:52 smithi165 ceph-mon[32696]: 10.1d scrub ok 2024-09-16T10:28:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:52 smithi165 ceph-mon[32696]: pgmap v823: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 4739/92634 objects degraded (5.116%); 12763/92634 objects misplaced (13.778%); 11 MiB/s, 301 objects/s recovering 2024-09-16T10:28:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:52.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:52.709 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:52.750 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:52.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:52.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:52.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:52.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:52.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:53.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:53.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:53.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:53.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:53.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:53.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:53.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:53.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:53.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:53.651 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:53.673 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:53.775 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:53.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:53.935 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:53.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:54.015 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:28:54.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:54.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:54.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:54.170 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:54.176 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T10:28:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:54.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:54.218 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:54.218 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:54.318 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:54.380 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:54.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:54.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: 10.21 scrub starts 2024-09-16T10:28:54.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: 10.21 scrub ok 2024-09-16T10:28:54.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: 10.20 scrub starts 2024-09-16T10:28:54.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: 10.20 scrub ok 2024-09-16T10:28:54.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: osdmap e452: 8 total, 8 up, 8 in 2024-09-16T10:28:54.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: pgmap v825: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 490 op/s; 4739/92634 objects degraded (5.116%); 12763/92634 objects misplaced (13.778%); 3.3 MiB/s, 152 objects/s recovering 2024-09-16T10:28:54.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3680288285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: 10.21 scrub starts 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: 10.21 scrub ok 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: 10.20 scrub starts 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: 10.20 scrub ok 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: osdmap e452: 8 total, 8 up, 8 in 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: pgmap v825: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 490 op/s; 4739/92634 objects degraded (5.116%); 12763/92634 objects misplaced (13.778%); 3.3 MiB/s, 152 objects/s recovering 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:54.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3680288285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:28:54.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:54.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:28:54.523 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:54.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: 10.21 scrub starts 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: 10.21 scrub ok 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: 10.20 scrub starts 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: 10.20 scrub ok 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: osdmap e452: 8 total, 8 up, 8 in 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: pgmap v825: 188 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 172 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 490 op/s; 4739/92634 objects degraded (5.116%); 12763/92634 objects misplaced (13.778%); 3.3 MiB/s, 152 objects/s recovering 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:28:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3680288285' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:28:54.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:54.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:54.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:54.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:54.806 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:54.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:54.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:54.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:54.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:54.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:28:55.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:55.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:55.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:55.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:55.174 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:55.178 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[29993]: 10.17 scrub starts 2024-09-16T10:28:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:55.224 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:55.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:55.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:55.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:55.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:55.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[39470]: 10.17 scrub starts 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[39470]: 10.17 scrub ok 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3680288285' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[39470]: osdmap e453: 8 total, 8 up, 8 in 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[29993]: 10.17 scrub ok 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3680288285' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[29993]: osdmap e453: 8 total, 8 up, 8 in 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:28:55.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:28:55.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:55.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:55.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:55 smithi165 ceph-mon[32696]: 10.17 scrub starts 2024-09-16T10:28:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:55 smithi165 ceph-mon[32696]: 10.17 scrub ok 2024-09-16T10:28:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:55 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3680288285' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:28:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:55 smithi165 ceph-mon[32696]: osdmap e453: 8 total, 8 up, 8 in 2024-09-16T10:28:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:28:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:28:55.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:55.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:55.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:55.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:55.794 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:55.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:55.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:55.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:55.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:56.037 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:56.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T10:28:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:56.115 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:56.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:56.186 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:56.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:56.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:56.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:56.333 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[29993]: 10.1 scrub starts 2024-09-16T10:28:56.333 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[29993]: 10.1 scrub ok 2024-09-16T10:28:56.333 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[29993]: osdmap e454: 8 total, 8 up, 8 in 2024-09-16T10:28:56.333 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[29993]: pgmap v828: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 1759/77217 objects degraded (2.278%); 10447/77217 objects misplaced (13.529%) 2024-09-16T10:28:56.333 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 4739/92634 objects degraded (5.116%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:56.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[39470]: 10.1 scrub starts 2024-09-16T10:28:56.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[39470]: 10.1 scrub ok 2024-09-16T10:28:56.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[39470]: osdmap e454: 8 total, 8 up, 8 in 2024-09-16T10:28:56.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[39470]: pgmap v828: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 1759/77217 objects degraded (2.278%); 10447/77217 objects misplaced (13.529%) 2024-09-16T10:28:56.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:56 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 4739/92634 objects degraded (5.116%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:56.345 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:56.375 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:56.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:56.477 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:56.527 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:56.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:56.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:56.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:56 smithi165 ceph-mon[32696]: 10.1 scrub starts 2024-09-16T10:28:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:56 smithi165 ceph-mon[32696]: 10.1 scrub ok 2024-09-16T10:28:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:56 smithi165 ceph-mon[32696]: osdmap e454: 8 total, 8 up, 8 in 2024-09-16T10:28:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:56 smithi165 ceph-mon[32696]: pgmap v828: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 1759/77217 objects degraded (2.278%); 10447/77217 objects misplaced (13.529%) 2024-09-16T10:28:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:56 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 4739/92634 objects degraded (5.116%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T10:28:56.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:56.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:56.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:56.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:56.806 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:56.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:56.878 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:28:56.908 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:56.961 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:56.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:57.062 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:57.110 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:57.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:57.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:57.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:57.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:57.332 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:57 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:57.332 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:28:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:28:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:28:57.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:57 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:57.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:57.405 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:57.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:57.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:57.514 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:28:57.514 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:28:57.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:57 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:28:57.559 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:57.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:57.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:57.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:57.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:57.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:28:57.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:57.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:28:57.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:57.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:57.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:58.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:58.106 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:58.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:58.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:58.232 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:28:58.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:58.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:58.275 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:28:58.348 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:58.393 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:58.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:58.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:28:58.541 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:58.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:28:58.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:58 smithi165 ceph-mon[32696]: pgmap v829: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 174 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 1759/77037 objects degraded (2.283%); 10447/77037 objects misplaced (13.561%) 2024-09-16T10:28:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:58 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3364106331' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:28:58.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:58.695 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:58.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:58.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:58 smithi063 ceph-mon[29993]: pgmap v829: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 174 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 1759/77037 objects degraded (2.283%); 10447/77037 objects misplaced (13.561%) 2024-09-16T10:28:58.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:58 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3364106331' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:28:58.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:58 smithi063 ceph-mon[39470]: pgmap v829: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 174 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 1759/77037 objects degraded (2.283%); 10447/77037 objects misplaced (13.561%) 2024-09-16T10:28:58.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:58 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3364106331' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:28:58.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:28:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:58.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:58.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:28:58.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:28:58.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:58.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:28:59.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:59.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:59.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:59.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:59.234 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:59.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:59 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3364106331' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:28:59.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:28:59 smithi165 ceph-mon[32696]: osdmap e455: 8 total, 8 up, 8 in 2024-09-16T10:28:59.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:28:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:28:59.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:28:59.380 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:59.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:28:59.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:28:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:59.504 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:59.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:28:59.605 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:28:59.647 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:28:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:59.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:28:59.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:59 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:59.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3364106331' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:28:59.679 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:28:59 smithi063 ceph-mon[29993]: osdmap e455: 8 total, 8 up, 8 in 2024-09-16T10:28:59.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:59 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:28:59.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3364106331' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:28:59.679 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:28:59 smithi063 ceph-mon[39470]: osdmap e455: 8 total, 8 up, 8 in 2024-09-16T10:28:59.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:59.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:59.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:28:59.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:28:59.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:28:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:28:59.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:00.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:00.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:00.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:00.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:00.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:00.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:00.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:00.343 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:00.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:00.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:00.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:00.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:00 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:29:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:00 smithi165 ceph-mon[32696]: 10.1c scrub starts 2024-09-16T10:29:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:00 smithi165 ceph-mon[32696]: pgmap v831: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 174 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1434/65085 objects degraded (2.203%); 9589/65085 objects misplaced (14.733%) 2024-09-16T10:29:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:00 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:29:00.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:00.626 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:00.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:00.638 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:29:00.638 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:00.638 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[29993]: 10.1c scrub starts 2024-09-16T10:29:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[29993]: pgmap v831: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 174 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1434/65085 objects degraded (2.203%); 9589/65085 objects misplaced (14.733%) 2024-09-16T10:29:00.639 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:29:00.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:29:00.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:00.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[39470]: 10.1c scrub starts 2024-09-16T10:29:00.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[39470]: pgmap v831: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 174 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1434/65085 objects degraded (2.203%); 9589/65085 objects misplaced (14.733%) 2024-09-16T10:29:00.639 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:00 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:29:00.727 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:00.762 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:00.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:00.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:00.903 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:00.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:01.004 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.053 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:01.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:01.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:01.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:01.302 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:01.342 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:01.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:01.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:01.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:01 smithi063 ceph-mon[29993]: 10.1e scrub starts 2024-09-16T10:29:01.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:01 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 1434/65085 objects degraded (2.203%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:01.551 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:01 smithi063 ceph-mon[29993]: osdmap e456: 8 total, 8 up, 8 in 2024-09-16T10:29:01.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:01 smithi063 ceph-mon[39470]: 10.1e scrub starts 2024-09-16T10:29:01.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:01 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 1434/65085 objects degraded (2.203%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:01.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:01 smithi063 ceph-mon[39470]: osdmap e456: 8 total, 8 up, 8 in 2024-09-16T10:29:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:01 smithi165 ceph-mon[32696]: 10.1e scrub starts 2024-09-16T10:29:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:01 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 1434/65085 objects degraded (2.203%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:01 smithi165 ceph-mon[32696]: osdmap e456: 8 total, 8 up, 8 in 2024-09-16T10:29:01.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.604 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:01.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:01.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:01.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:29:01.735 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:01.837 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:01.861 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:01.963 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:01.988 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:01.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:02.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:02.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:02.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:02.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:02.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:02.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:02.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:02.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:02.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:02.509 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:02 smithi063 ceph-mon[29993]: pgmap v833: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 296 op/s; 1274/60654 objects degraded (2.100%); 7481/60654 objects misplaced (12.334%); 2.3 MiB/s, 27 objects/s recovering 2024-09-16T10:29:02.510 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:02 smithi063 ceph-mon[29993]: osdmap e457: 8 total, 8 up, 8 in 2024-09-16T10:29:02.510 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:02 smithi063 ceph-mon[39470]: pgmap v833: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 296 op/s; 1274/60654 objects degraded (2.100%); 7481/60654 objects misplaced (12.334%); 2.3 MiB/s, 27 objects/s recovering 2024-09-16T10:29:02.510 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:02 smithi063 ceph-mon[39470]: osdmap e457: 8 total, 8 up, 8 in 2024-09-16T10:29:02.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:02.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:02 smithi165 ceph-mon[32696]: pgmap v833: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 296 op/s; 1274/60654 objects degraded (2.100%); 7481/60654 objects misplaced (12.334%); 2.3 MiB/s, 27 objects/s recovering 2024-09-16T10:29:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:02 smithi165 ceph-mon[32696]: osdmap e457: 8 total, 8 up, 8 in 2024-09-16T10:29:02.628 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:02.673 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:02.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:02.775 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:02.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:02.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:02.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:02.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:02.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:03.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:03.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:03.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:03.093 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:29:03.093 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:29:03.093 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:29:03.093 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:29:03.093 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:29:03.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:29:03.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:03.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:03.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:03.332 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:03.377 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:03.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:03 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2071827344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:03 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2071827344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:03.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:03.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:03.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:03 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2071827344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:03.605 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:03.648 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:03.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:03.749 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:03.790 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:03.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:03.834 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:03.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:03.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:03.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:04.031 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:04.082 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:04.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:04.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:04.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:04.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:04.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:04.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:04.365 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:04.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:04.453 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:04.466 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:04.533 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:04.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:04 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:04 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:04 smithi165 ceph-mon[32696]: 10.16 scrub starts 2024-09-16T10:29:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:04 smithi165 ceph-mon[32696]: osdmap e458: 8 total, 8 up, 8 in 2024-09-16T10:29:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:04 smithi165 ceph-mon[32696]: pgmap v836: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 986 B/s rd, 0 B/s wr, 1.05k op/s; 1274/60654 objects degraded (2.100%); 7481/60654 objects misplaced (12.334%); 9.6 MiB/s, 312 objects/s recovering 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[29993]: 10.16 scrub starts 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[29993]: osdmap e458: 8 total, 8 up, 8 in 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[29993]: pgmap v836: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 986 B/s rd, 0 B/s wr, 1.05k op/s; 1274/60654 objects degraded (2.100%); 7481/60654 objects misplaced (12.334%); 9.6 MiB/s, 312 objects/s recovering 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:04.568 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[39470]: 10.16 scrub starts 2024-09-16T10:29:04.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[39470]: osdmap e458: 8 total, 8 up, 8 in 2024-09-16T10:29:04.569 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:04 smithi063 ceph-mon[39470]: pgmap v836: 188 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 173 active+clean; 1.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 986 B/s rd, 0 B/s wr, 1.05k op/s; 1274/60654 objects degraded (2.100%); 7481/60654 objects misplaced (12.334%); 9.6 MiB/s, 312 objects/s recovering 2024-09-16T10:29:04.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:04.680 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:04.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:04.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:04.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:04.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:04.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:04.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:04.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:05.044 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:05.089 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:05.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:05.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:05.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:05.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:05.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:05.350 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:05.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:05.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:05.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:05.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:05.588 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:05.627 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:05.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:05.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:05.776 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:05.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:05.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:05.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:05.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:06.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:06.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:06.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:06.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:06.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:06.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:06.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:06.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:06.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:06.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:06.485 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:06.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:06.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:29:06.558 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:06.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:06.594 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:29:06.594 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:29:06.594 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:29:06.594 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9:dc100d6c:::benchmark_data_smithi063_124717_object60003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69542607, 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1260944720000001, 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:29:06.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881601+0000", 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.841e-06 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881621+0000", 2024-09-16T10:29:06.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9976e-05 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881629+0000", 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2120000000000003e-06 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881636+0000", 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.472e-06 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881700+0000", 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4152000000000005e-05 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881707+0000", 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882746+0000", 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001038574 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882755+0000", 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0480000000000004e-06 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007615+0000", 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.124860408 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.600 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007694+0000", 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8479000000000005e-05 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9:d2f9f966:::benchmark_data_smithi063_124717_object60060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69537894600001, 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1652373410000001, 2024-09-16T10:29:06.601 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:29:06.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881647+0000", 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7599999999999997e-07 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881671+0000", 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4284e-05 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881677+0000", 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3619999999999997e-06 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881681+0000", 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1980000000000003e-06 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881840+0000", 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015907000000000001 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting_for_map not empty", 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881846+0000", 2024-09-16T10:29:06.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6250000000000004e-06 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883410+0000", 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001564085 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883417+0000", 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4749999999999996e-06 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008717+0000", 2024-09-16T10:29:06.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.125299746 2024-09-16T10:29:06.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008819+0000", 2024-09-16T10:29:06.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102545 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008934+0000", 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000114923 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.608 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044911+0000", 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035976248000000002 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044965+0000", 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4756000000000002e-05 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045041+0000", 2024-09-16T10:29:06.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5828000000000007e-05 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045130+0000", 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9221000000000005e-05 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045181+0000", 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0691e-05 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046884+0000", 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001702708 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9:de195d7b:::benchmark_data_smithi063_124717_object60066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.611 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69532987700001, 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1671491920000001, 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.612 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5400000000000001e-07 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881718+0000", 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1948999999999999e-05 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881724+0000", 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7080000000000002e-06 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881728+0000", 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.946e-06 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883442+0000", 2024-09-16T10:29:06.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00171441 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883449+0000", 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4309999999999999e-06 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010618+0000", 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1271692849999999 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010714+0000", 2024-09-16T10:29:06.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6600000000000003e-05 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010840+0000", 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012543600000000001 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044920+0000", 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034079854 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045083+0000", 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163688 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047061+0000", 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019780269999999998 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047155+0000", 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3529999999999994e-05 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047228+0000", 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2822000000000002e-05 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048845+0000", 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016169520000000001 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9:d28fbaa4:::benchmark_data_smithi063_124717_object60144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69527318299998, 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.171973098, 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881753+0000", 2024-09-16T10:29:06.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8699999999999995e-07 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881772+0000", 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9159000000000001e-05 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881779+0000", 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.362e-06 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881784+0000", 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2580000000000003e-06 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883478+0000", 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016937180000000001 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883485+0000", 2024-09-16T10:29:06.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3000000000000004e-06 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012549+0000", 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.129063785 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012705+0000", 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015551399999999999 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012901+0000", 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019668299999999999 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046970+0000", 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034068909000000001 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.625 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047098+0000", 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012766899999999999 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048912+0000", 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001813797 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051278+0000", 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023664480000000002 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051344+0000", 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5415999999999998e-05 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053725+0000", 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002381593 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9:dfbc90fc:::benchmark_data_smithi063_124717_object60211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69523015700003, 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1744191719999999, 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881796+0000", 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3499999999999996e-07 2024-09-16T10:29:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881881+0000", 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5458999999999996e-05 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881889+0000", 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9060000000000002e-06 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881896+0000", 2024-09-16T10:29:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7830000000000001e-06 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883507+0000", 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016115249999999999 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883514+0000", 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-06 2024-09-16T10:29:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.014943+0000", 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1314289719999999 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015122+0000", 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000178982 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015285+0000", 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016376499999999999 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046978+0000", 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031692310000000001 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051202+0000", 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042246860000000001 2024-09-16T10:29:06.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053812+0000", 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026092720000000002 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053866+0000", 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4876000000000001e-05 2024-09-16T10:29:06.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053932+0000", 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5314000000000005e-05 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056214+0000", 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002282634 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9:dcccd893:::benchmark_data_smithi063_124717_object60240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69512091399997, 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2232440419999999, 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.638 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881905+0000", 2024-09-16T10:29:06.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2900000000000003e-07 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881928+0000", 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2747000000000001e-05 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881935+0000", 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3150000000000003e-06 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881941+0000", 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1639999999999998e-06 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883538+0000", 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015963939999999999 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883543+0000", 2024-09-16T10:29:06.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6540000000000004e-06 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056263+0000", 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.172719834 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056408+0000", 2024-09-16T10:29:06.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014513800000000001 2024-09-16T10:29:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056584+0000", 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017557299999999999 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064153+0000", 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0075692820000000001 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068713+0000", 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045601620000000004 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068767+0000", 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3881000000000001e-05 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101726+0000", 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032958901999999998 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101782+0000", 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5609999999999998e-05 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105149+0000", 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033666569999999999 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9:dbba527e:::benchmark_data_smithi063_124717_object60254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.695076599, 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2264516890000001, 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:29:06.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1399999999999997e-07 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881971+0000", 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1461999999999999e-05 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881979+0000", 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.259e-06 2024-09-16T10:29:06.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881985+0000", 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7320000000000003e-06 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883562+0000", 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001577171 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883568+0000", 2024-09-16T10:29:06.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7910000000000001e-06 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059019+0000", 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1754510929999999 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059188+0000", 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016928700000000001 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059360+0000", 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017208100000000001 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068663+0000", 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0093023089999999999 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068831+0000", 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016804999999999999 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082474+0000", 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013643597 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105299+0000", 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.022824816000000001 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105363+0000", 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4088e-05 2024-09-16T10:29:06.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108400+0000", 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003037339 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9:ddb5c386:::benchmark_data_smithi063_124717_object60260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69503340199998, 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2293183249999999, 2024-09-16T10:29:06.655 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:29:06.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881993+0000", 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9999999999999997e-07 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882015+0000", 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1946e-05 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882021+0000", 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9430000000000003e-06 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882028+0000", 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7599999999999997e-06 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883589+0000", 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001560875 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883595+0000", 2024-09-16T10:29:06.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0719999999999996e-06 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061597+0000", 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1780015660000001 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061781+0000", 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000183909 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.062032+0000", 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025166899999999999 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068729+0000", 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0066966940000000004 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068881+0000", 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000151965 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101683+0000", 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032802338 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108512+0000", 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0068289229999999998 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108574+0000", 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1630000000000005e-05 2024-09-16T10:29:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.111310+0000", 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027364350000000002 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60273 9.b 9:d6970cdd:::benchmark_data_smithi063_124717_object60272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69498961400001, 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2328018119999999, 2024-09-16T10:29:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60273 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3499999999999996e-07 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882058+0000", 2024-09-16T10:29:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1236000000000001e-05 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882064+0000", 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5230000000000001e-06 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882070+0000", 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4670000000000003e-06 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883625+0000", 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015556820000000001 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883632+0000", 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6610000000000003e-06 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064190+0000", 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1805575690000001 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064363+0000", 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000173124 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064567+0000", 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020472000000000001 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068929+0000", 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043613879999999999 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076738+0000", 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0078089300000000004 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105256+0000", 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.028517793 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112019+0000", 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067630110000000002 2024-09-16T10:29:06.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112083+0000", 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4938999999999996e-05 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114838+0000", 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027541340000000001 2024-09-16T10:29:06.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60317 9.b 9:d4f738e0:::benchmark_data_smithi063_124717_object60316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69494887600001, 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.235504929, 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60317 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:29:06.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5799999999999999e-07 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882098+0000", 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0758e-05 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882104+0000", 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3239999999999999e-06 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882110+0000", 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5879999999999997e-06 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883724+0000", 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016140670000000001 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883731+0000", 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068967+0000", 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1852353390000001 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069080+0000", 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113442 2024-09-16T10:29:06.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069174+0000", 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3623999999999997e-05 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079562+0000", 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010388029 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082423+0000", 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028609740000000001 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114907+0000", 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032484065999999999 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114978+0000", 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0919000000000003e-05 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.115036+0000", 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7878999999999999e-05 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117581+0000", 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025458019999999998 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.678 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60326 9.b 9:d0ff592b:::benchmark_data_smithi063_124717_object60325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69490241800003, 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2387679700000001, 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60326 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882124+0000", 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.99e-07 2024-09-16T10:29:06.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882145+0000", 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1650000000000001e-05 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882151+0000", 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2110000000000003e-06 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882158+0000", 2024-09-16T10:29:06.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6669999999999999e-06 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883760+0000", 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001601781 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883766+0000", 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4019999999999999e-06 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071354+0000", 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1875881050000001 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071501+0000", 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014622300000000001 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071673+0000", 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000172518 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082372+0000", 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010698874000000001 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082515+0000", 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000143409 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117647+0000", 2024-09-16T10:29:06.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035131837999999999 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117792+0000", 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014443299999999999 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117866+0000", 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4274000000000001e-05 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120891+0000", 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030249859999999999 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60333 9.b 9:d83577cf:::benchmark_data_smithi063_124717_object60332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.694860135, 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2414839360000001, 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.686 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60333 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882166+0000", 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0100000000000005e-07 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882188+0000", 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2252999999999999e-05 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882221+0000", 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2459999999999998e-05 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882227+0000", 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.015e-06 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883790+0000", 2024-09-16T10:29:06.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015635759999999999 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883797+0000", 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.567e-06 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076778+0000", 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1929816879999999 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076959+0000", 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000180607 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.077150+0000", 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019081600000000001 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082436+0000", 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0052863240000000002 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082555+0000", 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119305 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120947+0000", 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.038391740000000001 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120989+0000", 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2107999999999997e-05 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.121049+0000", 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9888e-05 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123649+0000", 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025999880000000001 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60362 9.b 9:d2c32f51:::benchmark_data_smithi063_124717_object60361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69478901399998, 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.244097462, 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60362 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.694 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.695 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882237+0000", 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0399999999999995e-07 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882261+0000", 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4269000000000001e-05 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882268+0000", 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8619999999999997e-06 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882274+0000", 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1829999999999999e-06 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883834+0000", 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015597930000000001 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883840+0000", 2024-09-16T10:29:06.697 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.251e-06 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079597+0000", 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1957566589999999 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079755+0000", 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015786399999999999 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.080192+0000", 2024-09-16T10:29:06.698 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00043717099999999998 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105209+0000", 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.025016842000000001 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114928+0000", 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0097189919999999992 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.699 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123709+0000", 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0087810240000000001 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123762+0000", 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2567e-05 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123828+0000", 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6799999999999997e-05 2024-09-16T10:29:06.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126334+0000", 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025054809999999999 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60366 9.b 9:d4ec99e6:::benchmark_data_smithi063_124717_object60365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:29:06.701 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69474351500003, 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.246752562, 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60366 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:29:06.702 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7899999999999998e-07 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.703 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883494+0000", 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012115959999999999 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883505+0000", 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.111e-05 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883513+0000", 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4320000000000002e-06 2024-09-16T10:29:06.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883866+0000", 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00035366599999999998 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883873+0000", 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2910000000000004e-06 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099038+0000", 2024-09-16T10:29:06.705 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2151655379999999 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099212+0000", 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017424000000000001 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099404+0000", 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000191366 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.706 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108466+0000", 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0090625159999999996 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117602+0000", 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0091353030000000009 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126413+0000", 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088110150000000002 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.707 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126572+0000", 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015944599999999999 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126628+0000", 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5819999999999997e-05 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129034+0000", 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024066439999999999 2024-09-16T10:29:06.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60423 9.b 9:db831613:::benchmark_data_smithi063_124717_object60422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.69347906199999, 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2882874870000001, 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.709 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60423 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883548+0000", 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.206e-06 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883603+0000", 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5036e-05 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.711 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883614+0000", 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1352000000000001e-05 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883624+0000", 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0519999999999999e-05 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883899+0000", 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027465699999999998 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.712 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883905+0000", 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3119999999999999e-06 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129084+0000", 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.245178672 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129268+0000", 2024-09-16T10:29:06.713 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018391999999999999 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129423+0000", 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015478000000000001 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136532+0000", 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071096270000000003 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.714 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136582+0000", 2024-09-16T10:29:06.715 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9962e-05 2024-09-16T10:29:06.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:06.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.152752+0000", 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016169444000000002 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168726+0000", 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015974559999999999 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.716 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168790+0000", 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3474999999999998e-05 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.171834+0000", 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030439640000000001 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60452 9.b 9:d848878f:::benchmark_data_smithi063_124717_object60451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 390.63451058999999, 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.233084683, 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60452 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.718 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.719 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942516+0000", 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5969999999999999e-06 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942536+0000", 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9970999999999999e-05 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942548+0000", 2024-09-16T10:29:06.720 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2057e-05 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942556+0000", 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0350000000000001e-06 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942605+0000", 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8770999999999998e-05 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.721 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942611+0000", 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1040000000000003e-06 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131546+0000", 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.188934811 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131704+0000", 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015802399999999999 2024-09-16T10:29:06.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131894+0000", 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018940499999999999 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136598+0000", 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047045409999999996 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136646+0000", 2024-09-16T10:29:06.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8204999999999997e-05 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168552+0000", 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031905880999999997 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172759+0000", 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042071809999999999 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172950+0000", 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019102799999999999 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.175599+0000", 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026490720000000001 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:88281 10.c 10:356f8e40:::benchmark_data_smithi063_149043_object88280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:29:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 159.03973837699999, 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.020604785, 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 88281 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537289+0000", 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3520000000000001e-06 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537316+0000", 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7038000000000001e-05 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537335+0000", 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8247e-05 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537343+0000", 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7239999999999998e-06 2024-09-16T10:29:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543029+0000", 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056852450000000002 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543036+0000", 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2749999999999998e-06 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.559029+0000", 2024-09-16T10:29:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015993265 2024-09-16T10:29:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.559039+0000", 2024-09-16T10:29:06.730 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:06 smithi063 ceph-mon[29993]: pgmap v837: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 174 active+clean; 934 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 690/45408 objects degraded (1.520%); 5339/45408 objects misplaced (11.758%); 10 MiB/s, 390 objects/s recovering 2024-09-16T10:29:06.730 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:06 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 690/45408 objects degraded (1.520%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:06.730 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:06 smithi063 ceph-mon[39470]: pgmap v837: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 174 active+clean; 934 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 690/45408 objects degraded (1.520%); 5339/45408 objects misplaced (11.758%); 10 MiB/s, 390 objects/s recovering 2024-09-16T10:29:06.731 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:06 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 690/45408 objects degraded (1.520%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0185e-05 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.556377+0000", 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99733790600000005 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.557892+0000", 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015145479999999999 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:88285 10.c 10:348a9cd0:::benchmark_data_smithi063_149043_object88284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 159.03966941499999, 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0211189380000001, 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 88285 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537357+0000", 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9500000000000002e-07 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537424+0000", 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7220999999999999e-05 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537459+0000", 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5497999999999998e-05 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537467+0000", 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6450000000000005e-06 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543085+0000", 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056183730000000003 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting_for_map not empty", 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543092+0000", 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2899999999999999e-06 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.562892+0000", 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.019800512999999999 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.562904+0000", 2024-09-16T10:29:06.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1368e-05 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.558397+0000", 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99549375600000001 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.558475+0000", 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7578999999999997e-05 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.737 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:98186 10.c 10:30a2e491:::benchmark_data_smithi063_149043_object98185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 149.60896546699999, 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0526488009999999, 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:06.738 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 98186 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968062+0000", 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.928e-06 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968086+0000", 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3671000000000001e-05 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968096+0000", 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0839000000000001e-05 2024-09-16T10:29:06.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968104+0000", 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8890000000000002e-06 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.974858+0000", 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067535399999999997 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.975006+0000", 2024-09-16T10:29:06.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014837499999999999 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.975142+0000", 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013551899999999999 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.986786+0000", 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011644408 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.742 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.992120+0000", 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0053334860000000001 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.012439+0000", 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.020319265999999999 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.012449+0000", 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0223e-05 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020605+0000", 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0081562230000001 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020709+0000", 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103434 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:98194 10.c 10:3f81be78:::benchmark_data_smithi063_149043_object98193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 149.59896504100001, 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.050357231, 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 98194 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.745 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978063+0000", 2024-09-16T10:29:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4260000000000002e-06 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978090+0000", 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7475e-05 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978103+0000", 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2544000000000001e-05 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978110+0000", 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0060000000000003e-06 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980486+0000", 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023762700000000002 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980605+0000", 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011847 2024-09-16T10:29:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980747+0000", 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014267800000000001 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.992126+0000", 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011379142 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.015722+0000", 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.023596004 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.015734+0000", 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2058999999999999e-05 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020985+0000", 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005250813 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.021617+0000", 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00063140899999999999 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.028349+0000", 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.006732839 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.028418+0000", 2024-09-16T10:29:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8096000000000007e-05 2024-09-16T10:29:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:06.752 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:29:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:06 smithi165 ceph-mon[32696]: pgmap v837: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 174 active+clean; 934 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 690/45408 objects degraded (1.520%); 5339/45408 objects misplaced (11.758%); 10 MiB/s, 390 objects/s recovering 2024-09-16T10:29:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:06 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 690/45408 objects degraded (1.520%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:06.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:06.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:06.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:06.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:06.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:06.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:07.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:07.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:07.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:07.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:07.274 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:07.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:07.376 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:07.421 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:07.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:07.425 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:29:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:29:07.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:07.530 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T10:29:07.577 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:07.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:07.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:07.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:07.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:07.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[29993]: osdmap e459: 8 total, 8 up, 8 in 2024-09-16T10:29:07.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2276132137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:29:07.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[29993]: from='client.46280 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:29:07.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:07.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:07.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[39470]: osdmap e459: 8 total, 8 up, 8 in 2024-09-16T10:29:07.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2276132137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:29:07.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:07 smithi063 ceph-mon[39470]: from='client.46280 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:29:07.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:07.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:07 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:07 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:07 smithi165 ceph-mon[32696]: osdmap e459: 8 total, 8 up, 8 in 2024-09-16T10:29:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:07 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2276132137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:29:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:07 smithi165 ceph-mon[32696]: from='client.46280 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:29:07.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:07.853 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:07.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:07.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:07.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:07.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:08.088 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:08.113 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:08.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:08.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:08.244 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:08.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:08.257 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:29:08.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:08.371 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:08.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:08.472 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:08.497 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:08.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:08.535 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:29:08.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:08.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:08.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:08.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:08.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:08.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:08.794 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[29993]: pgmap v839: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 928 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 690/45138 objects degraded (1.529%); 5339/45138 objects misplaced (11.828%); 5.0 MiB/s, 260 objects/s recovering 2024-09-16T10:29:08.794 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[29993]: from='client.46280 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:29:08.795 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[29993]: osdmap e460: 8 total, 8 up, 8 in 2024-09-16T10:29:08.795 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/927409308' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:08.795 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[29993]: from='client.46283 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:08.796 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[39470]: pgmap v839: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 928 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 690/45138 objects degraded (1.529%); 5339/45138 objects misplaced (11.828%); 5.0 MiB/s, 260 objects/s recovering 2024-09-16T10:29:08.796 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[39470]: from='client.46280 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:29:08.796 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[39470]: osdmap e460: 8 total, 8 up, 8 in 2024-09-16T10:29:08.796 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/927409308' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:08.796 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:08 smithi063 ceph-mon[39470]: from='client.46283 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:08.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:08 smithi165 ceph-mon[32696]: pgmap v839: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 928 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 690/45138 objects degraded (1.529%); 5339/45138 objects misplaced (11.828%); 5.0 MiB/s, 260 objects/s recovering 2024-09-16T10:29:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:08 smithi165 ceph-mon[32696]: from='client.46280 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:29:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:08 smithi165 ceph-mon[32696]: osdmap e460: 8 total, 8 up, 8 in 2024-09-16T10:29:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:08 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/927409308' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:08 smithi165 ceph-mon[32696]: from='client.46283 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:08.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:08.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:08.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:09.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:09.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:09.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:09.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:09.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:09.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:09.235 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T10:29:09.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:09.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:29:09 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:29:09.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:09.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:09.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:09.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:09.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:09.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:09.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:09.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:09.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:09.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:09.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:09.773 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:09.774 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[39470]: from='client.46283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:09.774 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[39470]: osdmap e461: 8 total, 8 up, 8 in 2024-09-16T10:29:09.774 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:09.774 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:09.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:09.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[29993]: from='client.46283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:09.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[29993]: osdmap e461: 8 total, 8 up, 8 in 2024-09-16T10:29:09.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:09.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:09.784 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:09.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:09 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:09 smithi165 ceph-mon[32696]: from='client.46283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:09 smithi165 ceph-mon[32696]: osdmap e461: 8 total, 8 up, 8 in 2024-09-16T10:29:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:09.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:09.918 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:09.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:10.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:10.067 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:10.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:10.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:10.216 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:10.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:10.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:10.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:10.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:10.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:10.499 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:10.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:10.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:10.654 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:10.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:10.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:10.798 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:10.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:10.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:10 smithi165 ceph-mon[32696]: 10.11 scrub starts 2024-09-16T10:29:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:10 smithi165 ceph-mon[32696]: pgmap v842: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 700 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.47k op/s; 622/34173 objects degraded (1.820%); 4582/34173 objects misplaced (13.408%); 5.7 MiB/s, 397 objects/s recovering 2024-09-16T10:29:10.825 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:10.826 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:10 smithi063 ceph-mon[29993]: 10.11 scrub starts 2024-09-16T10:29:10.826 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:10 smithi063 ceph-mon[29993]: pgmap v842: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 700 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.47k op/s; 622/34173 objects degraded (1.820%); 4582/34173 objects misplaced (13.408%); 5.7 MiB/s, 397 objects/s recovering 2024-09-16T10:29:10.826 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:10.826 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:10 smithi063 ceph-mon[39470]: 10.11 scrub starts 2024-09-16T10:29:10.826 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:10 smithi063 ceph-mon[39470]: pgmap v842: 188 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 700 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.47k op/s; 622/34173 objects degraded (1.820%); 4582/34173 objects misplaced (13.408%); 5.7 MiB/s, 397 objects/s recovering 2024-09-16T10:29:10.842 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:29:10.842 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:29:10.889 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:29:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:29:10.900 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:10.923 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:29:10.924 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:29:10.940 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:10.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:10.953 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:29:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:29:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.953 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.16152.0:257337 10.1d e461/449 10:bacce873:::benchmark_data_smithi063_149043_object124900:head v 461'14395, mlcod=461'14392)", 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:10.857867+0000", 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.012421781, 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012450904, 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857867+0000", 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857867+0000", 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857868+0000", 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3650000000000001e-06 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:10.955 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857878+0000", 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0047000000000001e-05 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857893+0000", 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4533e-05 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857901+0000", 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9039999999999992e-06 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857970+0000", 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8695999999999994e-05 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.857990+0000", 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0080000000000001e-05 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.957 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.16152.0:257340 10.15 e461/414 10:aa335a21:::benchmark_data_smithi063_149043_object124903:head v 461'13076, mlcod=461'13073)", 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:10.858469+0000", 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.011819774999999999, 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011905773, 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858469+0000", 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.958 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858469+0000", 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858470+0000", 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.164e-06 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858477+0000", 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8499999999999996e-06 2024-09-16T10:29:10.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858499+0000", 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2497000000000002e-05 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858508+0000", 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2470000000000008e-06 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858882+0000", 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000374459 2024-09-16T10:29:10.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.858903+0000", 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0482000000000001e-05 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:257350 10.1f 10:fe2c5e2c:::benchmark_data_smithi063_149043_object124913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:10.863620+0000", 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0066687630000000003, 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0068073600000000001, 2024-09-16T10:29:10.961 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 257350 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863620+0000", 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863620+0000", 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863622+0000", 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.401e-06 2024-09-16T10:29:10.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863629+0000", 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0099999999999998e-06 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863650+0000", 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0874e-05 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863659+0000", 2024-09-16T10:29:10.964 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9020000000000005e-06 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863727+0000", 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8003000000000006e-05 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.863873+0000", 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014629500000000001 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,3,7", 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.864260+0000", 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00038687500000000001 2024-09-16T10:29:10.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:10.868412+0000", 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041519499999999997 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 3 2024-09-16T10:29:10.966 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:29:10.967 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:29:10.967 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:29:11.002 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:29:11.003 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:29:11.041 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:29:11.041 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:29:11.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:11.078 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:29:11.078 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:29:11.083 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:11.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:11.114 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:29:11.115 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:29:11.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:11.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:11.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:11.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:11.354 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:11.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:11.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:11.475 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:11.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:11.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:11.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:11.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:11.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:11.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:11.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:11.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/336324147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:11 smithi165 ceph-mon[32696]: osdmap e462: 8 total, 8 up, 8 in 2024-09-16T10:29:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:11 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 622/34173 objects degraded (1.820%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:11.841 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:11.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:29:11.868 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:11.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:11.894 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/336324147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:11.894 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:11 smithi063 ceph-mon[29993]: osdmap e462: 8 total, 8 up, 8 in 2024-09-16T10:29:11.894 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:11 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 622/34173 objects degraded (1.820%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:11.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/336324147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:11.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:11 smithi063 ceph-mon[39470]: osdmap e462: 8 total, 8 up, 8 in 2024-09-16T10:29:11.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:11 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 622/34173 objects degraded (1.820%), 2 pgs degraded (PG_DEGRADED) 2024-09-16T10:29:11.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:29:11.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:11.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:12.101 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:12.141 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:12.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:12.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:12.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:12.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:12.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:12.530 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:12.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:12 smithi063 ceph-mon[29993]: pgmap v844: 188 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 623 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 B/s wr, 992 op/s; 261/30498 objects degraded (0.856%); 3664/30498 objects misplaced (12.014%); 4.1 MiB/s, 302 objects/s recovering 2024-09-16T10:29:12.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:12 smithi063 ceph-mon[29993]: osdmap e463: 8 total, 8 up, 8 in 2024-09-16T10:29:12.619 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:12 smithi063 ceph-mon[39470]: pgmap v844: 188 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 623 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 B/s wr, 992 op/s; 261/30498 objects degraded (0.856%); 3664/30498 objects misplaced (12.014%); 4.1 MiB/s, 302 objects/s recovering 2024-09-16T10:29:12.619 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:12 smithi063 ceph-mon[39470]: osdmap e463: 8 total, 8 up, 8 in 2024-09-16T10:29:12.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:12.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:12.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:12.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:12.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:12 smithi165 ceph-mon[32696]: pgmap v844: 188 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 175 active+clean; 623 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 B/s wr, 992 op/s; 261/30498 objects degraded (0.856%); 3664/30498 objects misplaced (12.014%); 4.1 MiB/s, 302 objects/s recovering 2024-09-16T10:29:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:12 smithi165 ceph-mon[32696]: osdmap e463: 8 total, 8 up, 8 in 2024-09-16T10:29:12.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:12.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:12.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:12.931 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:12.972 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:13.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:13.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:13.147 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:13.186 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:13.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:13.319 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:13.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:13.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:13.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:13.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:13.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:13.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:13.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:13.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:13.822 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:13.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:13.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:13.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:13.990 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T10:29:14.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:14.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:14.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:29:14.186 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:14.225 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:14.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:13 smithi165 ceph-mon[32696]: 10.c scrub starts 2024-09-16T10:29:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:13 smithi165 ceph-mon[32696]: osdmap e464: 8 total, 8 up, 8 in 2024-09-16T10:29:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1908415778' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:29:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:13 smithi165 ceph-mon[32696]: from='client.46292 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:29:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:13 smithi165 ceph-mon[32696]: pgmap v847: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 177 active+clean; 552 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 943 op/s; 261/27072 objects degraded (0.964%); 1823/27072 objects misplaced (6.734%); 12 MiB/s, 460 objects/s recovering 2024-09-16T10:29:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3856434296' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:29:14.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:14.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:14.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:14.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:29:14.434 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[39470]: 10.c scrub starts 2024-09-16T10:29:14.550 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[39470]: osdmap e464: 8 total, 8 up, 8 in 2024-09-16T10:29:14.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1908415778' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:29:14.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[39470]: from='client.46292 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:29:14.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[39470]: pgmap v847: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 177 active+clean; 552 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 943 op/s; 261/27072 objects degraded (0.964%); 1823/27072 objects misplaced (6.734%); 12 MiB/s, 460 objects/s recovering 2024-09-16T10:29:14.551 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3856434296' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:29:14.552 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[29993]: 10.c scrub starts 2024-09-16T10:29:14.552 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[29993]: osdmap e464: 8 total, 8 up, 8 in 2024-09-16T10:29:14.552 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1908415778' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:29:14.552 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[29993]: from='client.46292 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:29:14.552 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[29993]: pgmap v847: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 177 active+clean; 552 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 943 op/s; 261/27072 objects degraded (0.964%); 1823/27072 objects misplaced (6.734%); 12 MiB/s, 460 objects/s recovering 2024-09-16T10:29:14.553 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3856434296' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:29:14.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:14.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:14.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:14.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:14.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:14.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:14.841 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:14.883 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:14.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:14.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:15.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:15.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:15.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:15.254 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:15.289 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: 10.1b scrub starts 2024-09-16T10:29:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:29:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: from='client.46292 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:29:15.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3856434296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:29:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: osdmap e465: 8 total, 8 up, 8 in 2024-09-16T10:29:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: 10.23 scrub starts 2024-09-16T10:29:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: 10.23 scrub ok 2024-09-16T10:29:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: 10.1f scrub starts 2024-09-16T10:29:15.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:14 smithi165 ceph-mon[32696]: 10.1f scrub ok 2024-09-16T10:29:15.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:15.429 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: 10.1b scrub starts 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: from='client.46292 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3856434296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: osdmap e465: 8 total, 8 up, 8 in 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: 10.23 scrub starts 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: 10.23 scrub ok 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: 10.1f scrub starts 2024-09-16T10:29:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[29993]: 10.1f scrub ok 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: 10.1b scrub starts 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: from='client.46292 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3856434296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: osdmap e465: 8 total, 8 up, 8 in 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: 10.23 scrub starts 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: 10.23 scrub ok 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: 10.1f scrub starts 2024-09-16T10:29:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:14 smithi063 ceph-mon[39470]: 10.1f scrub ok 2024-09-16T10:29:15.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:15.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:15.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:15.709 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:15.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:15.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:15.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:15.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:15.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:16.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:16.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:16.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:16.227 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:16.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:16.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:15 smithi165 ceph-mon[32696]: 10.22 scrub starts 2024-09-16T10:29:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:15 smithi165 ceph-mon[32696]: 10.22 scrub ok 2024-09-16T10:29:16.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:15 smithi165 ceph-mon[32696]: pgmap v849: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 179 active+clean; 403 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 792 op/s; 236/19917 objects degraded (1.185%); 1175/19917 objects misplaced (5.899%); 14 MiB/s, 637 objects/s recovering 2024-09-16T10:29:16.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:16.413 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:16.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:16.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:15 smithi063 ceph-mon[29993]: 10.22 scrub starts 2024-09-16T10:29:16.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:15 smithi063 ceph-mon[29993]: 10.22 scrub ok 2024-09-16T10:29:16.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:15 smithi063 ceph-mon[29993]: pgmap v849: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 179 active+clean; 403 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 792 op/s; 236/19917 objects degraded (1.185%); 1175/19917 objects misplaced (5.899%); 14 MiB/s, 637 objects/s recovering 2024-09-16T10:29:16.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:15 smithi063 ceph-mon[39470]: 10.22 scrub starts 2024-09-16T10:29:16.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:15 smithi063 ceph-mon[39470]: 10.22 scrub ok 2024-09-16T10:29:16.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:15 smithi063 ceph-mon[39470]: pgmap v849: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 179 active+clean; 403 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 792 op/s; 236/19917 objects degraded (1.185%); 1175/19917 objects misplaced (5.899%); 14 MiB/s, 637 objects/s recovering 2024-09-16T10:29:16.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:16.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:16.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:16.652 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:16.693 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:16.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:16.795 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:16.834 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:16.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:16.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:16.977 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:16.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:17.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:17.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:17.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:17.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:17.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:17.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:17 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 236/19917 objects degraded (1.185%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:29:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:17 smithi165 ceph-mon[32696]: osdmap e466: 8 total, 8 up, 8 in 2024-09-16T10:29:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:17 smithi165 ceph-mon[32696]: 6.1e scrub starts 2024-09-16T10:29:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:17 smithi165 ceph-mon[32696]: 6.1e scrub ok 2024-09-16T10:29:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:17 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:17.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:17 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:17.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:17.399 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:17.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:29:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:29:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:29:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 236/19917 objects degraded (1.185%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:29:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[39470]: osdmap e466: 8 total, 8 up, 8 in 2024-09-16T10:29:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[39470]: 6.1e scrub starts 2024-09-16T10:29:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[39470]: 6.1e scrub ok 2024-09-16T10:29:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 236/19917 objects degraded (1.185%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:29:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[29993]: osdmap e466: 8 total, 8 up, 8 in 2024-09-16T10:29:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[29993]: 6.1e scrub starts 2024-09-16T10:29:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[29993]: 6.1e scrub ok 2024-09-16T10:29:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:16 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:17.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:17.538 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:17.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:17.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:17.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:17.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:17.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:17.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:17.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:17.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:17.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:17.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:18.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:18.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:18.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:18.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:18.225 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:18.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:18.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:18.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:18.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:18.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:18 smithi063 ceph-mon[29993]: osdmap e467: 8 total, 8 up, 8 in 2024-09-16T10:29:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:18 smithi063 ceph-mon[29993]: pgmap v852: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 179 active+clean; 399 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 805 op/s; 236/19749 objects degraded (1.195%); 1175/19749 objects misplaced (5.950%); 14 MiB/s, 637 objects/s recovering 2024-09-16T10:29:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:18 smithi063 ceph-mon[39470]: osdmap e467: 8 total, 8 up, 8 in 2024-09-16T10:29:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:18 smithi063 ceph-mon[39470]: pgmap v852: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 179 active+clean; 399 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 805 op/s; 236/19749 objects degraded (1.195%); 1175/19749 objects misplaced (5.950%); 14 MiB/s, 637 objects/s recovering 2024-09-16T10:29:18.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:18.509 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:18.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:18 smithi165 ceph-mon[32696]: osdmap e467: 8 total, 8 up, 8 in 2024-09-16T10:29:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:18 smithi165 ceph-mon[32696]: pgmap v852: 188 pgs: 1 active+recovering+undersized+remapped, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 179 active+clean; 399 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 805 op/s; 236/19749 objects degraded (1.195%); 1175/19749 objects misplaced (5.950%); 14 MiB/s, 637 objects/s recovering 2024-09-16T10:29:18.611 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:18.648 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:18.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:18.750 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:18.764 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Removed 132434 objects 2024-09-16T10:29:18.764 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Clean up completed and total clean up time :132.052 2024-09-16T10:29:18.777 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:18.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:18.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:18.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:18.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:18.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:29:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:29:19.004 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:19.029 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:19.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:19.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:19.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:19.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:19 smithi165 ceph-mon[32696]: 4.17 scrub starts 2024-09-16T10:29:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:19 smithi165 ceph-mon[32696]: 4.17 scrub ok 2024-09-16T10:29:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:19 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:29:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:19 smithi165 ceph-mon[32696]: 4.5 scrub starts 2024-09-16T10:29:19.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:19 smithi165 ceph-mon[32696]: 4.5 scrub ok 2024-09-16T10:29:19.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:29:19 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:29:19.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:19.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[29993]: 4.17 scrub starts 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[29993]: 4.17 scrub ok 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[29993]: 4.5 scrub starts 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[29993]: 4.5 scrub ok 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[39470]: 4.17 scrub starts 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[39470]: 4.17 scrub ok 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[39470]: 4.5 scrub starts 2024-09-16T10:29:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:19 smithi063 ceph-mon[39470]: 4.5 scrub ok 2024-09-16T10:29:19.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:19.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:19.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:19.656 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:19.758 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:19.785 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:19.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:19.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:19.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:20.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:20.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:20.139 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:20.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:20.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:20.291 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:20.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:20.417 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:20.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[29993]: osdmap e468: 8 total, 8 up, 8 in 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[29993]: 5.1e scrub starts 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[29993]: 5.1e scrub ok 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[29993]: pgmap v854: 188 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 183 active+clean; 104 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 939 B/s rd, 0 op/s; 236/5568 objects degraded (4.239%); 849/5568 objects misplaced (15.248%) 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[29993]: 4.12 scrub starts 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[29993]: 4.12 scrub ok 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:20.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[39470]: osdmap e468: 8 total, 8 up, 8 in 2024-09-16T10:29:20.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[39470]: 5.1e scrub starts 2024-09-16T10:29:20.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[39470]: 5.1e scrub ok 2024-09-16T10:29:20.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[39470]: pgmap v854: 188 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 183 active+clean; 104 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 939 B/s rd, 0 op/s; 236/5568 objects degraded (4.239%); 849/5568 objects misplaced (15.248%) 2024-09-16T10:29:20.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[39470]: 4.12 scrub starts 2024-09-16T10:29:20.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:20 smithi063 ceph-mon[39470]: 4.12 scrub ok 2024-09-16T10:29:20.519 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:20.545 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:20 smithi165 ceph-mon[32696]: osdmap e468: 8 total, 8 up, 8 in 2024-09-16T10:29:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:20 smithi165 ceph-mon[32696]: 5.1e scrub starts 2024-09-16T10:29:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:20 smithi165 ceph-mon[32696]: 5.1e scrub ok 2024-09-16T10:29:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:20 smithi165 ceph-mon[32696]: pgmap v854: 188 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 183 active+clean; 104 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 939 B/s rd, 0 op/s; 236/5568 objects degraded (4.239%); 849/5568 objects misplaced (15.248%) 2024-09-16T10:29:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:20 smithi165 ceph-mon[32696]: 4.12 scrub starts 2024-09-16T10:29:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:20 smithi165 ceph-mon[32696]: 4.12 scrub ok 2024-09-16T10:29:20.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:20.673 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:20.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:20.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:20.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:20.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:20.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:21.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:21.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:21.159 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.185 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:21.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:21.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:21.312 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:21.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:21.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:21.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 4.11 scrub starts 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 4.11 scrub ok 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 5.10 scrub starts 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 5.10 scrub ok 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 6.16 scrub starts 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 6.16 scrub ok 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 4.15 scrub starts 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 4.15 scrub ok 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: osdmap e469: 8 total, 8 up, 8 in 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 6.3 scrub starts 2024-09-16T10:29:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 6.3 scrub ok 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 3.0 scrub starts 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: 3.0 scrub ok 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: Health check update: Degraded data redundancy: 236/5568 objects degraded (4.239%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[39470]: osdmap e470: 8 total, 8 up, 8 in 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 4.11 scrub starts 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 4.11 scrub ok 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 5.10 scrub starts 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 5.10 scrub ok 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 6.16 scrub starts 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 6.16 scrub ok 2024-09-16T10:29:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 4.15 scrub starts 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 4.15 scrub ok 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: osdmap e469: 8 total, 8 up, 8 in 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 6.3 scrub starts 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 6.3 scrub ok 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 3.0 scrub starts 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: 3.0 scrub ok 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: Health check update: Degraded data redundancy: 236/5568 objects degraded (4.239%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:29:21.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:21 smithi063 ceph-mon[29993]: osdmap e470: 8 total, 8 up, 8 in 2024-09-16T10:29:21.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:21.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 4.11 scrub starts 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 4.11 scrub ok 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 5.10 scrub starts 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 5.10 scrub ok 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 6.16 scrub starts 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 6.16 scrub ok 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 4.15 scrub starts 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 4.15 scrub ok 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: osdmap e469: 8 total, 8 up, 8 in 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 6.3 scrub starts 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 6.3 scrub ok 2024-09-16T10:29:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 3.0 scrub starts 2024-09-16T10:29:21.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: 3.0 scrub ok 2024-09-16T10:29:21.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: Health check update: Degraded data redundancy: 236/5568 objects degraded (4.239%), 1 pg degraded (PG_DEGRADED) 2024-09-16T10:29:21.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:21 smithi165 ceph-mon[32696]: osdmap e470: 8 total, 8 up, 8 in 2024-09-16T10:29:21.567 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:21.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:21.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:21.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:21.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:21.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:21.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:21.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:22.050 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:22.077 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:22.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:22.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:22.279 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-16T10:29:22.279 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-16T10:29:22.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:22.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:22.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[29993]: 10.8 scrub starts 2024-09-16T10:29:22.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[29993]: 10.8 scrub ok 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[29993]: 1.a scrub starts 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[29993]: 1.a scrub ok 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[29993]: pgmap v857: 188 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 183 active+clean; 36 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 236/2283 objects degraded (10.337%); 361/2283 objects misplaced (15.813%) 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[29993]: 5.c scrub starts 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[29993]: 5.c scrub ok 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[39470]: 10.8 scrub starts 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[39470]: 10.8 scrub ok 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[39470]: 1.a scrub starts 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[39470]: 1.a scrub ok 2024-09-16T10:29:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[39470]: pgmap v857: 188 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 183 active+clean; 36 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 236/2283 objects degraded (10.337%); 361/2283 objects misplaced (15.813%) 2024-09-16T10:29:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[39470]: 5.c scrub starts 2024-09-16T10:29:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:22 smithi063 ceph-mon[39470]: 5.c scrub ok 2024-09-16T10:29:22.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:22.542 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:29:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:22 smithi165 ceph-mon[32696]: 10.8 scrub starts 2024-09-16T10:29:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:22 smithi165 ceph-mon[32696]: 10.8 scrub ok 2024-09-16T10:29:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:22 smithi165 ceph-mon[32696]: 1.a scrub starts 2024-09-16T10:29:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:22 smithi165 ceph-mon[32696]: 1.a scrub ok 2024-09-16T10:29:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:22 smithi165 ceph-mon[32696]: pgmap v857: 188 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 183 active+clean; 36 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 236/2283 objects degraded (10.337%); 361/2283 objects misplaced (15.813%) 2024-09-16T10:29:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:22 smithi165 ceph-mon[32696]: 5.c scrub starts 2024-09-16T10:29:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:22 smithi165 ceph-mon[32696]: 5.c scrub ok 2024-09-16T10:29:22.577 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.610 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:22.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:22.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:22.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:22.838 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:22.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:22.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:22.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:23.089 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:23.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T10:29:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.115 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:23.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:23.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:23.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:23.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:23.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:23.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 5.d scrub starts 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 5.d scrub ok 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 3.13 scrub starts 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 3.13 scrub ok 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 6.1b scrub starts 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 6.1b scrub ok 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 10.9 scrub starts 2024-09-16T10:29:23.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: 10.9 scrub ok 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 5.d scrub starts 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 5.d scrub ok 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 3.13 scrub starts 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 3.13 scrub ok 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 6.1b scrub starts 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 6.1b scrub ok 2024-09-16T10:29:23.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 10.9 scrub starts 2024-09-16T10:29:23.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: 10.9 scrub ok 2024-09-16T10:29:23.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:29:23.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:29:23.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:23 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:23.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:23.491 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 5.d scrub starts 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 5.d scrub ok 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 3.13 scrub starts 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 3.13 scrub ok 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 6.1b scrub starts 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 6.1b scrub ok 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 10.9 scrub starts 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: 10.9 scrub ok 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:29:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:23 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:23.590 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:23.592 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:23.593 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:23.593 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.623 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:23.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:23.724 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:23.746 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:23.848 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:23.871 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:23.973 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:23.995 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:23.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:24.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:24.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:24.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:24.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:24.247 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:24.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:24.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:24.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:24.424 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[29993]: 3.16 scrub starts 2024-09-16T10:29:24.424 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[29993]: 3.16 scrub ok 2024-09-16T10:29:24.424 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[29993]: pgmap v858: 188 pgs: 1 peering, 3 active+remapped+backfill_toofull, 184 active+clean; 18 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 307/1449 objects misplaced (21.187%) 2024-09-16T10:29:24.424 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:24.424 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[39470]: 3.16 scrub starts 2024-09-16T10:29:24.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[39470]: 3.16 scrub ok 2024-09-16T10:29:24.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[39470]: pgmap v858: 188 pgs: 1 peering, 3 active+remapped+backfill_toofull, 184 active+clean; 18 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 307/1449 objects misplaced (21.187%) 2024-09-16T10:29:24.425 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:24.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:24.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:24.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:24.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:24 smithi165 ceph-mon[32696]: 3.16 scrub starts 2024-09-16T10:29:24.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:24 smithi165 ceph-mon[32696]: 3.16 scrub ok 2024-09-16T10:29:24.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:24 smithi165 ceph-mon[32696]: pgmap v858: 188 pgs: 1 peering, 3 active+remapped+backfill_toofull, 184 active+clean; 18 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 307/1449 objects misplaced (21.187%) 2024-09-16T10:29:24.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:24.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:24.624 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:24.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:24.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:24.751 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:29:24.751 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:29:24.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:24.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:24.768 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:29:24.768 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:29:24.784 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:29:24.784 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:29:24.800 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:29:24.801 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:29:24.818 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:29:24.818 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:29:24.835 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:29:24.835 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:29:24.852 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:29:24.853 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:29:24.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:24.869 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:29:24.869 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:29:24.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:24.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:24.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:25.005 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:25.103 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_3' removed 2024-09-16T10:29:25.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:25.159 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:25.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:25.283 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:25.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:25.430 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:25.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 3.9 scrub starts 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 3.9 scrub ok 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 4.c scrub starts 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 4.c scrub ok 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 236/2283 objects degraded (10.337%), 1 pg degraded) 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3220683533' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: from='client.46304 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:25.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2511118629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 3.9 scrub starts 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 3.9 scrub ok 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 4.c scrub starts 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 4.c scrub ok 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 236/2283 objects degraded (10.337%), 1 pg degraded) 2024-09-16T10:29:25.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:29:25.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:29:25.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3220683533' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:29:25.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: from='client.46304 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:29:25.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:25.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:25.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2511118629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:25.527 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:29:25.532 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:25.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 3.9 scrub starts 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 3.9 scrub ok 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 4.c scrub starts 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 4.c scrub ok 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 236/2283 objects degraded (10.337%), 1 pg degraded) 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3220683533' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:29:25.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: from='client.46304 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:29:25.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:25.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:25.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2511118629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:25.592 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:25.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:25.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:25.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:25.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:25.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:25.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:25.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:26.045 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T10:29:26.054 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:26.065 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:26.089 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:26.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:26.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:26.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:26.337 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: 5.6 scrub starts 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: 5.6 scrub ok 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: from='client.46304 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: osdmap e471: 8 total, 8 up, 8 in 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3731327881' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: pgmap v860: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:29:26.357 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: 4.2 scrub starts 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: 4.2 scrub ok 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3731327881' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[29993]: osdmap e472: 8 total, 8 up, 8 in 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: 5.6 scrub starts 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: 5.6 scrub ok 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: from='client.46304 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: osdmap e471: 8 total, 8 up, 8 in 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3731327881' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:29:26.358 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: pgmap v860: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:29:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: 4.2 scrub starts 2024-09-16T10:29:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: 4.2 scrub ok 2024-09-16T10:29:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3731327881' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:29:26.359 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:26 smithi063 ceph-mon[39470]: osdmap e472: 8 total, 8 up, 8 in 2024-09-16T10:29:26.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:26.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: 5.6 scrub starts 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: 5.6 scrub ok 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: from='client.46304 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: osdmap e471: 8 total, 8 up, 8 in 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3731327881' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: pgmap v860: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: 4.2 scrub starts 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: 4.2 scrub ok 2024-09-16T10:29:26.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3731327881' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:29:26.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:26 smithi165 ceph-mon[32696]: osdmap e472: 8 total, 8 up, 8 in 2024-09-16T10:29:26.563 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:26.586 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:26.688 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:26.696 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T10:29:26.710 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:26.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:26.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:26.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:26.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:26.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:27.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:27.081 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:27.150 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:29:27.150 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:29:27.150 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:29:27.150 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDScrubReserve(6.19,QREQUEST,e:470,from:5,reservation_nonce:7)", 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:24.921383+0000", 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 2.2259641480000001, 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013151800000000001, 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921383+0000", 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921383+0000", 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921385+0000", 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.889e-06 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.152 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921392+0000", 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7429999999999996e-06 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921406+0000", 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4549000000000001e-05 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921413+0000", 2024-09-16T10:29:27.153 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9410000000000001e-06 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921474+0000", 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0906999999999998e-05 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921488+0000", 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.367e-05 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.154 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.921515+0000", 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6818999999999999e-05 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "replica_scrub(pg: 6.19,from:0'0,to:0'0,epoch:470/232,start:6:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:24.922141+0000", 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 2.2252059449999999, 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012308100000000001, 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.155 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922141+0000", 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922141+0000", 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922142+0000", 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8100000000000001e-07 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922150+0000", 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.267e-06 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.157 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922161+0000", 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1395e-05 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922167+0000", 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6609999999999997e-06 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922224+0000", 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7725e-05 2024-09-16T10:29:27.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922236+0000", 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1229999999999999e-05 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.922265+0000", 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8822e-05 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDScrubReserve(6.19,RELEASE,e:470,from:5,reservation_nonce:0)", 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:24.923073+0000", 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 2.2242739889999998, 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015432799999999999, 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923073+0000", 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923073+0000", 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923074+0000", 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4899999999999995e-07 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923077+0000", 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0319999999999999e-06 2024-09-16T10:29:27.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923083+0000", 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4580000000000001e-06 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923088+0000", 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7320000000000003e-06 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923196+0000", 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108185 2024-09-16T10:29:27.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923212+0000", 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5968e-05 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:24.923228+0000", 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5304000000000001e-05 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.163 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.24481.0:934 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e470)", 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:25.320699+0000", 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 1.8266483760000001, 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00039449800000000002, 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.24481", 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.165:0/2299456544", 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 934 2024-09-16T10:29:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320699+0000", 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320699+0000", 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.165 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.166 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320702+0000", 2024-09-16T10:29:27.166 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0529999999999998e-06 2024-09-16T10:29:27.166 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320714+0000", 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2036e-05 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320735+0000", 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1117000000000001e-05 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320745+0000", 2024-09-16T10:29:27.167 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2930000000000006e-06 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320880+0000", 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013569799999999999 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.320993+0000", 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112961 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.168 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.321094+0000", 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010034 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDScrubReserve(4.2,GRANT,e:471,from:4,reservation_nonce:0)", 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:25.661907+0000", 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 1.4854399119999999, 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119911, 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.169 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.661907+0000", 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.661907+0000", 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.661910+0000", 2024-09-16T10:29:27.170 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6400000000000001e-06 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.661917+0000", 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2599999999999999e-06 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.661932+0000", 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5089000000000001e-05 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.171 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.661946+0000", 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3528e-05 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.661986+0000", 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0444000000000001e-05 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.172 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662000+0000", 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3148e-05 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662027+0000", 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7801999999999999e-05 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.173 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDScrubReserve(4.2,GRANT,e:471,from:5,reservation_nonce:0)", 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:25.662271+0000", 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 1.485076209, 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016505299999999999, 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662271+0000", 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.174 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662271+0000", 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662272+0000", 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6199999999999996e-07 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662275+0000", 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0580000000000002e-06 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662285+0000", 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8009999999999998e-06 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662290+0000", 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6090000000000003e-06 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662319+0000", 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9079999999999999e-05 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662326+0000", 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2019999999999998e-06 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.662436+0000", 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000110441 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "rep_scrubmap(4.2 e60 from shard 4)", 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:25.663223+0000", 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 1.4841240710000001, 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0117999999999993e-05, 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663223+0000", 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.178 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663223+0000", 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663225+0000", 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1689999999999999e-06 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663232+0000", 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5560000000000002e-06 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663245+0000", 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2761e-05 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663251+0000", 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4280000000000001e-06 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663290+0000", 2024-09-16T10:29:27.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8373e-05 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663296+0000", 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9689999999999999e-06 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663313+0000", 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7861999999999999e-05 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "rep_scrubmap(4.2 e60 from shard 5)", 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:25.663271+0000", 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 1.484076621, 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012121900000000001, 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663271+0000", 2024-09-16T10:29:27.182 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663271+0000", 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663272+0000", 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2399999999999996e-07 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.183 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663276+0000", 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7199999999999997e-06 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663284+0000", 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8029999999999997e-06 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663289+0000", 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2059999999999998e-06 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663336+0000", 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6990999999999998e-05 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663350+0000", 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3695e-05 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:25.663392+0000", 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1879999999999999e-05 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.24481.0:935 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e471)", 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:26.321960+0000", 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.82538714899999999, 2024-09-16T10:29:27.186 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027741300000000001, 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.24481", 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.165:0/2299456544", 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 935 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.321960+0000", 2024-09-16T10:29:27.187 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.321960+0000", 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.321963+0000", 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1810000000000001e-06 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.188 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.321981+0000", 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7328000000000001e-05 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.322004+0000", 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3093000000000001e-05 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.322016+0000", 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2109999999999999e-05 2024-09-16T10:29:27.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.322112+0000", 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6107999999999995e-05 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.322175+0000", 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3159999999999998e-05 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.322238+0000", 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2433000000000003e-05 2024-09-16T10:29:27.190 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDScrubReserve(3.4,GRANT,e:472,from:2,reservation_nonce:0)", 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:26.684546+0000", 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.462801566, 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118919, 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.191 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684546+0000", 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684546+0000", 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684547+0000", 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0899999999999999e-06 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684551+0000", 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.602e-06 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684561+0000", 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0635000000000001e-05 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684569+0000", 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0120000000000005e-06 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684617+0000", 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7562000000000003e-05 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684633+0000", 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5947000000000002e-05 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.684665+0000", 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2070999999999998e-05 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDScrubReserve(3.4,GRANT,e:472,from:5,reservation_nonce:0)", 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:26.685005+0000", 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.46234202800000002, 2024-09-16T10:29:27.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000170939, 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685005+0000", 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685005+0000", 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685006+0000", 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2699999999999998e-07 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685010+0000", 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8319999999999999e-06 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685020+0000", 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8360000000000004e-06 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685025+0000", 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0139999999999998e-06 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685058+0000", 2024-09-16T10:29:27.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3352000000000002e-05 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685071+0000", 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2514000000000001e-05 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685176+0000", 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010556399999999999 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "rep_scrubmap(3.4 e281 from shard 2)", 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:26.685761+0000", 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.46158616600000002, 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2588000000000003e-05, 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685761+0000", 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685761+0000", 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685762+0000", 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0389999999999999e-06 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685766+0000", 2024-09-16T10:29:27.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0629999999999999e-06 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685775+0000", 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5180000000000004e-06 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685779+0000", 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2889999999999999e-06 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685829+0000", 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9734000000000002e-05 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685835+0000", 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4980000000000003e-06 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685844+0000", 2024-09-16T10:29:27.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4470000000000006e-06 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "rep_scrubmap(3.4 e281 from shard 5)", 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:26.685877+0000", 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.46147057600000002, 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4069999999999999e-05, 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:27.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685877+0000", 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685877+0000", 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685878+0000", 2024-09-16T10:29:27.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2099999999999995e-07 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685882+0000", 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1710000000000004e-06 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685890+0000", 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2239999999999995e-06 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:27.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685894+0000", 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1139999999999999e-06 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685935+0000", 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0592e-05 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685944+0000", 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7460000000000006e-06 2024-09-16T10:29:27.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:26.685971+0000", 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7302000000000001e-05 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:27.208 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:29:27.208 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:27.231 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:27.333 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:27.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:27.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:29:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:29:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[29993]: 6.c scrub starts 2024-09-16T10:29:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[29993]: 6.c scrub ok 2024-09-16T10:29:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[29993]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-16T10:29:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[29993]: 3.4 scrub starts 2024-09-16T10:29:27.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[29993]: 3.4 scrub ok 2024-09-16T10:29:27.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[39470]: 6.c scrub starts 2024-09-16T10:29:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[39470]: 6.c scrub ok 2024-09-16T10:29:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[39470]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-16T10:29:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[39470]: 3.4 scrub starts 2024-09-16T10:29:27.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:27 smithi063 ceph-mon[39470]: 3.4 scrub ok 2024-09-16T10:29:27.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:27.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:27.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:27 smithi165 ceph-mon[32696]: 6.c scrub starts 2024-09-16T10:29:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:27 smithi165 ceph-mon[32696]: 6.c scrub ok 2024-09-16T10:29:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:27 smithi165 ceph-mon[32696]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-16T10:29:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:27 smithi165 ceph-mon[32696]: 3.4 scrub starts 2024-09-16T10:29:27.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:27 smithi165 ceph-mon[32696]: 3.4 scrub ok 2024-09-16T10:29:27.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.604 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:27.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:27.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:29:27.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:27.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:27.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:27.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:27.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:27.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:27.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:28.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:29:28.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:28.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:28.198 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.221 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:28.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:28.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:28.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:28.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[29993]: 4.3 scrub starts 2024-09-16T10:29:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[29993]: 4.3 scrub ok 2024-09-16T10:29:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[29993]: pgmap v862: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-09-16T10:29:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:29:28.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[29993]: 1.b scrub ok 2024-09-16T10:29:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[39470]: 4.3 scrub starts 2024-09-16T10:29:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[39470]: 4.3 scrub ok 2024-09-16T10:29:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[39470]: pgmap v862: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-09-16T10:29:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:29:28.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:28 smithi063 ceph-mon[39470]: 1.b scrub ok 2024-09-16T10:29:28.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:28.537 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-16T10:29:28.538 INFO:tasks.radosbench:Beginning radosbench... 2024-09-16T10:29:28.539 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-16T10:29:28.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-16T10:29:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:28 smithi165 ceph-mon[32696]: 4.3 scrub starts 2024-09-16T10:29:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:28 smithi165 ceph-mon[32696]: 4.3 scrub ok 2024-09-16T10:29:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:28 smithi165 ceph-mon[32696]: pgmap v862: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-09-16T10:29:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:28 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:29:28.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:28 smithi165 ceph-mon[32696]: 1.b scrub ok 2024-09-16T10:29:28.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:28.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:28.603 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:28.603 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:28.645 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:28.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:28.715 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:28.818 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.842 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:28.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:28.943 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:28.965 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:28.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:29.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:29.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:29.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:29.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:29.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:29 smithi165 ceph-mon[32696]: 3.5 scrub starts 2024-09-16T10:29:29.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:29 smithi165 ceph-mon[32696]: 3.5 scrub ok 2024-09-16T10:29:29.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:29:29 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:29:29.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:29.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:29.395 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:29 smithi063 ceph-mon[29993]: 3.5 scrub starts 2024-09-16T10:29:29.395 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:29 smithi063 ceph-mon[29993]: 3.5 scrub ok 2024-09-16T10:29:29.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:29 smithi063 ceph-mon[39470]: 3.5 scrub starts 2024-09-16T10:29:29.396 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:29 smithi063 ceph-mon[39470]: 3.5 scrub ok 2024-09-16T10:29:29.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:29.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:29.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:29.682 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:29.705 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:29.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:29.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:29.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:29.953 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:29.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:30.009 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:29:30.009 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:29:30.009 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 0 2024-09-16T10:29:30.009 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:29:30.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:30.078 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:30.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:30.146 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:29:30.146 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T10:29:30.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:29:30.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:29:30.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:29:30.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.e scrub ok 2024-09-16T10:29:30.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:29:30.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.12 scrub ok 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 4.1e scrub starts 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 4.1e scrub ok 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.7 scrub ok 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: pgmap v863: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3651137086' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: from='client.46322 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1445460789' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: from='client.46328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:29:30.160 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[29993]: 1.d scrub ok 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.e scrub ok 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.12 scrub ok 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 4.1e scrub starts 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 4.1e scrub ok 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:29:30.161 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.7 scrub ok 2024-09-16T10:29:30.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: pgmap v863: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:29:30.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3651137086' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:29:30.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: from='client.46322 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:29:30.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1445460789' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:30.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: from='client.46328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:30.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:29:30.162 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:30 smithi063 ceph-mon[39470]: 1.d scrub ok 2024-09-16T10:29:30.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:30.202 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:30.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:30.327 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:30.428 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:30.451 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.e scrub ok 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.12 scrub ok 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 4.1e scrub starts 2024-09-16T10:29:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 4.1e scrub ok 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.7 scrub ok 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: pgmap v863: 152 pgs: 152 active+clean; 609 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3651137086' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: from='client.46322 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1445460789' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: from='client.46328 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:29:30.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:30 smithi165 ceph-mon[32696]: 1.d scrub ok 2024-09-16T10:29:30.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:30.575 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:30.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:30.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:30.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:30.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:30.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:30.955 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:30.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:31.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:31.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:31.134 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:29:31.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:31.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:31.310 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:31.333 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:31.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:31.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: from='client.46322 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: from='client.46328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: osdmap e473: 8 total, 8 up, 8 in 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[29993]: osdmap e474: 8 total, 8 up, 8 in 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:29:31.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: from='client.46322 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:29:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: from='client.46328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: osdmap e473: 8 total, 8 up, 8 in 2024-09-16T10:29:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:29:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:29:31.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:31 smithi063 ceph-mon[39470]: osdmap e474: 8 total, 8 up, 8 in 2024-09-16T10:29:31.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:31.521 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: from='client.46322 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: from='client.46328 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: osdmap e473: 8 total, 8 up, 8 in 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:29:31.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:31 smithi165 ceph-mon[32696]: osdmap e474: 8 total, 8 up, 8 in 2024-09-16T10:29:31.563 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:31.587 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:31.689 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.713 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:31.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:31.814 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:31.836 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:31.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:31.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:31.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:32.020 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:29:32.020 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:29:32.025 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:32.045 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:29:32.046 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:29:32.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:32.068 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:32.076 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:29:32.076 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:29:32.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:32.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:32.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:32.093 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:29:32.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:29:32.111 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:29:32.111 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:29:32.129 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:29:32.129 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:29:32.147 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:29:32.147 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:29:32.171 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:29:32.172 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:29:32.193 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:32.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:32.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:32.318 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:32.341 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:32.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:32.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:32.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:29:32.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:29:32.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: 5.11 scrub starts 2024-09-16T10:29:32.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: 5.11 scrub ok 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: pgmap v866: 152 pgs: 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/232052496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: 5.11 scrub starts 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: 5.11 scrub ok 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: pgmap v866: 152 pgs: 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/232052496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:32.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: 5.11 scrub starts 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: 5.11 scrub ok 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: pgmap v866: 152 pgs: 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:32.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/232052496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:32.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:32.590 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:32.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:32.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:32.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:32.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:32.820 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:32.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:32.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:33.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:33.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:33.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:33.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:33.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:33.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:33 smithi063 ceph-mon[29993]: osdmap e475: 8 total, 8 up, 8 in 2024-09-16T10:29:33.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:33 smithi063 ceph-mon[39470]: osdmap e475: 8 total, 8 up, 8 in 2024-09-16T10:29:33.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:33.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:33.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:33.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:33.488 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:33.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:33.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:33 smithi165 ceph-mon[32696]: osdmap e475: 8 total, 8 up, 8 in 2024-09-16T10:29:33.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:33.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:33.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:33.761 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:33.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:33.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:33.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:33.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:34.013 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:34.088 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:29:34.088 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:29:34.115 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.138 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:34.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:34.154 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_4' created 2024-09-16T10:29:34.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:34.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:34.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:34.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:29:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:29:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[29993]: pgmap v868: 152 pgs: 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-16T10:29:34.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2425078021' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-16T10:29:34.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:29:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:29:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[39470]: pgmap v868: 152 pgs: 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-16T10:29:34.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:34 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2425078021' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-16T10:29:34.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:34.517 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:34.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:34 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:29:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:34 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:29:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:34 smithi165 ceph-mon[32696]: pgmap v868: 152 pgs: 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-16T10:29:34.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:34 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2425078021' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-16T10:29:34.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:29:34.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:34.678 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:34.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:34.804 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:34.906 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:34.928 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:34.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:34.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-16T10:29:35.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:35.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:35.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:35.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:35.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2425078021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-16T10:29:35.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:35 smithi063 ceph-mon[29993]: osdmap e476: 8 total, 8 up, 8 in 2024-09-16T10:29:35.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2425078021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-16T10:29:35.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:35 smithi063 ceph-mon[39470]: osdmap e476: 8 total, 8 up, 8 in 2024-09-16T10:29:35.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:35.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:35.412 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:35.435 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:35.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:35.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2425078021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-16T10:29:35.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:35 smithi165 ceph-mon[32696]: osdmap e476: 8 total, 8 up, 8 in 2024-09-16T10:29:35.559 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:35.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:35.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:35.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:29:35.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:35.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:35.865 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:35.902 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:35.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:29:35.910 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:35.971 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:35.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:36.048 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T10:29:36.072 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:36.095 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:36.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:36.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: 1.10 scrub ok 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: osdmap e477: 8 total, 8 up, 8 in 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: pgmap v871: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/332320110' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: from='client.46346 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: from='client.46346 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[29993]: osdmap e478: 8 total, 8 up, 8 in 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: 1.10 scrub ok 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: osdmap e477: 8 total, 8 up, 8 in 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: pgmap v871: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/332320110' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:29:36.255 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: from='client.46346 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:29:36.256 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: from='client.46346 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:29:36.256 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:36 smithi063 ceph-mon[39470]: osdmap e478: 8 total, 8 up, 8 in 2024-09-16T10:29:36.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:36.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:36.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.474 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:36.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: 1.10 scrub ok 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: osdmap e477: 8 total, 8 up, 8 in 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: pgmap v871: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/332320110' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:29:36.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: from='client.46346 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:29:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: from='client.46346 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:29:36.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:36 smithi165 ceph-mon[32696]: osdmap e478: 8 total, 8 up, 8 in 2024-09-16T10:29:36.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:36.598 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:36.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:36.722 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:36.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:36.831 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T10:29:36.846 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:36.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:36.975 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:36.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:37.077 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:37.089 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:37.104 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:37.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:37.232 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:37.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:37.328 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:29:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:29:37.328 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:29:37.328 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[29993]: 1.14 scrub ok 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[29993]: 3.1f scrub starts 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[29993]: 3.1f scrub ok 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[29993]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[29993]: 1.4 scrub ok 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[39470]: 1.14 scrub ok 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[39470]: 3.1f scrub starts 2024-09-16T10:29:37.329 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[39470]: 3.1f scrub ok 2024-09-16T10:29:37.330 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[39470]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:29:37.330 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:29:37.330 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:37 smithi063 ceph-mon[39470]: 1.4 scrub ok 2024-09-16T10:29:37.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:37.359 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:37.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:37.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:37 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:29:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:37 smithi165 ceph-mon[32696]: 1.14 scrub ok 2024-09-16T10:29:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:37 smithi165 ceph-mon[32696]: 3.1f scrub starts 2024-09-16T10:29:37.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:37 smithi165 ceph-mon[32696]: 3.1f scrub ok 2024-09-16T10:29:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:37 smithi165 ceph-mon[32696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:29:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:37 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:29:37.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:37 smithi165 ceph-mon[32696]: 1.4 scrub ok 2024-09-16T10:29:37.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:37.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:37.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:37.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:37.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:37.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:37.950 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:29:37.950 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:29:37.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:37.967 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:29:37.967 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:29:37.983 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:29:37.983 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:29:37.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:37.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:37.997 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:38.003 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:29:38.004 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:29:38.023 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:29:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:29:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:38.041 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:29:38.041 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:29:38.067 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:29:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:29:38.084 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:29:38.084 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:29:38.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:38.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:38.177 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-16T10:29:38.181 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: 6.14 scrub starts 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: 6.14 scrub ok 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2074200053' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: from='client.46352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: 5.1f scrub starts 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: 5.1f scrub ok 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: pgmap v873: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/848767603' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:29:38.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/417759070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: 6.14 scrub starts 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: 6.14 scrub ok 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2074200053' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: from='client.46352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: 5.1f scrub starts 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: 5.1f scrub ok 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: pgmap v873: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/848767603' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:29:38.183 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/417759070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:38.184 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:29:38.213 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:38.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:38.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:38.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:38.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:38.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:38.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: 6.14 scrub starts 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: 6.14 scrub ok 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2074200053' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: from='client.46352 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: 5.1f scrub starts 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: 5.1f scrub ok 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: pgmap v873: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 583 KiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/848767603' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-16T10:29:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/417759070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:38.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:38.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:38.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:38.736 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:38.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:38.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:38.860 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:38.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:38.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:38.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:39.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:39.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:39.186 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T10:29:39.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:39.233 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:39.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:29:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: 3.12 scrub starts 2024-09-16T10:29:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: 3.12 scrub ok 2024-09-16T10:29:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: from='client.46352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:29:39.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/848767603' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:29:39.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: osdmap e479: 8 total, 8 up, 8 in 2024-09-16T10:29:39.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4079058689' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:29:39.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: from='client.46370 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:29:39.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:39.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:39.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:39.357 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:39.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: 3.12 scrub starts 2024-09-16T10:29:39.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: 3.12 scrub ok 2024-09-16T10:29:39.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:39.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: from='client.46352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:29:39.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/848767603' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:29:39.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: osdmap e479: 8 total, 8 up, 8 in 2024-09-16T10:29:39.421 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4079058689' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: from='client.46370 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: 3.12 scrub starts 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: 3.12 scrub ok 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: from='client.46352 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/848767603' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T10:29:39.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: osdmap e479: 8 total, 8 up, 8 in 2024-09-16T10:29:39.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4079058689' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:29:39.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: from='client.46370 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:29:39.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:39.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:39.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:39.480 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:39.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:39.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:39.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:39.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:39.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.855 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:39.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:39.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:39.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:39.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:40.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:40.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:40.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:40.195 DEBUG:teuthology.orchestra.run.smithi063:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-16T10:29:40.198 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-16T10:29:40.198 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-16T10:29:40.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:40.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:40.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:29:40.285 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9.36b0083b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.81137468700001, 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41720589800000002, 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.286 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463115+0000", 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7430000000000001e-06 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463140+0000", 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4477e-05 2024-09-16T10:29:40.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463151+0000", 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0891e-05 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463161+0000", 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0020000000000001e-05 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464130+0000", 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00096871499999999998 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464137+0000", 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1749999999999999e-06 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880306+0000", 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41616966999999999 2024-09-16T10:29:40.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880320+0000", 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3207000000000001e-05 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9.669f9f4b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.76992507300002, 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37579170000000001, 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.291 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:29:40.292 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.293 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.293 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.293 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:29:40.293 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.293 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504565+0000", 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9400000000000001e-06 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504589+0000", 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3971000000000001e-05 2024-09-16T10:29:40.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504602+0000", 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2539e-05 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504610+0000", 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4460000000000001e-06 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504674+0000", 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3895999999999999e-05 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504682+0000", 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0260000000000007e-06 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880348+0000", 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37566548300000002 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880355+0000", 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3989999999999998e-06 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.297 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9.deba987b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.76376414499998, 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36966423799999998, 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:29:40.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510727+0000", 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8370000000000001e-06 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510777+0000", 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9957999999999998e-05 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510794+0000", 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7356000000000001e-05 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510820+0000", 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6071000000000002e-05 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.301 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510892+0000", 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1520999999999995e-05 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510901+0000", 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5920000000000001e-06 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880380+0000", 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36947966100000001 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880388+0000", 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.242e-06 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9.255df14b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.71695330900002, 2024-09-16T10:29:40.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32288046799999998, 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557537+0000", 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9080000000000002e-06 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557562+0000", 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5131e-05 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557575+0000", 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3039e-05 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.306 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557582+0000", 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0110000000000003e-06 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557855+0000", 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027304500000000003 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557867+0000", 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1507e-05 2024-09-16T10:29:40.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880409+0000", 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32254223399999998 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880416+0000", 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5930000000000002e-06 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.308 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9.3f093dfb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.67277873, 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27873467899999999, 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.309 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:29:40.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601712+0000", 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4609999999999998e-06 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601740+0000", 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8008999999999999e-05 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.311 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601759+0000", 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8427000000000002e-05 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601767+0000", 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5539999999999998e-06 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601927+0000", 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015967299999999999 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601935+0000", 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8040000000000002e-06 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880436+0000", 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27850148000000002 2024-09-16T10:29:40.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880444+0000", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2709999999999992e-06 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9.c91b333b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:29:40.314 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.64746378400002, 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25344918100000002, 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627028+0000", 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.948e-06 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.316 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627081+0000", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3631999999999997e-05 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627100+0000", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9191999999999999e-05 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627110+0000", 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6409999999999997e-06 2024-09-16T10:29:40.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627207+0000", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6762999999999995e-05 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627215+0000", 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2260000000000005e-06 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880467+0000", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25325206 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880474+0000", 2024-09-16T10:29:40.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7190000000000004e-06 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9.7e4a5ddb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.63919049600003, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.245202226, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.320 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635300+0000", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2620000000000001e-06 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635352+0000", 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2079000000000001e-05 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.322 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635365+0000", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3081000000000001e-05 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635373+0000", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6909999999999997e-06 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635435+0000", 2024-09-16T10:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2271000000000001e-05 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635445+0000", 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6099999999999995e-06 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880493+0000", 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.24504827800000001 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880500+0000", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9539999999999998e-06 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9.61c3adbb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.63242793799998, 2024-09-16T10:29:40.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23846684600000001, 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:29:40.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642062+0000", 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0269999999999998e-06 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642104+0000", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1473000000000001e-05 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642120+0000", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6235e-05 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642127+0000", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2470000000000002e-06 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642184+0000", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6725999999999999e-05 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642192+0000", 2024-09-16T10:29:40.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1200000000000002e-06 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880520+0000", 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23832803799999999 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880527+0000", 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9800000000000001e-06 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60273 9.b 9.bb30e96b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:29:40.331 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.58388948200002, 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.189954294, 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60273 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690599+0000", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690601+0000", 2024-09-16T10:29:40.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2479999999999999e-06 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690656+0000", 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4951000000000003e-05 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690673+0000", 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6543999999999999e-05 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690680+0000", 2024-09-16T10:29:40.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8329999999999994e-06 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690741+0000", 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0491000000000001e-05 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.335 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.690754+0000", 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2806000000000001e-05 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880547+0000", 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18979368599999999 2024-09-16T10:29:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880553+0000", 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7350000000000001e-06 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60317 9.b 9.71cef2b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.55101390200002, 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.15710459700000001, 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60317 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723474+0000", 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723477+0000", 2024-09-16T10:29:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4119999999999999e-06 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723606+0000", 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012882700000000001 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723622+0000", 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6436e-05 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723630+0000", 2024-09-16T10:29:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.089e-06 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723701+0000", 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0729000000000007e-05 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.723710+0000", 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8140000000000007e-06 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880572+0000", 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.156862641 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880579+0000", 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6490000000000002e-06 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60326 9.b 9.d49aff0b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.54107274099999, 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14719354100000001, 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60326 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733416+0000", 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733418+0000", 2024-09-16T10:29:40.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.385e-06 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733480+0000", 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2331999999999999e-05 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733498+0000", 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8111e-05 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733506+0000", 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0129999999999993e-06 2024-09-16T10:29:40.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733571+0000", 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4437000000000007e-05 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.733582+0000", 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1073000000000001e-05 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880601+0000", 2024-09-16T10:29:40.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.147019074 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880609+0000", 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1159999999999999e-06 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60333 9.b 9.f3eeac1b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:29:40.348 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 424.53666941, 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14281871500000001, 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60333 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737819+0000", 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737822+0000", 2024-09-16T10:29:40.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.649e-06 2024-09-16T10:29:40.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737888+0000", 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6540999999999999e-05 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737908+0000", 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9419000000000002e-05 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.737917+0000", 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0790000000000007e-06 2024-09-16T10:29:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.738071+0000", 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015447900000000001 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.738079+0000", 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8590000000000005e-06 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880631+0000", 2024-09-16T10:29:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.142552229 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880638+0000", 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.46e-06 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14514.0:2964 5.d 5:b4812045:::notify.4:head [watch ping cookie 93976561472000 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-16T10:29:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:37.320904+0000", 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 182.953584616, 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68289200299999997, 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14514", 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 2964 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320904+0000", 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320904+0000", 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320907+0000", 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2569999999999999e-06 2024-09-16T10:29:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320919+0000", 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2444e-05 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320945+0000", 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5389e-05 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320956+0000", 2024-09-16T10:29:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0919e-05 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321365+0000", 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000409326 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321381+0000", 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6087999999999999e-05 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.003572+0000", 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68219090000000004 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.003725+0000", 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000153392 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.003796+0000", 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0288000000000002e-05 2024-09-16T10:29:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14514.0:2972 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 93976561472000 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:37.321458+0000", 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 182.95303037900001, 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68540705099999999, 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14514", 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:29:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 2972 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321458+0000", 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321458+0000", 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321459+0000", 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2589999999999999e-06 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321464+0000", 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4209999999999997e-06 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321476+0000", 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2704999999999999e-05 2024-09-16T10:29:40.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321484+0000", 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.447e-06 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321550+0000", 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6458000000000006e-05 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321559+0000", 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0140000000000004e-06 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.004206+0000", 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68264650900000001 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.004285+0000", 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8716999999999999e-05 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.006865+0000", 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025805210000000001 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:102282 10.5 10:a97fb381:::benchmark_data_smithi063_149043_object102281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:41.419016+0000", 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 178.855471994, 2024-09-16T10:29:40.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14995875, 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 102282 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419016+0000", 2024-09-16T10:29:40.367 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419016+0000", 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419018+0000", 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9099999999999999e-06 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419040+0000", 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1827999999999998e-05 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419055+0000", 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5184000000000001e-05 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419063+0000", 2024-09-16T10:29:40.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2060000000000001e-06 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419473+0000", 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00041080000000000001 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419620+0000", 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014705799999999999 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.419780+0000", 2024-09-16T10:29:40.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015935499999999999 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.423380+0000", 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0036000950000000002 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.424640+0000", 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012602550000000001 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.568861+0000", 2024-09-16T10:29:40.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14422088699999999 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.568925+0000", 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3534000000000005e-05 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.568975+0000", 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0637999999999999e-05 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.372 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:102287 10.3 10:cfa6868a:::benchmark_data_smithi063_149043_object102286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:41.421564+0000", 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 178.85292453299999, 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.15617345999999999, 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:40.373 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 102287 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421564+0000", 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421564+0000", 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421566+0000", 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.708e-06 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421588+0000", 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2379999999999999e-05 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421599+0000", 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1276e-05 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421607+0000", 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4340000000000003e-06 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421669+0000", 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2576000000000005e-05 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421797+0000", 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012805999999999999 2024-09-16T10:29:40.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.421955+0000", 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000157773 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.425139+0000", 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003184049 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.426446+0000", 2024-09-16T10:29:40.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001307253 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.577492+0000", 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.15104605500000001 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.577620+0000", 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012765699999999999 2024-09-16T10:29:40.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.577737+0000", 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011723899999999999 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:102296 10.3 10:ce8b8b15:::benchmark_data_smithi063_149043_object102295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:41.434100+0000", 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 178.84038809399999, 2024-09-16T10:29:40.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.14388963900000001, 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 102296 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434100+0000", 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434100+0000", 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434102+0000", 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1959999999999998e-06 2024-09-16T10:29:40.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434125+0000", 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2802e-05 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434137+0000", 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1996e-05 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434142+0000", 2024-09-16T10:29:40.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:40.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[39470]: from='client.46370 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:29:40.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[39470]: osdmap e480: 8 total, 8 up, 8 in 2024-09-16T10:29:40.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[39470]: pgmap v876: 168 pgs: 1 creating+peering, 167 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:29:40.383 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:40.383 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[29993]: from='client.46370 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:29:40.384 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[29993]: osdmap e480: 8 total, 8 up, 8 in 2024-09-16T10:29:40.384 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:40 smithi063 ceph-mon[29993]: pgmap v876: 168 pgs: 1 creating+peering, 167 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:29:40.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1660000000000002e-06 2024-09-16T10:29:40.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434197+0000", 2024-09-16T10:29:40.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5034999999999997e-05 2024-09-16T10:29:40.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434308+0000", 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000110859 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.434412+0000", 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103447 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.436817+0000", 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024053019999999998 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.443290+0000", 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00647249 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.577796+0000", 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13450662199999999 2024-09-16T10:29:40.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.577944+0000", 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014824799999999999 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:41.577990+0000", 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5476000000000003e-05 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:145025 10.3 10:c4cc0484:::benchmark_data_smithi063_149043_object12588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:17.342506+0000", 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 142.93198227900001, 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.15955150300000001, 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 145025 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.388 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.342506+0000", 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.342506+0000", 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.342507+0000", 2024-09-16T10:29:40.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1909999999999999e-06 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.342515+0000", 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7749999999999993e-06 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.342525+0000", 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0285000000000001e-05 2024-09-16T10:29:40.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.342532+0000", 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6309999999999997e-06 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.350301+0000", 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0077691380000000001 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.350381+0000", 2024-09-16T10:29:40.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9956999999999994e-05 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,5", 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.473737+0000", 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123356167 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.477450+0000", 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037123099999999999 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.485388+0000", 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079386360000000007 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.501969+0000", 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016581126000000002 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.502016+0000", 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6841000000000001e-05 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:17.502058+0000", 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1446000000000003e-05 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.394 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:188731 10.18 10:1aa8ee08:::benchmark_data_smithi063_149043_object56294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:27:55.951010+0000", 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 104.32347814800001, 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.158390171, 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 188731 2024-09-16T10:29:40.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.951010+0000", 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.951010+0000", 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.951012+0000", 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0250000000000001e-06 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.951018+0000", 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3650000000000003e-06 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.951029+0000", 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1304999999999999e-05 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.951037+0000", 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6650000000000003e-06 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.953424+0000", 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023877690000000001 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:55.953583+0000", 2024-09-16T10:29:40.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015861900000000001 2024-09-16T10:29:40.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,3", 2024-09-16T10:29:40.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:56.061057+0000", 2024-09-16T10:29:40.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10747448 2024-09-16T10:29:40.400 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:hints = 1 2024-09-16T10:29:40.400 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-16T10:29:40.400 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object prefix: benchmark_data_smithi063_176907 2024-09-16T10:29:40.400 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:29:40.400 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:29:40.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:56.068690+0000", 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0076328460000000004 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:56.068745+0000", 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4659e-05 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:56.069720+0000", 2024-09-16T10:29:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00097511199999999998 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:56.109201+0000", 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.039481341000000003 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:56.109281+0000", 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9963999999999995e-05 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:27:56.109400+0000", 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119021 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:229322 10.5 10:a0147a5a:::benchmark_data_smithi063_149043_object96885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e448)", 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:28:43.842105+0000", 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 56.432383475999998, 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.142390937, 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 229322 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.842105+0000", 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.842105+0000", 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.842106+0000", 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4100000000000001e-06 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:40.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.842111+0000", 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9629999999999997e-06 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.842122+0000", 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0553e-05 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.842129+0000", 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8460000000000002e-06 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.845399+0000", 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003269994 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.845509+0000", 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000110226 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.845935+0000", 2024-09-16T10:29:40.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000426248 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.848345+0000", 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024102939999999999 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.850638+0000", 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022930379999999998 2024-09-16T10:29:40.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.984238+0000", 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.133599777 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.984448+0000", 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020988100000000001 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:40.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:28:43.984496+0000", 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7707000000000002e-05 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:40.410 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:29:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:40.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:40.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:40.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:40.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:40.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:40 smithi165 ceph-mon[32696]: from='client.46370 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:29:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:40 smithi165 ceph-mon[32696]: osdmap e480: 8 total, 8 up, 8 in 2024-09-16T10:29:40.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:40 smithi165 ceph-mon[32696]: pgmap v876: 168 pgs: 1 creating+peering, 167 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:29:40.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:40.685 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:40.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:40.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:40.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:40.916 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:40.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:41.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:41.089 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:41.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:41.191 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:41.230 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:41.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:41.309 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 1 16 3266 3250 203.114 203.125 0.00438387 0.00490246 2024-09-16T10:29:41.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:41.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:41.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:41.447 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:41 smithi063 ceph-mon[29993]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:29:41.447 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:41 smithi063 ceph-mon[39470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:29:41.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:41.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:41.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:41.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:29:41.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:41 smithi165 ceph-mon[32696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T10:29:41.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:41.644 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:41.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:41.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:41.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:41.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:41.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:41.937 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:41.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:42.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:42.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:42.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:42.082 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:29:42.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:42.215 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:42.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:42.309 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 2 16 6281 6265 195.761 188.438 0.00320981 0.00509791 2024-09-16T10:29:42.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:42.358 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:42.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:42 smithi063 ceph-mon[39470]: pgmap v877: 168 pgs: 168 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:29:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:42 smithi063 ceph-mon[29993]: pgmap v877: 168 pgs: 168 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:29:42.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:42.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:42.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:42 smithi165 ceph-mon[32696]: pgmap v877: 168 pgs: 168 active+clean; 583 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:29:42.591 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:42.626 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:42.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:42.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:42.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:42.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:42.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:42.900 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:42.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:43.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:43.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:43.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:43.136 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:43.175 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:43.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:43.260 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:43.277 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:43.299 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T10:29:43.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:29:43.309 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 3 16 9133 9117 189.921 178.25 0.00927568 0.00525056 2024-09-16T10:29:43.324 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:43.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:43.426 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:43.474 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:43.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:43.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:43.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:43.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:43.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:43.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:43.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:43.838 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:43.864 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:43.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:43.965 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:44.007 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:44.109 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:44.151 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:44.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:44.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:29:44.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:44.310 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 4 16 11657 11641 181.859 157.75 0.00365709 0.00549184 2024-09-16T10:29:44.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:44.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:44.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:10508 11.b 11:d08fd0d5:::benchmark_data_smithi063_176907_object10507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:43.910576+0000", 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0047750450000000003, 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004810155, 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 10508 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.910576+0000", 2024-09-16T10:29:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.910576+0000", 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.910578+0000", 2024-09-16T10:29:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6759999999999999e-06 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.910630+0000", 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2571999999999997e-05 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.910644+0000", 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3534e-05 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.910651+0000", 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4850000000000003e-06 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.910850+0000", 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019811899999999999 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911005+0000", 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155678 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911142+0000", 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000136551 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.912790+0000", 2024-09-16T10:29:44.540 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016485110000000001 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:10516 11.9 11:9b036096:::benchmark_data_smithi063_176907_object10515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:43.911091+0000", 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0042604749999999997, 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043859390000000002, 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:44.541 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 10516 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911091+0000", 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911091+0000", 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911092+0000", 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.319e-06 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.543 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911142+0000", 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9673999999999999e-05 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911151+0000", 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2790000000000004e-06 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911158+0000", 2024-09-16T10:29:44.544 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.187e-06 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911213+0000", 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5013000000000003e-05 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911339+0000", 2024-09-16T10:29:44.545 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012552099999999999 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.911456+0000", 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116824 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.912777+0000", 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001321673 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.913716+0000", 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00093874900000000001 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:10520 11.9 11:9db2c2d0:::benchmark_data_smithi063_176907_object10519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:43.914819+0000", 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.00053267400000000002, 2024-09-16T10:29:44.547 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00074177200000000003, 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 10520 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:44.548 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.914819+0000", 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.914819+0000", 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.914821+0000", 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0829999999999998e-06 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.914845+0000", 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4735e-05 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.914861+0000", 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6064000000000001e-05 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:44.550 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.914870+0000", 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3659999999999992e-06 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915087+0000", 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000217421 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915213+0000", 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012561899999999999 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915348+0000", 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013520000000000001 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.26491.0:10508 11.b e480/476)", 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:43.915363+0000", 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4294967295.9999876, 2024-09-16T10:29:44.552 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026159099999999998, 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915363+0000", 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:29:44.553 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915363+0000", 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915365+0000", 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.221e-06 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915369+0000", 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.578e-06 2024-09-16T10:29:44.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915379+0000", 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.995e-06 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:43.915387+0000", 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8280000000000003e-06 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:29:44.555 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:29:44.556 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 4 2024-09-16T10:29:44.556 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:29:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.557 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:44.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:44.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:44.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:44 smithi063 ceph-mon[29993]: pgmap v878: 168 pgs: 168 active+clean; 326 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 44 MiB/s wr, 705 op/s 2024-09-16T10:29:44.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:44 smithi063 ceph-mon[39470]: pgmap v878: 168 pgs: 168 active+clean; 326 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 44 MiB/s wr, 705 op/s 2024-09-16T10:29:44.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:44.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:44 smithi165 ceph-mon[32696]: pgmap v878: 168 pgs: 168 active+clean; 326 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 44 MiB/s wr, 705 op/s 2024-09-16T10:29:44.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:44.883 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:44.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:44.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:44.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:45.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:45.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:45.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:45.185 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:45.281 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:45.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:45.310 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 5 16 13980 13964 174.523 145.188 0.00648215 0.00572307 2024-09-16T10:29:45.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:45.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:45.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:45.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:45.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:45.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:45.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:45.685 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:45.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:45.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:45.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:45.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:45.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:29:45.893 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:45.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:45.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:45.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:46.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:46.098 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:46.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:46.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:46.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:46.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:46.310 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 6 15 16428 16413 170.943 153.062 0.00432256 0.00584336 2024-09-16T10:29:46.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:46.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:46.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:46.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:46.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:46.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:46.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:46.594 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:29:46.594 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:29:46.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:46.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:46.635 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:29:46.635 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:29:46.694 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:29:46.694 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:29:46.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:46 smithi063 ceph-mon[29993]: pgmap v879: 168 pgs: 168 active+clean; 580 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-16T10:29:46.695 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:46 smithi063 ceph-mon[39470]: pgmap v879: 168 pgs: 168 active+clean; 580 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-16T10:29:46.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:46.729 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:29:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:29:46.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:46.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:46.762 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:29:46.763 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:29:46.795 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:29:46.796 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:29:46.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:46 smithi165 ceph-mon[32696]: pgmap v879: 168 pgs: 168 active+clean; 580 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-16T10:29:46.802 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:29:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:29:46.730928939Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-16T10:29:46.802 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:29:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=sqlstore.transactions t=2024-09-16T10:29:46.742221051Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-16T10:29:46.829 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:29:46.829 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:29:46.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:46.862 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:29:46.862 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:29:46.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:46.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:46.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:47.026 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:47.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:47.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:47.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:47.302 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:47.310 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 7 16 18824 18808 167.905 149.688 0.014043 0.00594778 2024-09-16T10:29:47.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:47.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:47.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:29:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:29:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:29:47.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.574 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:47.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:47.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:47.714 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:47.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3656597082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:47.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:47.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3656597082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:47.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3656597082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:29:47.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:47.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:47.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:48.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:48.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:48.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:48.153 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:48.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:48.257 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:48.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:48.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:48.310 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 8 15 20936 20921 163.423 132.062 0.00339764 0.00611109 2024-09-16T10:29:48.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:48.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:48.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:48.488 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:48.533 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:48.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:48.578 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:29:48.579 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:48 smithi063 ceph-mon[29993]: pgmap v880: 168 pgs: 168 active+clean; 580 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 552 B/s rd, 63 MiB/s wr, 1k op/s 2024-09-16T10:29:48.579 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:48 smithi063 ceph-mon[39470]: pgmap v880: 168 pgs: 168 active+clean; 580 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 552 B/s rd, 63 MiB/s wr, 1k op/s 2024-09-16T10:29:48.588 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:29:48.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:48.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:48.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:48.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:48.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:48.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:48 smithi165 ceph-mon[32696]: pgmap v880: 168 pgs: 168 active+clean; 580 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 552 B/s rd, 63 MiB/s wr, 1k op/s 2024-09-16T10:29:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:48.807 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:48.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:48.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:48.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:48.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:49.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:49.082 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:49.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:49.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:49.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:49.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:49.310 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 9 16 23285 23269 161.569 146.75 0.00685789 0.00618318 2024-09-16T10:29:49.318 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:49.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:49.404 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":880,"stamp":"2024-09-16T10:29:47.441211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":607787547,"num_objects":9464,"num_object_clones":0,"num_object_copies":28392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9464,"num_whiteouts":0,"num_read":1414,"num_read_kb":1166,"num_write":9987,"num_write_kb":593005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":527794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10279,"ondisk_log_size":10279,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6355612,"kb_used_data":1843996,"kb_used_omap":18,"kb_used_meta":4511469,"kb_avail":743605604,"statfs":{"total":767960285184,"available":761452138496,"internally_reserved":0,"allocated":1888251904,"data_stored":1879330465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19173,"internal_metadata":4619744539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,11,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":607191040,"num_objects":9265,"num_object_clones":0,"num_object_copies":27795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9265,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9265,"num_write_kb":592960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.264049"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":914,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030461+0000","last_change":"2024-09-16T10:29:36.039702+0000","last_active":"2024-09-16T10:29:40.030461+0000","last_peered":"2024-09-16T10:29:40.030461+0000","last_clean":"2024-09-16T10:29:40.030461+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:29:40.030461+0000","last_undegraded":"2024-09-16T10:29:40.030461+0000","last_fullsized":"2024-09-16T10:29:40.030461+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.039652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.039652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:56.261928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":852,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292274+0000","last_change":"2024-09-16T10:26:53.392698+0000","last_active":"2024-09-16T10:29:39.292274+0000","last_peered":"2024-09-16T10:29:39.292274+0000","last_clean":"2024-09-16T10:29:39.292274+0000","last_became_active":"2024-09-16T10:26:28.557096+0000","last_became_peered":"2024-09-16T10:26:28.557096+0000","last_unstale":"2024-09-16T10:29:39.292274+0000","last_undegraded":"2024-09-16T10:29:39.292274+0000","last_fullsized":"2024-09-16T10:29:39.292274+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:53.392646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:53.392646+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.181687+0000","last_change":"2024-09-16T10:28:28.072016+0000","last_active":"2024-09-16T10:29:39.181687+0000","last_peered":"2024-09-16T10:29:39.181687+0000","last_clean":"2024-09-16T10:29:39.181687+0000","last_became_active":"2024-09-16T10:28:28.071724+0000","last_became_peered":"2024-09-16T10:28:28.071724+0000","last_unstale":"2024-09-16T10:29:39.181687+0000","last_undegraded":"2024-09-16T10:29:39.181687+0000","last_fullsized":"2024-09-16T10:29:39.181687+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.940459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.940459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038591+0000","last_change":"2024-09-16T10:28:07.975203+0000","last_active":"2024-09-16T10:29:40.038591+0000","last_peered":"2024-09-16T10:29:40.038591+0000","last_clean":"2024-09-16T10:29:40.038591+0000","last_became_active":"2024-09-16T10:26:28.551847+0000","last_became_peered":"2024-09-16T10:26:28.551847+0000","last_unstale":"2024-09-16T10:29:40.038591+0000","last_undegraded":"2024-09-16T10:29:40.038591+0000","last_fullsized":"2024-09-16T10:29:40.038591+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:28:07.975153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:28:07.975153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":889,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704943+0000","last_change":"2024-09-16T10:29:21.916724+0000","last_active":"2024-09-16T10:29:39.704943+0000","last_peered":"2024-09-16T10:29:39.704943+0000","last_clean":"2024-09-16T10:29:39.704943+0000","last_became_active":"2024-09-16T10:28:28.094751+0000","last_became_peered":"2024-09-16T10:28:28.094751+0000","last_unstale":"2024-09-16T10:29:39.704943+0000","last_undegraded":"2024-09-16T10:29:39.704943+0000","last_fullsized":"2024-09-16T10:29:39.704943+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.675562+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":914,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705867+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:29:39.705867+0000","last_peered":"2024-09-16T10:29:39.705867+0000","last_clean":"2024-09-16T10:29:39.705867+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:29:39.705867+0000","last_undegraded":"2024-09-16T10:29:39.705867+0000","last_fullsized":"2024-09-16T10:29:39.705867+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":983,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705243+0000","last_change":"2024-09-16T10:26:12.334813+0000","last_active":"2024-09-16T10:29:39.705243+0000","last_peered":"2024-09-16T10:29:39.705243+0000","last_clean":"2024-09-16T10:29:39.705243+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:29:39.705243+0000","last_undegraded":"2024-09-16T10:29:39.705243+0000","last_fullsized":"2024-09-16T10:29:39.705243+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:12.334720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.334720+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292357+0000","last_change":"2024-09-16T10:26:45.470165+0000","last_active":"2024-09-16T10:29:39.292357+0000","last_peered":"2024-09-16T10:29:39.292357+0000","last_clean":"2024-09-16T10:29:39.292357+0000","last_became_active":"2024-09-16T10:26:28.556509+0000","last_became_peered":"2024-09-16T10:26:28.556509+0000","last_unstale":"2024-09-16T10:29:39.292357+0000","last_undegraded":"2024-09-16T10:29:39.292357+0000","last_fullsized":"2024-09-16T10:29:39.292357+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.470014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.470014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039408+0000","last_change":"2024-09-16T10:26:48.078096+0000","last_active":"2024-09-16T10:29:40.039408+0000","last_peered":"2024-09-16T10:29:40.039408+0000","last_clean":"2024-09-16T10:29:40.039408+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:29:40.039408+0000","last_undegraded":"2024-09-16T10:29:40.039408+0000","last_fullsized":"2024-09-16T10:29:40.039408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:48.078028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:48.078028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":818,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705170+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:29:39.705170+0000","last_peered":"2024-09-16T10:29:39.705170+0000","last_clean":"2024-09-16T10:29:39.705170+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:29:39.705170+0000","last_undegraded":"2024-09-16T10:29:39.705170+0000","last_fullsized":"2024-09-16T10:29:39.705170+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:53.512693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038822+0000","last_change":"2024-09-16T10:28:28.182357+0000","last_active":"2024-09-16T10:29:40.038822+0000","last_peered":"2024-09-16T10:29:40.038822+0000","last_clean":"2024-09-16T10:29:40.038822+0000","last_became_active":"2024-09-16T10:28:28.182043+0000","last_became_peered":"2024-09-16T10:28:28.182043+0000","last_unstale":"2024-09-16T10:29:40.038822+0000","last_undegraded":"2024-09-16T10:29:40.038822+0000","last_fullsized":"2024-09-16T10:29:40.038822+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.834823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.834823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.037375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":898,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029647+0000","last_change":"2024-09-16T10:28:28.090355+0000","last_active":"2024-09-16T10:29:40.029647+0000","last_peered":"2024-09-16T10:29:40.029647+0000","last_clean":"2024-09-16T10:29:40.029647+0000","last_became_active":"2024-09-16T10:28:28.089459+0000","last_became_peered":"2024-09-16T10:28:28.089459+0000","last_unstale":"2024-09-16T10:29:40.029647+0000","last_undegraded":"2024-09-16T10:29:40.029647+0000","last_fullsized":"2024-09-16T10:29:40.029647+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:08.436243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.436243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":868,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039004+0000","last_change":"2024-09-16T10:27:58.999207+0000","last_active":"2024-09-16T10:29:40.039004+0000","last_peered":"2024-09-16T10:29:40.039004+0000","last_clean":"2024-09-16T10:29:40.039004+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:29:40.039004+0000","last_undegraded":"2024-09-16T10:29:40.039004+0000","last_fullsized":"2024-09-16T10:29:40.039004+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:58.999143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:58.999143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":798,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705274+0000","last_change":"2024-09-16T10:26:28.557316+0000","last_active":"2024-09-16T10:29:39.705274+0000","last_peered":"2024-09-16T10:29:39.705274+0000","last_clean":"2024-09-16T10:29:39.705274+0000","last_became_active":"2024-09-16T10:26:28.556933+0000","last_became_peered":"2024-09-16T10:26:28.556933+0000","last_unstale":"2024-09-16T10:29:39.705274+0000","last_undegraded":"2024-09-16T10:29:39.705274+0000","last_fullsized":"2024-09-16T10:29:39.705274+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.715967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":739,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.394923+0000","last_change":"2024-09-16T10:29:23.841536+0000","last_active":"2024-09-16T10:29:39.394923+0000","last_peered":"2024-09-16T10:29:39.394923+0000","last_clean":"2024-09-16T10:29:39.394923+0000","last_became_active":"2024-09-16T10:28:28.071768+0000","last_became_peered":"2024-09-16T10:28:28.071768+0000","last_unstale":"2024-09-16T10:29:39.394923+0000","last_undegraded":"2024-09-16T10:29:39.394923+0000","last_fullsized":"2024-09-16T10:29:39.394923+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.841474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.841474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.210925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038629+0000","last_change":"2024-09-16T10:29:24.922917+0000","last_active":"2024-09-16T10:29:40.038629+0000","last_peered":"2024-09-16T10:29:40.038629+0000","last_clean":"2024-09-16T10:29:40.038629+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:29:40.038629+0000","last_undegraded":"2024-09-16T10:29:40.038629+0000","last_fullsized":"2024-09-16T10:29:40.038629+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.922862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.922862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:45.215312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292860+0000","last_change":"2024-09-16T10:29:16.166206+0000","last_active":"2024-09-16T10:29:39.292860+0000","last_peered":"2024-09-16T10:29:39.292860+0000","last_clean":"2024-09-16T10:29:39.292860+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:29:39.292860+0000","last_undegraded":"2024-09-16T10:29:39.292860+0000","last_fullsized":"2024-09-16T10:29:39.292860+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:16.166144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:16.166144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:19.152416+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":894,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030565+0000","last_change":"2024-09-16T10:28:28.081914+0000","last_active":"2024-09-16T10:29:40.030565+0000","last_peered":"2024-09-16T10:29:40.030565+0000","last_clean":"2024-09-16T10:29:40.030565+0000","last_became_active":"2024-09-16T10:28:28.081584+0000","last_became_peered":"2024-09-16T10:28:28.081584+0000","last_unstale":"2024-09-16T10:29:40.030565+0000","last_undegraded":"2024-09-16T10:29:40.030565+0000","last_fullsized":"2024-09-16T10:29:40.030565+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.031297+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":460,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.185031+0000","last_change":"2024-09-16T10:26:17.566065+0000","last_active":"2024-09-16T10:29:39.185031+0000","last_peered":"2024-09-16T10:29:39.185031+0000","last_clean":"2024-09-16T10:29:39.185031+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:29:39.185031+0000","last_undegraded":"2024-09-16T10:29:39.185031+0000","last_fullsized":"2024-09-16T10:29:39.185031+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:17.565970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.565970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":884,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182271+0000","last_change":"2024-09-16T10:26:22.230127+0000","last_active":"2024-09-16T10:29:39.182271+0000","last_peered":"2024-09-16T10:29:39.182271+0000","last_clean":"2024-09-16T10:29:39.182271+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:29:39.182271+0000","last_undegraded":"2024-09-16T10:29:39.182271+0000","last_fullsized":"2024-09-16T10:29:39.182271+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.230054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.230054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":844,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705625+0000","last_change":"2024-09-16T10:27:18.318288+0000","last_active":"2024-09-16T10:29:39.705625+0000","last_peered":"2024-09-16T10:29:39.705625+0000","last_clean":"2024-09-16T10:29:39.705625+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:29:39.705625+0000","last_undegraded":"2024-09-16T10:29:39.705625+0000","last_fullsized":"2024-09-16T10:29:39.705625+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:18.318214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:18.318214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":680,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182908+0000","last_change":"2024-09-16T10:26:28.551407+0000","last_active":"2024-09-16T10:29:39.182908+0000","last_peered":"2024-09-16T10:29:39.182908+0000","last_clean":"2024-09-16T10:29:39.182908+0000","last_became_active":"2024-09-16T10:26:28.551058+0000","last_became_peered":"2024-09-16T10:26:28.551058+0000","last_unstale":"2024-09-16T10:29:39.182908+0000","last_undegraded":"2024-09-16T10:29:39.182908+0000","last_fullsized":"2024-09-16T10:29:39.182908+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":953,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705487+0000","last_change":"2024-09-16T10:29:28.855774+0000","last_active":"2024-09-16T10:29:39.705487+0000","last_peered":"2024-09-16T10:29:39.705487+0000","last_clean":"2024-09-16T10:29:39.705487+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:29:39.705487+0000","last_undegraded":"2024-09-16T10:29:39.705487+0000","last_fullsized":"2024-09-16T10:29:39.705487+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:29:28.855720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.855720+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:34.642453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":688,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705714+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:29:39.705714+0000","last_peered":"2024-09-16T10:29:39.705714+0000","last_clean":"2024-09-16T10:29:39.705714+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:29:39.705714+0000","last_undegraded":"2024-09-16T10:29:39.705714+0000","last_fullsized":"2024-09-16T10:29:39.705714+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":302,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038847+0000","last_change":"2024-09-16T10:28:28.095829+0000","last_active":"2024-09-16T10:29:40.038847+0000","last_peered":"2024-09-16T10:29:40.038847+0000","last_clean":"2024-09-16T10:29:40.038847+0000","last_became_active":"2024-09-16T10:28:28.095455+0000","last_became_peered":"2024-09-16T10:28:28.095455+0000","last_unstale":"2024-09-16T10:29:40.038847+0000","last_undegraded":"2024-09-16T10:29:40.038847+0000","last_fullsized":"2024-09-16T10:29:40.038847+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.012515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":899,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.181873+0000","last_change":"2024-09-16T10:26:17.204523+0000","last_active":"2024-09-16T10:29:39.181873+0000","last_peered":"2024-09-16T10:29:39.181873+0000","last_clean":"2024-09-16T10:29:39.181873+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:29:39.181873+0000","last_undegraded":"2024-09-16T10:29:39.181873+0000","last_fullsized":"2024-09-16T10:29:39.181873+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:17.204441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.204441+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":938,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030662+0000","last_change":"2024-09-16T10:29:29.106834+0000","last_active":"2024-09-16T10:29:40.030662+0000","last_peered":"2024-09-16T10:29:40.030662+0000","last_clean":"2024-09-16T10:29:40.030662+0000","last_became_active":"2024-09-16T10:28:28.082261+0000","last_became_peered":"2024-09-16T10:28:28.082261+0000","last_unstale":"2024-09-16T10:29:40.030662+0000","last_undegraded":"2024-09-16T10:29:40.030662+0000","last_fullsized":"2024-09-16T10:29:40.030662+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:29.106770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.106770+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.311277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"442'10","reported_seq":1109,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.293118+0000","last_change":"2024-09-16T10:29:37.281159+0000","last_active":"2024-09-16T10:29:39.293118+0000","last_peered":"2024-09-16T10:29:39.293118+0000","last_clean":"2024-09-16T10:29:39.293118+0000","last_became_active":"2024-09-16T10:28:28.089383+0000","last_became_peered":"2024-09-16T10:28:28.089383+0000","last_unstale":"2024-09-16T10:29:39.293118+0000","last_undegraded":"2024-09-16T10:29:39.293118+0000","last_fullsized":"2024-09-16T10:29:39.293118+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.602654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"480'510","reported_seq":529,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.007491+0000","last_change":"2024-09-16T10:29:35.280092+0000","last_active":"2024-09-16T10:29:43.007491+0000","last_peered":"2024-09-16T10:29:43.007491+0000","last_clean":"2024-09-16T10:29:43.007491+0000","last_became_active":"2024-09-16T10:29:35.279496+0000","last_became_peered":"2024-09-16T10:29:35.279496+0000","last_unstale":"2024-09-16T10:29:43.007491+0000","last_undegraded":"2024-09-16T10:29:43.007491+0000","last_fullsized":"2024-09-16T10:29:43.007491+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:55.065966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":510,"num_object_clones":0,"num_object_copies":1530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":510,"num_write_kb":32640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":746,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038536+0000","last_change":"2024-09-16T10:29:21.013569+0000","last_active":"2024-09-16T10:29:40.038536+0000","last_peered":"2024-09-16T10:29:40.038536+0000","last_clean":"2024-09-16T10:29:40.038536+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:29:40.038536+0000","last_undegraded":"2024-09-16T10:29:40.038536+0000","last_fullsized":"2024-09-16T10:29:40.038536+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:21.013503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.013503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.013620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":932,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184246+0000","last_change":"2024-09-16T10:27:12.407528+0000","last_active":"2024-09-16T10:29:39.184246+0000","last_peered":"2024-09-16T10:29:39.184246+0000","last_clean":"2024-09-16T10:29:39.184246+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:29:39.184246+0000","last_undegraded":"2024-09-16T10:29:39.184246+0000","last_fullsized":"2024-09-16T10:29:39.184246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:27:12.407441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.407441+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705045+0000","last_change":"2024-09-16T10:28:28.066368+0000","last_active":"2024-09-16T10:29:39.705045+0000","last_peered":"2024-09-16T10:29:39.705045+0000","last_clean":"2024-09-16T10:29:39.705045+0000","last_became_active":"2024-09-16T10:28:28.065738+0000","last_became_peered":"2024-09-16T10:28:28.065738+0000","last_unstale":"2024-09-16T10:29:39.705045+0000","last_undegraded":"2024-09-16T10:29:39.705045+0000","last_fullsized":"2024-09-16T10:29:39.705045+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":281,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039274+0000","last_change":"2024-09-16T10:28:03.060402+0000","last_active":"2024-09-16T10:29:40.039274+0000","last_peered":"2024-09-16T10:29:40.039274+0000","last_clean":"2024-09-16T10:29:40.039274+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:29:40.039274+0000","last_undegraded":"2024-09-16T10:29:40.039274+0000","last_fullsized":"2024-09-16T10:29:40.039274+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:28:03.060327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:28:03.060327+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":889,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039052+0000","last_change":"2024-09-16T10:28:07.007838+0000","last_active":"2024-09-16T10:29:40.039052+0000","last_peered":"2024-09-16T10:29:40.039052+0000","last_clean":"2024-09-16T10:29:40.039052+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:29:40.039052+0000","last_undegraded":"2024-09-16T10:29:40.039052+0000","last_fullsized":"2024-09-16T10:29:40.039052+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:28:07.007777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:28:07.007777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"480'528","reported_seq":545,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.007345+0000","last_change":"2024-09-16T10:29:35.279709+0000","last_active":"2024-09-16T10:29:43.007345+0000","last_peered":"2024-09-16T10:29:43.007345+0000","last_clean":"2024-09-16T10:29:43.007345+0000","last_became_active":"2024-09-16T10:29:35.279416+0000","last_became_peered":"2024-09-16T10:29:35.279416+0000","last_unstale":"2024-09-16T10:29:43.007345+0000","last_undegraded":"2024-09-16T10:29:43.007345+0000","last_fullsized":"2024-09-16T10:29:43.007345+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.688136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603008,"num_objects":528,"num_object_clones":0,"num_object_copies":1584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":528,"num_write_kb":33792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":320,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.293034+0000","last_change":"2024-09-16T10:27:18.552950+0000","last_active":"2024-09-16T10:29:39.293034+0000","last_peered":"2024-09-16T10:29:39.293034+0000","last_clean":"2024-09-16T10:29:39.293034+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:29:39.293034+0000","last_undegraded":"2024-09-16T10:29:39.293034+0000","last_fullsized":"2024-09-16T10:29:39.293034+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:27:17.482427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:27:17.482427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":281,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292121+0000","last_change":"2024-09-16T10:26:44.445756+0000","last_active":"2024-09-16T10:29:39.292121+0000","last_peered":"2024-09-16T10:29:39.292121+0000","last_clean":"2024-09-16T10:29:39.292121+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:29:39.292121+0000","last_undegraded":"2024-09-16T10:29:39.292121+0000","last_fullsized":"2024-09-16T10:29:39.292121+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:26:44.445661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.445661+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":875,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705913+0000","last_change":"2024-09-16T10:28:28.098490+0000","last_active":"2024-09-16T10:29:39.705913+0000","last_peered":"2024-09-16T10:29:39.705913+0000","last_clean":"2024-09-16T10:29:39.705913+0000","last_became_active":"2024-09-16T10:28:28.097354+0000","last_became_peered":"2024-09-16T10:28:28.097354+0000","last_unstale":"2024-09-16T10:29:39.705913+0000","last_undegraded":"2024-09-16T10:29:39.705913+0000","last_fullsized":"2024-09-16T10:29:39.705913+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.404383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.404383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292265+0000","last_change":"2024-09-16T10:28:28.074218+0000","last_active":"2024-09-16T10:29:39.292265+0000","last_peered":"2024-09-16T10:29:39.292265+0000","last_clean":"2024-09-16T10:29:39.292265+0000","last_became_active":"2024-09-16T10:28:28.073750+0000","last_became_peered":"2024-09-16T10:28:28.073750+0000","last_unstale":"2024-09-16T10:29:39.292265+0000","last_undegraded":"2024-09-16T10:29:39.292265+0000","last_fullsized":"2024-09-16T10:29:39.292265+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.521586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.521586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":700,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704983+0000","last_change":"2024-09-16T10:27:40.298188+0000","last_active":"2024-09-16T10:29:39.704983+0000","last_peered":"2024-09-16T10:29:39.704983+0000","last_clean":"2024-09-16T10:29:39.704983+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:29:39.704983+0000","last_undegraded":"2024-09-16T10:29:39.704983+0000","last_fullsized":"2024-09-16T10:29:39.704983+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"480'508","reported_seq":527,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:42.986355+0000","last_change":"2024-09-16T10:29:35.280049+0000","last_active":"2024-09-16T10:29:42.986355+0000","last_peered":"2024-09-16T10:29:42.986355+0000","last_clean":"2024-09-16T10:29:42.986355+0000","last_became_active":"2024-09-16T10:29:35.279303+0000","last_became_peered":"2024-09-16T10:29:35.279303+0000","last_unstale":"2024-09-16T10:29:42.986355+0000","last_undegraded":"2024-09-16T10:29:42.986355+0000","last_fullsized":"2024-09-16T10:29:42.986355+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.356324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33292288,"num_objects":508,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":508,"num_write_kb":32512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":800,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705905+0000","last_change":"2024-09-16T10:29:36.701757+0000","last_active":"2024-09-16T10:29:39.705905+0000","last_peered":"2024-09-16T10:29:39.705905+0000","last_clean":"2024-09-16T10:29:39.705905+0000","last_became_active":"2024-09-16T10:28:28.095330+0000","last_became_peered":"2024-09-16T10:28:28.095330+0000","last_unstale":"2024-09-16T10:29:39.705905+0000","last_undegraded":"2024-09-16T10:29:39.705905+0000","last_fullsized":"2024-09-16T10:29:39.705905+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:36.701703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.701703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.846267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":837,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038651+0000","last_change":"2024-09-16T10:27:53.065126+0000","last_active":"2024-09-16T10:29:40.038651+0000","last_peered":"2024-09-16T10:29:40.038651+0000","last_clean":"2024-09-16T10:29:40.038651+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:29:40.038651+0000","last_undegraded":"2024-09-16T10:29:40.038651+0000","last_fullsized":"2024-09-16T10:29:40.038651+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:27:53.065069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:53.065069+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":893,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030743+0000","last_change":"2024-09-16T10:27:44.624946+0000","last_active":"2024-09-16T10:29:40.030743+0000","last_peered":"2024-09-16T10:29:40.030743+0000","last_clean":"2024-09-16T10:29:40.030743+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:29:40.030743+0000","last_undegraded":"2024-09-16T10:29:40.030743+0000","last_fullsized":"2024-09-16T10:29:40.030743+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":1096,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704767+0000","last_change":"2024-09-16T10:27:24.281038+0000","last_active":"2024-09-16T10:29:39.704767+0000","last_peered":"2024-09-16T10:29:39.704767+0000","last_clean":"2024-09-16T10:29:39.704767+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:29:39.704767+0000","last_undegraded":"2024-09-16T10:29:39.704767+0000","last_fullsized":"2024-09-16T10:29:39.704767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:27:24.280960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:24.280960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":751,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292730+0000","last_change":"2024-09-16T10:29:20.267647+0000","last_active":"2024-09-16T10:29:39.292730+0000","last_peered":"2024-09-16T10:29:39.292730+0000","last_clean":"2024-09-16T10:29:39.292730+0000","last_became_active":"2024-09-16T10:28:28.071989+0000","last_became_peered":"2024-09-16T10:28:28.071989+0000","last_unstale":"2024-09-16T10:29:39.292730+0000","last_undegraded":"2024-09-16T10:29:39.292730+0000","last_fullsized":"2024-09-16T10:29:39.292730+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.267597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.267597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:29.879063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"480'666","reported_seq":685,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:44.165259+0000","last_change":"2024-09-16T10:29:35.170854+0000","last_active":"2024-09-16T10:29:44.165259+0000","last_peered":"2024-09-16T10:29:44.165259+0000","last_clean":"2024-09-16T10:29:44.165259+0000","last_became_active":"2024-09-16T10:29:35.170400+0000","last_became_peered":"2024-09-16T10:29:35.170400+0000","last_unstale":"2024-09-16T10:29:44.165259+0000","last_undegraded":"2024-09-16T10:29:44.165259+0000","last_fullsized":"2024-09-16T10:29:44.165259+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.870813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43646976,"num_objects":666,"num_object_clones":0,"num_object_copies":1998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":42624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":850,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292928+0000","last_change":"2024-09-16T10:29:29.272700+0000","last_active":"2024-09-16T10:29:39.292928+0000","last_peered":"2024-09-16T10:29:39.292928+0000","last_clean":"2024-09-16T10:29:39.292928+0000","last_became_active":"2024-09-16T10:28:28.089337+0000","last_became_peered":"2024-09-16T10:28:28.089337+0000","last_unstale":"2024-09-16T10:29:39.292928+0000","last_undegraded":"2024-09-16T10:29:39.292928+0000","last_fullsized":"2024-09-16T10:29:39.292928+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.272650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.272650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.748781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":935,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182521+0000","last_change":"2024-09-16T10:29:25.663951+0000","last_active":"2024-09-16T10:29:39.182521+0000","last_peered":"2024-09-16T10:29:39.182521+0000","last_clean":"2024-09-16T10:29:39.182521+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:29:39.182521+0000","last_undegraded":"2024-09-16T10:29:39.182521+0000","last_fullsized":"2024-09-16T10:29:39.182521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.992176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":915,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039462+0000","last_change":"2024-09-16T10:29:27.891418+0000","last_active":"2024-09-16T10:29:40.039462+0000","last_peered":"2024-09-16T10:29:40.039462+0000","last_clean":"2024-09-16T10:29:40.039462+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:29:40.039462+0000","last_undegraded":"2024-09-16T10:29:40.039462+0000","last_fullsized":"2024-09-16T10:29:40.039462+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.891362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.891362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.339178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1078,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030223+0000","last_change":"2024-09-16T10:26:46.635924+0000","last_active":"2024-09-16T10:29:40.030223+0000","last_peered":"2024-09-16T10:29:40.030223+0000","last_clean":"2024-09-16T10:29:40.030223+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:29:40.030223+0000","last_undegraded":"2024-09-16T10:29:40.030223+0000","last_fullsized":"2024-09-16T10:29:40.030223+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:46.635855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.635855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029770+0000","last_change":"2024-09-16T10:26:52.648107+0000","last_active":"2024-09-16T10:29:40.029770+0000","last_peered":"2024-09-16T10:29:40.029770+0000","last_clean":"2024-09-16T10:29:40.029770+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:29:40.029770+0000","last_undegraded":"2024-09-16T10:29:40.029770+0000","last_fullsized":"2024-09-16T10:29:40.029770+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:52.648041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:52.648041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"480'490","reported_seq":509,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:42.983001+0000","last_change":"2024-09-16T10:29:35.168446+0000","last_active":"2024-09-16T10:29:42.983001+0000","last_peered":"2024-09-16T10:29:42.983001+0000","last_clean":"2024-09-16T10:29:42.983001+0000","last_became_active":"2024-09-16T10:29:35.168175+0000","last_became_peered":"2024-09-16T10:29:35.168175+0000","last_unstale":"2024-09-16T10:29:42.983001+0000","last_undegraded":"2024-09-16T10:29:42.983001+0000","last_fullsized":"2024-09-16T10:29:42.983001+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.317930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32112640,"num_objects":490,"num_object_clones":0,"num_object_copies":1470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":490,"num_write_kb":31360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":844,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292764+0000","last_change":"2024-09-16T10:26:51.422615+0000","last_active":"2024-09-16T10:29:39.292764+0000","last_peered":"2024-09-16T10:29:39.292764+0000","last_clean":"2024-09-16T10:29:39.292764+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:29:39.292764+0000","last_undegraded":"2024-09-16T10:29:39.292764+0000","last_fullsized":"2024-09-16T10:29:39.292764+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:51.422550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:51.422550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":958,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030299+0000","last_change":"2024-09-16T10:29:27.111777+0000","last_active":"2024-09-16T10:29:40.030299+0000","last_peered":"2024-09-16T10:29:40.030299+0000","last_clean":"2024-09-16T10:29:40.030299+0000","last_became_active":"2024-09-16T10:28:28.112972+0000","last_became_peered":"2024-09-16T10:28:28.112972+0000","last_unstale":"2024-09-16T10:29:40.030299+0000","last_undegraded":"2024-09-16T10:29:40.030299+0000","last_fullsized":"2024-09-16T10:29:40.030299+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:29:27.111715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.111715+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.694926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":915,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183649+0000","last_change":"2024-09-16T10:29:26.686298+0000","last_active":"2024-09-16T10:29:39.183649+0000","last_peered":"2024-09-16T10:29:39.183649+0000","last_clean":"2024-09-16T10:29:39.183649+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:29:39.183649+0000","last_undegraded":"2024-09-16T10:29:39.183649+0000","last_fullsized":"2024-09-16T10:29:39.183649+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.882005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":900,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292413+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:29:39.292413+0000","last_peered":"2024-09-16T10:29:39.292413+0000","last_clean":"2024-09-16T10:29:39.292413+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:29:39.292413+0000","last_undegraded":"2024-09-16T10:29:39.292413+0000","last_fullsized":"2024-09-16T10:29:39.292413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":895,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.181581+0000","last_change":"2024-09-16T10:27:41.587855+0000","last_active":"2024-09-16T10:29:39.181581+0000","last_peered":"2024-09-16T10:29:39.181581+0000","last_clean":"2024-09-16T10:29:39.181581+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:29:39.181581+0000","last_undegraded":"2024-09-16T10:29:39.181581+0000","last_fullsized":"2024-09-16T10:29:39.181581+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"480'589","reported_seq":607,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.048681+0000","last_change":"2024-09-16T10:29:35.171242+0000","last_active":"2024-09-16T10:29:43.048681+0000","last_peered":"2024-09-16T10:29:43.048681+0000","last_clean":"2024-09-16T10:29:43.048681+0000","last_became_active":"2024-09-16T10:29:35.171007+0000","last_became_peered":"2024-09-16T10:29:35.171007+0000","last_unstale":"2024-09-16T10:29:43.048681+0000","last_undegraded":"2024-09-16T10:29:43.048681+0000","last_fullsized":"2024-09-16T10:29:43.048681+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.125636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":589,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":790,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038455+0000","last_change":"2024-09-16T10:26:09.992070+0000","last_active":"2024-09-16T10:29:40.038455+0000","last_peered":"2024-09-16T10:29:40.038455+0000","last_clean":"2024-09-16T10:29:40.038455+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:29:40.038455+0000","last_undegraded":"2024-09-16T10:29:40.038455+0000","last_fullsized":"2024-09-16T10:29:40.038455+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.992000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.992000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":1231,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183591+0000","last_change":"2024-09-16T10:27:14.409646+0000","last_active":"2024-09-16T10:29:39.183591+0000","last_peered":"2024-09-16T10:29:39.183591+0000","last_clean":"2024-09-16T10:29:39.183591+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:29:39.183591+0000","last_undegraded":"2024-09-16T10:29:39.183591+0000","last_fullsized":"2024-09-16T10:29:39.183591+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:27:14.409552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:14.409552+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030545+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:29:40.030545+0000","last_peered":"2024-09-16T10:29:40.030545+0000","last_clean":"2024-09-16T10:29:40.030545+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:29:40.030545+0000","last_undegraded":"2024-09-16T10:29:40.030545+0000","last_fullsized":"2024-09-16T10:29:40.030545+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":911,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038890+0000","last_change":"2024-09-16T10:27:23.145340+0000","last_active":"2024-09-16T10:29:40.038890+0000","last_peered":"2024-09-16T10:29:40.038890+0000","last_clean":"2024-09-16T10:29:40.038890+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:29:40.038890+0000","last_undegraded":"2024-09-16T10:29:40.038890+0000","last_fullsized":"2024-09-16T10:29:40.038890+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:27:23.145278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:27:23.145278+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":876,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030390+0000","last_change":"2024-09-16T10:25:37.359150+0000","last_active":"2024-09-16T10:29:40.030390+0000","last_peered":"2024-09-16T10:29:40.030390+0000","last_clean":"2024-09-16T10:29:40.030390+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:29:40.030390+0000","last_undegraded":"2024-09-16T10:29:40.030390+0000","last_fullsized":"2024-09-16T10:29:40.030390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:37.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":906,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705578+0000","last_change":"2024-09-16T10:28:28.097003+0000","last_active":"2024-09-16T10:29:39.705578+0000","last_peered":"2024-09-16T10:29:39.705578+0000","last_clean":"2024-09-16T10:29:39.705578+0000","last_became_active":"2024-09-16T10:28:28.096691+0000","last_became_peered":"2024-09-16T10:28:28.096691+0000","last_unstale":"2024-09-16T10:29:39.705578+0000","last_undegraded":"2024-09-16T10:29:39.705578+0000","last_fullsized":"2024-09-16T10:29:39.705578+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:26.349415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:26.349415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"480'519","reported_seq":538,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.045207+0000","last_change":"2024-09-16T10:29:35.167788+0000","last_active":"2024-09-16T10:29:43.045207+0000","last_peered":"2024-09-16T10:29:43.045207+0000","last_clean":"2024-09-16T10:29:43.045207+0000","last_became_active":"2024-09-16T10:29:35.167515+0000","last_became_peered":"2024-09-16T10:29:35.167515+0000","last_unstale":"2024-09-16T10:29:43.045207+0000","last_undegraded":"2024-09-16T10:29:43.045207+0000","last_fullsized":"2024-09-16T10:29:43.045207+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.448815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":519,"num_write_kb":33216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":366,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029898+0000","last_change":"2024-09-16T10:28:28.073697+0000","last_active":"2024-09-16T10:29:40.029898+0000","last_peered":"2024-09-16T10:29:40.029898+0000","last_clean":"2024-09-16T10:29:40.029898+0000","last_became_active":"2024-09-16T10:28:28.073341+0000","last_became_peered":"2024-09-16T10:28:28.073341+0000","last_unstale":"2024-09-16T10:29:40.029898+0000","last_undegraded":"2024-09-16T10:29:40.029898+0000","last_fullsized":"2024-09-16T10:29:40.029898+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.028638+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":906,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184336+0000","last_change":"2024-09-16T10:25:13.649143+0000","last_active":"2024-09-16T10:29:39.184336+0000","last_peered":"2024-09-16T10:29:39.184336+0000","last_clean":"2024-09-16T10:29:39.184336+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:29:39.184336+0000","last_undegraded":"2024-09-16T10:29:39.184336+0000","last_fullsized":"2024-09-16T10:29:39.184336+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:25:13.648981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.648981+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":900,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.181406+0000","last_change":"2024-09-16T10:29:20.691013+0000","last_active":"2024-09-16T10:29:39.181406+0000","last_peered":"2024-09-16T10:29:39.181406+0000","last_clean":"2024-09-16T10:29:39.181406+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:29:39.181406+0000","last_undegraded":"2024-09-16T10:29:39.181406+0000","last_fullsized":"2024-09-16T10:29:39.181406+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:41.420313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":430,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182949+0000","last_change":"2024-09-16T10:26:18.549126+0000","last_active":"2024-09-16T10:29:39.182949+0000","last_peered":"2024-09-16T10:29:39.182949+0000","last_clean":"2024-09-16T10:29:39.182949+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:29:39.182949+0000","last_undegraded":"2024-09-16T10:29:39.182949+0000","last_fullsized":"2024-09-16T10:29:39.182949+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:26:18.549005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:26:18.549005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":437,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184816+0000","last_change":"2024-09-16T10:29:24.484142+0000","last_active":"2024-09-16T10:29:39.184816+0000","last_peered":"2024-09-16T10:29:39.184816+0000","last_clean":"2024-09-16T10:29:39.184816+0000","last_became_active":"2024-09-16T10:28:28.081619+0000","last_became_peered":"2024-09-16T10:28:28.081619+0000","last_unstale":"2024-09-16T10:29:39.184816+0000","last_undegraded":"2024-09-16T10:29:39.184816+0000","last_fullsized":"2024-09-16T10:29:39.184816+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.484086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.484086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.572572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":291,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292740+0000","last_change":"2024-09-16T10:28:28.073905+0000","last_active":"2024-09-16T10:29:39.292740+0000","last_peered":"2024-09-16T10:29:39.292740+0000","last_clean":"2024-09-16T10:29:39.292740+0000","last_became_active":"2024-09-16T10:28:28.073357+0000","last_became_peered":"2024-09-16T10:28:28.073357+0000","last_unstale":"2024-09-16T10:29:39.292740+0000","last_undegraded":"2024-09-16T10:29:39.292740+0000","last_fullsized":"2024-09-16T10:29:39.292740+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"480'509","reported_seq":528,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.010734+0000","last_change":"2024-09-16T10:29:35.171912+0000","last_active":"2024-09-16T10:29:43.010734+0000","last_peered":"2024-09-16T10:29:43.010734+0000","last_clean":"2024-09-16T10:29:43.010734+0000","last_became_active":"2024-09-16T10:29:35.171449+0000","last_became_peered":"2024-09-16T10:29:35.171449+0000","last_unstale":"2024-09-16T10:29:43.010734+0000","last_undegraded":"2024-09-16T10:29:43.010734+0000","last_fullsized":"2024-09-16T10:29:43.010734+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":32576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":778,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182950+0000","last_change":"2024-09-16T10:26:09.201694+0000","last_active":"2024-09-16T10:29:39.182950+0000","last_peered":"2024-09-16T10:29:39.182950+0000","last_clean":"2024-09-16T10:29:39.182950+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:29:39.182950+0000","last_undegraded":"2024-09-16T10:29:39.182950+0000","last_fullsized":"2024-09-16T10:29:39.182950+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.201559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.201559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":990,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030686+0000","last_change":"2024-09-16T10:27:09.622319+0000","last_active":"2024-09-16T10:29:40.030686+0000","last_peered":"2024-09-16T10:29:40.030686+0000","last_clean":"2024-09-16T10:29:40.030686+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:29:40.030686+0000","last_undegraded":"2024-09-16T10:29:40.030686+0000","last_fullsized":"2024-09-16T10:29:40.030686+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:27:09.622239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.622239+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029969+0000","last_change":"2024-09-16T10:27:10.642822+0000","last_active":"2024-09-16T10:29:40.029969+0000","last_peered":"2024-09-16T10:29:40.029969+0000","last_clean":"2024-09-16T10:29:40.029969+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:29:40.029969+0000","last_undegraded":"2024-09-16T10:29:40.029969+0000","last_fullsized":"2024-09-16T10:29:40.029969+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:10.642739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:10.642739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":613,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.345232+0000","last_change":"2024-09-16T10:29:30.751685+0000","last_active":"2024-09-16T10:29:41.345232+0000","last_peered":"2024-09-16T10:29:41.345232+0000","last_clean":"2024-09-16T10:29:41.345232+0000","last_became_active":"2024-09-16T10:28:28.070293+0000","last_became_peered":"2024-09-16T10:28:28.070293+0000","last_unstale":"2024-09-16T10:29:41.345232+0000","last_undegraded":"2024-09-16T10:29:41.345232+0000","last_fullsized":"2024-09-16T10:29:41.345232+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:29:30.751623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.751623+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:34.435816+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":285,"num_read_kb":280,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":901,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039461+0000","last_change":"2024-09-16T10:27:20.130780+0000","last_active":"2024-09-16T10:29:40.039461+0000","last_peered":"2024-09-16T10:29:40.039461+0000","last_clean":"2024-09-16T10:29:40.039461+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:29:40.039461+0000","last_undegraded":"2024-09-16T10:29:40.039461+0000","last_fullsized":"2024-09-16T10:29:40.039461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:20.130715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.130715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":905,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183288+0000","last_change":"2024-09-16T10:27:19.374485+0000","last_active":"2024-09-16T10:29:39.183288+0000","last_peered":"2024-09-16T10:29:39.183288+0000","last_clean":"2024-09-16T10:29:39.183288+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:29:39.183288+0000","last_undegraded":"2024-09-16T10:29:39.183288+0000","last_fullsized":"2024-09-16T10:29:39.183288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"480'548","reported_seq":567,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.040876+0000","last_change":"2024-09-16T10:29:35.169178+0000","last_active":"2024-09-16T10:29:43.040876+0000","last_peered":"2024-09-16T10:29:43.040876+0000","last_clean":"2024-09-16T10:29:43.040876+0000","last_became_active":"2024-09-16T10:29:35.168851+0000","last_became_peered":"2024-09-16T10:29:35.168851+0000","last_unstale":"2024-09-16T10:29:43.040876+0000","last_undegraded":"2024-09-16T10:29:43.040876+0000","last_fullsized":"2024-09-16T10:29:43.040876+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.747588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"442'30","reported_seq":1126,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292955+0000","last_change":"2024-09-16T10:29:18.202869+0000","last_active":"2024-09-16T10:29:39.292955+0000","last_peered":"2024-09-16T10:29:39.292955+0000","last_clean":"2024-09-16T10:29:39.292955+0000","last_became_active":"2024-09-16T10:28:28.072900+0000","last_became_peered":"2024-09-16T10:28:28.072900+0000","last_unstale":"2024-09-16T10:29:39.292955+0000","last_undegraded":"2024-09-16T10:29:39.292955+0000","last_fullsized":"2024-09-16T10:29:39.292955+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:29:18.202771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:18.202771+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:45.201112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":864,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705380+0000","last_change":"2024-09-16T10:26:47.493915+0000","last_active":"2024-09-16T10:29:39.705380+0000","last_peered":"2024-09-16T10:29:39.705380+0000","last_clean":"2024-09-16T10:29:39.705380+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:29:39.705380+0000","last_undegraded":"2024-09-16T10:29:39.705380+0000","last_fullsized":"2024-09-16T10:29:39.705380+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:47.493848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:47.493848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":366,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029935+0000","last_change":"2024-09-16T10:28:28.090815+0000","last_active":"2024-09-16T10:29:40.029935+0000","last_peered":"2024-09-16T10:29:40.029935+0000","last_clean":"2024-09-16T10:29:40.029935+0000","last_became_active":"2024-09-16T10:28:28.090031+0000","last_became_peered":"2024-09-16T10:28:28.090031+0000","last_unstale":"2024-09-16T10:29:40.029935+0000","last_undegraded":"2024-09-16T10:29:40.029935+0000","last_fullsized":"2024-09-16T10:29:40.029935+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":762,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184658+0000","last_change":"2024-09-16T10:28:28.072722+0000","last_active":"2024-09-16T10:29:39.184658+0000","last_peered":"2024-09-16T10:29:39.184658+0000","last_clean":"2024-09-16T10:29:39.184658+0000","last_became_active":"2024-09-16T10:28:28.072192+0000","last_became_peered":"2024-09-16T10:28:28.072192+0000","last_unstale":"2024-09-16T10:29:39.184658+0000","last_undegraded":"2024-09-16T10:29:39.184658+0000","last_fullsized":"2024-09-16T10:29:39.184658+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.591323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":889,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039357+0000","last_change":"2024-09-16T10:26:19.084467+0000","last_active":"2024-09-16T10:29:40.039357+0000","last_peered":"2024-09-16T10:29:40.039357+0000","last_clean":"2024-09-16T10:29:40.039357+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:29:40.039357+0000","last_undegraded":"2024-09-16T10:29:40.039357+0000","last_fullsized":"2024-09-16T10:29:40.039357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:19.084409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:19.084409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"480'568","reported_seq":587,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.558785+0000","last_change":"2024-09-16T10:29:35.171216+0000","last_active":"2024-09-16T10:29:43.558785+0000","last_peered":"2024-09-16T10:29:43.558785+0000","last_clean":"2024-09-16T10:29:43.558785+0000","last_became_active":"2024-09-16T10:29:35.170866+0000","last_became_peered":"2024-09-16T10:29:35.170866+0000","last_unstale":"2024-09-16T10:29:43.558785+0000","last_undegraded":"2024-09-16T10:29:43.558785+0000","last_fullsized":"2024-09-16T10:29:43.558785+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.526803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37224448,"num_objects":568,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":568,"num_write_kb":36352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":786,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182722+0000","last_change":"2024-09-16T10:29:27.703960+0000","last_active":"2024-09-16T10:29:39.182722+0000","last_peered":"2024-09-16T10:29:39.182722+0000","last_clean":"2024-09-16T10:29:39.182722+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:29:39.182722+0000","last_undegraded":"2024-09-16T10:29:39.182722+0000","last_fullsized":"2024-09-16T10:29:39.182722+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:27.703908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.703908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.239882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":829,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292878+0000","last_change":"2024-09-16T10:26:28.556079+0000","last_active":"2024-09-16T10:29:39.292878+0000","last_peered":"2024-09-16T10:29:39.292878+0000","last_clean":"2024-09-16T10:29:39.292878+0000","last_became_active":"2024-09-16T10:26:28.554604+0000","last_became_peered":"2024-09-16T10:26:28.554604+0000","last_unstale":"2024-09-16T10:29:39.292878+0000","last_undegraded":"2024-09-16T10:29:39.292878+0000","last_fullsized":"2024-09-16T10:29:39.292878+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:26:10.386128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.386128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":706,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182369+0000","last_change":"2024-09-16T10:29:23.474783+0000","last_active":"2024-09-16T10:29:39.182369+0000","last_peered":"2024-09-16T10:29:39.182369+0000","last_clean":"2024-09-16T10:29:39.182369+0000","last_became_active":"2024-09-16T10:28:28.071386+0000","last_became_peered":"2024-09-16T10:28:28.071386+0000","last_unstale":"2024-09-16T10:29:39.182369+0000","last_undegraded":"2024-09-16T10:29:39.182369+0000","last_fullsized":"2024-09-16T10:29:39.182369+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.474733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.474733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:52.069257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":901,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039288+0000","last_change":"2024-09-16T10:29:30.955964+0000","last_active":"2024-09-16T10:29:40.039288+0000","last_peered":"2024-09-16T10:29:40.039288+0000","last_clean":"2024-09-16T10:29:40.039288+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:29:40.039288+0000","last_undegraded":"2024-09-16T10:29:40.039288+0000","last_fullsized":"2024-09-16T10:29:40.039288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:30.955877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.955877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:51.333008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":835,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705833+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:29:39.705833+0000","last_peered":"2024-09-16T10:29:39.705833+0000","last_clean":"2024-09-16T10:29:39.705833+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:29:39.705833+0000","last_undegraded":"2024-09-16T10:29:39.705833+0000","last_fullsized":"2024-09-16T10:29:39.705833+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.037961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"480'720","reported_seq":739,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:44.174634+0000","last_change":"2024-09-16T10:29:35.171033+0000","last_active":"2024-09-16T10:29:44.174634+0000","last_peered":"2024-09-16T10:29:44.174634+0000","last_clean":"2024-09-16T10:29:44.174634+0000","last_became_active":"2024-09-16T10:29:35.170730+0000","last_became_peered":"2024-09-16T10:29:35.170730+0000","last_unstale":"2024-09-16T10:29:44.174634+0000","last_undegraded":"2024-09-16T10:29:44.174634+0000","last_fullsized":"2024-09-16T10:29:44.174634+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:36.397873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47185920,"num_objects":720,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":46080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":753,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038432+0000","last_change":"2024-09-16T10:26:49.093325+0000","last_active":"2024-09-16T10:29:40.038432+0000","last_peered":"2024-09-16T10:29:40.038432+0000","last_clean":"2024-09-16T10:29:40.038432+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:29:40.038432+0000","last_undegraded":"2024-09-16T10:29:40.038432+0000","last_fullsized":"2024-09-16T10:29:40.038432+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:49.093245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:49.093245+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":893,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184430+0000","last_change":"2024-09-16T10:27:21.459062+0000","last_active":"2024-09-16T10:29:39.184430+0000","last_peered":"2024-09-16T10:29:39.184430+0000","last_clean":"2024-09-16T10:29:39.184430+0000","last_became_active":"2024-09-16T10:26:28.553171+0000","last_became_peered":"2024-09-16T10:26:28.553171+0000","last_unstale":"2024-09-16T10:29:39.184430+0000","last_undegraded":"2024-09-16T10:29:39.184430+0000","last_fullsized":"2024-09-16T10:29:39.184430+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:27:21.458969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:21.458969+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":892,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292857+0000","last_change":"2024-09-16T10:26:42.490866+0000","last_active":"2024-09-16T10:29:39.292857+0000","last_peered":"2024-09-16T10:29:39.292857+0000","last_clean":"2024-09-16T10:29:39.292857+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:29:39.292857+0000","last_undegraded":"2024-09-16T10:29:39.292857+0000","last_fullsized":"2024-09-16T10:29:39.292857+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:42.490731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.490731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":885,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183319+0000","last_change":"2024-09-16T10:29:21.720973+0000","last_active":"2024-09-16T10:29:39.183319+0000","last_peered":"2024-09-16T10:29:39.183319+0000","last_clean":"2024-09-16T10:29:39.183319+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:29:39.183319+0000","last_undegraded":"2024-09-16T10:29:39.183319+0000","last_fullsized":"2024-09-16T10:29:39.183319+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.720910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.720910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.270176+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184922+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:29:39.184922+0000","last_peered":"2024-09-16T10:29:39.184922+0000","last_clean":"2024-09-16T10:29:39.184922+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:29:39.184922+0000","last_undegraded":"2024-09-16T10:29:39.184922+0000","last_fullsized":"2024-09-16T10:29:39.184922+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"480'631","reported_seq":650,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.528944+0000","last_change":"2024-09-16T10:29:35.279567+0000","last_active":"2024-09-16T10:29:43.528944+0000","last_peered":"2024-09-16T10:29:43.528944+0000","last_clean":"2024-09-16T10:29:43.528944+0000","last_became_active":"2024-09-16T10:29:35.279253+0000","last_became_peered":"2024-09-16T10:29:35.279253+0000","last_unstale":"2024-09-16T10:29:43.528944+0000","last_undegraded":"2024-09-16T10:29:43.528944+0000","last_fullsized":"2024-09-16T10:29:43.528944+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.137735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":631,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":566,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704810+0000","last_change":"2024-09-16T10:28:28.096386+0000","last_active":"2024-09-16T10:29:39.704810+0000","last_peered":"2024-09-16T10:29:39.704810+0000","last_clean":"2024-09-16T10:29:39.704810+0000","last_became_active":"2024-09-16T10:28:28.095730+0000","last_became_peered":"2024-09-16T10:28:28.095730+0000","last_unstale":"2024-09-16T10:29:39.704810+0000","last_undegraded":"2024-09-16T10:29:39.704810+0000","last_fullsized":"2024-09-16T10:29:39.704810+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:16.433199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:16.433199+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":809,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705534+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:29:39.705534+0000","last_peered":"2024-09-16T10:29:39.705534+0000","last_clean":"2024-09-16T10:29:39.705534+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:29:39.705534+0000","last_undegraded":"2024-09-16T10:29:39.705534+0000","last_fullsized":"2024-09-16T10:29:39.705534+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.568656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705387+0000","last_change":"2024-09-16T10:27:12.155297+0000","last_active":"2024-09-16T10:29:39.705387+0000","last_peered":"2024-09-16T10:29:39.705387+0000","last_clean":"2024-09-16T10:29:39.705387+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:29:39.705387+0000","last_undegraded":"2024-09-16T10:29:39.705387+0000","last_fullsized":"2024-09-16T10:29:39.705387+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:12.155186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.155186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"442'12","reported_seq":535,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184665+0000","last_change":"2024-09-16T10:29:21.445504+0000","last_active":"2024-09-16T10:29:39.184665+0000","last_peered":"2024-09-16T10:29:39.184665+0000","last_clean":"2024-09-16T10:29:39.184665+0000","last_became_active":"2024-09-16T10:28:28.072074+0000","last_became_peered":"2024-09-16T10:28:28.072074+0000","last_unstale":"2024-09-16T10:29:39.184665+0000","last_undegraded":"2024-09-16T10:29:39.184665+0000","last_fullsized":"2024-09-16T10:29:39.184665+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:29:21.445439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.445439+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.856000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":692,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183464+0000","last_change":"2024-09-16T10:27:13.405307+0000","last_active":"2024-09-16T10:29:39.183464+0000","last_peered":"2024-09-16T10:29:39.183464+0000","last_clean":"2024-09-16T10:29:39.183464+0000","last_became_active":"2024-09-16T10:26:28.550959+0000","last_became_peered":"2024-09-16T10:26:28.550959+0000","last_unstale":"2024-09-16T10:29:39.183464+0000","last_undegraded":"2024-09-16T10:29:39.183464+0000","last_fullsized":"2024-09-16T10:29:39.183464+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.405239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.405239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"480'612","reported_seq":631,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.550608+0000","last_change":"2024-09-16T10:29:35.280282+0000","last_active":"2024-09-16T10:29:43.550608+0000","last_peered":"2024-09-16T10:29:43.550608+0000","last_clean":"2024-09-16T10:29:43.550608+0000","last_became_active":"2024-09-16T10:29:35.279983+0000","last_became_peered":"2024-09-16T10:29:35.279983+0000","last_unstale":"2024-09-16T10:29:43.550608+0000","last_undegraded":"2024-09-16T10:29:43.550608+0000","last_fullsized":"2024-09-16T10:29:43.550608+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.246468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40108032,"num_objects":612,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":612,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":231,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.395296+0000","last_change":"2024-09-16T10:29:28.900601+0000","last_active":"2024-09-16T10:29:39.395296+0000","last_peered":"2024-09-16T10:29:39.395296+0000","last_clean":"2024-09-16T10:29:39.395296+0000","last_became_active":"2024-09-16T10:28:28.076677+0000","last_became_peered":"2024-09-16T10:28:28.076677+0000","last_unstale":"2024-09-16T10:29:39.395296+0000","last_undegraded":"2024-09-16T10:29:39.395296+0000","last_fullsized":"2024-09-16T10:29:39.395296+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.900551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.900551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:32.135504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":913,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029537+0000","last_change":"2024-09-16T10:27:20.653839+0000","last_active":"2024-09-16T10:29:40.029537+0000","last_peered":"2024-09-16T10:29:40.029537+0000","last_clean":"2024-09-16T10:29:40.029537+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:29:40.029537+0000","last_undegraded":"2024-09-16T10:29:40.029537+0000","last_fullsized":"2024-09-16T10:29:40.029537+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:27:20.653762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.653762+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":909,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039097+0000","last_change":"2024-09-16T10:26:44.070285+0000","last_active":"2024-09-16T10:29:40.039097+0000","last_peered":"2024-09-16T10:29:40.039097+0000","last_clean":"2024-09-16T10:29:40.039097+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:29:40.039097+0000","last_undegraded":"2024-09-16T10:29:40.039097+0000","last_fullsized":"2024-09-16T10:29:40.039097+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:44.070218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.070218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184487+0000","last_change":"2024-09-16T10:26:22.501028+0000","last_active":"2024-09-16T10:29:39.184487+0000","last_peered":"2024-09-16T10:29:39.184487+0000","last_clean":"2024-09-16T10:29:39.184487+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:29:39.184487+0000","last_undegraded":"2024-09-16T10:29:39.184487+0000","last_fullsized":"2024-09-16T10:29:39.184487+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.500963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.500963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":928,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029995+0000","last_change":"2024-09-16T10:28:28.075186+0000","last_active":"2024-09-16T10:29:40.029995+0000","last_peered":"2024-09-16T10:29:40.029995+0000","last_clean":"2024-09-16T10:29:40.029995+0000","last_became_active":"2024-09-16T10:28:28.074693+0000","last_became_peered":"2024-09-16T10:28:28.074693+0000","last_unstale":"2024-09-16T10:29:40.029995+0000","last_undegraded":"2024-09-16T10:29:40.029995+0000","last_fullsized":"2024-09-16T10:29:40.029995+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:26:20.555476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.555476+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"480'514","reported_seq":533,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.006368+0000","last_change":"2024-09-16T10:29:35.279583+0000","last_active":"2024-09-16T10:29:43.006368+0000","last_peered":"2024-09-16T10:29:43.006368+0000","last_clean":"2024-09-16T10:29:43.006368+0000","last_became_active":"2024-09-16T10:29:35.279260+0000","last_became_peered":"2024-09-16T10:29:35.279260+0000","last_unstale":"2024-09-16T10:29:43.006368+0000","last_undegraded":"2024-09-16T10:29:43.006368+0000","last_fullsized":"2024-09-16T10:29:43.006368+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.395721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":514,"num_object_clones":0,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":514,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":857,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.293152+0000","last_change":"2024-09-16T10:25:40.594525+0000","last_active":"2024-09-16T10:29:39.293152+0000","last_peered":"2024-09-16T10:29:39.293152+0000","last_clean":"2024-09-16T10:29:39.293152+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:29:39.293152+0000","last_undegraded":"2024-09-16T10:29:39.293152+0000","last_fullsized":"2024-09-16T10:29:39.293152+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:40.594454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.594454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.350308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":970,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292531+0000","last_change":"2024-09-16T10:28:28.088838+0000","last_active":"2024-09-16T10:29:39.292531+0000","last_peered":"2024-09-16T10:29:39.292531+0000","last_clean":"2024-09-16T10:29:39.292531+0000","last_became_active":"2024-09-16T10:28:28.088338+0000","last_became_peered":"2024-09-16T10:28:28.088338+0000","last_unstale":"2024-09-16T10:29:39.292531+0000","last_undegraded":"2024-09-16T10:29:39.292531+0000","last_fullsized":"2024-09-16T10:29:39.292531+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":290,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039276+0000","last_change":"2024-09-16T10:28:28.101048+0000","last_active":"2024-09-16T10:29:40.039276+0000","last_peered":"2024-09-16T10:29:40.039276+0000","last_clean":"2024-09-16T10:29:40.039276+0000","last_became_active":"2024-09-16T10:28:28.100755+0000","last_became_peered":"2024-09-16T10:28:28.100755+0000","last_unstale":"2024-09-16T10:29:40.039276+0000","last_undegraded":"2024-09-16T10:29:40.039276+0000","last_fullsized":"2024-09-16T10:29:40.039276+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.031940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.031940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182145+0000","last_change":"2024-09-16T10:26:43.199824+0000","last_active":"2024-09-16T10:29:39.182145+0000","last_peered":"2024-09-16T10:29:39.182145+0000","last_clean":"2024-09-16T10:29:39.182145+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:29:39.182145+0000","last_undegraded":"2024-09-16T10:29:39.182145+0000","last_fullsized":"2024-09-16T10:29:39.182145+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:43.199752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:43.199752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":766,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183930+0000","last_change":"2024-09-16T10:28:28.065678+0000","last_active":"2024-09-16T10:29:39.183930+0000","last_peered":"2024-09-16T10:29:39.183930+0000","last_clean":"2024-09-16T10:29:39.183930+0000","last_became_active":"2024-09-16T10:28:28.065386+0000","last_became_peered":"2024-09-16T10:28:28.065386+0000","last_unstale":"2024-09-16T10:29:39.183930+0000","last_undegraded":"2024-09-16T10:29:39.183930+0000","last_fullsized":"2024-09-16T10:29:39.183930+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:34.550928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:34.550928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"480'864","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:44.944147+0000","last_change":"2024-09-16T10:29:35.172946+0000","last_active":"2024-09-16T10:29:44.944147+0000","last_peered":"2024-09-16T10:29:44.944147+0000","last_clean":"2024-09-16T10:29:44.944147+0000","last_became_active":"2024-09-16T10:29:35.172649+0000","last_became_peered":"2024-09-16T10:29:35.172649+0000","last_unstale":"2024-09-16T10:29:44.944147+0000","last_undegraded":"2024-09-16T10:29:44.944147+0000","last_fullsized":"2024-09-16T10:29:44.944147+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.718311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56623104,"num_objects":864,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":864,"num_write_kb":55296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":808,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705118+0000","last_change":"2024-09-16T10:26:10.271093+0000","last_active":"2024-09-16T10:29:39.705118+0000","last_peered":"2024-09-16T10:29:39.705118+0000","last_clean":"2024-09-16T10:29:39.705118+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:29:39.705118+0000","last_undegraded":"2024-09-16T10:29:39.705118+0000","last_fullsized":"2024-09-16T10:29:39.705118+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:10.270833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.270833+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":817,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.181781+0000","last_change":"2024-09-16T10:26:42.132944+0000","last_active":"2024-09-16T10:29:39.181781+0000","last_peered":"2024-09-16T10:29:39.181781+0000","last_clean":"2024-09-16T10:29:39.181781+0000","last_became_active":"2024-09-16T10:26:28.553750+0000","last_became_peered":"2024-09-16T10:26:28.553750+0000","last_unstale":"2024-09-16T10:29:39.181781+0000","last_undegraded":"2024-09-16T10:29:39.181781+0000","last_fullsized":"2024-09-16T10:29:39.181781+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:26:42.132853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.132853+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":239,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705470+0000","last_change":"2024-09-16T10:27:13.285419+0000","last_active":"2024-09-16T10:29:39.705470+0000","last_peered":"2024-09-16T10:29:39.705470+0000","last_clean":"2024-09-16T10:29:39.705470+0000","last_became_active":"2024-09-16T10:25:41.990483+0000","last_became_peered":"2024-09-16T10:25:41.990483+0000","last_unstale":"2024-09-16T10:29:39.705470+0000","last_undegraded":"2024-09-16T10:29:39.705470+0000","last_fullsized":"2024-09-16T10:29:39.705470+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184031+0000","last_change":"2024-09-16T10:26:21.479904+0000","last_active":"2024-09-16T10:29:39.184031+0000","last_peered":"2024-09-16T10:29:39.184031+0000","last_clean":"2024-09-16T10:29:39.184031+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:29:39.184031+0000","last_undegraded":"2024-09-16T10:29:39.184031+0000","last_fullsized":"2024-09-16T10:29:39.184031+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":898,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705122+0000","last_change":"2024-09-16T10:28:28.095795+0000","last_active":"2024-09-16T10:29:39.705122+0000","last_peered":"2024-09-16T10:29:39.705122+0000","last_clean":"2024-09-16T10:29:39.705122+0000","last_became_active":"2024-09-16T10:28:28.095174+0000","last_became_peered":"2024-09-16T10:28:28.095174+0000","last_unstale":"2024-09-16T10:29:39.705122+0000","last_undegraded":"2024-09-16T10:29:39.705122+0000","last_fullsized":"2024-09-16T10:29:39.705122+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:41.478267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:41.478267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"480'489","reported_seq":508,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:43.011705+0000","last_change":"2024-09-16T10:29:35.165138+0000","last_active":"2024-09-16T10:29:43.011705+0000","last_peered":"2024-09-16T10:29:43.011705+0000","last_clean":"2024-09-16T10:29:43.011705+0000","last_became_active":"2024-09-16T10:29:35.164843+0000","last_became_peered":"2024-09-16T10:29:35.164843+0000","last_unstale":"2024-09-16T10:29:43.011705+0000","last_undegraded":"2024-09-16T10:29:43.011705+0000","last_fullsized":"2024-09-16T10:29:43.011705+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.727993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":489,"num_object_clones":0,"num_object_copies":1467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":489,"num_write_kb":31296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":617,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.181501+0000","last_change":"2024-09-16T10:29:29.763437+0000","last_active":"2024-09-16T10:29:39.181501+0000","last_peered":"2024-09-16T10:29:39.181501+0000","last_clean":"2024-09-16T10:29:39.181501+0000","last_became_active":"2024-09-16T10:28:28.071881+0000","last_became_peered":"2024-09-16T10:28:28.071881+0000","last_unstale":"2024-09-16T10:29:39.181501+0000","last_undegraded":"2024-09-16T10:29:39.181501+0000","last_fullsized":"2024-09-16T10:29:39.181501+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:32.911633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":935,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038539+0000","last_change":"2024-09-16T10:28:28.097779+0000","last_active":"2024-09-16T10:29:40.038539+0000","last_peered":"2024-09-16T10:29:40.038539+0000","last_clean":"2024-09-16T10:29:40.038539+0000","last_became_active":"2024-09-16T10:28:28.097478+0000","last_became_peered":"2024-09-16T10:28:28.097478+0000","last_unstale":"2024-09-16T10:29:40.038539+0000","last_undegraded":"2024-09-16T10:29:40.038539+0000","last_fullsized":"2024-09-16T10:29:40.038539+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:20.104708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.104708+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":229,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030350+0000","last_change":"2024-09-16T10:28:28.081893+0000","last_active":"2024-09-16T10:29:40.030350+0000","last_peered":"2024-09-16T10:29:40.030350+0000","last_clean":"2024-09-16T10:29:40.030350+0000","last_became_active":"2024-09-16T10:28:28.081512+0000","last_became_peered":"2024-09-16T10:28:28.081512+0000","last_unstale":"2024-09-16T10:29:40.030350+0000","last_undegraded":"2024-09-16T10:29:40.030350+0000","last_fullsized":"2024-09-16T10:29:40.030350+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:08.381010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.381010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"442'11","reported_seq":374,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292431+0000","last_change":"2024-09-16T10:28:28.074072+0000","last_active":"2024-09-16T10:29:39.292431+0000","last_peered":"2024-09-16T10:29:39.292431+0000","last_clean":"2024-09-16T10:29:39.292431+0000","last_became_active":"2024-09-16T10:28:28.073675+0000","last_became_peered":"2024-09-16T10:28:28.073675+0000","last_unstale":"2024-09-16T10:29:39.292431+0000","last_undegraded":"2024-09-16T10:29:39.292431+0000","last_fullsized":"2024-09-16T10:29:39.292431+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'9","last_scrub_stamp":"2024-09-16T10:25:37.549468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.549468+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":897,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038665+0000","last_change":"2024-09-16T10:27:09.163203+0000","last_active":"2024-09-16T10:29:40.038665+0000","last_peered":"2024-09-16T10:29:40.038665+0000","last_clean":"2024-09-16T10:29:40.038665+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:29:40.038665+0000","last_undegraded":"2024-09-16T10:29:40.038665+0000","last_fullsized":"2024-09-16T10:29:40.038665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:09.163139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.163139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":473,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038819+0000","last_change":"2024-09-16T10:29:28.935051+0000","last_active":"2024-09-16T10:29:40.038819+0000","last_peered":"2024-09-16T10:29:40.038819+0000","last_clean":"2024-09-16T10:29:40.038819+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:29:40.038819+0000","last_undegraded":"2024-09-16T10:29:40.038819+0000","last_fullsized":"2024-09-16T10:29:40.038819+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.935001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.935001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.004175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292444+0000","last_change":"2024-09-16T10:26:20.469286+0000","last_active":"2024-09-16T10:29:39.292444+0000","last_peered":"2024-09-16T10:29:39.292444+0000","last_clean":"2024-09-16T10:29:39.292444+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:29:39.292444+0000","last_undegraded":"2024-09-16T10:29:39.292444+0000","last_fullsized":"2024-09-16T10:29:39.292444+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:20.469240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.469240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":915,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705871+0000","last_change":"2024-09-16T10:29:17.916063+0000","last_active":"2024-09-16T10:29:39.705871+0000","last_peered":"2024-09-16T10:29:39.705871+0000","last_clean":"2024-09-16T10:29:39.705871+0000","last_became_active":"2024-09-16T10:28:28.096974+0000","last_became_peered":"2024-09-16T10:28:28.096974+0000","last_unstale":"2024-09-16T10:29:39.705871+0000","last_undegraded":"2024-09-16T10:29:39.705871+0000","last_fullsized":"2024-09-16T10:29:39.705871+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:29:17.915982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:17.915982+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.961249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":884,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039345+0000","last_change":"2024-09-16T10:26:46.061085+0000","last_active":"2024-09-16T10:29:40.039345+0000","last_peered":"2024-09-16T10:29:40.039345+0000","last_clean":"2024-09-16T10:29:40.039345+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:29:40.039345+0000","last_undegraded":"2024-09-16T10:29:40.039345+0000","last_fullsized":"2024-09-16T10:29:40.039345+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":711,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.395349+0000","last_change":"2024-09-16T10:28:28.077871+0000","last_active":"2024-09-16T10:29:39.395349+0000","last_peered":"2024-09-16T10:29:39.395349+0000","last_clean":"2024-09-16T10:29:39.395349+0000","last_became_active":"2024-09-16T10:28:28.072905+0000","last_became_peered":"2024-09-16T10:28:28.072905+0000","last_unstale":"2024-09-16T10:29:39.395349+0000","last_undegraded":"2024-09-16T10:29:39.395349+0000","last_fullsized":"2024-09-16T10:29:39.395349+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.018433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":786,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182779+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:29:39.182779+0000","last_peered":"2024-09-16T10:29:39.182779+0000","last_clean":"2024-09-16T10:29:39.182779+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:29:39.182779+0000","last_undegraded":"2024-09-16T10:29:39.182779+0000","last_fullsized":"2024-09-16T10:29:39.182779+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":923,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030692+0000","last_change":"2024-09-16T10:28:28.083403+0000","last_active":"2024-09-16T10:29:40.030692+0000","last_peered":"2024-09-16T10:29:40.030692+0000","last_clean":"2024-09-16T10:29:40.030692+0000","last_became_active":"2024-09-16T10:28:28.082315+0000","last_became_peered":"2024-09-16T10:28:28.082315+0000","last_unstale":"2024-09-16T10:29:40.030692+0000","last_undegraded":"2024-09-16T10:29:40.030692+0000","last_fullsized":"2024-09-16T10:29:40.030692+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:24.517639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.517639+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":229,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292588+0000","last_change":"2024-09-16T10:29:24.266598+0000","last_active":"2024-09-16T10:29:39.292588+0000","last_peered":"2024-09-16T10:29:39.292588+0000","last_clean":"2024-09-16T10:29:39.292588+0000","last_became_active":"2024-09-16T10:28:28.071950+0000","last_became_peered":"2024-09-16T10:28:28.071950+0000","last_unstale":"2024-09-16T10:29:39.292588+0000","last_undegraded":"2024-09-16T10:29:39.292588+0000","last_fullsized":"2024-09-16T10:29:39.292588+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.266543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.266543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:41.996180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":893,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705766+0000","last_change":"2024-09-16T10:28:28.098654+0000","last_active":"2024-09-16T10:29:39.705766+0000","last_peered":"2024-09-16T10:29:39.705766+0000","last_clean":"2024-09-16T10:29:39.705766+0000","last_became_active":"2024-09-16T10:28:28.097898+0000","last_became_peered":"2024-09-16T10:28:28.097898+0000","last_unstale":"2024-09-16T10:29:39.705766+0000","last_undegraded":"2024-09-16T10:29:39.705766+0000","last_fullsized":"2024-09-16T10:29:39.705766+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.419085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.419085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":437,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.185122+0000","last_change":"2024-09-16T10:29:36.541047+0000","last_active":"2024-09-16T10:29:39.185122+0000","last_peered":"2024-09-16T10:29:39.185122+0000","last_clean":"2024-09-16T10:29:39.185122+0000","last_became_active":"2024-09-16T10:28:28.072347+0000","last_became_peered":"2024-09-16T10:28:28.072347+0000","last_unstale":"2024-09-16T10:29:39.185122+0000","last_undegraded":"2024-09-16T10:29:39.185122+0000","last_fullsized":"2024-09-16T10:29:39.185122+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.541001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.541001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.828112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":474,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.038384+0000","last_change":"2024-09-16T10:29:34.955215+0000","last_active":"2024-09-16T10:29:40.038384+0000","last_peered":"2024-09-16T10:29:40.038384+0000","last_clean":"2024-09-16T10:29:40.038384+0000","last_became_active":"2024-09-16T10:28:28.096538+0000","last_became_peered":"2024-09-16T10:28:28.096538+0000","last_unstale":"2024-09-16T10:29:40.038384+0000","last_undegraded":"2024-09-16T10:29:40.038384+0000","last_fullsized":"2024-09-16T10:29:40.038384+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:59.442771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039367+0000","last_change":"2024-09-16T10:29:20.061250+0000","last_active":"2024-09-16T10:29:40.039367+0000","last_peered":"2024-09-16T10:29:40.039367+0000","last_clean":"2024-09-16T10:29:40.039367+0000","last_became_active":"2024-09-16T10:28:28.101644+0000","last_became_peered":"2024-09-16T10:28:28.101644+0000","last_unstale":"2024-09-16T10:29:40.039367+0000","last_undegraded":"2024-09-16T10:29:40.039367+0000","last_fullsized":"2024-09-16T10:29:40.039367+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:29:20.061184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.061184+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:21.959259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030148+0000","last_change":"2024-09-16T10:29:38.060471+0000","last_active":"2024-09-16T10:29:40.030148+0000","last_peered":"2024-09-16T10:29:40.030148+0000","last_clean":"2024-09-16T10:29:40.030148+0000","last_became_active":"2024-09-16T10:28:28.072930+0000","last_became_peered":"2024-09-16T10:28:28.072930+0000","last_unstale":"2024-09-16T10:29:40.030148+0000","last_undegraded":"2024-09-16T10:29:40.030148+0000","last_fullsized":"2024-09-16T10:29:40.030148+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:38.060395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:38.060395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:05.957375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"442'12","reported_seq":1134,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705819+0000","last_change":"2024-09-16T10:28:28.095653+0000","last_active":"2024-09-16T10:29:39.705819+0000","last_peered":"2024-09-16T10:29:39.705819+0000","last_clean":"2024-09-16T10:29:39.705819+0000","last_became_active":"2024-09-16T10:28:28.094870+0000","last_became_peered":"2024-09-16T10:28:28.094870+0000","last_unstale":"2024-09-16T10:29:39.705819+0000","last_undegraded":"2024-09-16T10:29:39.705819+0000","last_fullsized":"2024-09-16T10:29:39.705819+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.703651+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":691,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184799+0000","last_change":"2024-09-16T10:26:45.194577+0000","last_active":"2024-09-16T10:29:39.184799+0000","last_peered":"2024-09-16T10:29:39.184799+0000","last_clean":"2024-09-16T10:29:39.184799+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:29:39.184799+0000","last_undegraded":"2024-09-16T10:29:39.184799+0000","last_fullsized":"2024-09-16T10:29:39.184799+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.194454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.194454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":469,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030086+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:29:40.030086+0000","last_peered":"2024-09-16T10:29:40.030086+0000","last_clean":"2024-09-16T10:29:40.030086+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:29:40.030086+0000","last_undegraded":"2024-09-16T10:29:40.030086+0000","last_fullsized":"2024-09-16T10:29:40.030086+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":943,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705569+0000","last_change":"2024-09-16T10:28:28.096614+0000","last_active":"2024-09-16T10:29:39.705569+0000","last_peered":"2024-09-16T10:29:39.705569+0000","last_clean":"2024-09-16T10:29:39.705569+0000","last_became_active":"2024-09-16T10:28:28.095319+0000","last_became_peered":"2024-09-16T10:28:28.095319+0000","last_unstale":"2024-09-16T10:29:39.705569+0000","last_undegraded":"2024-09-16T10:29:39.705569+0000","last_fullsized":"2024-09-16T10:29:39.705569+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:26:20.405507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.405507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":799,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.395093+0000","last_change":"2024-09-16T10:29:21.812647+0000","last_active":"2024-09-16T10:29:39.395093+0000","last_peered":"2024-09-16T10:29:39.395093+0000","last_clean":"2024-09-16T10:29:39.395093+0000","last_became_active":"2024-09-16T10:28:28.071944+0000","last_became_peered":"2024-09-16T10:28:28.071944+0000","last_unstale":"2024-09-16T10:29:39.395093+0000","last_undegraded":"2024-09-16T10:29:39.395093+0000","last_fullsized":"2024-09-16T10:29:39.395093+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.812585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.812585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:23.180131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1094,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039212+0000","last_change":"2024-09-16T10:26:11.007271+0000","last_active":"2024-09-16T10:29:40.039212+0000","last_peered":"2024-09-16T10:29:40.039212+0000","last_clean":"2024-09-16T10:29:40.039212+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:29:40.039212+0000","last_undegraded":"2024-09-16T10:29:40.039212+0000","last_fullsized":"2024-09-16T10:29:40.039212+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:11.007191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:11.007191+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":890,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030365+0000","last_change":"2024-09-16T10:29:20.053166+0000","last_active":"2024-09-16T10:29:40.030365+0000","last_peered":"2024-09-16T10:29:40.030365+0000","last_clean":"2024-09-16T10:29:40.030365+0000","last_became_active":"2024-09-16T10:28:28.081510+0000","last_became_peered":"2024-09-16T10:28:28.081510+0000","last_unstale":"2024-09-16T10:29:40.030365+0000","last_undegraded":"2024-09-16T10:29:40.030365+0000","last_fullsized":"2024-09-16T10:29:40.030365+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.053102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.053102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:31.409241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":829,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292861+0000","last_change":"2024-09-16T10:28:02.318109+0000","last_active":"2024-09-16T10:29:39.292861+0000","last_peered":"2024-09-16T10:29:39.292861+0000","last_clean":"2024-09-16T10:29:39.292861+0000","last_became_active":"2024-09-16T10:26:28.557003+0000","last_became_peered":"2024-09-16T10:26:28.557003+0000","last_unstale":"2024-09-16T10:29:39.292861+0000","last_undegraded":"2024-09-16T10:29:39.292861+0000","last_fullsized":"2024-09-16T10:29:39.292861+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:02.318062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:02.318062+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":353,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183224+0000","last_change":"2024-09-16T10:28:28.105143+0000","last_active":"2024-09-16T10:29:39.183224+0000","last_peered":"2024-09-16T10:29:39.183224+0000","last_clean":"2024-09-16T10:29:39.183224+0000","last_became_active":"2024-09-16T10:28:28.104604+0000","last_became_peered":"2024-09-16T10:28:28.104604+0000","last_unstale":"2024-09-16T10:29:39.183224+0000","last_undegraded":"2024-09-16T10:29:39.183224+0000","last_fullsized":"2024-09-16T10:29:39.183224+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":884,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182325+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:29:39.182325+0000","last_peered":"2024-09-16T10:29:39.182325+0000","last_clean":"2024-09-16T10:29:39.182325+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:29:39.182325+0000","last_undegraded":"2024-09-16T10:29:39.182325+0000","last_fullsized":"2024-09-16T10:29:39.182325+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":878,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704723+0000","last_change":"2024-09-16T10:26:14.413104+0000","last_active":"2024-09-16T10:29:39.704723+0000","last_peered":"2024-09-16T10:29:39.704723+0000","last_clean":"2024-09-16T10:29:39.704723+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:29:39.704723+0000","last_undegraded":"2024-09-16T10:29:39.704723+0000","last_fullsized":"2024-09-16T10:29:39.704723+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:14.413019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:14.413019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":947,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182744+0000","last_change":"2024-09-16T10:29:19.698059+0000","last_active":"2024-09-16T10:29:39.182744+0000","last_peered":"2024-09-16T10:29:39.182744+0000","last_clean":"2024-09-16T10:29:39.182744+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:29:39.182744+0000","last_undegraded":"2024-09-16T10:29:39.182744+0000","last_fullsized":"2024-09-16T10:29:39.182744+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.530423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":288,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705623+0000","last_change":"2024-09-16T10:28:28.098266+0000","last_active":"2024-09-16T10:29:39.705623+0000","last_peered":"2024-09-16T10:29:39.705623+0000","last_clean":"2024-09-16T10:29:39.705623+0000","last_became_active":"2024-09-16T10:28:28.094896+0000","last_became_peered":"2024-09-16T10:28:28.094896+0000","last_unstale":"2024-09-16T10:29:39.705623+0000","last_undegraded":"2024-09-16T10:29:39.705623+0000","last_fullsized":"2024-09-16T10:29:39.705623+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.368081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.368081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":847,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705122+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:29:39.705122+0000","last_peered":"2024-09-16T10:29:39.705122+0000","last_clean":"2024-09-16T10:29:39.705122+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:29:39.705122+0000","last_undegraded":"2024-09-16T10:29:39.705122+0000","last_fullsized":"2024-09-16T10:29:39.705122+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":863,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029567+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:29:40.029567+0000","last_peered":"2024-09-16T10:29:40.029567+0000","last_clean":"2024-09-16T10:29:40.029567+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:29:40.029567+0000","last_undegraded":"2024-09-16T10:29:40.029567+0000","last_fullsized":"2024-09-16T10:29:40.029567+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":809,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184491+0000","last_change":"2024-09-16T10:29:35.573887+0000","last_active":"2024-09-16T10:29:39.184491+0000","last_peered":"2024-09-16T10:29:39.184491+0000","last_clean":"2024-09-16T10:29:39.184491+0000","last_became_active":"2024-09-16T10:28:28.071473+0000","last_became_peered":"2024-09-16T10:28:28.071473+0000","last_unstale":"2024-09-16T10:29:39.184491+0000","last_undegraded":"2024-09-16T10:29:39.184491+0000","last_fullsized":"2024-09-16T10:29:39.184491+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:35.573835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.573835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.601451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":912,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704927+0000","last_change":"2024-09-16T10:29:19.861104+0000","last_active":"2024-09-16T10:29:39.704927+0000","last_peered":"2024-09-16T10:29:39.704927+0000","last_clean":"2024-09-16T10:29:39.704927+0000","last_became_active":"2024-09-16T10:28:28.095283+0000","last_became_peered":"2024-09-16T10:28:28.095283+0000","last_unstale":"2024-09-16T10:29:39.704927+0000","last_undegraded":"2024-09-16T10:29:39.704927+0000","last_fullsized":"2024-09-16T10:29:39.704927+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.496265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":911,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.039347+0000","last_change":"2024-09-16T10:29:23.002575+0000","last_active":"2024-09-16T10:29:40.039347+0000","last_peered":"2024-09-16T10:29:40.039347+0000","last_clean":"2024-09-16T10:29:40.039347+0000","last_became_active":"2024-09-16T10:28:28.109359+0000","last_became_peered":"2024-09-16T10:28:28.109359+0000","last_unstale":"2024-09-16T10:29:40.039347+0000","last_undegraded":"2024-09-16T10:29:40.039347+0000","last_fullsized":"2024-09-16T10:29:40.039347+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.002518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.002518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.887462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":723,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.394677+0000","last_change":"2024-09-16T10:29:19.866083+0000","last_active":"2024-09-16T10:29:39.394677+0000","last_peered":"2024-09-16T10:29:39.394677+0000","last_clean":"2024-09-16T10:29:39.394677+0000","last_became_active":"2024-09-16T10:28:28.091060+0000","last_became_peered":"2024-09-16T10:28:28.091060+0000","last_unstale":"2024-09-16T10:29:39.394677+0000","last_undegraded":"2024-09-16T10:29:39.394677+0000","last_fullsized":"2024-09-16T10:29:39.394677+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.866022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:36.779287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":894,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.705335+0000","last_change":"2024-09-16T10:29:34.789497+0000","last_active":"2024-09-16T10:29:39.705335+0000","last_peered":"2024-09-16T10:29:39.705335+0000","last_clean":"2024-09-16T10:29:39.705335+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:29:39.705335+0000","last_undegraded":"2024-09-16T10:29:39.705335+0000","last_fullsized":"2024-09-16T10:29:39.705335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.789441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.789441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.777028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":932,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704762+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:29:39.704762+0000","last_peered":"2024-09-16T10:29:39.704762+0000","last_clean":"2024-09-16T10:29:39.704762+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:29:39.704762+0000","last_undegraded":"2024-09-16T10:29:39.704762+0000","last_fullsized":"2024-09-16T10:29:39.704762+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":889,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030487+0000","last_change":"2024-09-16T10:27:27.598364+0000","last_active":"2024-09-16T10:29:40.030487+0000","last_peered":"2024-09-16T10:29:40.030487+0000","last_clean":"2024-09-16T10:29:40.030487+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:29:40.030487+0000","last_undegraded":"2024-09-16T10:29:40.030487+0000","last_fullsized":"2024-09-16T10:29:40.030487+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:27.598299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:27.598299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292438+0000","last_change":"2024-09-16T10:29:31.282374+0000","last_active":"2024-09-16T10:29:39.292438+0000","last_peered":"2024-09-16T10:29:39.292438+0000","last_clean":"2024-09-16T10:29:39.292438+0000","last_became_active":"2024-09-16T10:28:28.072759+0000","last_became_peered":"2024-09-16T10:28:28.072759+0000","last_unstale":"2024-09-16T10:29:39.292438+0000","last_undegraded":"2024-09-16T10:29:39.292438+0000","last_fullsized":"2024-09-16T10:29:39.292438+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:31.282323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:31.282323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.735338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":521,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184284+0000","last_change":"2024-09-16T10:29:32.563307+0000","last_active":"2024-09-16T10:29:39.184284+0000","last_peered":"2024-09-16T10:29:39.184284+0000","last_clean":"2024-09-16T10:29:39.184284+0000","last_became_active":"2024-09-16T10:28:28.066479+0000","last_became_peered":"2024-09-16T10:28:28.066479+0000","last_unstale":"2024-09-16T10:29:39.184284+0000","last_undegraded":"2024-09-16T10:29:39.184284+0000","last_fullsized":"2024-09-16T10:29:39.184284+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:32.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:32.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.751962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":880,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.181603+0000","last_change":"2024-09-16T10:26:23.240494+0000","last_active":"2024-09-16T10:29:39.181603+0000","last_peered":"2024-09-16T10:29:39.181603+0000","last_clean":"2024-09-16T10:29:39.181603+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:29:39.181603+0000","last_undegraded":"2024-09-16T10:29:39.181603+0000","last_fullsized":"2024-09-16T10:29:39.181603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:23.240435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:23.240435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":898,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.704900+0000","last_change":"2024-09-16T10:26:13.362416+0000","last_active":"2024-09-16T10:29:39.704900+0000","last_peered":"2024-09-16T10:29:39.704900+0000","last_clean":"2024-09-16T10:29:39.704900+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:29:39.704900+0000","last_undegraded":"2024-09-16T10:29:39.704900+0000","last_fullsized":"2024-09-16T10:29:39.704900+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.362347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.362347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":935,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292087+0000","last_change":"2024-09-16T10:28:06.383385+0000","last_active":"2024-09-16T10:29:39.292087+0000","last_peered":"2024-09-16T10:29:39.292087+0000","last_clean":"2024-09-16T10:29:39.292087+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:29:39.292087+0000","last_undegraded":"2024-09-16T10:29:39.292087+0000","last_fullsized":"2024-09-16T10:29:39.292087+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:06.383318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:06.383318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":909,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030502+0000","last_change":"2024-09-16T10:29:19.094876+0000","last_active":"2024-09-16T10:29:40.030502+0000","last_peered":"2024-09-16T10:29:40.030502+0000","last_clean":"2024-09-16T10:29:40.030502+0000","last_became_active":"2024-09-16T10:28:28.081905+0000","last_became_peered":"2024-09-16T10:28:28.081905+0000","last_unstale":"2024-09-16T10:29:40.030502+0000","last_undegraded":"2024-09-16T10:29:40.030502+0000","last_fullsized":"2024-09-16T10:29:40.030502+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.094826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.094826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.628616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":607191040,"num_objects":9265,"num_object_clones":0,"num_object_copies":27795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9265,"num_write_kb":592960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2040004608,"data_stored":2040004608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9265,"ondisk_log_size":9265,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":285,"num_read_kb":280,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":441,"seq":1894080577553,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":489648,"kb_used_data":322336,"kb_used_omap":7,"kb_used_meta":167288,"kb_avail":93255504,"statfs":{"total":95995035648,"available":95493636096,"internally_reserved":0,"allocated":330072064,"data_stored":328968085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8012,"internal_metadata":171303092},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528521,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909736,"kb_used_data":296856,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92835416,"statfs":{"total":95995035648,"available":95063465984,"internally_reserved":0,"allocated":303980544,"data_stored":302884099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":627571142},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790159,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793464,"kb_used_data":176424,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92951688,"statfs":{"total":95995035648,"available":95182528512,"internally_reserved":0,"allocated":180658176,"data_stored":179547353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986388,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692872,"kb_used_data":102200,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93052280,"statfs":{"total":95995035648,"available":95285534720,"internally_reserved":0,"allocated":104652800,"data_stored":103524628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215321,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907848,"kb_used_data":214456,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":92837304,"statfs":{"total":95995035648,"available":95065399296,"internally_reserved":0,"allocated":219602944,"data_stored":218476155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":710015432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411551,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":685104,"kb_used_data":230496,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93060048,"statfs":{"total":95995035648,"available":95293489152,"internally_reserved":0,"allocated":236027904,"data_stored":234924082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575074,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912660,"kb_used_data":231748,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92832492,"statfs":{"total":95995035648,"available":95060471808,"internally_reserved":0,"allocated":237309952,"data_stored":236170012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738600,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964280,"kb_used_data":269480,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":92780872,"statfs":{"total":95995035648,"available":95007612928,"internally_reserved":0,"allocated":275947520,"data_stored":274836051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":273022976,"data_stored":273022976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275972096,"data_stored":275972096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":276234240,"data_stored":276234240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242089984,"data_stored":242089984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":145686528,"data_stored":145686528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":135004160,"data_stored":135004160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":316080128,"data_stored":316080128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":375914496,"data_stored":375914496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:49.412 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:29:49.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:49.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:49.517 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:29:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:29:49.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:49.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:49.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:49.692 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:49 smithi063 ceph-mon[29993]: from='client.16578 v1:172.21.15.63:0/3597225444' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:29:49.692 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:49 smithi063 ceph-mon[39470]: from='client.16578 v1:172.21.15.63:0/3597225444' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:29:49.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:49.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:49.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:49.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:49 smithi165 ceph-mon[32696]: from='client.16578 v1:172.21.15.63:0/3597225444' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:29:49.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:49.936 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:49.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:50.035 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:50.038 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:50.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:50.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:50.240 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:50.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:50.311 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 10 16 25209 25193 157.429 120.25 0.00699967 0.0063449 2024-09-16T10:29:50.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:50.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:50.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:50.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:50.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:50.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:50.545 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:29:50.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:50.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:50.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[29993]: pgmap v881: 168 pgs: 168 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 997 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-16T10:29:50.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2526971144' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:50.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[29993]: from='client.46397 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:50.606 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:50.607 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[39470]: pgmap v881: 168 pgs: 168 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 997 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-16T10:29:50.607 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2526971144' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:50.607 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:50 smithi063 ceph-mon[39470]: from='client.46397 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:50.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:50.676 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:29:50.676 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:29:50.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:29:50.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:50.813 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:50.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:50.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:50.944 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:50.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:51.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:51.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:29:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:50 smithi165 ceph-mon[32696]: pgmap v881: 168 pgs: 168 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 997 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-16T10:29:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:50 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2526971144' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:50 smithi165 ceph-mon[32696]: from='client.46397 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:29:51.097 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:51.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:51.198 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:51.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:51.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:51.311 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 11 16 27223 27207 154.56 125.875 0.0060689 0.0064644 2024-09-16T10:29:51.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:51.378 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:51.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:51.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:51.524 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:51.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:51.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:51.632 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:51.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:51.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:51.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[29993]: from='client.46397 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:51.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[29993]: osdmap e481: 8 total, 8 up, 8 in 2024-09-16T10:29:51.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[29993]: osdmap e482: 8 total, 8 up, 8 in 2024-09-16T10:29:51.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:51.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[39470]: from='client.46397 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:51.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[39470]: osdmap e481: 8 total, 8 up, 8 in 2024-09-16T10:29:51.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:51 smithi063 ceph-mon[39470]: osdmap e482: 8 total, 8 up, 8 in 2024-09-16T10:29:51.786 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:51.824 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:51.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:51.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:51.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:51.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:52.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:51 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:29:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:51 smithi165 ceph-mon[32696]: from='client.46397 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:29:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:51 smithi165 ceph-mon[32696]: osdmap e481: 8 total, 8 up, 8 in 2024-09-16T10:29:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:51 smithi165 ceph-mon[32696]: osdmap e482: 8 total, 8 up, 8 in 2024-09-16T10:29:52.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:52.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:52.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:52.115 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:29:52.187 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:52.241 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:52.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:52.312 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 12 16 29285 29269 152.41 128.875 0.00802565 0.00655555 2024-09-16T10:29:52.343 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:52.385 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:52.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:52.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:52.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:52.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:52.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:52.642 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:52.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:52.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:52.775 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:52.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:52.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:52.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:52.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:52.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:52 smithi063 ceph-mon[39470]: pgmap v884: 168 pgs: 168 active+clean; 1.3 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-16T10:29:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:52 smithi063 ceph-mon[29993]: pgmap v884: 168 pgs: 168 active+clean; 1.3 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-16T10:29:53.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:53.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:53.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:53.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:52 smithi165 ceph-mon[32696]: pgmap v884: 168 pgs: 168 active+clean; 1.3 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-16T10:29:53.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:53.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:53.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:53.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:53.312 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 13 16 30903 30887 148.465 101.125 0.00488912 0.00673078 2024-09-16T10:29:53.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:53.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:53.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:53.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:53.441 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:29:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:53.458 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:53.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:53.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:53.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:53.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:53.686 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:53.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:53.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:53.878 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:53.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:53.979 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.003 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:54.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:54.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:54.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:54.236 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:54.272 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:54.312 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 14 16 32586 32570 145.374 105.188 0.00361358 0.00687306 2024-09-16T10:29:54.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:54.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:54.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:54.526 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:54.628 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:54.668 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:54.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:54.793 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:54.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:54.932 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:54.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:54 smithi063 ceph-mon[29993]: pgmap v885: 168 pgs: 168 active+clean; 1.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-16T10:29:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:54 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:54 smithi063 ceph-mon[39470]: pgmap v885: 168 pgs: 168 active+clean; 1.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-16T10:29:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:54 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:55.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:55.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:54 smithi165 ceph-mon[32696]: pgmap v885: 168 pgs: 168 active+clean; 1.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-16T10:29:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:54 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:29:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:29:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.072 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:55.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:55.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:55.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:55.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:55.312 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 15 16 34182 34166 142.329 99.75 0.0144527 0.00701853 2024-09-16T10:29:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.337 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:55.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:55.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:55.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:55.564 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.589 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:55.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:55.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:55.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:55.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:55.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:55.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:55.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:55.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:55.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:56.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:56.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:56.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:56.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:56.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:56.241 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:29:56.244 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:29:56.255 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:29:56.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:56.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:56.312 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 16 15 35642 35627 139.139 91.3125 0.00442693 0.00718123 2024-09-16T10:29:56.358 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:56.384 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:56.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:56.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:56.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:56.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:56.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:56.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:56.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:56.643 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:56.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:56.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:56.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:56.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:56.874 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:56.896 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:56.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:56.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[29993]: from='client.46403 v1:172.21.15.63:0/846400797' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:29:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[29993]: pgmap v886: 168 pgs: 168 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s 2024-09-16T10:29:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[29993]: osdmap e483: 8 total, 8 up, 8 in 2024-09-16T10:29:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[29993]: osdmap e484: 8 total, 8 up, 8 in 2024-09-16T10:29:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[39470]: from='client.46403 v1:172.21.15.63:0/846400797' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:29:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[39470]: pgmap v886: 168 pgs: 168 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s 2024-09-16T10:29:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[39470]: osdmap e483: 8 total, 8 up, 8 in 2024-09-16T10:29:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:56 smithi063 ceph-mon[39470]: osdmap e484: 8 total, 8 up, 8 in 2024-09-16T10:29:56.998 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:57.034 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:57.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:56 smithi165 ceph-mon[32696]: from='client.46403 v1:172.21.15.63:0/846400797' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:29:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:56 smithi165 ceph-mon[32696]: pgmap v886: 168 pgs: 168 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s 2024-09-16T10:29:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:56 smithi165 ceph-mon[32696]: osdmap e483: 8 total, 8 up, 8 in 2024-09-16T10:29:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:56 smithi165 ceph-mon[32696]: osdmap e484: 8 total, 8 up, 8 in 2024-09-16T10:29:57.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.168 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:57.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:57.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:57.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:57.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 17 16 37465 37449 137.653 113.875 0.00840288 0.00725966 2024-09-16T10:29:57.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:57.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:57.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:29:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:29:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:29:57.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:57.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:57.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:29:57.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:29:57.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:29:57.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:29:57.926 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:57.951 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:57.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:58.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:58.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:58.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:58.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:58.229 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:58.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:58.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 18 16 39145 39129 135.838 105 0.012809 0.00735617 2024-09-16T10:29:58.331 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:58.372 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:58.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:58.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:29:58.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:58.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:29:58.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:58.649 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:58.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:58.754 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:58.795 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:58.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:58.896 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:29:58.936 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:58.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:29:58.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:58.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:58.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[29993]: pgmap v889: 168 pgs: 168 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 801 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-16T10:29:58.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[29993]: osdmap e485: 8 total, 8 up, 8 in 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[39470]: pgmap v889: 168 pgs: 168 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 801 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[39470]: osdmap e485: 8 total, 8 up, 8 in 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:29:58.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:58 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:29:59.038 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:59.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:58 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:59.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:58 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:59.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:58 smithi165 ceph-mon[32696]: pgmap v889: 168 pgs: 168 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 801 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-16T10:29:59.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:58 smithi165 ceph-mon[32696]: osdmap e485: 8 total, 8 up, 8 in 2024-09-16T10:29:59.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:58 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:29:59.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:58 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:29:59.078 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:59.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:59.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:59.205 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:59.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:59.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:29:59.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 19 16 40722 40706 133.876 98.5625 0.00525552 0.00746475 2024-09-16T10:29:59.345 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:59.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:29:59.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:29:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:59.478 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:59.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:29:59.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:29:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:29:59.579 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:29:59.620 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:29:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:59.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:29:59.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:59.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:59.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:59.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:59.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:29:59.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:29:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:29:59.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:29:59.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:29:59 smithi063 ceph-mon[29993]: osdmap e486: 8 total, 8 up, 8 in 2024-09-16T10:29:59.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:29:59 smithi063 ceph-mon[39470]: osdmap e486: 8 total, 8 up, 8 in 2024-09-16T10:29:59.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:00.003 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:00.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:29:59 smithi165 ceph-mon[32696]: osdmap e486: 8 total, 8 up, 8 in 2024-09-16T10:30:00.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:00.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:00.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:00.266 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:00.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:30:00.312341+0000 min lat: 0.00120603 max lat: 0.224582 avg lat: 0.00765995 2024-09-16T10:30:00.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:30:00.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 20 16 41769 41753 130.454 65.4375 0.00465063 0.00765995 2024-09-16T10:30:00.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:00.408 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:00.510 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.564 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:00.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:00.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:00.691 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":885,"stamp":"2024-09-16T10:29:53.444210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1797724699,"num_objects":27621,"num_object_clones":0,"num_object_copies":82863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27621,"num_whiteouts":0,"num_read":1424,"num_read_kb":1176,"num_write":28144,"num_write_kb":1755053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":527794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28436,"ondisk_log_size":28436,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9711112,"kb_used_data":5087036,"kb_used_omap":18,"kb_used_meta":4623917,"kb_avail":740250104,"statfs":{"total":767960285184,"available":758016106496,"internally_reserved":0,"allocated":5209124864,"data_stored":5200161785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19173,"internal_metadata":4734891291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1189937152,"num_objects":18157,"num_object_clones":0,"num_object_copies":54471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18157,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":18157,"num_write_kb":1162048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004055"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":914,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030461+0000","last_change":"2024-09-16T10:29:36.039702+0000","last_active":"2024-09-16T10:29:40.030461+0000","last_peered":"2024-09-16T10:29:40.030461+0000","last_clean":"2024-09-16T10:29:40.030461+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:29:40.030461+0000","last_undegraded":"2024-09-16T10:29:40.030461+0000","last_fullsized":"2024-09-16T10:29:40.030461+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.039652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.039652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:56.261928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":852,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292274+0000","last_change":"2024-09-16T10:26:53.392698+0000","last_active":"2024-09-16T10:29:39.292274+0000","last_peered":"2024-09-16T10:29:39.292274+0000","last_clean":"2024-09-16T10:29:39.292274+0000","last_became_active":"2024-09-16T10:26:28.557096+0000","last_became_peered":"2024-09-16T10:26:28.557096+0000","last_unstale":"2024-09-16T10:29:39.292274+0000","last_undegraded":"2024-09-16T10:29:39.292274+0000","last_fullsized":"2024-09-16T10:29:39.292274+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:53.392646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:53.392646+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":907,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059683+0000","last_change":"2024-09-16T10:28:28.072016+0000","last_active":"2024-09-16T10:29:51.059683+0000","last_peered":"2024-09-16T10:29:51.059683+0000","last_clean":"2024-09-16T10:29:51.059683+0000","last_became_active":"2024-09-16T10:28:28.071724+0000","last_became_peered":"2024-09-16T10:28:28.071724+0000","last_unstale":"2024-09-16T10:29:51.059683+0000","last_undegraded":"2024-09-16T10:29:51.059683+0000","last_fullsized":"2024-09-16T10:29:51.059683+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.940459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.940459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":220,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061714+0000","last_change":"2024-09-16T10:28:07.975203+0000","last_active":"2024-09-16T10:29:51.061714+0000","last_peered":"2024-09-16T10:29:51.061714+0000","last_clean":"2024-09-16T10:29:51.061714+0000","last_became_active":"2024-09-16T10:26:28.551847+0000","last_became_peered":"2024-09-16T10:26:28.551847+0000","last_unstale":"2024-09-16T10:29:51.061714+0000","last_undegraded":"2024-09-16T10:29:51.061714+0000","last_fullsized":"2024-09-16T10:29:51.061714+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:28:07.975153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:28:07.975153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":893,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059308+0000","last_change":"2024-09-16T10:29:21.916724+0000","last_active":"2024-09-16T10:29:51.059308+0000","last_peered":"2024-09-16T10:29:51.059308+0000","last_clean":"2024-09-16T10:29:51.059308+0000","last_became_active":"2024-09-16T10:28:28.094751+0000","last_became_peered":"2024-09-16T10:28:28.094751+0000","last_unstale":"2024-09-16T10:29:51.059308+0000","last_undegraded":"2024-09-16T10:29:51.059308+0000","last_fullsized":"2024-09-16T10:29:51.059308+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.675562+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":918,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062679+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:29:51.062679+0000","last_peered":"2024-09-16T10:29:51.062679+0000","last_clean":"2024-09-16T10:29:51.062679+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:29:51.062679+0000","last_undegraded":"2024-09-16T10:29:51.062679+0000","last_fullsized":"2024-09-16T10:29:51.062679+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.825253+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":987,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061670+0000","last_change":"2024-09-16T10:26:12.334813+0000","last_active":"2024-09-16T10:29:51.061670+0000","last_peered":"2024-09-16T10:29:51.061670+0000","last_clean":"2024-09-16T10:29:51.061670+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:29:51.061670+0000","last_undegraded":"2024-09-16T10:29:51.061670+0000","last_fullsized":"2024-09-16T10:29:51.061670+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:12.334720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.334720+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292357+0000","last_change":"2024-09-16T10:26:45.470165+0000","last_active":"2024-09-16T10:29:39.292357+0000","last_peered":"2024-09-16T10:29:39.292357+0000","last_clean":"2024-09-16T10:29:39.292357+0000","last_became_active":"2024-09-16T10:26:28.556509+0000","last_became_peered":"2024-09-16T10:26:28.556509+0000","last_unstale":"2024-09-16T10:29:39.292357+0000","last_undegraded":"2024-09-16T10:29:39.292357+0000","last_fullsized":"2024-09-16T10:29:39.292357+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.470014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.470014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":920,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062439+0000","last_change":"2024-09-16T10:26:48.078096+0000","last_active":"2024-09-16T10:29:51.062439+0000","last_peered":"2024-09-16T10:29:51.062439+0000","last_clean":"2024-09-16T10:29:51.062439+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:29:51.062439+0000","last_undegraded":"2024-09-16T10:29:51.062439+0000","last_fullsized":"2024-09-16T10:29:51.062439+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:48.078028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:48.078028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":822,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062680+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:29:51.062680+0000","last_peered":"2024-09-16T10:29:51.062680+0000","last_clean":"2024-09-16T10:29:51.062680+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:29:51.062680+0000","last_undegraded":"2024-09-16T10:29:51.062680+0000","last_fullsized":"2024-09-16T10:29:51.062680+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:53.512693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":907,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061889+0000","last_change":"2024-09-16T10:28:28.182357+0000","last_active":"2024-09-16T10:29:51.061889+0000","last_peered":"2024-09-16T10:29:51.061889+0000","last_clean":"2024-09-16T10:29:51.061889+0000","last_became_active":"2024-09-16T10:28:28.182043+0000","last_became_peered":"2024-09-16T10:28:28.182043+0000","last_unstale":"2024-09-16T10:29:51.061889+0000","last_undegraded":"2024-09-16T10:29:51.061889+0000","last_fullsized":"2024-09-16T10:29:51.061889+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.834823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.834823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.037375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":898,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029647+0000","last_change":"2024-09-16T10:28:28.090355+0000","last_active":"2024-09-16T10:29:40.029647+0000","last_peered":"2024-09-16T10:29:40.029647+0000","last_clean":"2024-09-16T10:29:40.029647+0000","last_became_active":"2024-09-16T10:28:28.089459+0000","last_became_peered":"2024-09-16T10:28:28.089459+0000","last_unstale":"2024-09-16T10:29:40.029647+0000","last_undegraded":"2024-09-16T10:29:40.029647+0000","last_fullsized":"2024-09-16T10:29:40.029647+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:08.436243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.436243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":872,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062095+0000","last_change":"2024-09-16T10:27:58.999207+0000","last_active":"2024-09-16T10:29:51.062095+0000","last_peered":"2024-09-16T10:29:51.062095+0000","last_clean":"2024-09-16T10:29:51.062095+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:29:51.062095+0000","last_undegraded":"2024-09-16T10:29:51.062095+0000","last_fullsized":"2024-09-16T10:29:51.062095+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:58.999143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:58.999143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":802,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059723+0000","last_change":"2024-09-16T10:26:28.557316+0000","last_active":"2024-09-16T10:29:51.059723+0000","last_peered":"2024-09-16T10:29:51.059723+0000","last_clean":"2024-09-16T10:29:51.059723+0000","last_became_active":"2024-09-16T10:26:28.556933+0000","last_became_peered":"2024-09-16T10:26:28.556933+0000","last_unstale":"2024-09-16T10:29:51.059723+0000","last_undegraded":"2024-09-16T10:29:51.059723+0000","last_fullsized":"2024-09-16T10:29:51.059723+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.715967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":741,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.550868+0000","last_change":"2024-09-16T10:29:23.841536+0000","last_active":"2024-09-16T10:29:50.550868+0000","last_peered":"2024-09-16T10:29:50.550868+0000","last_clean":"2024-09-16T10:29:50.550868+0000","last_became_active":"2024-09-16T10:28:28.071768+0000","last_became_peered":"2024-09-16T10:28:28.071768+0000","last_unstale":"2024-09-16T10:29:50.550868+0000","last_undegraded":"2024-09-16T10:29:50.550868+0000","last_fullsized":"2024-09-16T10:29:50.550868+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.841474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.841474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.210925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":907,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061829+0000","last_change":"2024-09-16T10:29:24.922917+0000","last_active":"2024-09-16T10:29:51.061829+0000","last_peered":"2024-09-16T10:29:51.061829+0000","last_clean":"2024-09-16T10:29:51.061829+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:29:51.061829+0000","last_undegraded":"2024-09-16T10:29:51.061829+0000","last_fullsized":"2024-09-16T10:29:51.061829+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.922862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.922862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:45.215312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292860+0000","last_change":"2024-09-16T10:29:16.166206+0000","last_active":"2024-09-16T10:29:39.292860+0000","last_peered":"2024-09-16T10:29:39.292860+0000","last_clean":"2024-09-16T10:29:39.292860+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:29:39.292860+0000","last_undegraded":"2024-09-16T10:29:39.292860+0000","last_fullsized":"2024-09-16T10:29:39.292860+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:16.166144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:16.166144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:19.152416+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":894,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030565+0000","last_change":"2024-09-16T10:28:28.081914+0000","last_active":"2024-09-16T10:29:40.030565+0000","last_peered":"2024-09-16T10:29:40.030565+0000","last_clean":"2024-09-16T10:29:40.030565+0000","last_became_active":"2024-09-16T10:28:28.081584+0000","last_became_peered":"2024-09-16T10:28:28.081584+0000","last_unstale":"2024-09-16T10:29:40.030565+0000","last_undegraded":"2024-09-16T10:29:40.030565+0000","last_fullsized":"2024-09-16T10:29:40.030565+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.031297+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":460,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.185031+0000","last_change":"2024-09-16T10:26:17.566065+0000","last_active":"2024-09-16T10:29:39.185031+0000","last_peered":"2024-09-16T10:29:39.185031+0000","last_clean":"2024-09-16T10:29:39.185031+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:29:39.185031+0000","last_undegraded":"2024-09-16T10:29:39.185031+0000","last_fullsized":"2024-09-16T10:29:39.185031+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:17.565970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.565970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":888,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061143+0000","last_change":"2024-09-16T10:26:22.230127+0000","last_active":"2024-09-16T10:29:51.061143+0000","last_peered":"2024-09-16T10:29:51.061143+0000","last_clean":"2024-09-16T10:29:51.061143+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:29:51.061143+0000","last_undegraded":"2024-09-16T10:29:51.061143+0000","last_fullsized":"2024-09-16T10:29:51.061143+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.230054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.230054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":848,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059961+0000","last_change":"2024-09-16T10:27:18.318288+0000","last_active":"2024-09-16T10:29:51.059961+0000","last_peered":"2024-09-16T10:29:51.059961+0000","last_clean":"2024-09-16T10:29:51.059961+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:29:51.059961+0000","last_undegraded":"2024-09-16T10:29:51.059961+0000","last_fullsized":"2024-09-16T10:29:51.059961+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:18.318214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:18.318214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":684,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.063064+0000","last_change":"2024-09-16T10:26:28.551407+0000","last_active":"2024-09-16T10:29:51.063064+0000","last_peered":"2024-09-16T10:29:51.063064+0000","last_clean":"2024-09-16T10:29:51.063064+0000","last_became_active":"2024-09-16T10:26:28.551058+0000","last_became_peered":"2024-09-16T10:26:28.551058+0000","last_unstale":"2024-09-16T10:29:51.063064+0000","last_undegraded":"2024-09-16T10:29:51.063064+0000","last_fullsized":"2024-09-16T10:29:51.063064+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":957,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060732+0000","last_change":"2024-09-16T10:29:28.855774+0000","last_active":"2024-09-16T10:29:51.060732+0000","last_peered":"2024-09-16T10:29:51.060732+0000","last_clean":"2024-09-16T10:29:51.060732+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:29:51.060732+0000","last_undegraded":"2024-09-16T10:29:51.060732+0000","last_fullsized":"2024-09-16T10:29:51.060732+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:29:28.855720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.855720+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:34.642453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":692,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060187+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:29:51.060187+0000","last_peered":"2024-09-16T10:29:51.060187+0000","last_clean":"2024-09-16T10:29:51.060187+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:29:51.060187+0000","last_undegraded":"2024-09-16T10:29:51.060187+0000","last_fullsized":"2024-09-16T10:29:51.060187+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":306,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061895+0000","last_change":"2024-09-16T10:28:28.095829+0000","last_active":"2024-09-16T10:29:51.061895+0000","last_peered":"2024-09-16T10:29:51.061895+0000","last_clean":"2024-09-16T10:29:51.061895+0000","last_became_active":"2024-09-16T10:28:28.095455+0000","last_became_peered":"2024-09-16T10:28:28.095455+0000","last_unstale":"2024-09-16T10:29:51.061895+0000","last_undegraded":"2024-09-16T10:29:51.061895+0000","last_fullsized":"2024-09-16T10:29:51.061895+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.012515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":903,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061465+0000","last_change":"2024-09-16T10:26:17.204523+0000","last_active":"2024-09-16T10:29:51.061465+0000","last_peered":"2024-09-16T10:29:51.061465+0000","last_clean":"2024-09-16T10:29:51.061465+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:29:51.061465+0000","last_undegraded":"2024-09-16T10:29:51.061465+0000","last_fullsized":"2024-09-16T10:29:51.061465+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:17.204441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.204441+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":938,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030662+0000","last_change":"2024-09-16T10:29:29.106834+0000","last_active":"2024-09-16T10:29:40.030662+0000","last_peered":"2024-09-16T10:29:40.030662+0000","last_clean":"2024-09-16T10:29:40.030662+0000","last_became_active":"2024-09-16T10:28:28.082261+0000","last_became_peered":"2024-09-16T10:28:28.082261+0000","last_unstale":"2024-09-16T10:29:40.030662+0000","last_undegraded":"2024-09-16T10:29:40.030662+0000","last_fullsized":"2024-09-16T10:29:40.030662+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:29.106770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.106770+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.311277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"442'10","reported_seq":1110,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:42.372422+0000","last_change":"2024-09-16T10:29:37.281159+0000","last_active":"2024-09-16T10:29:42.372422+0000","last_peered":"2024-09-16T10:29:42.372422+0000","last_clean":"2024-09-16T10:29:42.372422+0000","last_became_active":"2024-09-16T10:28:28.089383+0000","last_became_peered":"2024-09-16T10:28:28.089383+0000","last_unstale":"2024-09-16T10:29:42.372422+0000","last_undegraded":"2024-09-16T10:29:42.372422+0000","last_fullsized":"2024-09-16T10:29:42.372422+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.602654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"482'1884","reported_seq":1907,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.907647+0000","last_change":"2024-09-16T10:29:35.280092+0000","last_active":"2024-09-16T10:29:52.907647+0000","last_peered":"2024-09-16T10:29:52.907647+0000","last_clean":"2024-09-16T10:29:52.907647+0000","last_became_active":"2024-09-16T10:29:35.279496+0000","last_became_peered":"2024-09-16T10:29:35.279496+0000","last_unstale":"2024-09-16T10:29:52.907647+0000","last_undegraded":"2024-09-16T10:29:52.907647+0000","last_fullsized":"2024-09-16T10:29:52.907647+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1884,"log_dups_size":0,"ondisk_log_size":1884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:55.065966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123469824,"num_objects":1884,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1884,"num_write_kb":120576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":750,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061657+0000","last_change":"2024-09-16T10:29:21.013569+0000","last_active":"2024-09-16T10:29:51.061657+0000","last_peered":"2024-09-16T10:29:51.061657+0000","last_clean":"2024-09-16T10:29:51.061657+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:29:51.061657+0000","last_undegraded":"2024-09-16T10:29:51.061657+0000","last_fullsized":"2024-09-16T10:29:51.061657+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:21.013503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.013503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.013620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":936,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060024+0000","last_change":"2024-09-16T10:27:12.407528+0000","last_active":"2024-09-16T10:29:51.060024+0000","last_peered":"2024-09-16T10:29:51.060024+0000","last_clean":"2024-09-16T10:29:51.060024+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:29:51.060024+0000","last_undegraded":"2024-09-16T10:29:51.060024+0000","last_fullsized":"2024-09-16T10:29:51.060024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:27:12.407441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.407441+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":914,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.063185+0000","last_change":"2024-09-16T10:28:28.066368+0000","last_active":"2024-09-16T10:29:51.063185+0000","last_peered":"2024-09-16T10:29:51.063185+0000","last_clean":"2024-09-16T10:29:51.063185+0000","last_became_active":"2024-09-16T10:28:28.065738+0000","last_became_peered":"2024-09-16T10:28:28.065738+0000","last_unstale":"2024-09-16T10:29:51.063185+0000","last_undegraded":"2024-09-16T10:29:51.063185+0000","last_fullsized":"2024-09-16T10:29:51.063185+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.851080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":287,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062309+0000","last_change":"2024-09-16T10:28:03.060402+0000","last_active":"2024-09-16T10:29:51.062309+0000","last_peered":"2024-09-16T10:29:51.062309+0000","last_clean":"2024-09-16T10:29:51.062309+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:29:51.062309+0000","last_undegraded":"2024-09-16T10:29:51.062309+0000","last_fullsized":"2024-09-16T10:29:51.062309+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:28:03.060327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:28:03.060327+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":893,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062023+0000","last_change":"2024-09-16T10:28:07.007838+0000","last_active":"2024-09-16T10:29:51.062023+0000","last_peered":"2024-09-16T10:29:51.062023+0000","last_clean":"2024-09-16T10:29:51.062023+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:29:51.062023+0000","last_undegraded":"2024-09-16T10:29:51.062023+0000","last_fullsized":"2024-09-16T10:29:51.062023+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:28:07.007777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:28:07.007777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"482'1959","reported_seq":1982,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.928600+0000","last_change":"2024-09-16T10:29:35.279709+0000","last_active":"2024-09-16T10:29:52.928600+0000","last_peered":"2024-09-16T10:29:52.928600+0000","last_clean":"2024-09-16T10:29:52.928600+0000","last_became_active":"2024-09-16T10:29:35.279416+0000","last_became_peered":"2024-09-16T10:29:35.279416+0000","last_unstale":"2024-09-16T10:29:52.928600+0000","last_undegraded":"2024-09-16T10:29:52.928600+0000","last_fullsized":"2024-09-16T10:29:52.928600+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1959,"log_dups_size":0,"ondisk_log_size":1959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.688136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128385024,"num_objects":1959,"num_object_clones":0,"num_object_copies":5877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":125376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":320,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.293034+0000","last_change":"2024-09-16T10:27:18.552950+0000","last_active":"2024-09-16T10:29:39.293034+0000","last_peered":"2024-09-16T10:29:39.293034+0000","last_clean":"2024-09-16T10:29:39.293034+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:29:39.293034+0000","last_undegraded":"2024-09-16T10:29:39.293034+0000","last_fullsized":"2024-09-16T10:29:39.293034+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:27:17.482427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:27:17.482427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":281,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292121+0000","last_change":"2024-09-16T10:26:44.445756+0000","last_active":"2024-09-16T10:29:39.292121+0000","last_peered":"2024-09-16T10:29:39.292121+0000","last_clean":"2024-09-16T10:29:39.292121+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:29:39.292121+0000","last_undegraded":"2024-09-16T10:29:39.292121+0000","last_fullsized":"2024-09-16T10:29:39.292121+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:26:44.445661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.445661+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":879,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060417+0000","last_change":"2024-09-16T10:28:28.098490+0000","last_active":"2024-09-16T10:29:51.060417+0000","last_peered":"2024-09-16T10:29:51.060417+0000","last_clean":"2024-09-16T10:29:51.060417+0000","last_became_active":"2024-09-16T10:28:28.097354+0000","last_became_peered":"2024-09-16T10:28:28.097354+0000","last_unstale":"2024-09-16T10:29:51.060417+0000","last_undegraded":"2024-09-16T10:29:51.060417+0000","last_fullsized":"2024-09-16T10:29:51.060417+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.404383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.404383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292265+0000","last_change":"2024-09-16T10:28:28.074218+0000","last_active":"2024-09-16T10:29:39.292265+0000","last_peered":"2024-09-16T10:29:39.292265+0000","last_clean":"2024-09-16T10:29:39.292265+0000","last_became_active":"2024-09-16T10:28:28.073750+0000","last_became_peered":"2024-09-16T10:28:28.073750+0000","last_unstale":"2024-09-16T10:29:39.292265+0000","last_undegraded":"2024-09-16T10:29:39.292265+0000","last_fullsized":"2024-09-16T10:29:39.292265+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.521586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.521586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":704,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062581+0000","last_change":"2024-09-16T10:27:40.298188+0000","last_active":"2024-09-16T10:29:51.062581+0000","last_peered":"2024-09-16T10:29:51.062581+0000","last_clean":"2024-09-16T10:29:51.062581+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:29:51.062581+0000","last_undegraded":"2024-09-16T10:29:51.062581+0000","last_fullsized":"2024-09-16T10:29:51.062581+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"482'1986","reported_seq":2009,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.933321+0000","last_change":"2024-09-16T10:29:35.280049+0000","last_active":"2024-09-16T10:29:52.933321+0000","last_peered":"2024-09-16T10:29:52.933321+0000","last_clean":"2024-09-16T10:29:52.933321+0000","last_became_active":"2024-09-16T10:29:35.279303+0000","last_became_peered":"2024-09-16T10:29:35.279303+0000","last_unstale":"2024-09-16T10:29:52.933321+0000","last_undegraded":"2024-09-16T10:29:52.933321+0000","last_fullsized":"2024-09-16T10:29:52.933321+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1986,"log_dups_size":0,"ondisk_log_size":1986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.356324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130154496,"num_objects":1986,"num_object_clones":0,"num_object_copies":5958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1986,"num_write_kb":127104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":804,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060437+0000","last_change":"2024-09-16T10:29:36.701757+0000","last_active":"2024-09-16T10:29:51.060437+0000","last_peered":"2024-09-16T10:29:51.060437+0000","last_clean":"2024-09-16T10:29:51.060437+0000","last_became_active":"2024-09-16T10:28:28.095330+0000","last_became_peered":"2024-09-16T10:28:28.095330+0000","last_unstale":"2024-09-16T10:29:51.060437+0000","last_undegraded":"2024-09-16T10:29:51.060437+0000","last_fullsized":"2024-09-16T10:29:51.060437+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:36.701703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.701703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.846267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":841,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061813+0000","last_change":"2024-09-16T10:27:53.065126+0000","last_active":"2024-09-16T10:29:51.061813+0000","last_peered":"2024-09-16T10:29:51.061813+0000","last_clean":"2024-09-16T10:29:51.061813+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:29:51.061813+0000","last_undegraded":"2024-09-16T10:29:51.061813+0000","last_fullsized":"2024-09-16T10:29:51.061813+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:27:53.065069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:53.065069+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":893,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030743+0000","last_change":"2024-09-16T10:27:44.624946+0000","last_active":"2024-09-16T10:29:40.030743+0000","last_peered":"2024-09-16T10:29:40.030743+0000","last_clean":"2024-09-16T10:29:40.030743+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:29:40.030743+0000","last_undegraded":"2024-09-16T10:29:40.030743+0000","last_fullsized":"2024-09-16T10:29:40.030743+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":1103,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059555+0000","last_change":"2024-09-16T10:27:24.281038+0000","last_active":"2024-09-16T10:29:51.059555+0000","last_peered":"2024-09-16T10:29:51.059555+0000","last_clean":"2024-09-16T10:29:51.059555+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:29:51.059555+0000","last_undegraded":"2024-09-16T10:29:51.059555+0000","last_fullsized":"2024-09-16T10:29:51.059555+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:27:24.280960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:24.280960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":751,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292730+0000","last_change":"2024-09-16T10:29:20.267647+0000","last_active":"2024-09-16T10:29:39.292730+0000","last_peered":"2024-09-16T10:29:39.292730+0000","last_clean":"2024-09-16T10:29:39.292730+0000","last_became_active":"2024-09-16T10:28:28.071989+0000","last_became_peered":"2024-09-16T10:28:28.071989+0000","last_unstale":"2024-09-16T10:29:39.292730+0000","last_undegraded":"2024-09-16T10:29:39.292730+0000","last_fullsized":"2024-09-16T10:29:39.292730+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.267597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.267597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:29.879063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"480'1384","reported_seq":1403,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:49.165029+0000","last_change":"2024-09-16T10:29:35.170854+0000","last_active":"2024-09-16T10:29:49.165029+0000","last_peered":"2024-09-16T10:29:49.165029+0000","last_clean":"2024-09-16T10:29:49.165029+0000","last_became_active":"2024-09-16T10:29:35.170400+0000","last_became_peered":"2024-09-16T10:29:35.170400+0000","last_unstale":"2024-09-16T10:29:49.165029+0000","last_undegraded":"2024-09-16T10:29:49.165029+0000","last_fullsized":"2024-09-16T10:29:49.165029+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1384,"log_dups_size":0,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.870813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90701824,"num_objects":1384,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":88576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":850,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292928+0000","last_change":"2024-09-16T10:29:29.272700+0000","last_active":"2024-09-16T10:29:39.292928+0000","last_peered":"2024-09-16T10:29:39.292928+0000","last_clean":"2024-09-16T10:29:39.292928+0000","last_became_active":"2024-09-16T10:28:28.089337+0000","last_became_peered":"2024-09-16T10:28:28.089337+0000","last_unstale":"2024-09-16T10:29:39.292928+0000","last_undegraded":"2024-09-16T10:29:39.292928+0000","last_fullsized":"2024-09-16T10:29:39.292928+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.272650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.272650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.748781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":939,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062853+0000","last_change":"2024-09-16T10:29:25.663951+0000","last_active":"2024-09-16T10:29:51.062853+0000","last_peered":"2024-09-16T10:29:51.062853+0000","last_clean":"2024-09-16T10:29:51.062853+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:29:51.062853+0000","last_undegraded":"2024-09-16T10:29:51.062853+0000","last_fullsized":"2024-09-16T10:29:51.062853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.992176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":919,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062506+0000","last_change":"2024-09-16T10:29:27.891418+0000","last_active":"2024-09-16T10:29:51.062506+0000","last_peered":"2024-09-16T10:29:51.062506+0000","last_clean":"2024-09-16T10:29:51.062506+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:29:51.062506+0000","last_undegraded":"2024-09-16T10:29:51.062506+0000","last_fullsized":"2024-09-16T10:29:51.062506+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.891362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.891362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.339178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1079,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:42.372620+0000","last_change":"2024-09-16T10:26:46.635924+0000","last_active":"2024-09-16T10:29:42.372620+0000","last_peered":"2024-09-16T10:29:42.372620+0000","last_clean":"2024-09-16T10:29:42.372620+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:29:42.372620+0000","last_undegraded":"2024-09-16T10:29:42.372620+0000","last_fullsized":"2024-09-16T10:29:42.372620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:46.635855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.635855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":903,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029841+0000","last_change":"2024-09-16T10:26:52.648107+0000","last_active":"2024-09-16T10:29:40.029841+0000","last_peered":"2024-09-16T10:29:40.029841+0000","last_clean":"2024-09-16T10:29:40.029841+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:29:40.029841+0000","last_undegraded":"2024-09-16T10:29:40.029841+0000","last_fullsized":"2024-09-16T10:29:40.029841+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:52.648041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:52.648041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"482'1852","reported_seq":1875,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.926814+0000","last_change":"2024-09-16T10:29:35.168446+0000","last_active":"2024-09-16T10:29:52.926814+0000","last_peered":"2024-09-16T10:29:52.926814+0000","last_clean":"2024-09-16T10:29:52.926814+0000","last_became_active":"2024-09-16T10:29:35.168175+0000","last_became_peered":"2024-09-16T10:29:35.168175+0000","last_unstale":"2024-09-16T10:29:52.926814+0000","last_undegraded":"2024-09-16T10:29:52.926814+0000","last_fullsized":"2024-09-16T10:29:52.926814+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.317930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121372672,"num_objects":1852,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1852,"num_write_kb":118528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":844,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292764+0000","last_change":"2024-09-16T10:26:51.422615+0000","last_active":"2024-09-16T10:29:39.292764+0000","last_peered":"2024-09-16T10:29:39.292764+0000","last_clean":"2024-09-16T10:29:39.292764+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:29:39.292764+0000","last_undegraded":"2024-09-16T10:29:39.292764+0000","last_fullsized":"2024-09-16T10:29:39.292764+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:51.422550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:51.422550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":958,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030299+0000","last_change":"2024-09-16T10:29:27.111777+0000","last_active":"2024-09-16T10:29:40.030299+0000","last_peered":"2024-09-16T10:29:40.030299+0000","last_clean":"2024-09-16T10:29:40.030299+0000","last_became_active":"2024-09-16T10:28:28.112972+0000","last_became_peered":"2024-09-16T10:28:28.112972+0000","last_unstale":"2024-09-16T10:29:40.030299+0000","last_undegraded":"2024-09-16T10:29:40.030299+0000","last_fullsized":"2024-09-16T10:29:40.030299+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:29:27.111715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.111715+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.694926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":919,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.057245+0000","last_change":"2024-09-16T10:29:26.686298+0000","last_active":"2024-09-16T10:29:51.057245+0000","last_peered":"2024-09-16T10:29:51.057245+0000","last_clean":"2024-09-16T10:29:51.057245+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:29:51.057245+0000","last_undegraded":"2024-09-16T10:29:51.057245+0000","last_fullsized":"2024-09-16T10:29:51.057245+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.882005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":900,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292413+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:29:39.292413+0000","last_peered":"2024-09-16T10:29:39.292413+0000","last_clean":"2024-09-16T10:29:39.292413+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:29:39.292413+0000","last_undegraded":"2024-09-16T10:29:39.292413+0000","last_fullsized":"2024-09-16T10:29:39.292413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":899,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.058513+0000","last_change":"2024-09-16T10:27:41.587855+0000","last_active":"2024-09-16T10:29:51.058513+0000","last_peered":"2024-09-16T10:29:51.058513+0000","last_clean":"2024-09-16T10:29:51.058513+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:29:51.058513+0000","last_undegraded":"2024-09-16T10:29:51.058513+0000","last_fullsized":"2024-09-16T10:29:51.058513+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"482'1908","reported_seq":1931,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:53.038647+0000","last_change":"2024-09-16T10:29:35.171242+0000","last_active":"2024-09-16T10:29:53.038647+0000","last_peered":"2024-09-16T10:29:53.038647+0000","last_clean":"2024-09-16T10:29:53.038647+0000","last_became_active":"2024-09-16T10:29:35.171007+0000","last_became_peered":"2024-09-16T10:29:35.171007+0000","last_unstale":"2024-09-16T10:29:53.038647+0000","last_undegraded":"2024-09-16T10:29:53.038647+0000","last_fullsized":"2024-09-16T10:29:53.038647+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1908,"log_dups_size":0,"ondisk_log_size":1908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.125636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125042688,"num_objects":1908,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1908,"num_write_kb":122112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":794,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061659+0000","last_change":"2024-09-16T10:26:09.992070+0000","last_active":"2024-09-16T10:29:51.061659+0000","last_peered":"2024-09-16T10:29:51.061659+0000","last_clean":"2024-09-16T10:29:51.061659+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:29:51.061659+0000","last_undegraded":"2024-09-16T10:29:51.061659+0000","last_fullsized":"2024-09-16T10:29:51.061659+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.992000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.992000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.967589+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":1237,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.057034+0000","last_change":"2024-09-16T10:27:14.409646+0000","last_active":"2024-09-16T10:29:51.057034+0000","last_peered":"2024-09-16T10:29:51.057034+0000","last_clean":"2024-09-16T10:29:51.057034+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:29:51.057034+0000","last_undegraded":"2024-09-16T10:29:51.057034+0000","last_fullsized":"2024-09-16T10:29:51.057034+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:27:14.409552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:14.409552+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030545+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:29:40.030545+0000","last_peered":"2024-09-16T10:29:40.030545+0000","last_clean":"2024-09-16T10:29:40.030545+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:29:40.030545+0000","last_undegraded":"2024-09-16T10:29:40.030545+0000","last_fullsized":"2024-09-16T10:29:40.030545+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":915,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062315+0000","last_change":"2024-09-16T10:27:23.145340+0000","last_active":"2024-09-16T10:29:51.062315+0000","last_peered":"2024-09-16T10:29:51.062315+0000","last_clean":"2024-09-16T10:29:51.062315+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:29:51.062315+0000","last_undegraded":"2024-09-16T10:29:51.062315+0000","last_fullsized":"2024-09-16T10:29:51.062315+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:27:23.145278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:27:23.145278+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":876,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030390+0000","last_change":"2024-09-16T10:25:37.359150+0000","last_active":"2024-09-16T10:29:40.030390+0000","last_peered":"2024-09-16T10:29:40.030390+0000","last_clean":"2024-09-16T10:29:40.030390+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:29:40.030390+0000","last_undegraded":"2024-09-16T10:29:40.030390+0000","last_fullsized":"2024-09-16T10:29:40.030390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:37.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":910,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062435+0000","last_change":"2024-09-16T10:28:28.097003+0000","last_active":"2024-09-16T10:29:51.062435+0000","last_peered":"2024-09-16T10:29:51.062435+0000","last_clean":"2024-09-16T10:29:51.062435+0000","last_became_active":"2024-09-16T10:28:28.096691+0000","last_became_peered":"2024-09-16T10:28:28.096691+0000","last_unstale":"2024-09-16T10:29:51.062435+0000","last_undegraded":"2024-09-16T10:29:51.062435+0000","last_fullsized":"2024-09-16T10:29:51.062435+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:26.349415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:26.349415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"482'1890","reported_seq":1913,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:53.055528+0000","last_change":"2024-09-16T10:29:35.167788+0000","last_active":"2024-09-16T10:29:53.055528+0000","last_peered":"2024-09-16T10:29:53.055528+0000","last_clean":"2024-09-16T10:29:53.055528+0000","last_became_active":"2024-09-16T10:29:35.167515+0000","last_became_peered":"2024-09-16T10:29:35.167515+0000","last_unstale":"2024-09-16T10:29:53.055528+0000","last_undegraded":"2024-09-16T10:29:53.055528+0000","last_fullsized":"2024-09-16T10:29:53.055528+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1890,"log_dups_size":0,"ondisk_log_size":1890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.448815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123863040,"num_objects":1890,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1890,"num_write_kb":120960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":366,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029898+0000","last_change":"2024-09-16T10:28:28.073697+0000","last_active":"2024-09-16T10:29:40.029898+0000","last_peered":"2024-09-16T10:29:40.029898+0000","last_clean":"2024-09-16T10:29:40.029898+0000","last_became_active":"2024-09-16T10:28:28.073341+0000","last_became_peered":"2024-09-16T10:28:28.073341+0000","last_unstale":"2024-09-16T10:29:40.029898+0000","last_undegraded":"2024-09-16T10:29:40.029898+0000","last_fullsized":"2024-09-16T10:29:40.029898+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.028638+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":910,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062358+0000","last_change":"2024-09-16T10:25:13.649143+0000","last_active":"2024-09-16T10:29:51.062358+0000","last_peered":"2024-09-16T10:29:51.062358+0000","last_clean":"2024-09-16T10:29:51.062358+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:29:51.062358+0000","last_undegraded":"2024-09-16T10:29:51.062358+0000","last_fullsized":"2024-09-16T10:29:51.062358+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:25:13.648981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:13.648981+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":904,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.056189+0000","last_change":"2024-09-16T10:29:20.691013+0000","last_active":"2024-09-16T10:29:51.056189+0000","last_peered":"2024-09-16T10:29:51.056189+0000","last_clean":"2024-09-16T10:29:51.056189+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:29:51.056189+0000","last_undegraded":"2024-09-16T10:29:51.056189+0000","last_fullsized":"2024-09-16T10:29:51.056189+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:41.420313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":430,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182949+0000","last_change":"2024-09-16T10:26:18.549126+0000","last_active":"2024-09-16T10:29:39.182949+0000","last_peered":"2024-09-16T10:29:39.182949+0000","last_clean":"2024-09-16T10:29:39.182949+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:29:39.182949+0000","last_undegraded":"2024-09-16T10:29:39.182949+0000","last_fullsized":"2024-09-16T10:29:39.182949+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:26:18.549005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:26:18.549005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":437,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184816+0000","last_change":"2024-09-16T10:29:24.484142+0000","last_active":"2024-09-16T10:29:39.184816+0000","last_peered":"2024-09-16T10:29:39.184816+0000","last_clean":"2024-09-16T10:29:39.184816+0000","last_became_active":"2024-09-16T10:28:28.081619+0000","last_became_peered":"2024-09-16T10:28:28.081619+0000","last_unstale":"2024-09-16T10:29:39.184816+0000","last_undegraded":"2024-09-16T10:29:39.184816+0000","last_fullsized":"2024-09-16T10:29:39.184816+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.484086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.484086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.572572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":291,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292740+0000","last_change":"2024-09-16T10:28:28.073905+0000","last_active":"2024-09-16T10:29:39.292740+0000","last_peered":"2024-09-16T10:29:39.292740+0000","last_clean":"2024-09-16T10:29:39.292740+0000","last_became_active":"2024-09-16T10:28:28.073357+0000","last_became_peered":"2024-09-16T10:28:28.073357+0000","last_unstale":"2024-09-16T10:29:39.292740+0000","last_undegraded":"2024-09-16T10:29:39.292740+0000","last_fullsized":"2024-09-16T10:29:39.292740+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"482'1895","reported_seq":1918,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:53.049847+0000","last_change":"2024-09-16T10:29:35.171912+0000","last_active":"2024-09-16T10:29:53.049847+0000","last_peered":"2024-09-16T10:29:53.049847+0000","last_clean":"2024-09-16T10:29:53.049847+0000","last_became_active":"2024-09-16T10:29:35.171449+0000","last_became_peered":"2024-09-16T10:29:35.171449+0000","last_unstale":"2024-09-16T10:29:53.049847+0000","last_undegraded":"2024-09-16T10:29:53.049847+0000","last_fullsized":"2024-09-16T10:29:53.049847+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1895,"log_dups_size":0,"ondisk_log_size":1895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124190720,"num_objects":1895,"num_object_clones":0,"num_object_copies":5685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1895,"num_write_kb":121280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":782,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061594+0000","last_change":"2024-09-16T10:26:09.201694+0000","last_active":"2024-09-16T10:29:51.061594+0000","last_peered":"2024-09-16T10:29:51.061594+0000","last_clean":"2024-09-16T10:29:51.061594+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:29:51.061594+0000","last_undegraded":"2024-09-16T10:29:51.061594+0000","last_fullsized":"2024-09-16T10:29:51.061594+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.201559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.201559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":990,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030686+0000","last_change":"2024-09-16T10:27:09.622319+0000","last_active":"2024-09-16T10:29:40.030686+0000","last_peered":"2024-09-16T10:29:40.030686+0000","last_clean":"2024-09-16T10:29:40.030686+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:29:40.030686+0000","last_undegraded":"2024-09-16T10:29:40.030686+0000","last_fullsized":"2024-09-16T10:29:40.030686+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:27:09.622239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.622239+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029969+0000","last_change":"2024-09-16T10:27:10.642822+0000","last_active":"2024-09-16T10:29:40.029969+0000","last_peered":"2024-09-16T10:29:40.029969+0000","last_clean":"2024-09-16T10:29:40.029969+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:29:40.029969+0000","last_undegraded":"2024-09-16T10:29:40.029969+0000","last_fullsized":"2024-09-16T10:29:40.029969+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:10.642739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:10.642739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":627,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.404550+0000","last_change":"2024-09-16T10:29:30.751685+0000","last_active":"2024-09-16T10:29:51.404550+0000","last_peered":"2024-09-16T10:29:51.404550+0000","last_clean":"2024-09-16T10:29:51.404550+0000","last_became_active":"2024-09-16T10:28:28.070293+0000","last_became_peered":"2024-09-16T10:28:28.070293+0000","last_unstale":"2024-09-16T10:29:51.404550+0000","last_undegraded":"2024-09-16T10:29:51.404550+0000","last_fullsized":"2024-09-16T10:29:51.404550+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:29:30.751623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.751623+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:34.435816+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":905,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062540+0000","last_change":"2024-09-16T10:27:20.130780+0000","last_active":"2024-09-16T10:29:51.062540+0000","last_peered":"2024-09-16T10:29:51.062540+0000","last_clean":"2024-09-16T10:29:51.062540+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:29:51.062540+0000","last_undegraded":"2024-09-16T10:29:51.062540+0000","last_fullsized":"2024-09-16T10:29:51.062540+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:20.130715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.130715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":909,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.056583+0000","last_change":"2024-09-16T10:27:19.374485+0000","last_active":"2024-09-16T10:29:51.056583+0000","last_peered":"2024-09-16T10:29:51.056583+0000","last_clean":"2024-09-16T10:29:51.056583+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:29:51.056583+0000","last_undegraded":"2024-09-16T10:29:51.056583+0000","last_fullsized":"2024-09-16T10:29:51.056583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"482'1947","reported_seq":1970,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:53.054646+0000","last_change":"2024-09-16T10:29:35.169178+0000","last_active":"2024-09-16T10:29:53.054646+0000","last_peered":"2024-09-16T10:29:53.054646+0000","last_clean":"2024-09-16T10:29:53.054646+0000","last_became_active":"2024-09-16T10:29:35.168851+0000","last_became_peered":"2024-09-16T10:29:35.168851+0000","last_unstale":"2024-09-16T10:29:53.054646+0000","last_undegraded":"2024-09-16T10:29:53.054646+0000","last_fullsized":"2024-09-16T10:29:53.054646+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1947,"log_dups_size":0,"ondisk_log_size":1947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.747588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127598592,"num_objects":1947,"num_object_clones":0,"num_object_copies":5841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1947,"num_write_kb":124608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"442'30","reported_seq":1127,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:42.372451+0000","last_change":"2024-09-16T10:29:18.202869+0000","last_active":"2024-09-16T10:29:42.372451+0000","last_peered":"2024-09-16T10:29:42.372451+0000","last_clean":"2024-09-16T10:29:42.372451+0000","last_became_active":"2024-09-16T10:28:28.072900+0000","last_became_peered":"2024-09-16T10:28:28.072900+0000","last_unstale":"2024-09-16T10:29:42.372451+0000","last_undegraded":"2024-09-16T10:29:42.372451+0000","last_fullsized":"2024-09-16T10:29:42.372451+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:29:18.202771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:18.202771+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:45.201112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":868,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.067617+0000","last_change":"2024-09-16T10:26:47.493915+0000","last_active":"2024-09-16T10:29:51.067617+0000","last_peered":"2024-09-16T10:29:51.067617+0000","last_clean":"2024-09-16T10:29:51.067617+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:29:51.067617+0000","last_undegraded":"2024-09-16T10:29:51.067617+0000","last_fullsized":"2024-09-16T10:29:51.067617+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:47.493848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:47.493848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":366,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029935+0000","last_change":"2024-09-16T10:28:28.090815+0000","last_active":"2024-09-16T10:29:40.029935+0000","last_peered":"2024-09-16T10:29:40.029935+0000","last_clean":"2024-09-16T10:29:40.029935+0000","last_became_active":"2024-09-16T10:28:28.090031+0000","last_became_peered":"2024-09-16T10:28:28.090031+0000","last_unstale":"2024-09-16T10:29:40.029935+0000","last_undegraded":"2024-09-16T10:29:40.029935+0000","last_fullsized":"2024-09-16T10:29:40.029935+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":762,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184658+0000","last_change":"2024-09-16T10:28:28.072722+0000","last_active":"2024-09-16T10:29:39.184658+0000","last_peered":"2024-09-16T10:29:39.184658+0000","last_clean":"2024-09-16T10:29:39.184658+0000","last_became_active":"2024-09-16T10:28:28.072192+0000","last_became_peered":"2024-09-16T10:28:28.072192+0000","last_unstale":"2024-09-16T10:29:39.184658+0000","last_undegraded":"2024-09-16T10:29:39.184658+0000","last_fullsized":"2024-09-16T10:29:39.184658+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.591323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":893,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062464+0000","last_change":"2024-09-16T10:26:19.084467+0000","last_active":"2024-09-16T10:29:51.062464+0000","last_peered":"2024-09-16T10:29:51.062464+0000","last_clean":"2024-09-16T10:29:51.062464+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:29:51.062464+0000","last_undegraded":"2024-09-16T10:29:51.062464+0000","last_fullsized":"2024-09-16T10:29:51.062464+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:19.084409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:19.084409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.011496+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"480'1288","reported_seq":1307,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:48.549509+0000","last_change":"2024-09-16T10:29:35.171216+0000","last_active":"2024-09-16T10:29:48.549509+0000","last_peered":"2024-09-16T10:29:48.549509+0000","last_clean":"2024-09-16T10:29:48.549509+0000","last_became_active":"2024-09-16T10:29:35.170866+0000","last_became_peered":"2024-09-16T10:29:35.170866+0000","last_unstale":"2024-09-16T10:29:48.549509+0000","last_undegraded":"2024-09-16T10:29:48.549509+0000","last_fullsized":"2024-09-16T10:29:48.549509+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.526803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84410368,"num_objects":1288,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1288,"num_write_kb":82432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":790,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061371+0000","last_change":"2024-09-16T10:29:27.703960+0000","last_active":"2024-09-16T10:29:51.061371+0000","last_peered":"2024-09-16T10:29:51.061371+0000","last_clean":"2024-09-16T10:29:51.061371+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:29:51.061371+0000","last_undegraded":"2024-09-16T10:29:51.061371+0000","last_fullsized":"2024-09-16T10:29:51.061371+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:27.703908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.703908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.239882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":829,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292878+0000","last_change":"2024-09-16T10:26:28.556079+0000","last_active":"2024-09-16T10:29:39.292878+0000","last_peered":"2024-09-16T10:29:39.292878+0000","last_clean":"2024-09-16T10:29:39.292878+0000","last_became_active":"2024-09-16T10:26:28.554604+0000","last_became_peered":"2024-09-16T10:26:28.554604+0000","last_unstale":"2024-09-16T10:29:39.292878+0000","last_undegraded":"2024-09-16T10:29:39.292878+0000","last_fullsized":"2024-09-16T10:29:39.292878+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:26:10.386128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.386128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":706,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182369+0000","last_change":"2024-09-16T10:29:23.474783+0000","last_active":"2024-09-16T10:29:39.182369+0000","last_peered":"2024-09-16T10:29:39.182369+0000","last_clean":"2024-09-16T10:29:39.182369+0000","last_became_active":"2024-09-16T10:28:28.071386+0000","last_became_peered":"2024-09-16T10:28:28.071386+0000","last_unstale":"2024-09-16T10:29:39.182369+0000","last_undegraded":"2024-09-16T10:29:39.182369+0000","last_fullsized":"2024-09-16T10:29:39.182369+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.474733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.474733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:52.069257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":905,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062416+0000","last_change":"2024-09-16T10:29:30.955964+0000","last_active":"2024-09-16T10:29:51.062416+0000","last_peered":"2024-09-16T10:29:51.062416+0000","last_clean":"2024-09-16T10:29:51.062416+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:29:51.062416+0000","last_undegraded":"2024-09-16T10:29:51.062416+0000","last_fullsized":"2024-09-16T10:29:51.062416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:30.955877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.955877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:51.333008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":840,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060349+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:29:51.060349+0000","last_peered":"2024-09-16T10:29:51.060349+0000","last_clean":"2024-09-16T10:29:51.060349+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:29:51.060349+0000","last_undegraded":"2024-09-16T10:29:51.060349+0000","last_fullsized":"2024-09-16T10:29:51.060349+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.037961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"480'1437","reported_seq":1453,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:49.161289+0000","last_change":"2024-09-16T10:29:35.171033+0000","last_active":"2024-09-16T10:29:49.161289+0000","last_peered":"2024-09-16T10:29:49.161289+0000","last_clean":"2024-09-16T10:29:49.161289+0000","last_became_active":"2024-09-16T10:29:35.170730+0000","last_became_peered":"2024-09-16T10:29:35.170730+0000","last_unstale":"2024-09-16T10:29:49.161289+0000","last_undegraded":"2024-09-16T10:29:49.161289+0000","last_fullsized":"2024-09-16T10:29:49.161289+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1437,"log_dups_size":0,"ondisk_log_size":1437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:36.397873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94175232,"num_objects":1437,"num_object_clones":0,"num_object_copies":4311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1437,"num_write_kb":91968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":757,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061608+0000","last_change":"2024-09-16T10:26:49.093325+0000","last_active":"2024-09-16T10:29:51.061608+0000","last_peered":"2024-09-16T10:29:51.061608+0000","last_clean":"2024-09-16T10:29:51.061608+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:29:51.061608+0000","last_undegraded":"2024-09-16T10:29:51.061608+0000","last_fullsized":"2024-09-16T10:29:51.061608+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:49.093245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:49.093245+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":893,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184430+0000","last_change":"2024-09-16T10:27:21.459062+0000","last_active":"2024-09-16T10:29:39.184430+0000","last_peered":"2024-09-16T10:29:39.184430+0000","last_clean":"2024-09-16T10:29:39.184430+0000","last_became_active":"2024-09-16T10:26:28.553171+0000","last_became_peered":"2024-09-16T10:26:28.553171+0000","last_unstale":"2024-09-16T10:29:39.184430+0000","last_undegraded":"2024-09-16T10:29:39.184430+0000","last_fullsized":"2024-09-16T10:29:39.184430+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:27:21.458969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:21.458969+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":892,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292857+0000","last_change":"2024-09-16T10:26:42.490866+0000","last_active":"2024-09-16T10:29:39.292857+0000","last_peered":"2024-09-16T10:29:39.292857+0000","last_clean":"2024-09-16T10:29:39.292857+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:29:39.292857+0000","last_undegraded":"2024-09-16T10:29:39.292857+0000","last_fullsized":"2024-09-16T10:29:39.292857+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:42.490731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.490731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":889,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.056798+0000","last_change":"2024-09-16T10:29:21.720973+0000","last_active":"2024-09-16T10:29:51.056798+0000","last_peered":"2024-09-16T10:29:51.056798+0000","last_clean":"2024-09-16T10:29:51.056798+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:29:51.056798+0000","last_undegraded":"2024-09-16T10:29:51.056798+0000","last_fullsized":"2024-09-16T10:29:51.056798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.720910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.720910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.270176+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184922+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:29:39.184922+0000","last_peered":"2024-09-16T10:29:39.184922+0000","last_clean":"2024-09-16T10:29:39.184922+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:29:39.184922+0000","last_undegraded":"2024-09-16T10:29:39.184922+0000","last_fullsized":"2024-09-16T10:29:39.184922+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"480'1368","reported_seq":1387,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:48.553554+0000","last_change":"2024-09-16T10:29:35.279567+0000","last_active":"2024-09-16T10:29:48.553554+0000","last_peered":"2024-09-16T10:29:48.553554+0000","last_clean":"2024-09-16T10:29:48.553554+0000","last_became_active":"2024-09-16T10:29:35.279253+0000","last_became_peered":"2024-09-16T10:29:35.279253+0000","last_unstale":"2024-09-16T10:29:48.553554+0000","last_undegraded":"2024-09-16T10:29:48.553554+0000","last_fullsized":"2024-09-16T10:29:48.553554+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.137735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89653248,"num_objects":1368,"num_object_clones":0,"num_object_copies":4104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1368,"num_write_kb":87552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":570,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061354+0000","last_change":"2024-09-16T10:28:28.096386+0000","last_active":"2024-09-16T10:29:51.061354+0000","last_peered":"2024-09-16T10:29:51.061354+0000","last_clean":"2024-09-16T10:29:51.061354+0000","last_became_active":"2024-09-16T10:28:28.095730+0000","last_became_peered":"2024-09-16T10:28:28.095730+0000","last_unstale":"2024-09-16T10:29:51.061354+0000","last_undegraded":"2024-09-16T10:29:51.061354+0000","last_fullsized":"2024-09-16T10:29:51.061354+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:16.433199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:16.433199+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":813,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059953+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:29:51.059953+0000","last_peered":"2024-09-16T10:29:51.059953+0000","last_clean":"2024-09-16T10:29:51.059953+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:29:51.059953+0000","last_undegraded":"2024-09-16T10:29:51.059953+0000","last_fullsized":"2024-09-16T10:29:51.059953+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.568656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":887,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062054+0000","last_change":"2024-09-16T10:27:12.155297+0000","last_active":"2024-09-16T10:29:51.062054+0000","last_peered":"2024-09-16T10:29:51.062054+0000","last_clean":"2024-09-16T10:29:51.062054+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:29:51.062054+0000","last_undegraded":"2024-09-16T10:29:51.062054+0000","last_fullsized":"2024-09-16T10:29:51.062054+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:12.155186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.155186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"442'12","reported_seq":536,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:42.373736+0000","last_change":"2024-09-16T10:29:21.445504+0000","last_active":"2024-09-16T10:29:42.373736+0000","last_peered":"2024-09-16T10:29:42.373736+0000","last_clean":"2024-09-16T10:29:42.373736+0000","last_became_active":"2024-09-16T10:28:28.072074+0000","last_became_peered":"2024-09-16T10:28:28.072074+0000","last_unstale":"2024-09-16T10:29:42.373736+0000","last_undegraded":"2024-09-16T10:29:42.373736+0000","last_fullsized":"2024-09-16T10:29:42.373736+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:29:21.445439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.445439+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.856000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":696,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.056503+0000","last_change":"2024-09-16T10:27:13.405307+0000","last_active":"2024-09-16T10:29:51.056503+0000","last_peered":"2024-09-16T10:29:51.056503+0000","last_clean":"2024-09-16T10:29:51.056503+0000","last_became_active":"2024-09-16T10:26:28.550959+0000","last_became_peered":"2024-09-16T10:26:28.550959+0000","last_unstale":"2024-09-16T10:29:51.056503+0000","last_undegraded":"2024-09-16T10:29:51.056503+0000","last_fullsized":"2024-09-16T10:29:51.056503+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.405239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.405239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"480'1326","reported_seq":1345,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:48.539617+0000","last_change":"2024-09-16T10:29:35.280282+0000","last_active":"2024-09-16T10:29:48.539617+0000","last_peered":"2024-09-16T10:29:48.539617+0000","last_clean":"2024-09-16T10:29:48.539617+0000","last_became_active":"2024-09-16T10:29:35.279983+0000","last_became_peered":"2024-09-16T10:29:35.279983+0000","last_unstale":"2024-09-16T10:29:48.539617+0000","last_undegraded":"2024-09-16T10:29:48.539617+0000","last_fullsized":"2024-09-16T10:29:48.539617+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1326,"log_dups_size":0,"ondisk_log_size":1326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.246468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":84864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":233,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.551314+0000","last_change":"2024-09-16T10:29:28.900601+0000","last_active":"2024-09-16T10:29:50.551314+0000","last_peered":"2024-09-16T10:29:50.551314+0000","last_clean":"2024-09-16T10:29:50.551314+0000","last_became_active":"2024-09-16T10:28:28.076677+0000","last_became_peered":"2024-09-16T10:28:28.076677+0000","last_unstale":"2024-09-16T10:29:50.551314+0000","last_undegraded":"2024-09-16T10:29:50.551314+0000","last_fullsized":"2024-09-16T10:29:50.551314+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.900551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.900551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:32.135504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":913,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029537+0000","last_change":"2024-09-16T10:27:20.653839+0000","last_active":"2024-09-16T10:29:40.029537+0000","last_peered":"2024-09-16T10:29:40.029537+0000","last_clean":"2024-09-16T10:29:40.029537+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:29:40.029537+0000","last_undegraded":"2024-09-16T10:29:40.029537+0000","last_fullsized":"2024-09-16T10:29:40.029537+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:27:20.653762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:20.653762+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":913,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062191+0000","last_change":"2024-09-16T10:26:44.070285+0000","last_active":"2024-09-16T10:29:51.062191+0000","last_peered":"2024-09-16T10:29:51.062191+0000","last_clean":"2024-09-16T10:29:51.062191+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:29:51.062191+0000","last_undegraded":"2024-09-16T10:29:51.062191+0000","last_fullsized":"2024-09-16T10:29:51.062191+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:44.070218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.070218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184487+0000","last_change":"2024-09-16T10:26:22.501028+0000","last_active":"2024-09-16T10:29:39.184487+0000","last_peered":"2024-09-16T10:29:39.184487+0000","last_clean":"2024-09-16T10:29:39.184487+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:29:39.184487+0000","last_undegraded":"2024-09-16T10:29:39.184487+0000","last_fullsized":"2024-09-16T10:29:39.184487+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.500963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.500963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":928,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029995+0000","last_change":"2024-09-16T10:28:28.075186+0000","last_active":"2024-09-16T10:29:40.029995+0000","last_peered":"2024-09-16T10:29:40.029995+0000","last_clean":"2024-09-16T10:29:40.029995+0000","last_became_active":"2024-09-16T10:28:28.074693+0000","last_became_peered":"2024-09-16T10:28:28.074693+0000","last_unstale":"2024-09-16T10:29:40.029995+0000","last_undegraded":"2024-09-16T10:29:40.029995+0000","last_fullsized":"2024-09-16T10:29:40.029995+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:26:20.555476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.555476+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"482'1867","reported_seq":1890,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.929050+0000","last_change":"2024-09-16T10:29:35.279583+0000","last_active":"2024-09-16T10:29:52.929050+0000","last_peered":"2024-09-16T10:29:52.929050+0000","last_clean":"2024-09-16T10:29:52.929050+0000","last_became_active":"2024-09-16T10:29:35.279260+0000","last_became_peered":"2024-09-16T10:29:35.279260+0000","last_unstale":"2024-09-16T10:29:52.929050+0000","last_undegraded":"2024-09-16T10:29:52.929050+0000","last_fullsized":"2024-09-16T10:29:52.929050+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1867,"log_dups_size":0,"ondisk_log_size":1867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.395721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122355712,"num_objects":1867,"num_object_clones":0,"num_object_copies":5601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1867,"num_write_kb":119488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":857,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.293152+0000","last_change":"2024-09-16T10:25:40.594525+0000","last_active":"2024-09-16T10:29:39.293152+0000","last_peered":"2024-09-16T10:29:39.293152+0000","last_clean":"2024-09-16T10:29:39.293152+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:29:39.293152+0000","last_undegraded":"2024-09-16T10:29:39.293152+0000","last_fullsized":"2024-09-16T10:29:39.293152+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:40.594454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.594454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.350308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":970,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292531+0000","last_change":"2024-09-16T10:28:28.088838+0000","last_active":"2024-09-16T10:29:39.292531+0000","last_peered":"2024-09-16T10:29:39.292531+0000","last_clean":"2024-09-16T10:29:39.292531+0000","last_became_active":"2024-09-16T10:28:28.088338+0000","last_became_peered":"2024-09-16T10:28:28.088338+0000","last_unstale":"2024-09-16T10:29:39.292531+0000","last_undegraded":"2024-09-16T10:29:39.292531+0000","last_fullsized":"2024-09-16T10:29:39.292531+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":294,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062339+0000","last_change":"2024-09-16T10:28:28.101048+0000","last_active":"2024-09-16T10:29:51.062339+0000","last_peered":"2024-09-16T10:29:51.062339+0000","last_clean":"2024-09-16T10:29:51.062339+0000","last_became_active":"2024-09-16T10:28:28.100755+0000","last_became_peered":"2024-09-16T10:28:28.100755+0000","last_unstale":"2024-09-16T10:29:51.062339+0000","last_undegraded":"2024-09-16T10:29:51.062339+0000","last_fullsized":"2024-09-16T10:29:51.062339+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.031940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.031940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.981262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.182145+0000","last_change":"2024-09-16T10:26:43.199824+0000","last_active":"2024-09-16T10:29:39.182145+0000","last_peered":"2024-09-16T10:29:39.182145+0000","last_clean":"2024-09-16T10:29:39.182145+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:29:39.182145+0000","last_undegraded":"2024-09-16T10:29:39.182145+0000","last_fullsized":"2024-09-16T10:29:39.182145+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:43.199752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:43.199752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":766,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.183930+0000","last_change":"2024-09-16T10:28:28.065678+0000","last_active":"2024-09-16T10:29:39.183930+0000","last_peered":"2024-09-16T10:29:39.183930+0000","last_clean":"2024-09-16T10:29:39.183930+0000","last_became_active":"2024-09-16T10:28:28.065386+0000","last_became_peered":"2024-09-16T10:28:28.065386+0000","last_unstale":"2024-09-16T10:29:39.183930+0000","last_undegraded":"2024-09-16T10:29:39.183930+0000","last_fullsized":"2024-09-16T10:29:39.183930+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:34.550928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:34.550928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"480'1561","reported_seq":1579,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.006388+0000","last_change":"2024-09-16T10:29:35.172946+0000","last_active":"2024-09-16T10:29:50.006388+0000","last_peered":"2024-09-16T10:29:50.006388+0000","last_clean":"2024-09-16T10:29:50.006388+0000","last_became_active":"2024-09-16T10:29:35.172649+0000","last_became_peered":"2024-09-16T10:29:35.172649+0000","last_unstale":"2024-09-16T10:29:50.006388+0000","last_undegraded":"2024-09-16T10:29:50.006388+0000","last_fullsized":"2024-09-16T10:29:50.006388+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.718311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102301696,"num_objects":1561,"num_object_clones":0,"num_object_copies":4683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1561,"num_write_kb":99904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":812,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059643+0000","last_change":"2024-09-16T10:26:10.271093+0000","last_active":"2024-09-16T10:29:51.059643+0000","last_peered":"2024-09-16T10:29:51.059643+0000","last_clean":"2024-09-16T10:29:51.059643+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:29:51.059643+0000","last_undegraded":"2024-09-16T10:29:51.059643+0000","last_fullsized":"2024-09-16T10:29:51.059643+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:10.270833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.270833+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":821,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060394+0000","last_change":"2024-09-16T10:26:42.132944+0000","last_active":"2024-09-16T10:29:51.060394+0000","last_peered":"2024-09-16T10:29:51.060394+0000","last_clean":"2024-09-16T10:29:51.060394+0000","last_became_active":"2024-09-16T10:26:28.553750+0000","last_became_peered":"2024-09-16T10:26:28.553750+0000","last_unstale":"2024-09-16T10:29:51.060394+0000","last_undegraded":"2024-09-16T10:29:51.060394+0000","last_fullsized":"2024-09-16T10:29:51.060394+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:26:42.132853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.132853+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":243,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062315+0000","last_change":"2024-09-16T10:27:13.285419+0000","last_active":"2024-09-16T10:29:51.062315+0000","last_peered":"2024-09-16T10:29:51.062315+0000","last_clean":"2024-09-16T10:29:51.062315+0000","last_became_active":"2024-09-16T10:25:41.990483+0000","last_became_peered":"2024-09-16T10:25:41.990483+0000","last_unstale":"2024-09-16T10:29:51.062315+0000","last_undegraded":"2024-09-16T10:29:51.062315+0000","last_fullsized":"2024-09-16T10:29:51.062315+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":426,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184031+0000","last_change":"2024-09-16T10:26:21.479904+0000","last_active":"2024-09-16T10:29:39.184031+0000","last_peered":"2024-09-16T10:29:39.184031+0000","last_clean":"2024-09-16T10:29:39.184031+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:29:39.184031+0000","last_undegraded":"2024-09-16T10:29:39.184031+0000","last_fullsized":"2024-09-16T10:29:39.184031+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":902,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059500+0000","last_change":"2024-09-16T10:28:28.095795+0000","last_active":"2024-09-16T10:29:51.059500+0000","last_peered":"2024-09-16T10:29:51.059500+0000","last_clean":"2024-09-16T10:29:51.059500+0000","last_became_active":"2024-09-16T10:28:28.095174+0000","last_became_peered":"2024-09-16T10:28:28.095174+0000","last_unstale":"2024-09-16T10:29:51.059500+0000","last_undegraded":"2024-09-16T10:29:51.059500+0000","last_fullsized":"2024-09-16T10:29:51.059500+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:41.478267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:41.478267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"482'1870","reported_seq":1893,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.918800+0000","last_change":"2024-09-16T10:29:35.165138+0000","last_active":"2024-09-16T10:29:52.918800+0000","last_peered":"2024-09-16T10:29:52.918800+0000","last_clean":"2024-09-16T10:29:52.918800+0000","last_became_active":"2024-09-16T10:29:35.164843+0000","last_became_peered":"2024-09-16T10:29:35.164843+0000","last_unstale":"2024-09-16T10:29:52.918800+0000","last_undegraded":"2024-09-16T10:29:52.918800+0000","last_fullsized":"2024-09-16T10:29:52.918800+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.727993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122552320,"num_objects":1870,"num_object_clones":0,"num_object_copies":5610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1870,"num_write_kb":119680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":621,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.058385+0000","last_change":"2024-09-16T10:29:29.763437+0000","last_active":"2024-09-16T10:29:51.058385+0000","last_peered":"2024-09-16T10:29:51.058385+0000","last_clean":"2024-09-16T10:29:51.058385+0000","last_became_active":"2024-09-16T10:28:28.071881+0000","last_became_peered":"2024-09-16T10:28:28.071881+0000","last_unstale":"2024-09-16T10:29:51.058385+0000","last_undegraded":"2024-09-16T10:29:51.058385+0000","last_fullsized":"2024-09-16T10:29:51.058385+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:32.911633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":939,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061756+0000","last_change":"2024-09-16T10:28:28.097779+0000","last_active":"2024-09-16T10:29:51.061756+0000","last_peered":"2024-09-16T10:29:51.061756+0000","last_clean":"2024-09-16T10:29:51.061756+0000","last_became_active":"2024-09-16T10:28:28.097478+0000","last_became_peered":"2024-09-16T10:28:28.097478+0000","last_unstale":"2024-09-16T10:29:51.061756+0000","last_undegraded":"2024-09-16T10:29:51.061756+0000","last_fullsized":"2024-09-16T10:29:51.061756+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:20.104708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.104708+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.925950+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":229,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030350+0000","last_change":"2024-09-16T10:28:28.081893+0000","last_active":"2024-09-16T10:29:40.030350+0000","last_peered":"2024-09-16T10:29:40.030350+0000","last_clean":"2024-09-16T10:29:40.030350+0000","last_became_active":"2024-09-16T10:28:28.081512+0000","last_became_peered":"2024-09-16T10:28:28.081512+0000","last_unstale":"2024-09-16T10:29:40.030350+0000","last_undegraded":"2024-09-16T10:29:40.030350+0000","last_fullsized":"2024-09-16T10:29:40.030350+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:08.381010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.381010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"442'11","reported_seq":375,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:42.372574+0000","last_change":"2024-09-16T10:28:28.074072+0000","last_active":"2024-09-16T10:29:42.372574+0000","last_peered":"2024-09-16T10:29:42.372574+0000","last_clean":"2024-09-16T10:29:42.372574+0000","last_became_active":"2024-09-16T10:28:28.073675+0000","last_became_peered":"2024-09-16T10:28:28.073675+0000","last_unstale":"2024-09-16T10:29:42.372574+0000","last_undegraded":"2024-09-16T10:29:42.372574+0000","last_fullsized":"2024-09-16T10:29:42.372574+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'9","last_scrub_stamp":"2024-09-16T10:25:37.549468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.549468+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":901,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061849+0000","last_change":"2024-09-16T10:27:09.163203+0000","last_active":"2024-09-16T10:29:51.061849+0000","last_peered":"2024-09-16T10:29:51.061849+0000","last_clean":"2024-09-16T10:29:51.061849+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:29:51.061849+0000","last_undegraded":"2024-09-16T10:29:51.061849+0000","last_fullsized":"2024-09-16T10:29:51.061849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:09.163139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.163139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":477,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062017+0000","last_change":"2024-09-16T10:29:28.935051+0000","last_active":"2024-09-16T10:29:51.062017+0000","last_peered":"2024-09-16T10:29:51.062017+0000","last_clean":"2024-09-16T10:29:51.062017+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:29:51.062017+0000","last_undegraded":"2024-09-16T10:29:51.062017+0000","last_fullsized":"2024-09-16T10:29:51.062017+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.935001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.935001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.004175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292444+0000","last_change":"2024-09-16T10:26:20.469286+0000","last_active":"2024-09-16T10:29:39.292444+0000","last_peered":"2024-09-16T10:29:39.292444+0000","last_clean":"2024-09-16T10:29:39.292444+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:29:39.292444+0000","last_undegraded":"2024-09-16T10:29:39.292444+0000","last_fullsized":"2024-09-16T10:29:39.292444+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:20.469240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.469240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":919,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060263+0000","last_change":"2024-09-16T10:29:17.916063+0000","last_active":"2024-09-16T10:29:51.060263+0000","last_peered":"2024-09-16T10:29:51.060263+0000","last_clean":"2024-09-16T10:29:51.060263+0000","last_became_active":"2024-09-16T10:28:28.096974+0000","last_became_peered":"2024-09-16T10:28:28.096974+0000","last_unstale":"2024-09-16T10:29:51.060263+0000","last_undegraded":"2024-09-16T10:29:51.060263+0000","last_fullsized":"2024-09-16T10:29:51.060263+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:29:17.915982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:17.915982+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.961249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":888,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062394+0000","last_change":"2024-09-16T10:26:46.061085+0000","last_active":"2024-09-16T10:29:51.062394+0000","last_peered":"2024-09-16T10:29:51.062394+0000","last_clean":"2024-09-16T10:29:51.062394+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:29:51.062394+0000","last_undegraded":"2024-09-16T10:29:51.062394+0000","last_fullsized":"2024-09-16T10:29:51.062394+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":713,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.551286+0000","last_change":"2024-09-16T10:28:28.077871+0000","last_active":"2024-09-16T10:29:50.551286+0000","last_peered":"2024-09-16T10:29:50.551286+0000","last_clean":"2024-09-16T10:29:50.551286+0000","last_became_active":"2024-09-16T10:28:28.072905+0000","last_became_peered":"2024-09-16T10:28:28.072905+0000","last_unstale":"2024-09-16T10:29:50.551286+0000","last_undegraded":"2024-09-16T10:29:50.551286+0000","last_fullsized":"2024-09-16T10:29:50.551286+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.018433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":790,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061316+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:29:51.061316+0000","last_peered":"2024-09-16T10:29:51.061316+0000","last_clean":"2024-09-16T10:29:51.061316+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:29:51.061316+0000","last_undegraded":"2024-09-16T10:29:51.061316+0000","last_fullsized":"2024-09-16T10:29:51.061316+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.632169+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":923,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030692+0000","last_change":"2024-09-16T10:28:28.083403+0000","last_active":"2024-09-16T10:29:40.030692+0000","last_peered":"2024-09-16T10:29:40.030692+0000","last_clean":"2024-09-16T10:29:40.030692+0000","last_became_active":"2024-09-16T10:28:28.082315+0000","last_became_peered":"2024-09-16T10:28:28.082315+0000","last_unstale":"2024-09-16T10:29:40.030692+0000","last_undegraded":"2024-09-16T10:29:40.030692+0000","last_fullsized":"2024-09-16T10:29:40.030692+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:26:24.517639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.517639+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":229,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292588+0000","last_change":"2024-09-16T10:29:24.266598+0000","last_active":"2024-09-16T10:29:39.292588+0000","last_peered":"2024-09-16T10:29:39.292588+0000","last_clean":"2024-09-16T10:29:39.292588+0000","last_became_active":"2024-09-16T10:28:28.071950+0000","last_became_peered":"2024-09-16T10:28:28.071950+0000","last_unstale":"2024-09-16T10:29:39.292588+0000","last_undegraded":"2024-09-16T10:29:39.292588+0000","last_fullsized":"2024-09-16T10:29:39.292588+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.266543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.266543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:41.996180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":897,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060040+0000","last_change":"2024-09-16T10:28:28.098654+0000","last_active":"2024-09-16T10:29:51.060040+0000","last_peered":"2024-09-16T10:29:51.060040+0000","last_clean":"2024-09-16T10:29:51.060040+0000","last_became_active":"2024-09-16T10:28:28.097898+0000","last_became_peered":"2024-09-16T10:28:28.097898+0000","last_unstale":"2024-09-16T10:29:51.060040+0000","last_undegraded":"2024-09-16T10:29:51.060040+0000","last_fullsized":"2024-09-16T10:29:51.060040+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.419085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.419085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":437,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.185122+0000","last_change":"2024-09-16T10:29:36.541047+0000","last_active":"2024-09-16T10:29:39.185122+0000","last_peered":"2024-09-16T10:29:39.185122+0000","last_clean":"2024-09-16T10:29:39.185122+0000","last_became_active":"2024-09-16T10:28:28.072347+0000","last_became_peered":"2024-09-16T10:28:28.072347+0000","last_unstale":"2024-09-16T10:29:39.185122+0000","last_undegraded":"2024-09-16T10:29:39.185122+0000","last_fullsized":"2024-09-16T10:29:39.185122+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.541001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.541001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.828112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":478,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061557+0000","last_change":"2024-09-16T10:29:34.955215+0000","last_active":"2024-09-16T10:29:51.061557+0000","last_peered":"2024-09-16T10:29:51.061557+0000","last_clean":"2024-09-16T10:29:51.061557+0000","last_became_active":"2024-09-16T10:28:28.096538+0000","last_became_peered":"2024-09-16T10:28:28.096538+0000","last_unstale":"2024-09-16T10:29:51.061557+0000","last_undegraded":"2024-09-16T10:29:51.061557+0000","last_fullsized":"2024-09-16T10:29:51.061557+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:59.442771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":920,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062399+0000","last_change":"2024-09-16T10:29:20.061250+0000","last_active":"2024-09-16T10:29:51.062399+0000","last_peered":"2024-09-16T10:29:51.062399+0000","last_clean":"2024-09-16T10:29:51.062399+0000","last_became_active":"2024-09-16T10:28:28.101644+0000","last_became_peered":"2024-09-16T10:28:28.101644+0000","last_unstale":"2024-09-16T10:29:51.062399+0000","last_undegraded":"2024-09-16T10:29:51.062399+0000","last_fullsized":"2024-09-16T10:29:51.062399+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:29:20.061184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.061184+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:21.959259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030148+0000","last_change":"2024-09-16T10:29:38.060471+0000","last_active":"2024-09-16T10:29:40.030148+0000","last_peered":"2024-09-16T10:29:40.030148+0000","last_clean":"2024-09-16T10:29:40.030148+0000","last_became_active":"2024-09-16T10:28:28.072930+0000","last_became_peered":"2024-09-16T10:28:28.072930+0000","last_unstale":"2024-09-16T10:29:40.030148+0000","last_undegraded":"2024-09-16T10:29:40.030148+0000","last_fullsized":"2024-09-16T10:29:40.030148+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:38.060395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:38.060395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:05.957375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"442'12","reported_seq":1140,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060305+0000","last_change":"2024-09-16T10:28:28.095653+0000","last_active":"2024-09-16T10:29:51.060305+0000","last_peered":"2024-09-16T10:29:51.060305+0000","last_clean":"2024-09-16T10:29:51.060305+0000","last_became_active":"2024-09-16T10:28:28.094870+0000","last_became_peered":"2024-09-16T10:28:28.094870+0000","last_unstale":"2024-09-16T10:29:51.060305+0000","last_undegraded":"2024-09-16T10:29:51.060305+0000","last_fullsized":"2024-09-16T10:29:51.060305+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.703651+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":691,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184799+0000","last_change":"2024-09-16T10:26:45.194577+0000","last_active":"2024-09-16T10:29:39.184799+0000","last_peered":"2024-09-16T10:29:39.184799+0000","last_clean":"2024-09-16T10:29:39.184799+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:29:39.184799+0000","last_undegraded":"2024-09-16T10:29:39.184799+0000","last_fullsized":"2024-09-16T10:29:39.184799+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.194454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.194454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":469,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030086+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:29:40.030086+0000","last_peered":"2024-09-16T10:29:40.030086+0000","last_clean":"2024-09-16T10:29:40.030086+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:29:40.030086+0000","last_undegraded":"2024-09-16T10:29:40.030086+0000","last_fullsized":"2024-09-16T10:29:40.030086+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":947,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059863+0000","last_change":"2024-09-16T10:28:28.096614+0000","last_active":"2024-09-16T10:29:51.059863+0000","last_peered":"2024-09-16T10:29:51.059863+0000","last_clean":"2024-09-16T10:29:51.059863+0000","last_became_active":"2024-09-16T10:28:28.095319+0000","last_became_peered":"2024-09-16T10:28:28.095319+0000","last_unstale":"2024-09-16T10:29:51.059863+0000","last_undegraded":"2024-09-16T10:29:51.059863+0000","last_fullsized":"2024-09-16T10:29:51.059863+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:26:20.405507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.405507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":801,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.551074+0000","last_change":"2024-09-16T10:29:21.812647+0000","last_active":"2024-09-16T10:29:50.551074+0000","last_peered":"2024-09-16T10:29:50.551074+0000","last_clean":"2024-09-16T10:29:50.551074+0000","last_became_active":"2024-09-16T10:28:28.071944+0000","last_became_peered":"2024-09-16T10:28:28.071944+0000","last_unstale":"2024-09-16T10:29:50.551074+0000","last_undegraded":"2024-09-16T10:29:50.551074+0000","last_fullsized":"2024-09-16T10:29:50.551074+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.812585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.812585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:23.180131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1100,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062187+0000","last_change":"2024-09-16T10:26:11.007271+0000","last_active":"2024-09-16T10:29:51.062187+0000","last_peered":"2024-09-16T10:29:51.062187+0000","last_clean":"2024-09-16T10:29:51.062187+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:29:51.062187+0000","last_undegraded":"2024-09-16T10:29:51.062187+0000","last_fullsized":"2024-09-16T10:29:51.062187+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:11.007191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:11.007191+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":890,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030365+0000","last_change":"2024-09-16T10:29:20.053166+0000","last_active":"2024-09-16T10:29:40.030365+0000","last_peered":"2024-09-16T10:29:40.030365+0000","last_clean":"2024-09-16T10:29:40.030365+0000","last_became_active":"2024-09-16T10:28:28.081510+0000","last_became_peered":"2024-09-16T10:28:28.081510+0000","last_unstale":"2024-09-16T10:29:40.030365+0000","last_undegraded":"2024-09-16T10:29:40.030365+0000","last_fullsized":"2024-09-16T10:29:40.030365+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.053102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.053102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:31.409241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":829,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292861+0000","last_change":"2024-09-16T10:28:02.318109+0000","last_active":"2024-09-16T10:29:39.292861+0000","last_peered":"2024-09-16T10:29:39.292861+0000","last_clean":"2024-09-16T10:29:39.292861+0000","last_became_active":"2024-09-16T10:26:28.557003+0000","last_became_peered":"2024-09-16T10:26:28.557003+0000","last_unstale":"2024-09-16T10:29:39.292861+0000","last_undegraded":"2024-09-16T10:29:39.292861+0000","last_fullsized":"2024-09-16T10:29:39.292861+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:02.318062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:02.318062+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":357,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.056473+0000","last_change":"2024-09-16T10:28:28.105143+0000","last_active":"2024-09-16T10:29:51.056473+0000","last_peered":"2024-09-16T10:29:51.056473+0000","last_clean":"2024-09-16T10:29:51.056473+0000","last_became_active":"2024-09-16T10:28:28.104604+0000","last_became_peered":"2024-09-16T10:28:28.104604+0000","last_unstale":"2024-09-16T10:29:51.056473+0000","last_undegraded":"2024-09-16T10:29:51.056473+0000","last_fullsized":"2024-09-16T10:29:51.056473+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.718457+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":888,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062615+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:29:51.062615+0000","last_peered":"2024-09-16T10:29:51.062615+0000","last_clean":"2024-09-16T10:29:51.062615+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:29:51.062615+0000","last_undegraded":"2024-09-16T10:29:51.062615+0000","last_fullsized":"2024-09-16T10:29:51.062615+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":882,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060548+0000","last_change":"2024-09-16T10:26:14.413104+0000","last_active":"2024-09-16T10:29:51.060548+0000","last_peered":"2024-09-16T10:29:51.060548+0000","last_clean":"2024-09-16T10:29:51.060548+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:29:51.060548+0000","last_undegraded":"2024-09-16T10:29:51.060548+0000","last_fullsized":"2024-09-16T10:29:51.060548+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:14.413019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:14.413019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":951,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062940+0000","last_change":"2024-09-16T10:29:19.698059+0000","last_active":"2024-09-16T10:29:51.062940+0000","last_peered":"2024-09-16T10:29:51.062940+0000","last_clean":"2024-09-16T10:29:51.062940+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:29:51.062940+0000","last_undegraded":"2024-09-16T10:29:51.062940+0000","last_fullsized":"2024-09-16T10:29:51.062940+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.530423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":293,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.061280+0000","last_change":"2024-09-16T10:28:28.098266+0000","last_active":"2024-09-16T10:29:51.061280+0000","last_peered":"2024-09-16T10:29:51.061280+0000","last_clean":"2024-09-16T10:29:51.061280+0000","last_became_active":"2024-09-16T10:28:28.094896+0000","last_became_peered":"2024-09-16T10:28:28.094896+0000","last_unstale":"2024-09-16T10:29:51.061280+0000","last_undegraded":"2024-09-16T10:29:51.061280+0000","last_fullsized":"2024-09-16T10:29:51.061280+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.368081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.368081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":851,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059579+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:29:51.059579+0000","last_peered":"2024-09-16T10:29:51.059579+0000","last_clean":"2024-09-16T10:29:51.059579+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:29:51.059579+0000","last_undegraded":"2024-09-16T10:29:51.059579+0000","last_fullsized":"2024-09-16T10:29:51.059579+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":863,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.029567+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:29:40.029567+0000","last_peered":"2024-09-16T10:29:40.029567+0000","last_clean":"2024-09-16T10:29:40.029567+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:29:40.029567+0000","last_undegraded":"2024-09-16T10:29:40.029567+0000","last_fullsized":"2024-09-16T10:29:40.029567+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":809,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184491+0000","last_change":"2024-09-16T10:29:35.573887+0000","last_active":"2024-09-16T10:29:39.184491+0000","last_peered":"2024-09-16T10:29:39.184491+0000","last_clean":"2024-09-16T10:29:39.184491+0000","last_became_active":"2024-09-16T10:28:28.071473+0000","last_became_peered":"2024-09-16T10:28:28.071473+0000","last_unstale":"2024-09-16T10:29:39.184491+0000","last_undegraded":"2024-09-16T10:29:39.184491+0000","last_fullsized":"2024-09-16T10:29:39.184491+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:35.573835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.573835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.601451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":916,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060661+0000","last_change":"2024-09-16T10:29:19.861104+0000","last_active":"2024-09-16T10:29:51.060661+0000","last_peered":"2024-09-16T10:29:51.060661+0000","last_clean":"2024-09-16T10:29:51.060661+0000","last_became_active":"2024-09-16T10:28:28.095283+0000","last_became_peered":"2024-09-16T10:28:28.095283+0000","last_unstale":"2024-09-16T10:29:51.060661+0000","last_undegraded":"2024-09-16T10:29:51.060661+0000","last_fullsized":"2024-09-16T10:29:51.060661+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.496265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":915,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.062433+0000","last_change":"2024-09-16T10:29:23.002575+0000","last_active":"2024-09-16T10:29:51.062433+0000","last_peered":"2024-09-16T10:29:51.062433+0000","last_clean":"2024-09-16T10:29:51.062433+0000","last_became_active":"2024-09-16T10:28:28.109359+0000","last_became_peered":"2024-09-16T10:28:28.109359+0000","last_unstale":"2024-09-16T10:29:51.062433+0000","last_undegraded":"2024-09-16T10:29:51.062433+0000","last_fullsized":"2024-09-16T10:29:51.062433+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.002518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.002518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.887462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":725,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.550498+0000","last_change":"2024-09-16T10:29:19.866083+0000","last_active":"2024-09-16T10:29:50.550498+0000","last_peered":"2024-09-16T10:29:50.550498+0000","last_clean":"2024-09-16T10:29:50.550498+0000","last_became_active":"2024-09-16T10:28:28.091060+0000","last_became_peered":"2024-09-16T10:28:28.091060+0000","last_unstale":"2024-09-16T10:29:50.550498+0000","last_undegraded":"2024-09-16T10:29:50.550498+0000","last_fullsized":"2024-09-16T10:29:50.550498+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.866022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:36.779287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":898,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060617+0000","last_change":"2024-09-16T10:29:34.789497+0000","last_active":"2024-09-16T10:29:51.060617+0000","last_peered":"2024-09-16T10:29:51.060617+0000","last_clean":"2024-09-16T10:29:51.060617+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:29:51.060617+0000","last_undegraded":"2024-09-16T10:29:51.060617+0000","last_fullsized":"2024-09-16T10:29:51.060617+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.789441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.789441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.777028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":936,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059617+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:29:51.059617+0000","last_peered":"2024-09-16T10:29:51.059617+0000","last_clean":"2024-09-16T10:29:51.059617+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:29:51.059617+0000","last_undegraded":"2024-09-16T10:29:51.059617+0000","last_fullsized":"2024-09-16T10:29:51.059617+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":889,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030487+0000","last_change":"2024-09-16T10:27:27.598364+0000","last_active":"2024-09-16T10:29:40.030487+0000","last_peered":"2024-09-16T10:29:40.030487+0000","last_clean":"2024-09-16T10:29:40.030487+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:29:40.030487+0000","last_undegraded":"2024-09-16T10:29:40.030487+0000","last_fullsized":"2024-09-16T10:29:40.030487+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:27.598299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:27.598299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292438+0000","last_change":"2024-09-16T10:29:31.282374+0000","last_active":"2024-09-16T10:29:39.292438+0000","last_peered":"2024-09-16T10:29:39.292438+0000","last_clean":"2024-09-16T10:29:39.292438+0000","last_became_active":"2024-09-16T10:28:28.072759+0000","last_became_peered":"2024-09-16T10:28:28.072759+0000","last_unstale":"2024-09-16T10:29:39.292438+0000","last_undegraded":"2024-09-16T10:29:39.292438+0000","last_fullsized":"2024-09-16T10:29:39.292438+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:31.282323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:31.282323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.735338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":521,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.184284+0000","last_change":"2024-09-16T10:29:32.563307+0000","last_active":"2024-09-16T10:29:39.184284+0000","last_peered":"2024-09-16T10:29:39.184284+0000","last_clean":"2024-09-16T10:29:39.184284+0000","last_became_active":"2024-09-16T10:28:28.066479+0000","last_became_peered":"2024-09-16T10:28:28.066479+0000","last_unstale":"2024-09-16T10:29:39.184284+0000","last_undegraded":"2024-09-16T10:29:39.184284+0000","last_fullsized":"2024-09-16T10:29:39.184284+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:32.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:32.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.751962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":884,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.060892+0000","last_change":"2024-09-16T10:26:23.240494+0000","last_active":"2024-09-16T10:29:51.060892+0000","last_peered":"2024-09-16T10:29:51.060892+0000","last_clean":"2024-09-16T10:29:51.060892+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:29:51.060892+0000","last_undegraded":"2024-09-16T10:29:51.060892+0000","last_fullsized":"2024-09-16T10:29:51.060892+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:23.240435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:23.240435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":902,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.059778+0000","last_change":"2024-09-16T10:26:13.362416+0000","last_active":"2024-09-16T10:29:51.059778+0000","last_peered":"2024-09-16T10:29:51.059778+0000","last_clean":"2024-09-16T10:29:51.059778+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:29:51.059778+0000","last_undegraded":"2024-09-16T10:29:51.059778+0000","last_fullsized":"2024-09-16T10:29:51.059778+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.362347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.362347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":935,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:39.292087+0000","last_change":"2024-09-16T10:28:06.383385+0000","last_active":"2024-09-16T10:29:39.292087+0000","last_peered":"2024-09-16T10:29:39.292087+0000","last_clean":"2024-09-16T10:29:39.292087+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:29:39.292087+0000","last_undegraded":"2024-09-16T10:29:39.292087+0000","last_fullsized":"2024-09-16T10:29:39.292087+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:06.383318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:06.383318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.030539+0000","last_change":"2024-09-16T10:29:19.094876+0000","last_active":"2024-09-16T10:29:40.030539+0000","last_peered":"2024-09-16T10:29:40.030539+0000","last_clean":"2024-09-16T10:29:40.030539+0000","last_became_active":"2024-09-16T10:28:28.081905+0000","last_became_peered":"2024-09-16T10:28:28.081905+0000","last_unstale":"2024-09-16T10:29:40.030539+0000","last_undegraded":"2024-09-16T10:29:40.030539+0000","last_fullsized":"2024-09-16T10:29:40.030539+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.094826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.094826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.628616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1797128192,"num_objects":27422,"num_object_clones":0,"num_object_copies":82266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27422,"num_write_kb":1755008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5283840000,"data_stored":5283840000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27422,"ondisk_log_size":27422,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":441,"seq":1894080577554,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761136,"kb_used_data":593824,"kb_used_omap":7,"kb_used_meta":167288,"kb_avail":92984016,"statfs":{"total":95995035648,"available":95215632384,"internally_reserved":0,"allocated":608075776,"data_stored":606971797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8012,"internal_metadata":171303092},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528522,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304808,"kb_used_data":669720,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92440344,"statfs":{"total":95995035648,"available":94658912256,"internally_reserved":0,"allocated":685793280,"data_stored":684696835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":650312134},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790161,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117140,"kb_used_data":508416,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92628012,"statfs":{"total":95995035648,"available":94851084288,"internally_reserved":0,"allocated":520617984,"data_stored":519496751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986390,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110432,"kb_used_data":493392,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92634720,"statfs":{"total":95995035648,"available":94857953280,"internally_reserved":0,"allocated":505233408,"data_stored":504094826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215323,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573856,"kb_used_data":842960,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92171296,"statfs":{"total":95995035648,"available":94383407104,"internally_reserved":0,"allocated":863191040,"data_stored":862053841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":748419528},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411552,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":967344,"kb_used_data":512736,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92777808,"statfs":{"total":95995035648,"available":95004475392,"internally_reserved":0,"allocated":525041664,"data_stored":523937842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1674932,"kb_used_data":971804,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92070220,"statfs":{"total":95995035648,"available":94279905280,"internally_reserved":0,"allocated":995127296,"data_stored":993976946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738601,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201464,"kb_used_data":494184,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92543688,"statfs":{"total":95995035648,"available":94764736512,"internally_reserved":0,"allocated":506044416,"data_stored":504932947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":506003456,"data_stored":506003456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1005256704,"data_stored":1005256704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":560726016,"data_stored":560726016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":871432192,"data_stored":871432192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":503382016,"data_stored":503382016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":495452160,"data_stored":495452160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":697499648,"data_stored":697499648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":644087808,"data_stored":644087808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:00.698 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T10:30:00.698 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:00.699 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:30:00.699 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T10:30:00.699 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-16T10:30:00.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-16T10:30:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:00.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:00.718 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:00.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:00.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:00.925 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:00.926 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[39470]: pgmap v892: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-16T10:30:00.926 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[39470]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-16T10:30:00.926 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[39470]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-16T10:30:00.926 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:00.926 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[29993]: pgmap v892: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-16T10:30:00.926 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[29993]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-16T10:30:00.927 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:00 smithi063 ceph-mon[29993]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-16T10:30:00.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:00.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:00.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:01.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:00 smithi165 ceph-mon[32696]: pgmap v892: 168 pgs: 168 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-16T10:30:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:00 smithi165 ceph-mon[32696]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-16T10:30:01.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:00 smithi165 ceph-mon[32696]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-16T10:30:01.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:01.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:01.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:01.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:01.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:01.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:01.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 21 16 43401 43385 129.099 102 0.00544766 0.00774102 2024-09-16T10:30:01.317 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:01.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:01.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:01.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:01.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:01.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:01.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:01.636 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:01.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:01.661 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:01.703 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:01.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:01.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:01.739 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:01.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:01.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:01.775 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:01.775 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:01.809 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:01.809 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:01.842 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:01.842 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:01.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:01.880 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:01.880 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:01.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:01.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:01.918 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:01.918 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:02.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:02.039 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:02.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:02.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:02.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:02.180 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:02.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:02.282 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:02.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 22 16 45043 45027 127.895 102.625 0.00987498 0.0078135 2024-09-16T10:30:02.320 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:02.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:02.333 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:02 smithi063 ceph-mon[29993]: pgmap v893: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 871 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-16T10:30:02.333 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3068392793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:02.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:02 smithi063 ceph-mon[39470]: pgmap v893: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 871 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-16T10:30:02.334 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3068392793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:02.422 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:02.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:02.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:02.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:02 smithi165 ceph-mon[32696]: pgmap v893: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 871 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-16T10:30:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3068392793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:02.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:02.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:02.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:02.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:02.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:02.755 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:02.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:02.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:02.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:02.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:02.989 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:03.027 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:03.078 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:30:03.129 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:03.152 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:03.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:03.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:03.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 23 16 46409 46393 126.047 85.375 0.00271646 0.00792782 2024-09-16T10:30:03.383 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:03.422 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:03.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:03 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4186623260' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:03 smithi063 ceph-mon[29993]: from='client.26527 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:03 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4186623260' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:03 smithi063 ceph-mon[39470]: from='client.26527 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:03.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.548 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:03.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:03 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4186623260' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:03 smithi165 ceph-mon[32696]: from='client.26527 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:03.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:03.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:03.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:03.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:03.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:03.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:03.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:04.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:04.077 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:04.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:04.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:04.218 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:04.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:04.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 24 16 47768 47752 124.333 84.9375 0.00715869 0.00803685 2024-09-16T10:30:04.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:04.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:04.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:04.371 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:30:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:30:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:30:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4108 9.2 9:4f3b5670:::benchmark_data_smithi063_124717_object4107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.903454+0000", 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.23724025500002, 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.000099557, 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4108 2024-09-16T10:30:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903454+0000", 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903454+0000", 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903455+0000", 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.356e-06 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903478+0000", 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2731999999999999e-05 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903488+0000", 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0638999999999999e-05 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903496+0000", 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2629999999999997e-06 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903549+0000", 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3103999999999998e-05 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903558+0000", 2024-09-16T10:30:04.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5349999999999993e-06 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.901312+0000", 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.997753899 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.901469+0000", 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015720600000000001 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.901640+0000", 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017031699999999999 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903362+0000", 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017224199999999999 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903381+0000", 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8668000000000002e-05 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903444+0000", 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3017999999999999e-05 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903505+0000", 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1208000000000001e-05 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903553+0000", 2024-09-16T10:30:04.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8192000000000002e-05 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4116 9.2 9:4536b227:::benchmark_data_smithi063_124717_object4115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.905723+0000", 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.234970847, 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99950095800000005, 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.379 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4116 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905723+0000", 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905723+0000", 2024-09-16T10:30:04.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905724+0000", 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9699999999999994e-07 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905749+0000", 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5154999999999999e-05 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905759+0000", 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6139999999999998e-06 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905766+0000", 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4329999999999998e-06 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905839+0000", 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3011999999999998e-05 2024-09-16T10:30:04.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905849+0000", 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5349999999999993e-06 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.902620+0000", 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99677087200000003 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.902752+0000", 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013278899999999999 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.902922+0000", 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000169996 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905057+0000", 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0021347720000000001 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905120+0000", 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2590999999999998e-05 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905134+0000", 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3942e-05 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905182+0000", 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8699000000000002e-05 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905224+0000", 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1551000000000002e-05 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4127 9.2 9:4fce5579:::benchmark_data_smithi063_124717_object4126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.386 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.909448+0000", 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.23124584800001, 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99610295500000001, 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4127 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909448+0000", 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909448+0000", 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909449+0000", 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.156e-06 2024-09-16T10:30:04.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909477+0000", 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7736999999999999e-05 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909486+0000", 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0450000000000006e-06 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909492+0000", 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7740000000000001e-06 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909614+0000", 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000122077 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909623+0000", 2024-09-16T10:30:04.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1390000000000001e-06 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903602+0000", 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99397868300000003 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903723+0000", 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121278 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903920+0000", 2024-09-16T10:30:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000197586 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905237+0000", 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001316291 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905278+0000", 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1561999999999999e-05 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905355+0000", 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6755999999999995e-05 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905410+0000", 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5198000000000002e-05 2024-09-16T10:30:04.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905551+0000", 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014067300000000001 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4173 9.2 9:48ddd8d1:::benchmark_data_smithi063_124717_object4172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.924332+0000", 2024-09-16T10:30:04.394 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.21636214500001, 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98297970700000004, 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4173 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924332+0000", 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924332+0000", 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924333+0000", 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.624e-06 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924356+0000", 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3155000000000001e-05 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924367+0000", 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0560000000000001e-05 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924374+0000", 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0929999999999997e-06 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924435+0000", 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1192000000000006e-05 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924446+0000", 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1032999999999999e-05 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.904397+0000", 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97995030599999999 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.904498+0000", 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000101488 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.904635+0000", 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013644899999999999 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905295+0000", 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00066009 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907059+0000", 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017646669999999999 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907221+0000", 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016126899999999999 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907270+0000", 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9811999999999997e-05 2024-09-16T10:30:04.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907311+0000", 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0969e-05 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4200 9.2 9:433bfede:::benchmark_data_smithi063_124717_object4199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.402 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.931343+0000", 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.20935131300001, 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97620721099999996, 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4200 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931343+0000", 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931343+0000", 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931344+0000", 2024-09-16T10:30:04.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7090000000000001e-06 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931368+0000", 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3689999999999998e-05 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931378+0000", 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9289999999999993e-06 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931386+0000", 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8359999999999992e-06 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931493+0000", 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000107453 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931502+0000", 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8349999999999993e-06 2024-09-16T10:30:04.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905594+0000", 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97409171800000005 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905741+0000", 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000147429 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905851+0000", 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011026 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907329+0000", 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014775350000000001 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907379+0000", 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0488000000000002e-05 2024-09-16T10:30:04.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907461+0000", 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1686000000000004e-05 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907510+0000", 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9320000000000002e-05 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907550+0000", 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9323000000000003e-05 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4225 9.2 9:4ec3d5f5:::benchmark_data_smithi063_124717_object4224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.939134+0000", 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.20155943499998, 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.970788706, 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.410 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4225 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939134+0000", 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939134+0000", 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939136+0000", 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3799999999999999e-06 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939160+0000", 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4354000000000001e-05 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939169+0000", 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2629999999999992e-06 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939177+0000", 2024-09-16T10:30:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2960000000000001e-06 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939251+0000", 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.394e-05 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939262+0000", 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1351e-05 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.906274+0000", 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96701167600000004 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.906399+0000", 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012484 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.906535+0000", 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000136148 2024-09-16T10:30:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907562+0000", 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001027575 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909783+0000", 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022207220000000001 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909836+0000", 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2769999999999998e-05 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909885+0000", 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9212000000000002e-05 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909923+0000", 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8179000000000002e-05 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.418 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4254 9.2 9:42392fd0:::benchmark_data_smithi063_124717_object4253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.948542+0000", 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.19215139300002, 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96154861800000002, 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.419 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4254 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948542+0000", 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948542+0000", 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948544+0000", 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.398e-06 2024-09-16T10:30:04.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948568+0000", 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3935e-05 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948577+0000", 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7960000000000001e-06 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948584+0000", 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2189999999999998e-06 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948710+0000", 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000126444 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948720+0000", 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9340000000000001e-06 2024-09-16T10:30:04.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907698+0000", 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95897798199999995 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907844+0000", 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014559100000000001 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907997+0000", 2024-09-16T10:30:04.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015361699999999999 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909932+0000", 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019350719999999999 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909967+0000", 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4382999999999999e-05 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910006+0000", 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8677000000000003e-05 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910050+0000", 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4124e-05 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910091+0000", 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1446000000000003e-05 2024-09-16T10:30:04.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4301 9.2 9:4a35399d:::benchmark_data_smithi063_124717_object4300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.964920+0000", 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.175773773, 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.949930108, 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.427 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4301 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964920+0000", 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964920+0000", 2024-09-16T10:30:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964922+0000", 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6500000000000001e-06 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964945+0000", 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3027000000000002e-05 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964959+0000", 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4579999999999999e-05 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964968+0000", 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3930000000000008e-06 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.965093+0000", 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125265 2024-09-16T10:30:04.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.965106+0000", 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3213e-05 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.908907+0000", 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94380035100000004 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909214+0000", 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030766700000000002 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909372+0000", 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015809999999999999 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911971+0000", 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025982969999999998 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914587+0000", 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026161579999999999 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914742+0000", 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155119 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914796+0000", 2024-09-16T10:30:04.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4169000000000003e-05 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914850+0000", 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4119000000000001e-05 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4315 9.2 9:47b319fd:::benchmark_data_smithi063_124717_object4314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.968925+0000", 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.171768536, 2024-09-16T10:30:04.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94607211800000002, 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4315 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.968925+0000", 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.968925+0000", 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.968927+0000", 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3599999999999999e-06 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969031+0000", 2024-09-16T10:30:04.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103829 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969047+0000", 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6626e-05 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969055+0000", 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2700000000000004e-06 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969477+0000", 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00042144699999999998 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969489+0000", 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2435e-05 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910125+0000", 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94063537699999999 2024-09-16T10:30:04.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910236+0000", 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111457 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910371+0000", 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013445199999999999 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914644+0000", 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042737440000000003 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914665+0000", 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.086e-05 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914905+0000", 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000239478 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914959+0000", 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3859000000000001e-05 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914997+0000", 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8924000000000002e-05 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4318 9.2 9:4dd0366d:::benchmark_data_smithi063_124717_object4317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.970502+0000", 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.17019208900001, 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94779669700000002, 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.442 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4318 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970502+0000", 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970502+0000", 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970503+0000", 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.514e-06 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970526+0000", 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2463000000000002e-05 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970536+0000", 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0229e-05 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970543+0000", 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4449999999999999e-06 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970611+0000", 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7148000000000002e-05 2024-09-16T10:30:04.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970621+0000", 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0356999999999999e-05 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910752+0000", 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94013116200000002 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910872+0000", 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120359 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911003+0000", 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013043899999999999 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914672+0000", 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0036691419999999998 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917195+0000", 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025227549999999998 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918197+0000", 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001002544 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918251+0000", 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3609999999999997e-05 2024-09-16T10:30:04.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918298+0000", 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7530000000000001e-05 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4321 9.2 9:4f79c027:::benchmark_data_smithi063_124717_object4320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.449 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.971410+0000", 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.16928408000001, 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.947013997, 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4321 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.450 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971410+0000", 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971410+0000", 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971411+0000", 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.528e-06 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971434+0000", 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2580999999999999e-05 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971448+0000", 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4082e-05 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971457+0000", 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0969999999999995e-06 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971523+0000", 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6204000000000005e-05 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971533+0000", 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.91e-06 2024-09-16T10:30:04.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911382+0000", 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93984913299999995 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911500+0000", 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117928 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911624+0000", 2024-09-16T10:30:04.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000124067 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914678+0000", 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030535549999999999 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917292+0000", 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002613804 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918346+0000", 2024-09-16T10:30:04.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00105437 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918388+0000", 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2024000000000003e-05 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918424+0000", 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5713999999999997e-05 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4331 9.2 9:4a8271f9:::benchmark_data_smithi063_124717_object4330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.975610+0000", 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.165083713, 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94326413600000003, 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4331 2024-09-16T10:30:04.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975610+0000", 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975610+0000", 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975612+0000", 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.807e-06 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975634+0000", 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.232e-05 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975645+0000", 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0861e-05 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975650+0000", 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0520000000000001e-06 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975706+0000", 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5744e-05 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975712+0000", 2024-09-16T10:30:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1670000000000004e-06 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912011+0000", 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93629880399999998 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912132+0000", 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120968 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912264+0000", 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013233599999999999 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914684+0000", 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00241939 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917385+0000", 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027012970000000001 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.462 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918768+0000", 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013833090000000001 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918833+0000", 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5068999999999997e-05 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918874+0000", 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1012e-05 2024-09-16T10:30:04.463 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4346 9.2 9:45bbf4f4:::benchmark_data_smithi063_124717_object4345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.981605+0000", 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.15908902699999, 2024-09-16T10:30:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.937441992, 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4346 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981605+0000", 2024-09-16T10:30:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981605+0000", 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981606+0000", 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6619999999999999e-06 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981630+0000", 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3337e-05 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981639+0000", 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.516e-06 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981647+0000", 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2699999999999999e-06 2024-09-16T10:30:04.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981743+0000", 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6416999999999995e-05 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981753+0000", 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0362e-05 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912607+0000", 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93085387799999997 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912708+0000", 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010064 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912853+0000", 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014459799999999999 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914690+0000", 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001837588 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917480+0000", 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027903310000000001 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918961+0000", 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014803699999999999 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919008+0000", 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7469999999999998e-05 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919047+0000", 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8553000000000002e-05 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.471 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4360 9.2 9:4df79284:::benchmark_data_smithi063_124717_object4359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.989021+0000", 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.15167265000002, 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93028860800000002, 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4360 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989021+0000", 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989021+0000", 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989023+0000", 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.875e-06 2024-09-16T10:30:04.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989050+0000", 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6750999999999998e-05 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989062+0000", 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2133999999999999e-05 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.474 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989071+0000", 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1710000000000008e-06 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989139+0000", 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7929000000000006e-05 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989146+0000", 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8159999999999996e-06 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913223+0000", 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92407729800000005 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913347+0000", 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000124194 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913480+0000", 2024-09-16T10:30:04.476 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013293100000000001 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914696+0000", 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012155709999999999 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917578+0000", 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028823519999999999 2024-09-16T10:30:04.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919215+0000", 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016364369999999999 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919268+0000", 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3423e-05 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.478 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919310+0000", 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1726e-05 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4366 9.2 9:4b5f36f4:::benchmark_data_smithi063_124717_object4365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.992754+0000", 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 478.147939726, 2024-09-16T10:30:04.479 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92669086099999998, 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4366 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992754+0000", 2024-09-16T10:30:04.480 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992754+0000", 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992755+0000", 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1999999999999999e-06 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.481 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992775+0000", 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9598000000000001e-05 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992786+0000", 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1532000000000001e-05 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992794+0000", 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8469999999999996e-06 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.482 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992850+0000", 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5696999999999999e-05 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992857+0000", 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3849999999999996e-06 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913839+0000", 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92098122199999999 2024-09-16T10:30:04.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913950+0000", 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111778 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914103+0000", 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000152492 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.915901+0000", 2024-09-16T10:30:04.484 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001797739 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917675+0000", 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00177462 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919363+0000", 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016879569999999999 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919405+0000", 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1628999999999997e-05 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919445+0000", 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0164999999999999e-05 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:13236 11.a 11:5a714180:::benchmark_data_smithi063_176907_object13235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:44.943785+0000", 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 18.196908780000001, 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.111629466, 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 13236 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.487 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.943785+0000", 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.943785+0000", 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.943786+0000", 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.285e-06 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.943819+0000", 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2762999999999998e-05 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.943828+0000", 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9420000000000001e-06 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.943835+0000", 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6170000000000003e-06 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.944086+0000", 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025127199999999998 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.944198+0000", 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111648 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.944318+0000", 2024-09-16T10:30:04.490 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120803 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.945442+0000", 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001124051 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.945813+0000", 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00037066699999999998 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.491 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055306+0000", 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10949296 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055359+0000", 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2456000000000001e-05 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055415+0000", 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6001999999999997e-05 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:13242 11.a 11:548a2fc7:::benchmark_data_smithi063_176907_object13241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:44.946565+0000", 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 18.194128559999999, 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.109067831, 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:30:04.493 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 13242 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.946565+0000", 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.946565+0000", 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.946567+0000", 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.125e-06 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.946592+0000", 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5086999999999999e-05 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.946602+0000", 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6490000000000003e-06 2024-09-16T10:30:04.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.946610+0000", 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5580000000000003e-06 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.946757+0000", 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014699099999999999 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.496 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947012+0000", 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025535199999999998 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947134+0000", 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012209300000000001 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055486+0000", 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10835170199999999 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.497 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055538+0000", 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2065999999999999e-05 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055550+0000", 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1647999999999999e-05 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055594+0000", 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.426e-05 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055633+0000", 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.93e-05 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:13245 11.a 11:5ab6aed7:::benchmark_data_smithi063_176907_object13244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:44.947415+0000", 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 18.193279059999998, 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.112651271, 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.499 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 13245 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947415+0000", 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947415+0000", 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947416+0000", 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1739999999999999e-06 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947441+0000", 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4649000000000001e-05 2024-09-16T10:30:04.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947454+0000", 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2891000000000001e-05 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.947463+0000", 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1090000000000004e-06 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.054196+0000", 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10673295300000001 2024-09-16T10:30:04.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.054320+0000", 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012444999999999999 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.054452+0000", 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013149599999999999 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.503 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055641+0000", 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0011891899999999999 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.057189+0000", 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015487509999999999 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.059969+0000", 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027793739999999998 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.060025+0000", 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6073999999999999e-05 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.060066+0000", 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1159999999999999e-05 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:13247 11.8 11:1da11562:::benchmark_data_smithi063_176907_object13246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:44.948141+0000", 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 18.192552866, 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.108637364, 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 13247 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948141+0000", 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948141+0000", 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948143+0000", 2024-09-16T10:30:04.507 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.733e-06 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948289+0000", 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000146479 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948297+0000", 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.259e-06 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.508 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948305+0000", 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.272e-06 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.053087+0000", 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.10478256900000001 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.053256+0000", 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000168989 2024-09-16T10:30:04.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.053409+0000", 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015310000000000001 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.054576+0000", 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001166729 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.510 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.056460+0000", 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018841540000000001 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.056682+0000", 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000222174 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.056738+0000", 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5364000000000001e-05 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.511 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.056778+0000", 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0541999999999999e-05 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:13251 11.a 11:5c447c7f:::benchmark_data_smithi063_176907_object13250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:29:44.948327+0000", 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 18.192366756999999, 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.11186367699999999, 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:04.512 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 13251 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948327+0000", 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.513 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948327+0000", 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948328+0000", 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0189999999999999e-06 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:44.948352+0000", 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3762999999999999e-05 2024-09-16T10:30:04.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.052790+0000", 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.104437629 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.052841+0000", 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1363000000000003e-05 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:04.515 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055669+0000", 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028286180000000002 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055798+0000", 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012847600000000001 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 5,6", 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.055939+0000", 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014054200000000001 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.058802+0000", 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028633249999999999 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.058828+0000", 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6188999999999999e-05 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:04.517 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.060102+0000", 2024-09-16T10:30:04.518 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012738490000000001 2024-09-16T10:30:04.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.518 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:04.518 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:04.518 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[29993]: from='client.26527 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:30:04.518 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[29993]: osdmap e487: 8 total, 8 up, 8 in 2024-09-16T10:30:04.518 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[29993]: pgmap v895: 168 pgs: 168 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[29993]: 3.a scrub starts 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[29993]: 3.a scrub ok 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[39470]: from='client.26527 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[39470]: osdmap e487: 8 total, 8 up, 8 in 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[39470]: pgmap v895: 168 pgs: 168 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[39470]: 3.a scrub starts 2024-09-16T10:30:04.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:04 smithi063 ceph-mon[39470]: 3.a scrub ok 2024-09-16T10:30:04.519 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.060155+0000", 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2951000000000002e-05 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:29:45.060191+0000", 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5952999999999999e-05 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:04.520 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:30:04.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:04 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:04 smithi165 ceph-mon[32696]: from='client.26527 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:30:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:04 smithi165 ceph-mon[32696]: osdmap e487: 8 total, 8 up, 8 in 2024-09-16T10:30:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:04 smithi165 ceph-mon[32696]: pgmap v895: 168 pgs: 168 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-16T10:30:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:04 smithi165 ceph-mon[32696]: 3.a scrub starts 2024-09-16T10:30:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:04 smithi165 ceph-mon[32696]: 3.a scrub ok 2024-09-16T10:30:04.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:04.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:04.655 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:04.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:04.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:04.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:04.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:04.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:04.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:04.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:04.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:05.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:05.072 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:05.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:05.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:05.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:05.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:05.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:05.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 25 16 49483 49467 123.647 107.188 0.00460697 0.00808228 2024-09-16T10:30:05.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:05.378 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:05.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:05.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:05 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:30:05.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:05 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:30:05.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:05 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:30:05.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:05 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:30:05.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:05.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:05.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:05.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:05 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:30:05.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:05 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:30:05.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:05.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:05.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:05.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:05.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:05.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:05.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:05.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:05.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:06.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:06.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:06.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:06.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:06.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:06.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:06.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:06.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:06.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:06.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 26 16 50971 50955 122.468 93 0.00884087 0.00815969 2024-09-16T10:30:06.413 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:06.451 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: 6.9 scrub starts 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: 6.9 scrub ok 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: pgmap v896: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-16T10:30:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[29993]: osdmap e488: 8 total, 8 up, 8 in 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: 6.9 scrub starts 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: 6.9 scrub ok 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: pgmap v896: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-16T10:30:06.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:06 smithi063 ceph-mon[39470]: osdmap e488: 8 total, 8 up, 8 in 2024-09-16T10:30:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:06.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: 6.9 scrub starts 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: 6.9 scrub ok 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: pgmap v896: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-16T10:30:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:06 smithi165 ceph-mon[32696]: osdmap e488: 8 total, 8 up, 8 in 2024-09-16T10:30:06.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:06.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:06.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:06.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:06.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:06.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:06.821 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:06.858 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:06.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:06.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:30:06.961 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.008 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:07.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:07.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:07.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:07.185 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[29993]: 1.8 scrub starts 2024-09-16T10:30:07.185 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[29993]: 1.8 scrub ok 2024-09-16T10:30:07.185 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[29993]: 4.7 scrub starts 2024-09-16T10:30:07.186 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[29993]: 4.7 scrub ok 2024-09-16T10:30:07.186 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:30:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:30:07.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[39470]: 1.8 scrub starts 2024-09-16T10:30:07.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[39470]: 1.8 scrub ok 2024-09-16T10:30:07.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[39470]: 4.7 scrub starts 2024-09-16T10:30:07.186 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:07 smithi063 ceph-mon[39470]: 4.7 scrub ok 2024-09-16T10:30:07.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:07.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:07 smithi165 ceph-mon[32696]: 1.8 scrub starts 2024-09-16T10:30:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:07 smithi165 ceph-mon[32696]: 1.8 scrub ok 2024-09-16T10:30:07.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:07 smithi165 ceph-mon[32696]: 4.7 scrub starts 2024-09-16T10:30:07.303 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:07 smithi165 ceph-mon[32696]: 4.7 scrub ok 2024-09-16T10:30:07.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 27 16 52558 52542 121.606 99.1875 0.0136287 0.00821761 2024-09-16T10:30:07.316 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:07.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:07.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:07.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:07.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:07.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:07.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:07.841 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:07.943 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:07.990 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:07.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:08.093 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:08.108 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:53788 11.d e488/476 11:b7150307:::benchmark_data_smithi063_176907_object53787:head v 488'3252, mlcod=488'3250)", 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:08.105067+0000", 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00073507100000000001, 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00075948300000000003, 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:30:08.109 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:08.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:08.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105067+0000", 2024-09-16T10:30:08.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105067+0000", 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105068+0000", 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1510000000000001e-06 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105184+0000", 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116218 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105197+0000", 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2721000000000001e-05 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:08.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105204+0000", 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2960000000000001e-06 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105267+0000", 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2829000000000004e-05 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:08.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:08.105286+0000", 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8785000000000001e-05 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-16T10:30:08.114 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:30:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:08.133 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:08.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:08.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:08.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:08.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:08.313 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 28 16 54091 54075 120.684 95.8125 0.0066413 0.00828042 2024-09-16T10:30:08.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:08.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:08.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:08.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[29993]: 4.16 scrub starts 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[29993]: 4.16 scrub ok 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[29993]: pgmap v898: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 986 op/s 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[39470]: 4.16 scrub starts 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[39470]: 4.16 scrub ok 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[39470]: pgmap v898: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 986 op/s 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:30:08.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:08 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:30:08.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:08.521 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:30:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:08 smithi165 ceph-mon[32696]: 4.16 scrub starts 2024-09-16T10:30:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:08 smithi165 ceph-mon[32696]: 4.16 scrub ok 2024-09-16T10:30:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:08 smithi165 ceph-mon[32696]: pgmap v898: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 986 op/s 2024-09-16T10:30:08.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:08 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:30:08.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:08 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:30:08.541 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:08.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:08.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:08.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:08.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:08.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:08.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:08.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:08.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:08.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:08.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:09.017 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:09.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:09.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:09.131 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T10:30:09.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:09.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:09.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:09.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:09.314 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 29 16 55485 55469 119.524 87.125 0.00648891 0.00836143 2024-09-16T10:30:09.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:30:09.338 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:09.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: 5.3 scrub starts 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: 5.3 scrub ok 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1625652002' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:30:09.351 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:09.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:30:09 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:30:09.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:30:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: 5.3 scrub starts 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: 5.3 scrub ok 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1625652002' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:30:09.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:30:09.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: 5.3 scrub starts 2024-09-16T10:30:09.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: 5.3 scrub ok 2024-09-16T10:30:09.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1625652002' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:30:09.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:09.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:09.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:09.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:09.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:09.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:09.697 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:09.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:09.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:09.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:09.776 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:30:09.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:09.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:09.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:09.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:10.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:10.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:10.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:10.257 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.294 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:10.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:10.314 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 30 16 56895 56879 118.477 88.125 0.0158826 0.00843501 2024-09-16T10:30:10.327 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:30:10.327 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:30:10.327 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:30:10.327 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:30:10.327 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:30:10.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:10.418 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:10.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:30:10.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:30:10.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: 5.19 scrub starts 2024-09-16T10:30:10.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: 5.19 scrub ok 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1625652002' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: osdmap e489: 8 total, 8 up, 8 in 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[29993]: pgmap v900: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 802 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: 5.19 scrub starts 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: 5.19 scrub ok 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1625652002' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:30:10.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: osdmap e489: 8 total, 8 up, 8 in 2024-09-16T10:30:10.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:10 smithi063 ceph-mon[39470]: pgmap v900: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 802 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-16T10:30:10.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:10.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: 5.19 scrub starts 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: 5.19 scrub ok 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1625652002' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: osdmap e489: 8 total, 8 up, 8 in 2024-09-16T10:30:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:10 smithi165 ceph-mon[32696]: pgmap v900: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 802 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-16T10:30:10.652 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:10.692 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:10.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:10.830 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:10.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:10.957 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:10.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:11.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:11.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:11.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:11.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:11.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:11.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:11.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:11.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 31 16 58178 58162 117.242 80.1875 0.00908021 0.00852373 2024-09-16T10:30:11.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:11.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:11.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:11.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:11.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:11 smithi063 ceph-mon[29993]: 5.1d scrub starts 2024-09-16T10:30:11.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:11 smithi063 ceph-mon[29993]: 5.1d scrub ok 2024-09-16T10:30:11.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:11 smithi063 ceph-mon[29993]: osdmap e490: 8 total, 8 up, 8 in 2024-09-16T10:30:11.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:11 smithi063 ceph-mon[39470]: 5.1d scrub starts 2024-09-16T10:30:11.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:11 smithi063 ceph-mon[39470]: 5.1d scrub ok 2024-09-16T10:30:11.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:11 smithi063 ceph-mon[39470]: osdmap e490: 8 total, 8 up, 8 in 2024-09-16T10:30:11.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:11 smithi165 ceph-mon[32696]: 5.1d scrub starts 2024-09-16T10:30:11.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:11 smithi165 ceph-mon[32696]: 5.1d scrub ok 2024-09-16T10:30:11.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:11 smithi165 ceph-mon[32696]: osdmap e490: 8 total, 8 up, 8 in 2024-09-16T10:30:11.474 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:11.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:11.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:30:11.576 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:11.613 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:11.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:11.714 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:11.751 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:11.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:11.842 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:11.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:11.875 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:11.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:11.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:12.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:12.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:12.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:12.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:12.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:12.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:12.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:12.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:12 smithi165 ceph-mon[32696]: pgmap v902: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-16T10:30:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:12 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:30:12.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:12 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:30:12.314 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 32 16 59402 59386 115.969 76.5 0.0101901 0.00861741 2024-09-16T10:30:12.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:12.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:12.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:12.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:12 smithi063 ceph-mon[29993]: pgmap v902: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-16T10:30:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:12 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:30:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:12 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:30:12.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:12 smithi063 ceph-mon[39470]: pgmap v902: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-16T10:30:12.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:12 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:30:12.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:12 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:30:12.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:12.538 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:12.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:12.616 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.15333.0:231096 8.2 8:4cceeac5:::benchmark_data_smithi063_101290_object103776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 527.85905815800004, 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115767351, 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.617 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.15333", 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/214958900", 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 231096 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:30:12.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754721+0000", 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754723+0000", 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6330000000000001e-06 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754727+0000", 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2300000000000002e-06 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754734+0000", 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7850000000000003e-06 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754739+0000", 2024-09-16T10:30:12.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7979999999999996e-06 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754843+0000", 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010442000000000001 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.754957+0000", 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113748 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.755066+0000", 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109279 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758427+0000", 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033604519999999999 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.758995+0000", 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056835100000000003 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870387+0000", 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11139234100000001 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870444+0000", 2024-09-16T10:30:12.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6450999999999999e-05 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:21:24.870489+0000", 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4863000000000001e-05 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212280 9.e 9:7a91a301:::benchmark_data_smithi063_124717_object96049:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.943633992, 2024-09-16T10:30:12.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30939322400000002, 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212280 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.625 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670148+0000", 2024-09-16T10:30:12.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.283e-06 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670154+0000", 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8370000000000002e-06 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670167+0000", 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3243e-05 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670176+0000", 2024-09-16T10:30:12.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5969999999999992e-06 2024-09-16T10:30:12.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670244+0000", 2024-09-16T10:30:12.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8726999999999994e-05 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670254+0000", 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9210000000000004e-06 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.978436+0000", 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30818199899999998 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979539+0000", 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001102617 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212292 9.e 9:780c7a09:::benchmark_data_smithi063_124717_object96061:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.94358227100003, 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31685680300000002, 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.630 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212292 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670198+0000", 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9599999999999999e-07 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670201+0000", 2024-09-16T10:30:12.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1130000000000001e-06 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670207+0000", 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2689999999999996e-06 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670213+0000", 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.164e-06 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670284+0000", 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1136000000000005e-05 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670291+0000", 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9210000000000003e-06 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979884+0000", 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30959349200000003 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980142+0000", 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025797 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980215+0000", 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3054999999999998e-05 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985118+0000", 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049030380000000002 2024-09-16T10:30:12.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985153+0000", 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4496999999999999e-05 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986997+0000", 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018447450000000001 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987028+0000", 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.099e-05 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987054+0000", 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5717000000000001e-05 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212294 9.e 9:724d3208:::benchmark_data_smithi063_124717_object96063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:30:12.637 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.94355050199999, 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.32240128099999998, 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212294 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670230+0000", 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7999999999999995e-07 2024-09-16T10:30:12.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670233+0000", 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9919999999999999e-06 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670239+0000", 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3779999999999998e-06 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670245+0000", 2024-09-16T10:30:12.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670316+0000", 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1147000000000002e-05 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670323+0000", 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7209999999999997e-06 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981661+0000", 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31133887799999999 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981742+0000", 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0136000000000007e-05 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981814+0000", 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2065999999999997e-05 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985187+0000", 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003373048 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985193+0000", 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7170000000000002e-06 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992481+0000", 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072876870000000002 2024-09-16T10:30:12.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992565+0000", 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3726000000000004e-05 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992630+0000", 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5581999999999997e-05 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212314 9.e 9:73a00d32:::benchmark_data_smithi063_124717_object96083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.91850362600002, 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29758563799999999, 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212314 2024-09-16T10:30:12.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695278+0000", 2024-09-16T10:30:12.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9800000000000001e-06 2024-09-16T10:30:12.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:12.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695284+0000", 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9349999999999999e-06 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695295+0000", 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0849000000000001e-05 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695303+0000", 2024-09-16T10:30:12.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2130000000000008e-06 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695359+0000", 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5859999999999997e-05 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695366+0000", 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6580000000000002e-06 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982882+0000", 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.287515939 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982949+0000", 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6370000000000003e-05 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.983024+0000", 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5723000000000007e-05 2024-09-16T10:30:12.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985197+0000", 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002172195 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986274+0000", 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001077733 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992764+0000", 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064893520000000003 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992820+0000", 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5903000000000003e-05 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992862+0000", 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1928000000000002e-05 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.653 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212323 9.e 9:7cb576e6:::benchmark_data_smithi063_124717_object96092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.91300949499998, 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29218721399999997, 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212323 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700773+0000", 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5299999999999999e-06 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700779+0000", 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1399999999999997e-06 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700801+0000", 2024-09-16T10:30:12.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1812999999999999e-05 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700809+0000", 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1929999999999993e-06 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700877+0000", 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8199000000000002e-05 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700889+0000", 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1904e-05 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984122+0000", 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28323306799999998 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984181+0000", 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8847000000000001e-05 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984241+0000", 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0056000000000002e-05 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986252+0000", 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020111399999999998 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986461+0000", 2024-09-16T10:30:12.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000209273 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992908+0000", 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006446523 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992935+0000", 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7016999999999999e-05 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992957+0000", 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2510999999999998e-05 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212342 9.e 9:700843e6:::benchmark_data_smithi063_124717_object96111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.89346929599998, 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27274648600000001, 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.661 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212342 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720312+0000", 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0949999999999998e-06 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720318+0000", 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1059999999999996e-06 2024-09-16T10:30:12.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720333+0000", 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4173e-05 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720340+0000", 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0389999999999999e-06 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720405+0000", 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5079000000000005e-05 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720415+0000", 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0351e-05 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985207+0000", 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.264792321 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985273+0000", 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5215999999999994e-05 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985329+0000", 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6403000000000002e-05 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986392+0000", 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010631939999999999 2024-09-16T10:30:12.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986576+0000", 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018346000000000001 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992994+0000", 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064186979999999996 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993027+0000", 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2262999999999999e-05 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993057+0000", 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0088000000000001e-05 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212688 9.e 9:7ab865c0:::benchmark_data_smithi063_124717_object96457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:30:12.668 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.39278295299999, 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12657935300000001, 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212688 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220999+0000", 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221004+0000", 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1880000000000001e-06 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221017+0000", 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2802e-05 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221023+0000", 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8789999999999997e-06 2024-09-16T10:30:12.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221090+0000", 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6854999999999996e-05 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221200+0000", 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010959399999999999 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221304+0000", 2024-09-16T10:30:12.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104456 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.223708+0000", 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024038449999999999 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.224936+0000", 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012280119999999999 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347460+0000", 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122523584 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347521+0000", 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1199999999999997e-05 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347576+0000", 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4972000000000002e-05 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.674 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:213104 9.e 9:709eca86:::benchmark_data_smithi063_124717_object96873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 310.86912971700002, 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12242840100000001, 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:30:12.675 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213104 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744652+0000", 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0219999999999999e-06 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744657+0000", 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3020000000000002e-06 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744668+0000", 2024-09-16T10:30:12.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.118e-05 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744674+0000", 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6500000000000001e-06 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744728+0000", 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3705000000000002e-05 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744820+0000", 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2052000000000006e-05 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744897+0000", 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7200999999999995e-05 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.746741+0000", 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018441250000000001 2024-09-16T10:30:12.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.747036+0000", 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029480100000000003 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.866958+0000", 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11992201700000001 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867027+0000", 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9314999999999996e-05 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867078+0000", 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1031e-05 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:41313 10.1 10:8e698f2c:::benchmark_data_smithi063_149043_object41312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:30:12.681 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 258.36985621500003, 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10554345399999999, 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 41313 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243925+0000", 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0329999999999998e-06 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243997+0000", 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1767000000000006e-05 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244022+0000", 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4737999999999999e-05 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244030+0000", 2024-09-16T10:30:12.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5699999999999993e-06 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244235+0000", 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020468299999999999 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244353+0000", 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011790699999999999 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.246158+0000", 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018052179999999999 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.251015+0000", 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004856595 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.251463+0000", 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000448369 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349318+0000", 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097854294999999994 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349407+0000", 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9024999999999996e-05 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349467+0000", 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0254000000000003e-05 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.687 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73195 10.e 10:780b6aaa:::benchmark_data_smithi063_149043_object73194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 237.17951316099999, 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858509, 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:12.688 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73195 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434267+0000", 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434306+0000", 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8250000000000001e-05 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434314+0000", 2024-09-16T10:30:12.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2949999999999994e-06 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434319+0000", 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5300000000000004e-06 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434368+0000", 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8321e-05 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434486+0000", 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117924 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434622+0000", 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013583099999999999 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.437851+0000", 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032291569999999999 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.438873+0000", 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00102191 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542761+0000", 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10388875 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542818+0000", 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6863000000000001e-05 2024-09-16T10:30:12.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542852+0000", 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73199 10.e 10:752bb1d5:::benchmark_data_smithi063_149043_object73198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 237.17171940599999, 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112679315, 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.694 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73199 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442062+0000", 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.367e-06 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442108+0000", 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6814999999999998e-05 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442117+0000", 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8410000000000006e-06 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442124+0000", 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6889999999999998e-06 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442179+0000", 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4756000000000002e-05 2024-09-16T10:30:12.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442288+0000", 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109372 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442378+0000", 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0488000000000005e-05 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.444289+0000", 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001910293 2024-09-16T10:30:12.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.444588+0000", 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000299602 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554621+0000", 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110032625 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554689+0000", 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8324000000000005e-05 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554739+0000", 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0142999999999997e-05 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73208 10.e 10:776621e8:::benchmark_data_smithi063_149043_object73207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:30:12.700 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 237.166595583, 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10772002999999999, 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73208 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447185+0000", 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2100000000000001e-06 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447333+0000", 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014735900000000001 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447344+0000", 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0994e-05 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447351+0000", 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1130000000000003e-06 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447410+0000", 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9219999999999999e-05 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447533+0000", 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123325 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447677+0000", 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143972 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.449865+0000", 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021879680000000002 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.450597+0000", 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073198000000000002 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554808+0000", 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104210409 2024-09-16T10:30:12.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554861+0000", 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3298999999999999e-05 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554904+0000", 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3180999999999999e-05 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73211 10.e 10:77f82def:::benchmark_data_smithi063_149043_object73210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 237.161033745, 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10288251299999999, 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73211 2024-09-16T10:30:12.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:30:12.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452747+0000", 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.643e-06 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452838+0000", 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0611000000000004e-05 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452849+0000", 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0519e-05 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452855+0000", 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454022+0000", 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001166582 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454155+0000", 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133013 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454307+0000", 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015232200000000001 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.459163+0000", 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048564769999999997 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.459272+0000", 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109069 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554980+0000", 2024-09-16T10:30:12.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095707424999999999 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.555077+0000", 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6832999999999999e-05 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.555628+0000", 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055162699999999996 2024-09-16T10:30:12.713 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:80880 10.1 10:8f0a7a66:::benchmark_data_smithi063_149043_object80879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 231.29895603599999, 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109496315, 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.714 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 80880 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314825+0000", 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.689e-06 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314901+0000", 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5549000000000005e-05 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314912+0000", 2024-09-16T10:30:12.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270999999999999e-05 2024-09-16T10:30:12.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314917+0000", 2024-09-16T10:30:12.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.169e-06 2024-09-16T10:30:12.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314973+0000", 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5448999999999997e-05 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315116+0000", 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142986 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315268+0000", 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015202799999999999 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.319831+0000", 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004563302 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.319869+0000", 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7737000000000001e-05 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424204+0000", 2024-09-16T10:30:12.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104335386 2024-09-16T10:30:12.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424277+0000", 2024-09-16T10:30:12.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2435999999999996e-05 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424320+0000", 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3312999999999997e-05 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:80882 10.1 10:8d1b387e:::benchmark_data_smithi063_149043_object80881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:30:12.724 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 231.29882707900001, 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111081265, 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 80882 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314954+0000", 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.319e-06 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315013+0000", 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9150000000000001e-05 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315022+0000", 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9309999999999997e-06 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315027+0000", 2024-09-16T10:30:12.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6360000000000002e-06 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317727+0000", 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027003840000000001 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317832+0000", 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010535999999999999 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317972+0000", 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014001700000000001 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.320350+0000", 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023780020000000002 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.320544+0000", 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019368200000000001 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.425942+0000", 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105398164 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.425992+0000", 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9570000000000001e-05 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.426034+0000", 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2049999999999999e-05 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:207268 10.2 e432/426 10:4040fde1:::benchmark_data_smithi063_149043_object74831:head v 432'11681, mlcod=432'11679)", 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:16.899956+0000", 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 115.713823682, 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058080294999999997, 2024-09-16T10:30:12.731 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.899956+0000", 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.899956+0000", 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.899957+0000", 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.23e-07 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.899961+0000", 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4909999999999999e-06 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.899968+0000", 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4239999999999998e-06 2024-09-16T10:30:12.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.899971+0000", 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.822e-06 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.900009+0000", 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7503000000000003e-05 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.900026+0000", 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.717e-05 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.957999+0000", 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.057973086 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:16.958036+0000", 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7076000000000002e-05 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221554 10.1b e446/441 10:d926993d:::benchmark_data_smithi063_149043_object89117:head v 446'13142, mlcod=446'13140)", 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.613186+0000", 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 96.000593795, 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078483563000000006, 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613186+0000", 2024-09-16T10:30:12.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613186+0000", 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613188+0000", 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7770000000000001e-06 2024-09-16T10:30:12.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613194+0000", 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1090000000000003e-06 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613203+0000", 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6919999999999997e-06 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613210+0000", 2024-09-16T10:30:12.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.601e-06 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613256+0000", 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6248000000000002e-05 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613271+0000", 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4963999999999999e-05 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.691625+0000", 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.078353364999999994 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.691669+0000", 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4807e-05 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221552 10.c e446/442 10:30346976:::benchmark_data_smithi063_149043_object89115:head v 446'13317, mlcod=446'13315)", 2024-09-16T10:30:12.740 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.613310+0000", 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 96.000469543999998, 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.085401337999999993, 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613310+0000", 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613310+0000", 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613311+0000", 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.006e-06 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613316+0000", 2024-09-16T10:30:12.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0100000000000003e-06 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613323+0000", 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6769999999999998e-06 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613328+0000", 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4060000000000004e-06 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.689983+0000", 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076655128000000003 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.690009+0000", 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5837e-05 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.698621+0000", 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086123359999999999 2024-09-16T10:30:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.698711+0000", 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9938000000000005e-05 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221553 10.12 e446/414 10:4f7e32bd:::benchmark_data_smithi063_149043_object89116:head v 446'12226, mlcod=446'12224)", 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.614067+0000", 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.999712106000004, 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.082785922999999997, 2024-09-16T10:30:12.745 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614067+0000", 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614067+0000", 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:12.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614069+0000", 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614075+0000", 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2489999999999998e-06 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.689968+0000", 2024-09-16T10:30:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.075892792000000001 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.689979+0000", 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1164e-05 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.691738+0000", 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017594970000000001 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:12.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.691761+0000", 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.283e-05 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.696716+0000", 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049551359999999997 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.696853+0000", 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137107 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:12.750 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:12.750 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:30:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:12.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:12.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:12.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:12.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:12.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:12.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:13.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:13.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:13.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:13.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.272 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:13.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:13.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 33 16 60252 60236 114.064 53.125 0.00554149 0.00876191 2024-09-16T10:30:13.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:13.421 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:13.439 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:13.440 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:13.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[29993]: 2.2 scrub starts 2024-09-16T10:30:13.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[29993]: 2.2 scrub ok 2024-09-16T10:30:13.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[29993]: 6.0 scrub starts 2024-09-16T10:30:13.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[29993]: 6.0 scrub ok 2024-09-16T10:30:13.443 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[39470]: 2.2 scrub starts 2024-09-16T10:30:13.443 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[39470]: 2.2 scrub ok 2024-09-16T10:30:13.443 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[39470]: 6.0 scrub starts 2024-09-16T10:30:13.443 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:13 smithi063 ceph-mon[39470]: 6.0 scrub ok 2024-09-16T10:30:13.473 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:13.473 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:13.496 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:13.497 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:13.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:13.529 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:13.530 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:13.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:13 smithi165 ceph-mon[32696]: 2.2 scrub starts 2024-09-16T10:30:13.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:13 smithi165 ceph-mon[32696]: 2.2 scrub ok 2024-09-16T10:30:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:13 smithi165 ceph-mon[32696]: 6.0 scrub starts 2024-09-16T10:30:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:13 smithi165 ceph-mon[32696]: 6.0 scrub ok 2024-09-16T10:30:13.564 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:13.564 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:13.564 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:13.601 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:13.601 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:13.637 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:13.638 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:13.658 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:13.666 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:13.666 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:13.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:13.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:13.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:13.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:13.837 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:13.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:13.987 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:13.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:14.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:30:14.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:14.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:14.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:14.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:14.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:14.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:14.279 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:30:14.279 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:30:14.279 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[29993]: 3.19 scrub starts 2024-09-16T10:30:14.279 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[29993]: 3.19 scrub ok 2024-09-16T10:30:14.279 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2681954830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:14.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[29993]: pgmap v903: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-16T10:30:14.280 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:30:14.280 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:30:14.280 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[39470]: 3.19 scrub starts 2024-09-16T10:30:14.280 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[39470]: 3.19 scrub ok 2024-09-16T10:30:14.280 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2681954830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:14.280 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:14 smithi063 ceph-mon[39470]: pgmap v903: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-16T10:30:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:14 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:30:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:14 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:30:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:14 smithi165 ceph-mon[32696]: 3.19 scrub starts 2024-09-16T10:30:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:14 smithi165 ceph-mon[32696]: 3.19 scrub ok 2024-09-16T10:30:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2681954830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:14.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:14 smithi165 ceph-mon[32696]: pgmap v903: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-16T10:30:14.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 34 16 61497 61481 112.998 77.8125 0.0135577 0.0088445 2024-09-16T10:30:14.370 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:14.415 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:14.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:14.441 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:14.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:14.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:14.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:14.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:14.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:14.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:14.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:14.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:14.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:14.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:14.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:15.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:15.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:15.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:15.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:15.205 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:30:15.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:15.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:15.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 35 16 62843 62827 112.173 84.125 0.0139617 0.00891009 2024-09-16T10:30:15.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:15.365 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:15.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:15.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:15 smithi063 ceph-mon[29993]: 3.1c scrub starts 2024-09-16T10:30:15.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:15 smithi063 ceph-mon[29993]: 3.1c scrub ok 2024-09-16T10:30:15.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4139217472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:15.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:15 smithi063 ceph-mon[39470]: 3.1c scrub starts 2024-09-16T10:30:15.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:15 smithi063 ceph-mon[39470]: 3.1c scrub ok 2024-09-16T10:30:15.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4139217472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:15.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:15.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:15.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:15 smithi165 ceph-mon[32696]: 3.1c scrub starts 2024-09-16T10:30:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:15 smithi165 ceph-mon[32696]: 3.1c scrub ok 2024-09-16T10:30:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4139217472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:15.601 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:15.633 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:15.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:15.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:15.773 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:15.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:15.866 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:15.866 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:15.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:15.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:15.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:16.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:16.095 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:16.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:16.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:16.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:16.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:16.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:16.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 36 15 64272 64257 111.539 89.375 0.0158422 0.00896067 2024-09-16T10:30:16.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:16.364 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:16.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:16.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: 6.d scrub starts 2024-09-16T10:30:16.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: 6.d scrub ok 2024-09-16T10:30:16.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4139217472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: osdmap e491: 8 total, 8 up, 8 in 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: pgmap v905: 168 pgs: 168 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 810 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: 6.1d scrub starts 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: 6.1d scrub ok 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[29993]: osdmap e492: 8 total, 8 up, 8 in 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: 6.d scrub starts 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: 6.d scrub ok 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:30:16.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4139217472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: osdmap e491: 8 total, 8 up, 8 in 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: pgmap v905: 168 pgs: 168 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 810 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: 6.1d scrub starts 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: 6.1d scrub ok 2024-09-16T10:30:16.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:16 smithi063 ceph-mon[39470]: osdmap e492: 8 total, 8 up, 8 in 2024-09-16T10:30:16.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:16.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:16.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: 6.d scrub starts 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: 6.d scrub ok 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4139217472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: osdmap e491: 8 total, 8 up, 8 in 2024-09-16T10:30:16.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:30:16.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: pgmap v905: 168 pgs: 168 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 810 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-16T10:30:16.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: 6.1d scrub starts 2024-09-16T10:30:16.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: 6.1d scrub ok 2024-09-16T10:30:16.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:16 smithi165 ceph-mon[32696]: osdmap e492: 8 total, 8 up, 8 in 2024-09-16T10:30:16.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:30:16.603 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:16.645 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:16.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:16.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:16.781 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:16.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:16.845 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:30:16.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:16.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:16.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:17.022 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:17.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:17.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:17.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:17.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:17.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:17.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:17.205 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:30:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:30:17.301 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:17.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 37 16 65633 65617 110.822 85 0.0130746 0.00901803 2024-09-16T10:30:17.348 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:17.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:17.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:17.495 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:17.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:17.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:17 smithi165 ceph-mon[32696]: 4.19 scrub starts 2024-09-16T10:30:17.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:17 smithi165 ceph-mon[32696]: 4.19 scrub ok 2024-09-16T10:30:17.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:17 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:30:17.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:17 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:30:17.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:17 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:30:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:17 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:30:17.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:17.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:17.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:17.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[29993]: 4.19 scrub starts 2024-09-16T10:30:17.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[29993]: 4.19 scrub ok 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[39470]: 4.19 scrub starts 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[39470]: 4.19 scrub ok 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:30:17.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:17 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:30:17.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:17.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:17.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:17.860 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:17.898 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:17.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:17.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:30:18.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:18.037 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:18.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:18.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:18.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:18.293 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:18.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:18.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 38 16 66750 66734 109.742 69.8125 0.00534038 0.00909121 2024-09-16T10:30:18.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:18.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:18.442 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:18.443 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:18.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:18 smithi063 ceph-mon[39470]: 5.e scrub starts 2024-09-16T10:30:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:18 smithi063 ceph-mon[39470]: 5.e scrub ok 2024-09-16T10:30:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:18 smithi063 ceph-mon[39470]: pgmap v907: 168 pgs: 168 active+clean; 3.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 800 B/s rd, 60 MiB/s wr, 963 op/s 2024-09-16T10:30:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:18 smithi063 ceph-mon[29993]: 5.e scrub starts 2024-09-16T10:30:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:18 smithi063 ceph-mon[29993]: 5.e scrub ok 2024-09-16T10:30:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:18 smithi063 ceph-mon[29993]: pgmap v907: 168 pgs: 168 active+clean; 3.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 800 B/s rd, 60 MiB/s wr, 963 op/s 2024-09-16T10:30:18.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:18.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:18.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:18 smithi165 ceph-mon[32696]: 5.e scrub starts 2024-09-16T10:30:18.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:18 smithi165 ceph-mon[32696]: 5.e scrub ok 2024-09-16T10:30:18.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:18 smithi165 ceph-mon[32696]: pgmap v907: 168 pgs: 168 active+clean; 3.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 800 B/s rd, 60 MiB/s wr, 963 op/s 2024-09-16T10:30:18.625 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:30:18.625 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:30:18.626 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-09-16T10:30:18.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:30:18.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.720 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:18.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:18.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:18.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:18.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:18.968 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:18.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:19.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:19.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:19.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:19.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:19.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:19.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:19.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 39 15 67729 67714 108.499 61.25 0.0132848 0.00921109 2024-09-16T10:30:19.328 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:30:19.328 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:30:19.328 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:30:19.328 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[29993]: 1.5 scrub ok 2024-09-16T10:30:19.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:19.331 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:30:19.331 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:30:19.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:30:19.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:19 smithi063 ceph-mon[39470]: 1.5 scrub ok 2024-09-16T10:30:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:19.382 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:19.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:19.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:19.512 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:19.513 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:19.549 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:19.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:19.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:19 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:30:19.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:19 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:30:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:19 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:30:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:19 smithi165 ceph-mon[32696]: 1.5 scrub ok 2024-09-16T10:30:19.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:30:19 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:30:19.552 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:19.553 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:19.553 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:19.575 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:19.592 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:19.592 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:19.627 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:19.628 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:19.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:19.663 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:19.664 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:19.695 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:19.695 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:19.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:19.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:19.730 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:19.765 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:19.766 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:19.800 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:19.843 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:19.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:19.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:19.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:19.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:20.074 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:20.113 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:20.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:20.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:20.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:20.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:20.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:30:20.315196+0000 min lat: 0.00120603 max lat: 0.224582 avg lat: 0.009291 2024-09-16T10:30:20.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:30:20.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 40 16 68866 68850 107.561 71 0.0173819 0.009291 2024-09-16T10:30:20.343 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:20.423 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:20.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:20.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:20 smithi165 ceph-mon[32696]: 4.b scrub starts 2024-09-16T10:30:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:20 smithi165 ceph-mon[32696]: 4.b scrub ok 2024-09-16T10:30:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:20 smithi165 ceph-mon[32696]: pgmap v908: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-16T10:30:20.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1527101307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:20.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:20.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:20.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:20.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:20.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[29993]: 4.b scrub starts 2024-09-16T10:30:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[29993]: 4.b scrub ok 2024-09-16T10:30:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[29993]: pgmap v908: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-16T10:30:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1527101307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[39470]: 4.b scrub starts 2024-09-16T10:30:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[39470]: 4.b scrub ok 2024-09-16T10:30:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[39470]: pgmap v908: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-16T10:30:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1527101307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:20.791 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:20.830 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:20.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:20.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:20.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:20.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:21.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:21.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:21.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:21.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:21.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:21.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:21 smithi165 ceph-mon[32696]: 5.17 scrub starts 2024-09-16T10:30:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:21 smithi165 ceph-mon[32696]: 5.17 scrub ok 2024-09-16T10:30:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:21 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:30:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:21 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:30:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:21 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:30:21.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:21 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:30:21.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 41 16 70127 70111 106.859 78.8125 0.0142183 0.00935234 2024-09-16T10:30:21.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:21.374 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:21.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:21.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[39470]: 5.17 scrub starts 2024-09-16T10:30:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[39470]: 5.17 scrub ok 2024-09-16T10:30:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:30:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:30:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:30:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:30:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[29993]: 5.17 scrub starts 2024-09-16T10:30:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[29993]: 5.17 scrub ok 2024-09-16T10:30:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:30:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:30:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:30:21.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:21 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:30:21.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:21.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:21.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:21.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:30:21.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:21.656 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:30:21.659 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:30:21.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:21.660 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:21.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:21.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:21.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:21.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:21.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:21.935 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:21.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:21.949 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:22.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:22.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:22.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:22.170 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:22.210 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:22.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:22.272 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:30:22.312 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:22.315 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 42 16 71318 71302 106.087 74.4375 0.0106469 0.0094202 2024-09-16T10:30:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:22.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: 4.9 scrub starts 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: 4.9 scrub ok 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: 3.17 scrub starts 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: 3.17 scrub ok 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: pgmap v909: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3181757712' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: from='client.46439 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:22.398 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:22 smithi165 ceph-mon[32696]: from='client.16635 v1:172.21.15.63:0/455558704' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:30:22.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:22.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: 4.9 scrub starts 2024-09-16T10:30:22.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: 4.9 scrub ok 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: 3.17 scrub starts 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: 3.17 scrub ok 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: pgmap v909: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3181757712' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: from='client.46439 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[29993]: from='client.16635 v1:172.21.15.63:0/455558704' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: 4.9 scrub starts 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: 4.9 scrub ok 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: 3.17 scrub starts 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: 3.17 scrub ok 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: pgmap v909: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3181757712' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: from='client.46439 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:22.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:22 smithi063 ceph-mon[39470]: from='client.16635 v1:172.21.15.63:0/455558704' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:30:22.485 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:22.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:2951 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 93976561478912 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 235.01169983200001, 2024-09-16T10:30:22.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0211994769999999, 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2951 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:30:22.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536592+0000", 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2979999999999999e-06 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536598+0000", 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536615+0000", 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7496999999999999e-05 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536624+0000", 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9099999999999994e-06 2024-09-16T10:30:22.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536869+0000", 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024484799999999998 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536876+0000", 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1910000000000003e-06 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.547076+0000", 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010199542000000001 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.547092+0000", 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5818e-05 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.552328+0000", 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0052359230000001 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.552503+0000", 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017550000000000001 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.557789+0000", 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052855139999999998 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.559 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98204 10.9 10:9d9b818e:::benchmark_data_smithi063_149043_object98203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 225.56670019399999, 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.034086345, 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.560 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98204 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981591+0000", 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981670+0000", 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9151000000000002e-05 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981680+0000", 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8109999999999997e-06 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981687+0000", 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9959999999999996e-06 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981740+0000", 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3491e-05 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981886+0000", 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014617800000000001 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.982038+0000", 2024-09-16T10:30:22.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151927 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.990399+0000", 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083606689999999994 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999930+0000", 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095309130000000002 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:22.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999946+0000", 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6342999999999999e-05 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015600+0000", 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0156542 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015675+0000", 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.483e-05 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.566 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98205 10.9 10:919b6e78:::benchmark_data_smithi063_149043_object98204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 225.56658082000001, 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.046984186, 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:22.567 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98205 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981709+0000", 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4499999999999995e-07 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981760+0000", 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0654e-05 2024-09-16T10:30:22.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981766+0000", 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6089999999999997e-06 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981772+0000", 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7620000000000001e-06 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.985245+0000", 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034727030000000002 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.986079+0000", 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083413300000000003 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.986209+0000", 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012954199999999999 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999975+0000", 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013766733 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:22.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999982+0000", 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8759999999999999e-06 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015907+0000", 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0159249210000001 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016460+0000", 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055315400000000004 2024-09-16T10:30:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028602+0000", 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012142065 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028692+0000", 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0088999999999998e-05 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98225 10.9 10:9f9b788e:::benchmark_data_smithi063_149043_object98224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 225.52650419599999, 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0071949410000001, 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98225 2024-09-16T10:30:22.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021788+0000", 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2959999999999999e-06 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021880+0000", 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1638999999999995e-05 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021898+0000", 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873e-05 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021907+0000", 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1789999999999997e-06 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021961+0000", 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4026000000000001e-05 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021969+0000", 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9840000000000001e-06 2024-09-16T10:30:22.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016509+0000", 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99453982200000002 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016634+0000", 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012573099999999999 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016773+0000", 2024-09-16T10:30:22.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013850399999999999 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.021953+0000", 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051800379999999997 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028816+0000", 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068627699999999998 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028873+0000", 2024-09-16T10:30:22.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7655999999999998e-05 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028980+0000", 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010642299999999999 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98247 10.9 10:95b7428a:::benchmark_data_smithi063_149043_object98246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 225.511638086, 2024-09-16T10:30:22.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99248770799999997, 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98247 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:30:22.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036653+0000", 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9139999999999998e-06 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036702+0000", 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9490000000000002e-05 2024-09-16T10:30:22.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036716+0000", 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3521e-05 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036723+0000", 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4259999999999997e-06 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036784+0000", 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1013e-05 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036790+0000", 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4739999999999996e-06 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.018768+0000", 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98197859600000004 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.018953+0000", 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184358 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.019066+0000", 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011326 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.021961+0000", 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028944869999999998 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029022+0000", 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070617960000000004 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029056+0000", 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4079999999999999e-05 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029139+0000", 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2293000000000006e-05 2024-09-16T10:30:22.588 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:204177 10.23 10:c5a630ae:::benchmark_data_smithi063_149043_object71740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:13.524797+0000", 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 129.02349240999999, 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.922248611, 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.589 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 204177 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524797+0000", 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524797+0000", 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524798+0000", 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4839999999999999e-06 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524803+0000", 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7899999999999999e-06 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524813+0000", 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8570000000000007e-06 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524820+0000", 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3089999999999998e-06 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524865+0000", 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5302000000000001e-05 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524872+0000", 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.579e-06 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.441098+0000", 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91622587499999997 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.441189+0000", 2024-09-16T10:30:22.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1100000000000005e-05 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.441350+0000", 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016148700000000001 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.443761+0000", 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024104009999999999 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.443769+0000", 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8800000000000008e-06 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.446868+0000", 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030995279999999998 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.446899+0000", 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0568000000000003e-05 2024-09-16T10:30:22.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.447045+0000", 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146451 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:254094 10.11 10:8ae0866e:::benchmark_data_smithi063_149043_object121657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:07.537563+0000", 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.010725782999998, 2024-09-16T10:30:22.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0256367769999999, 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 254094 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537563+0000", 2024-09-16T10:30:22.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537563+0000", 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537565+0000", 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7910000000000001e-06 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537570+0000", 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.327e-06 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537581+0000", 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0631999999999999e-05 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537586+0000", 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.186e-06 2024-09-16T10:30:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.538277+0000", 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00069087 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.538287+0000", 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8849999999999994e-06 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.557618+0000", 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0193307949999999 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.557734+0000", 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011606400000000001 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.558062+0000", 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032821400000000001 2024-09-16T10:30:22.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.562925+0000", 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048629800000000003 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.562990+0000", 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4615000000000004e-05 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.563100+0000", 2024-09-16T10:30:22.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011081099999999999 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.563160+0000", 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9432000000000003e-05 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.563200+0000", 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0175000000000001e-05 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.603 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:17231 11.7 11:e64fab2c:::benchmark_data_smithi063_176907_object17230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:46.636130+0000", 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.912159426000002, 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.055837249999999998, 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.604 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 17231 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636130+0000", 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636130+0000", 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636131+0000", 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5260000000000001e-06 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636195+0000", 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3581e-05 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636209+0000", 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3842000000000001e-05 2024-09-16T10:30:22.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636217+0000", 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9950000000000005e-06 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636279+0000", 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2588000000000005e-05 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636409+0000", 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013001199999999999 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.636524+0000", 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115136 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.637347+0000", 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082246999999999995 2024-09-16T10:30:22.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.637649+0000", 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030235 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.691897+0000", 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247421999999997 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.691931+0000", 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4471999999999998e-05 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:46.691967+0000", 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5855999999999997e-05 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:46676 11.7 11:eb3f6f74:::benchmark_data_smithi063_176907_object46675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:03.463650+0000", 2024-09-16T10:30:22.610 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.084638673000001, 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190323599999999, 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46676 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.463650+0000", 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.463650+0000", 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.463653+0000", 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.463723+0000", 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0117000000000006e-05 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.463734+0000", 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1612e-05 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.463741+0000", 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.465839+0000", 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002098117 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.465932+0000", 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3148999999999999e-05 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.466013+0000", 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0572e-05 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467095+0000", 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010826760000000001 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467820+0000", 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072501300000000005 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.565450+0000", 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097629696000000002 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.565508+0000", 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7340999999999999e-05 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.565554+0000", 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6134000000000002e-05 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.616 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:46683 11.7 11:ece9a776:::benchmark_data_smithi063_176907_object46682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:03.467815+0000", 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.080474497000001, 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102975842, 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46683 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467815+0000", 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467815+0000", 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467816+0000", 2024-09-16T10:30:22.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.291e-06 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467879+0000", 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3081000000000001e-05 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467891+0000", 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1635e-05 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467898+0000", 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8890000000000002e-06 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.467937+0000", 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8062999999999998e-05 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.468061+0000", 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000124208 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.468221+0000", 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016039900000000001 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.469612+0000", 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001390613 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.469787+0000", 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017529500000000001 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.570684+0000", 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100897403 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.570743+0000", 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8875999999999997e-05 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.570790+0000", 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7089000000000003e-05 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.622 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:47368 11.7 11:e2d8ab9c:::benchmark_data_smithi063_176907_object47367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:03.952159+0000", 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.596129931, 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098958247999999999, 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.623 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 47368 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952159+0000", 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952159+0000", 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952162+0000", 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5569999999999998e-06 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952202+0000", 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0083999999999998e-05 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952213+0000", 2024-09-16T10:30:22.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1436e-05 2024-09-16T10:30:22.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:22.628 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":909,"stamp":"2024-09-16T10:30:21.455463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4384758299,"num_objects":67096,"num_object_clones":0,"num_object_copies":201288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67096,"num_whiteouts":0,"num_read":1449,"num_read_kb":1201,"num_write":67619,"num_write_kb":4281453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":527794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67911,"ondisk_log_size":67911,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17671720,"kb_used_data":12887964,"kb_used_omap":18,"kb_used_meta":4783597,"kb_avail":732289496,"statfs":{"total":767960285184,"available":749864443904,"internally_reserved":0,"allocated":13197275136,"data_stored":13187855281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19173,"internal_metadata":4898403611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":569901056,"num_objects":8696,"num_object_clones":0,"num_object_copies":26088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8696,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8696,"num_write_kb":556544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002449"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":939,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071457+0000","last_change":"2024-09-16T10:29:36.039702+0000","last_active":"2024-09-16T10:30:16.071457+0000","last_peered":"2024-09-16T10:30:16.071457+0000","last_clean":"2024-09-16T10:30:16.071457+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:30:16.071457+0000","last_undegraded":"2024-09-16T10:30:16.071457+0000","last_fullsized":"2024-09-16T10:30:16.071457+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.039652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.039652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:56.261928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":884,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.596737+0000","last_change":"2024-09-16T10:30:16.596737+0000","last_active":"2024-09-16T10:30:16.596737+0000","last_peered":"2024-09-16T10:30:16.596737+0000","last_clean":"2024-09-16T10:30:16.596737+0000","last_became_active":"2024-09-16T10:26:28.557096+0000","last_became_peered":"2024-09-16T10:26:28.557096+0000","last_unstale":"2024-09-16T10:30:16.596737+0000","last_undegraded":"2024-09-16T10:30:16.596737+0000","last_fullsized":"2024-09-16T10:30:16.596737+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:16.596655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.596655+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:38.661213+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":935,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062653+0000","last_change":"2024-09-16T10:30:08.872970+0000","last_active":"2024-09-16T10:30:16.062653+0000","last_peered":"2024-09-16T10:30:16.062653+0000","last_clean":"2024-09-16T10:30:16.062653+0000","last_became_active":"2024-09-16T10:28:28.071724+0000","last_became_peered":"2024-09-16T10:28:28.071724+0000","last_unstale":"2024-09-16T10:30:16.062653+0000","last_undegraded":"2024-09-16T10:30:16.062653+0000","last_fullsized":"2024-09-16T10:30:16.062653+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.872906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.872906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:16.262177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":248,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.879751+0000","last_change":"2024-09-16T10:30:17.879751+0000","last_active":"2024-09-16T10:30:17.879751+0000","last_peered":"2024-09-16T10:30:17.879751+0000","last_clean":"2024-09-16T10:30:17.879751+0000","last_became_active":"2024-09-16T10:26:28.551847+0000","last_became_peered":"2024-09-16T10:26:28.551847+0000","last_unstale":"2024-09-16T10:30:17.879751+0000","last_undegraded":"2024-09-16T10:30:17.879751+0000","last_fullsized":"2024-09-16T10:30:17.879751+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:17.879610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:17.879610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:38.628891+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":913,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073497+0000","last_change":"2024-09-16T10:29:21.916724+0000","last_active":"2024-09-16T10:30:16.073497+0000","last_peered":"2024-09-16T10:30:16.073497+0000","last_clean":"2024-09-16T10:30:16.073497+0000","last_became_active":"2024-09-16T10:28:28.094751+0000","last_became_peered":"2024-09-16T10:28:28.094751+0000","last_unstale":"2024-09-16T10:30:16.073497+0000","last_undegraded":"2024-09-16T10:30:16.073497+0000","last_fullsized":"2024-09-16T10:30:16.073497+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.675562+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":938,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075395+0000","last_change":"2024-09-16T10:25:17.422550+0000","last_active":"2024-09-16T10:30:16.075395+0000","last_peered":"2024-09-16T10:30:16.075395+0000","last_clean":"2024-09-16T10:30:16.075395+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:30:16.075395+0000","last_undegraded":"2024-09-16T10:30:16.075395+0000","last_fullsized":"2024-09-16T10:30:16.075395+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:17.422478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:17.422478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.825253+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":1015,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074297+0000","last_change":"2024-09-16T10:30:15.998707+0000","last_active":"2024-09-16T10:30:16.074297+0000","last_peered":"2024-09-16T10:30:16.074297+0000","last_clean":"2024-09-16T10:30:16.074297+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:30:16.074297+0000","last_undegraded":"2024-09-16T10:30:16.074297+0000","last_fullsized":"2024-09-16T10:30:16.074297+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:15.998614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.998614+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:33.187618+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":248,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062076+0000","last_change":"2024-09-16T10:30:07.591239+0000","last_active":"2024-09-16T10:30:16.062076+0000","last_peered":"2024-09-16T10:30:16.062076+0000","last_clean":"2024-09-16T10:30:16.062076+0000","last_became_active":"2024-09-16T10:26:28.556509+0000","last_became_peered":"2024-09-16T10:26:28.556509+0000","last_unstale":"2024-09-16T10:30:16.062076+0000","last_undegraded":"2024-09-16T10:30:16.062076+0000","last_fullsized":"2024-09-16T10:30:16.062076+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.591170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.591170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:12.070771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":948,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069842+0000","last_change":"2024-09-16T10:30:04.838255+0000","last_active":"2024-09-16T10:30:16.069842+0000","last_peered":"2024-09-16T10:30:16.069842+0000","last_clean":"2024-09-16T10:30:16.069842+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:30:16.069842+0000","last_undegraded":"2024-09-16T10:30:16.069842+0000","last_fullsized":"2024-09-16T10:30:16.069842+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.838182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.838182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:09.124154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":842,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.077394+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:30:16.077394+0000","last_peered":"2024-09-16T10:30:16.077394+0000","last_clean":"2024-09-16T10:30:16.077394+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:30:16.077394+0000","last_undegraded":"2024-09-16T10:30:16.077394+0000","last_fullsized":"2024-09-16T10:30:16.077394+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:53.512693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":927,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069433+0000","last_change":"2024-09-16T10:28:28.182357+0000","last_active":"2024-09-16T10:30:16.069433+0000","last_peered":"2024-09-16T10:30:16.069433+0000","last_clean":"2024-09-16T10:30:16.069433+0000","last_became_active":"2024-09-16T10:28:28.182043+0000","last_became_peered":"2024-09-16T10:28:28.182043+0000","last_unstale":"2024-09-16T10:30:16.069433+0000","last_undegraded":"2024-09-16T10:30:16.069433+0000","last_fullsized":"2024-09-16T10:30:16.069433+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.834823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.834823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.037375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":922,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070366+0000","last_change":"2024-09-16T10:28:28.090355+0000","last_active":"2024-09-16T10:30:16.070366+0000","last_peered":"2024-09-16T10:30:16.070366+0000","last_clean":"2024-09-16T10:30:16.070366+0000","last_became_active":"2024-09-16T10:28:28.089459+0000","last_became_peered":"2024-09-16T10:28:28.089459+0000","last_unstale":"2024-09-16T10:30:16.070366+0000","last_undegraded":"2024-09-16T10:30:16.070366+0000","last_fullsized":"2024-09-16T10:30:16.070366+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:08.436243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.436243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:00.119133+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":900,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069831+0000","last_change":"2024-09-16T10:30:13.941093+0000","last_active":"2024-09-16T10:30:16.069831+0000","last_peered":"2024-09-16T10:30:16.069831+0000","last_clean":"2024-09-16T10:30:16.069831+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:30:16.069831+0000","last_undegraded":"2024-09-16T10:30:16.069831+0000","last_fullsized":"2024-09-16T10:30:16.069831+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:13.941032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:13.941032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:18.622886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":822,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074046+0000","last_change":"2024-09-16T10:26:28.557316+0000","last_active":"2024-09-16T10:30:16.074046+0000","last_peered":"2024-09-16T10:30:16.074046+0000","last_clean":"2024-09-16T10:30:16.074046+0000","last_became_active":"2024-09-16T10:26:28.556933+0000","last_became_peered":"2024-09-16T10:26:28.556933+0000","last_unstale":"2024-09-16T10:30:16.074046+0000","last_undegraded":"2024-09-16T10:30:16.074046+0000","last_fullsized":"2024-09-16T10:30:16.074046+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:24:37.134640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:24:37.134640+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.715967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":763,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.059317+0000","last_change":"2024-09-16T10:29:23.841536+0000","last_active":"2024-09-16T10:30:16.059317+0000","last_peered":"2024-09-16T10:30:16.059317+0000","last_clean":"2024-09-16T10:30:16.059317+0000","last_became_active":"2024-09-16T10:28:28.071768+0000","last_became_peered":"2024-09-16T10:28:28.071768+0000","last_unstale":"2024-09-16T10:30:16.059317+0000","last_undegraded":"2024-09-16T10:30:16.059317+0000","last_fullsized":"2024-09-16T10:30:16.059317+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.841474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.841474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.210925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":927,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069231+0000","last_change":"2024-09-16T10:29:24.922917+0000","last_active":"2024-09-16T10:30:16.069231+0000","last_peered":"2024-09-16T10:30:16.069231+0000","last_clean":"2024-09-16T10:30:16.069231+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:30:16.069231+0000","last_undegraded":"2024-09-16T10:30:16.069231+0000","last_fullsized":"2024-09-16T10:30:16.069231+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.922862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.922862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:45.215312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":240,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063724+0000","last_change":"2024-09-16T10:29:16.166206+0000","last_active":"2024-09-16T10:30:16.063724+0000","last_peered":"2024-09-16T10:30:16.063724+0000","last_clean":"2024-09-16T10:30:16.063724+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:30:16.063724+0000","last_undegraded":"2024-09-16T10:30:16.063724+0000","last_fullsized":"2024-09-16T10:30:16.063724+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:16.166144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:16.166144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:19.152416+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":918,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.072526+0000","last_change":"2024-09-16T10:28:28.081914+0000","last_active":"2024-09-16T10:30:16.072526+0000","last_peered":"2024-09-16T10:30:16.072526+0000","last_clean":"2024-09-16T10:30:16.072526+0000","last_became_active":"2024-09-16T10:28:28.081584+0000","last_became_peered":"2024-09-16T10:28:28.081584+0000","last_unstale":"2024-09-16T10:30:16.072526+0000","last_undegraded":"2024-09-16T10:30:16.072526+0000","last_fullsized":"2024-09-16T10:30:16.072526+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:27.210465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:22:27.210465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.031297+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":484,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065623+0000","last_change":"2024-09-16T10:26:17.566065+0000","last_active":"2024-09-16T10:30:16.065623+0000","last_peered":"2024-09-16T10:30:16.065623+0000","last_clean":"2024-09-16T10:30:16.065623+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:30:16.065623+0000","last_undegraded":"2024-09-16T10:30:16.065623+0000","last_fullsized":"2024-09-16T10:30:16.065623+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:17.565970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:17.565970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.478474+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":916,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064092+0000","last_change":"2024-09-16T10:30:09.918436+0000","last_active":"2024-09-16T10:30:16.064092+0000","last_peered":"2024-09-16T10:30:16.064092+0000","last_clean":"2024-09-16T10:30:16.064092+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:30:16.064092+0000","last_undegraded":"2024-09-16T10:30:16.064092+0000","last_fullsized":"2024-09-16T10:30:16.064092+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:09.918371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:09.918371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:21.941703+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":868,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075372+0000","last_change":"2024-09-16T10:27:18.318288+0000","last_active":"2024-09-16T10:30:16.075372+0000","last_peered":"2024-09-16T10:30:16.075372+0000","last_clean":"2024-09-16T10:30:16.075372+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:30:16.075372+0000","last_undegraded":"2024-09-16T10:30:16.075372+0000","last_fullsized":"2024-09-16T10:30:16.075372+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:18.318214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:18.318214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":704,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063504+0000","last_change":"2024-09-16T10:26:28.551407+0000","last_active":"2024-09-16T10:30:16.063504+0000","last_peered":"2024-09-16T10:30:16.063504+0000","last_clean":"2024-09-16T10:30:16.063504+0000","last_became_active":"2024-09-16T10:26:28.551058+0000","last_became_peered":"2024-09-16T10:26:28.551058+0000","last_unstale":"2024-09-16T10:30:16.063504+0000","last_undegraded":"2024-09-16T10:30:16.063504+0000","last_fullsized":"2024-09-16T10:30:16.063504+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:56.830978+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":977,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074813+0000","last_change":"2024-09-16T10:29:28.855774+0000","last_active":"2024-09-16T10:30:16.074813+0000","last_peered":"2024-09-16T10:30:16.074813+0000","last_clean":"2024-09-16T10:30:16.074813+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:30:16.074813+0000","last_undegraded":"2024-09-16T10:30:16.074813+0000","last_fullsized":"2024-09-16T10:30:16.074813+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:29:28.855720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.855720+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:34.642453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":712,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074268+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:30:16.074268+0000","last_peered":"2024-09-16T10:30:16.074268+0000","last_clean":"2024-09-16T10:30:16.074268+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:30:16.074268+0000","last_undegraded":"2024-09-16T10:30:16.074268+0000","last_fullsized":"2024-09-16T10:30:16.074268+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:02.823725+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":327,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069694+0000","last_change":"2024-09-16T10:28:28.095829+0000","last_active":"2024-09-16T10:30:16.069694+0000","last_peered":"2024-09-16T10:30:16.069694+0000","last_clean":"2024-09-16T10:30:16.069694+0000","last_became_active":"2024-09-16T10:28:28.095455+0000","last_became_peered":"2024-09-16T10:28:28.095455+0000","last_unstale":"2024-09-16T10:30:16.069694+0000","last_undegraded":"2024-09-16T10:30:16.069694+0000","last_fullsized":"2024-09-16T10:30:16.069694+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:25:18.973746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.973746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.012515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":931,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062842+0000","last_change":"2024-09-16T10:30:12.928363+0000","last_active":"2024-09-16T10:30:16.062842+0000","last_peered":"2024-09-16T10:30:16.062842+0000","last_clean":"2024-09-16T10:30:16.062842+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:30:16.062842+0000","last_undegraded":"2024-09-16T10:30:16.062842+0000","last_fullsized":"2024-09-16T10:30:16.062842+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:30:12.928274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.928274+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:28.098737+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":962,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071353+0000","last_change":"2024-09-16T10:29:29.106834+0000","last_active":"2024-09-16T10:30:16.071353+0000","last_peered":"2024-09-16T10:30:16.071353+0000","last_clean":"2024-09-16T10:30:16.071353+0000","last_became_active":"2024-09-16T10:28:28.082261+0000","last_became_peered":"2024-09-16T10:28:28.082261+0000","last_unstale":"2024-09-16T10:30:16.071353+0000","last_undegraded":"2024-09-16T10:30:16.071353+0000","last_fullsized":"2024-09-16T10:30:16.071353+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:29.106770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.106770+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.311277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"442'10","reported_seq":1140,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063574+0000","last_change":"2024-09-16T10:29:37.281159+0000","last_active":"2024-09-16T10:30:16.063574+0000","last_peered":"2024-09-16T10:30:16.063574+0000","last_clean":"2024-09-16T10:30:16.063574+0000","last_became_active":"2024-09-16T10:28:28.089383+0000","last_became_peered":"2024-09-16T10:28:28.089383+0000","last_unstale":"2024-09-16T10:30:16.063574+0000","last_undegraded":"2024-09-16T10:30:16.063574+0000","last_fullsized":"2024-09-16T10:30:16.063574+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.602654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"492'4169","reported_seq":4212,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.024454+0000","last_change":"2024-09-16T10:29:35.280092+0000","last_active":"2024-09-16T10:30:18.024454+0000","last_peered":"2024-09-16T10:30:18.024454+0000","last_clean":"2024-09-16T10:30:18.024454+0000","last_became_active":"2024-09-16T10:29:35.279496+0000","last_became_peered":"2024-09-16T10:29:35.279496+0000","last_unstale":"2024-09-16T10:30:18.024454+0000","last_undegraded":"2024-09-16T10:30:18.024454+0000","last_fullsized":"2024-09-16T10:30:18.024454+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:55.065966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273219584,"num_objects":4169,"num_object_clones":0,"num_object_copies":12507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4169,"num_write_kb":266816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":770,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069663+0000","last_change":"2024-09-16T10:29:21.013569+0000","last_active":"2024-09-16T10:30:16.069663+0000","last_peered":"2024-09-16T10:30:16.069663+0000","last_clean":"2024-09-16T10:30:16.069663+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:30:16.069663+0000","last_undegraded":"2024-09-16T10:30:16.069663+0000","last_fullsized":"2024-09-16T10:30:16.069663+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:21.013503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.013503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.013620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":956,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.067238+0000","last_change":"2024-09-16T10:27:12.407528+0000","last_active":"2024-09-16T10:30:16.067238+0000","last_peered":"2024-09-16T10:30:16.067238+0000","last_clean":"2024-09-16T10:30:16.067238+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:30:16.067238+0000","last_undegraded":"2024-09-16T10:30:16.067238+0000","last_fullsized":"2024-09-16T10:30:16.067238+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:27:12.407441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.407441+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":934,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.076611+0000","last_change":"2024-09-16T10:28:28.066368+0000","last_active":"2024-09-16T10:30:16.076611+0000","last_peered":"2024-09-16T10:30:16.076611+0000","last_clean":"2024-09-16T10:30:16.076611+0000","last_became_active":"2024-09-16T10:28:28.065738+0000","last_became_peered":"2024-09-16T10:28:28.065738+0000","last_unstale":"2024-09-16T10:30:16.076611+0000","last_undegraded":"2024-09-16T10:30:16.076611+0000","last_fullsized":"2024-09-16T10:30:16.076611+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:33.376990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:33.376990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.851080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":320,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.857785+0000","last_change":"2024-09-16T10:30:16.857737+0000","last_active":"2024-09-16T10:30:16.857785+0000","last_peered":"2024-09-16T10:30:16.857785+0000","last_clean":"2024-09-16T10:30:16.857785+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:30:16.857785+0000","last_undegraded":"2024-09-16T10:30:16.857785+0000","last_fullsized":"2024-09-16T10:30:16.857785+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:30:16.857676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.857676+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:33.191176+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":921,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069194+0000","last_change":"2024-09-16T10:30:14.925349+0000","last_active":"2024-09-16T10:30:16.069194+0000","last_peered":"2024-09-16T10:30:16.069194+0000","last_clean":"2024-09-16T10:30:16.069194+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:30:16.069194+0000","last_undegraded":"2024-09-16T10:30:16.069194+0000","last_fullsized":"2024-09-16T10:30:16.069194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:14.925284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:14.925284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:39.491806+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"492'4177","reported_seq":4220,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.010915+0000","last_change":"2024-09-16T10:29:35.279709+0000","last_active":"2024-09-16T10:30:18.010915+0000","last_peered":"2024-09-16T10:30:18.010915+0000","last_clean":"2024-09-16T10:30:18.010915+0000","last_became_active":"2024-09-16T10:29:35.279416+0000","last_became_peered":"2024-09-16T10:29:35.279416+0000","last_unstale":"2024-09-16T10:30:18.010915+0000","last_undegraded":"2024-09-16T10:30:18.010915+0000","last_fullsized":"2024-09-16T10:30:18.010915+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4177,"log_dups_size":0,"ondisk_log_size":4177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.688136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273743872,"num_objects":4177,"num_object_clones":0,"num_object_copies":12531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4177,"num_write_kb":267328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":348,"reported_epoch":492,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:30:18.544624+0000","last_change":"2024-09-16T10:30:18.541589+0000","last_active":"2024-09-16T10:30:18.544624+0000","last_peered":"2024-09-16T10:30:18.544624+0000","last_clean":"2024-09-16T10:30:18.544624+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:30:18.544624+0000","last_undegraded":"2024-09-16T10:30:18.544624+0000","last_fullsized":"2024-09-16T10:30:18.544624+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:27:17.482427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:27:17.482427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":305,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.061573+0000","last_change":"2024-09-16T10:26:44.445756+0000","last_active":"2024-09-16T10:30:16.061573+0000","last_peered":"2024-09-16T10:30:16.061573+0000","last_clean":"2024-09-16T10:30:16.061573+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:30:16.061573+0000","last_undegraded":"2024-09-16T10:30:16.061573+0000","last_fullsized":"2024-09-16T10:30:16.061573+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:26:44.445661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.445661+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:02.579663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":899,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075714+0000","last_change":"2024-09-16T10:28:28.098490+0000","last_active":"2024-09-16T10:30:16.075714+0000","last_peered":"2024-09-16T10:30:16.075714+0000","last_clean":"2024-09-16T10:30:16.075714+0000","last_became_active":"2024-09-16T10:28:28.097354+0000","last_became_peered":"2024-09-16T10:28:28.097354+0000","last_unstale":"2024-09-16T10:30:16.075714+0000","last_undegraded":"2024-09-16T10:30:16.075714+0000","last_fullsized":"2024-09-16T10:30:16.075714+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:36.404383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:36.404383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:52.778100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":666,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062239+0000","last_change":"2024-09-16T10:28:28.074218+0000","last_active":"2024-09-16T10:30:16.062239+0000","last_peered":"2024-09-16T10:30:16.062239+0000","last_clean":"2024-09-16T10:30:16.062239+0000","last_became_active":"2024-09-16T10:28:28.073750+0000","last_became_peered":"2024-09-16T10:28:28.073750+0000","last_unstale":"2024-09-16T10:30:16.062239+0000","last_undegraded":"2024-09-16T10:30:16.062239+0000","last_fullsized":"2024-09-16T10:30:16.062239+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.521586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.521586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:00.557073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":724,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.077294+0000","last_change":"2024-09-16T10:27:40.298188+0000","last_active":"2024-09-16T10:30:16.077294+0000","last_peered":"2024-09-16T10:30:16.077294+0000","last_clean":"2024-09-16T10:30:16.077294+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:30:16.077294+0000","last_undegraded":"2024-09-16T10:30:16.077294+0000","last_fullsized":"2024-09-16T10:30:16.077294+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"492'4187","reported_seq":4230,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.053798+0000","last_change":"2024-09-16T10:29:35.280049+0000","last_active":"2024-09-16T10:30:18.053798+0000","last_peered":"2024-09-16T10:30:18.053798+0000","last_clean":"2024-09-16T10:30:18.053798+0000","last_became_active":"2024-09-16T10:29:35.279303+0000","last_became_peered":"2024-09-16T10:29:35.279303+0000","last_unstale":"2024-09-16T10:30:18.053798+0000","last_undegraded":"2024-09-16T10:30:18.053798+0000","last_fullsized":"2024-09-16T10:30:18.053798+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.356324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274399232,"num_objects":4187,"num_object_clones":0,"num_object_copies":12561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4187,"num_write_kb":267968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":824,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074561+0000","last_change":"2024-09-16T10:29:36.701757+0000","last_active":"2024-09-16T10:30:16.074561+0000","last_peered":"2024-09-16T10:30:16.074561+0000","last_clean":"2024-09-16T10:30:16.074561+0000","last_became_active":"2024-09-16T10:28:28.095330+0000","last_became_peered":"2024-09-16T10:28:28.095330+0000","last_unstale":"2024-09-16T10:30:16.074561+0000","last_undegraded":"2024-09-16T10:30:16.074561+0000","last_fullsized":"2024-09-16T10:30:16.074561+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:36.701703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.701703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.846267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":869,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069261+0000","last_change":"2024-09-16T10:30:12.893400+0000","last_active":"2024-09-16T10:30:16.069261+0000","last_peered":"2024-09-16T10:30:16.069261+0000","last_clean":"2024-09-16T10:30:16.069261+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:30:16.069261+0000","last_undegraded":"2024-09-16T10:30:16.069261+0000","last_fullsized":"2024-09-16T10:30:16.069261+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:30:12.893340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.893340+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:28.993798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":917,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071625+0000","last_change":"2024-09-16T10:27:44.624946+0000","last_active":"2024-09-16T10:30:16.071625+0000","last_peered":"2024-09-16T10:30:16.071625+0000","last_clean":"2024-09-16T10:30:16.071625+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:30:16.071625+0000","last_undegraded":"2024-09-16T10:30:16.071625+0000","last_fullsized":"2024-09-16T10:30:16.071625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":1128,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073840+0000","last_change":"2024-09-16T10:27:24.281038+0000","last_active":"2024-09-16T10:30:16.073840+0000","last_peered":"2024-09-16T10:30:16.073840+0000","last_clean":"2024-09-16T10:30:16.073840+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:30:16.073840+0000","last_undegraded":"2024-09-16T10:30:16.073840+0000","last_fullsized":"2024-09-16T10:30:16.073840+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:27:24.280960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:24.280960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":775,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062927+0000","last_change":"2024-09-16T10:29:20.267647+0000","last_active":"2024-09-16T10:30:16.062927+0000","last_peered":"2024-09-16T10:30:16.062927+0000","last_clean":"2024-09-16T10:30:16.062927+0000","last_became_active":"2024-09-16T10:28:28.071989+0000","last_became_peered":"2024-09-16T10:28:28.071989+0000","last_unstale":"2024-09-16T10:30:16.062927+0000","last_undegraded":"2024-09-16T10:30:16.062927+0000","last_fullsized":"2024-09-16T10:30:16.062927+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.267597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.267597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:29.879063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"492'4082","reported_seq":4125,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:19.216477+0000","last_change":"2024-09-16T10:29:35.170854+0000","last_active":"2024-09-16T10:30:19.216477+0000","last_peered":"2024-09-16T10:30:19.216477+0000","last_clean":"2024-09-16T10:30:19.216477+0000","last_became_active":"2024-09-16T10:29:35.170400+0000","last_became_peered":"2024-09-16T10:29:35.170400+0000","last_unstale":"2024-09-16T10:30:19.216477+0000","last_undegraded":"2024-09-16T10:30:19.216477+0000","last_fullsized":"2024-09-16T10:30:19.216477+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.870813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267517952,"num_objects":4082,"num_object_clones":0,"num_object_copies":12246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4082,"num_write_kb":261248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":874,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.061926+0000","last_change":"2024-09-16T10:29:29.272700+0000","last_active":"2024-09-16T10:30:16.061926+0000","last_peered":"2024-09-16T10:30:16.061926+0000","last_clean":"2024-09-16T10:30:16.061926+0000","last_became_active":"2024-09-16T10:28:28.089337+0000","last_became_peered":"2024-09-16T10:28:28.089337+0000","last_unstale":"2024-09-16T10:30:16.061926+0000","last_undegraded":"2024-09-16T10:30:16.061926+0000","last_fullsized":"2024-09-16T10:30:16.061926+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.272650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.272650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.748781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":959,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063260+0000","last_change":"2024-09-16T10:29:25.663951+0000","last_active":"2024-09-16T10:30:16.063260+0000","last_peered":"2024-09-16T10:30:16.063260+0000","last_clean":"2024-09-16T10:30:16.063260+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:30:16.063260+0000","last_undegraded":"2024-09-16T10:30:16.063260+0000","last_fullsized":"2024-09-16T10:30:16.063260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.992176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":939,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069924+0000","last_change":"2024-09-16T10:29:27.891418+0000","last_active":"2024-09-16T10:30:16.069924+0000","last_peered":"2024-09-16T10:30:16.069924+0000","last_clean":"2024-09-16T10:30:16.069924+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:30:16.069924+0000","last_undegraded":"2024-09-16T10:30:16.069924+0000","last_fullsized":"2024-09-16T10:30:16.069924+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.891362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.891362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.339178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1117,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.072123+0000","last_change":"2024-09-16T10:30:08.183366+0000","last_active":"2024-09-16T10:30:16.072123+0000","last_peered":"2024-09-16T10:30:16.072123+0000","last_clean":"2024-09-16T10:30:16.072123+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:30:16.072123+0000","last_undegraded":"2024-09-16T10:30:16.072123+0000","last_fullsized":"2024-09-16T10:30:16.072123+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:08.182949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.182949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:30.974057+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":935,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070470+0000","last_change":"2024-09-16T10:30:12.258492+0000","last_active":"2024-09-16T10:30:16.070470+0000","last_peered":"2024-09-16T10:30:16.070470+0000","last_clean":"2024-09-16T10:30:16.070470+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:30:16.070470+0000","last_undegraded":"2024-09-16T10:30:16.070470+0000","last_fullsized":"2024-09-16T10:30:16.070470+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:12.258426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.258426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:29.743926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"492'4099","reported_seq":4142,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.056233+0000","last_change":"2024-09-16T10:29:35.168446+0000","last_active":"2024-09-16T10:30:18.056233+0000","last_peered":"2024-09-16T10:30:18.056233+0000","last_clean":"2024-09-16T10:30:18.056233+0000","last_became_active":"2024-09-16T10:29:35.168175+0000","last_became_peered":"2024-09-16T10:29:35.168175+0000","last_unstale":"2024-09-16T10:30:18.056233+0000","last_undegraded":"2024-09-16T10:30:18.056233+0000","last_fullsized":"2024-09-16T10:30:18.056233+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.317930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268632064,"num_objects":4099,"num_object_clones":0,"num_object_copies":12297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":262336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":876,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063630+0000","last_change":"2024-09-16T10:30:11.662236+0000","last_active":"2024-09-16T10:30:16.063630+0000","last_peered":"2024-09-16T10:30:16.063630+0000","last_clean":"2024-09-16T10:30:16.063630+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:30:16.063630+0000","last_undegraded":"2024-09-16T10:30:16.063630+0000","last_fullsized":"2024-09-16T10:30:16.063630+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:11.662185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.662185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:23.457770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":983,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.072306+0000","last_change":"2024-09-16T10:29:27.111777+0000","last_active":"2024-09-16T10:30:16.072306+0000","last_peered":"2024-09-16T10:30:16.072306+0000","last_clean":"2024-09-16T10:30:16.072306+0000","last_became_active":"2024-09-16T10:28:28.112972+0000","last_became_peered":"2024-09-16T10:28:28.112972+0000","last_unstale":"2024-09-16T10:30:16.072306+0000","last_undegraded":"2024-09-16T10:30:16.072306+0000","last_fullsized":"2024-09-16T10:30:16.072306+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:29:27.111715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.111715+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.694926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":939,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065999+0000","last_change":"2024-09-16T10:29:26.686298+0000","last_active":"2024-09-16T10:30:16.065999+0000","last_peered":"2024-09-16T10:30:16.065999+0000","last_clean":"2024-09-16T10:30:16.065999+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:30:16.065999+0000","last_undegraded":"2024-09-16T10:30:16.065999+0000","last_fullsized":"2024-09-16T10:30:16.065999+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.882005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":924,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.061790+0000","last_change":"2024-09-16T10:25:31.451289+0000","last_active":"2024-09-16T10:30:16.061790+0000","last_peered":"2024-09-16T10:30:16.061790+0000","last_clean":"2024-09-16T10:30:16.061790+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:30:16.061790+0000","last_undegraded":"2024-09-16T10:30:16.061790+0000","last_fullsized":"2024-09-16T10:30:16.061790+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:31.451244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:31.451244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.466187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":919,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062558+0000","last_change":"2024-09-16T10:27:41.587855+0000","last_active":"2024-09-16T10:30:16.062558+0000","last_peered":"2024-09-16T10:30:16.062558+0000","last_clean":"2024-09-16T10:30:16.062558+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:30:16.062558+0000","last_undegraded":"2024-09-16T10:30:16.062558+0000","last_fullsized":"2024-09-16T10:30:16.062558+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"492'4132","reported_seq":4175,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.155036+0000","last_change":"2024-09-16T10:29:35.171242+0000","last_active":"2024-09-16T10:30:18.155036+0000","last_peered":"2024-09-16T10:30:18.155036+0000","last_clean":"2024-09-16T10:30:18.155036+0000","last_became_active":"2024-09-16T10:29:35.171007+0000","last_became_peered":"2024-09-16T10:29:35.171007+0000","last_unstale":"2024-09-16T10:30:18.155036+0000","last_undegraded":"2024-09-16T10:30:18.155036+0000","last_fullsized":"2024-09-16T10:30:18.155036+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4132,"log_dups_size":0,"ondisk_log_size":4132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.125636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270794752,"num_objects":4132,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":264448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":814,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069115+0000","last_change":"2024-09-16T10:26:09.992070+0000","last_active":"2024-09-16T10:30:16.069115+0000","last_peered":"2024-09-16T10:30:16.069115+0000","last_clean":"2024-09-16T10:30:16.069115+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:30:16.069115+0000","last_undegraded":"2024-09-16T10:30:16.069115+0000","last_fullsized":"2024-09-16T10:30:16.069115+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:09.992000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.992000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.967589+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":1262,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065792+0000","last_change":"2024-09-16T10:27:14.409646+0000","last_active":"2024-09-16T10:30:16.065792+0000","last_peered":"2024-09-16T10:30:16.065792+0000","last_clean":"2024-09-16T10:30:16.065792+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:30:16.065792+0000","last_undegraded":"2024-09-16T10:30:16.065792+0000","last_fullsized":"2024-09-16T10:30:16.065792+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:27:14.409552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:14.409552+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":240,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.072509+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:30:16.072509+0000","last_peered":"2024-09-16T10:30:16.072509+0000","last_clean":"2024-09-16T10:30:16.072509+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:30:16.072509+0000","last_undegraded":"2024-09-16T10:30:16.072509+0000","last_fullsized":"2024-09-16T10:30:16.072509+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":944,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070417+0000","last_change":"2024-09-16T10:30:11.895955+0000","last_active":"2024-09-16T10:30:16.070417+0000","last_peered":"2024-09-16T10:30:16.070417+0000","last_clean":"2024-09-16T10:30:16.070417+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:30:16.070417+0000","last_undegraded":"2024-09-16T10:30:16.070417+0000","last_fullsized":"2024-09-16T10:30:16.070417+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:30:11.895746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.895746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:41.723636+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":900,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070776+0000","last_change":"2024-09-16T10:25:37.359150+0000","last_active":"2024-09-16T10:30:16.070776+0000","last_peered":"2024-09-16T10:30:16.070776+0000","last_clean":"2024-09-16T10:30:16.070776+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:30:16.070776+0000","last_undegraded":"2024-09-16T10:30:16.070776+0000","last_fullsized":"2024-09-16T10:30:16.070776+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:37.359080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.359080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:52.955966+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":938,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075214+0000","last_change":"2024-09-16T10:30:04.877973+0000","last_active":"2024-09-16T10:30:16.075214+0000","last_peered":"2024-09-16T10:30:16.075214+0000","last_clean":"2024-09-16T10:30:16.075214+0000","last_became_active":"2024-09-16T10:28:28.096691+0000","last_became_peered":"2024-09-16T10:28:28.096691+0000","last_unstale":"2024-09-16T10:30:16.075214+0000","last_undegraded":"2024-09-16T10:30:16.075214+0000","last_fullsized":"2024-09-16T10:30:16.075214+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.877913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.877913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:10.686358+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"492'4216","reported_seq":4259,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.209508+0000","last_change":"2024-09-16T10:29:35.167788+0000","last_active":"2024-09-16T10:30:18.209508+0000","last_peered":"2024-09-16T10:30:18.209508+0000","last_clean":"2024-09-16T10:30:18.209508+0000","last_became_active":"2024-09-16T10:29:35.167515+0000","last_became_peered":"2024-09-16T10:29:35.167515+0000","last_unstale":"2024-09-16T10:30:18.209508+0000","last_undegraded":"2024-09-16T10:30:18.209508+0000","last_fullsized":"2024-09-16T10:30:18.209508+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4216,"log_dups_size":0,"ondisk_log_size":4216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.448815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276299776,"num_objects":4216,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":269824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":390,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070847+0000","last_change":"2024-09-16T10:28:28.073697+0000","last_active":"2024-09-16T10:30:16.070847+0000","last_peered":"2024-09-16T10:30:16.070847+0000","last_clean":"2024-09-16T10:30:16.070847+0000","last_became_active":"2024-09-16T10:28:28.073341+0000","last_became_peered":"2024-09-16T10:28:28.073341+0000","last_unstale":"2024-09-16T10:30:16.070847+0000","last_undegraded":"2024-09-16T10:30:16.070847+0000","last_fullsized":"2024-09-16T10:30:16.070847+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:02.365813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:02.365813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:40.028638+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":938,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.067313+0000","last_change":"2024-09-16T10:30:05.864251+0000","last_active":"2024-09-16T10:30:16.067313+0000","last_peered":"2024-09-16T10:30:16.067313+0000","last_clean":"2024-09-16T10:30:16.067313+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:30:16.067313+0000","last_undegraded":"2024-09-16T10:30:16.067313+0000","last_fullsized":"2024-09-16T10:30:16.067313+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:30:05.864159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.864159+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:16.545039+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":924,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065559+0000","last_change":"2024-09-16T10:29:20.691013+0000","last_active":"2024-09-16T10:30:16.065559+0000","last_peered":"2024-09-16T10:30:16.065559+0000","last_clean":"2024-09-16T10:30:16.065559+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:30:16.065559+0000","last_undegraded":"2024-09-16T10:30:16.065559+0000","last_fullsized":"2024-09-16T10:30:16.065559+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:41.420313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":454,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065167+0000","last_change":"2024-09-16T10:26:18.549126+0000","last_active":"2024-09-16T10:30:16.065167+0000","last_peered":"2024-09-16T10:30:16.065167+0000","last_clean":"2024-09-16T10:30:16.065167+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:30:16.065167+0000","last_undegraded":"2024-09-16T10:30:16.065167+0000","last_fullsized":"2024-09-16T10:30:16.065167+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:26:18.549005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:26:18.549005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.472864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":461,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063779+0000","last_change":"2024-09-16T10:29:24.484142+0000","last_active":"2024-09-16T10:30:16.063779+0000","last_peered":"2024-09-16T10:30:16.063779+0000","last_clean":"2024-09-16T10:30:16.063779+0000","last_became_active":"2024-09-16T10:28:28.081619+0000","last_became_peered":"2024-09-16T10:28:28.081619+0000","last_unstale":"2024-09-16T10:30:16.063779+0000","last_undegraded":"2024-09-16T10:30:16.063779+0000","last_fullsized":"2024-09-16T10:30:16.063779+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.484086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.484086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.572572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":315,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063992+0000","last_change":"2024-09-16T10:28:28.073905+0000","last_active":"2024-09-16T10:30:16.063992+0000","last_peered":"2024-09-16T10:30:16.063992+0000","last_clean":"2024-09-16T10:30:16.063992+0000","last_became_active":"2024-09-16T10:28:28.073357+0000","last_became_peered":"2024-09-16T10:28:28.073357+0000","last_unstale":"2024-09-16T10:30:16.063992+0000","last_undegraded":"2024-09-16T10:30:16.063992+0000","last_fullsized":"2024-09-16T10:30:16.063992+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:32.463551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:32.463551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.370256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"492'4144","reported_seq":4187,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.165889+0000","last_change":"2024-09-16T10:29:35.171912+0000","last_active":"2024-09-16T10:30:18.165889+0000","last_peered":"2024-09-16T10:30:18.165889+0000","last_clean":"2024-09-16T10:30:18.165889+0000","last_became_active":"2024-09-16T10:29:35.171449+0000","last_became_peered":"2024-09-16T10:29:35.171449+0000","last_unstale":"2024-09-16T10:30:18.165889+0000","last_undegraded":"2024-09-16T10:30:18.165889+0000","last_fullsized":"2024-09-16T10:30:18.165889+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4144,"log_dups_size":0,"ondisk_log_size":4144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:38.873549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271581184,"num_objects":4144,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4144,"num_write_kb":265216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":810,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.866299+0000","last_change":"2024-09-16T10:30:16.866299+0000","last_active":"2024-09-16T10:30:16.866299+0000","last_peered":"2024-09-16T10:30:16.866299+0000","last_clean":"2024-09-16T10:30:16.866299+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:30:16.866299+0000","last_undegraded":"2024-09-16T10:30:16.866299+0000","last_fullsized":"2024-09-16T10:30:16.866299+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:16.862456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.862456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:41.111359+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":1014,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071443+0000","last_change":"2024-09-16T10:27:09.622319+0000","last_active":"2024-09-16T10:30:16.071443+0000","last_peered":"2024-09-16T10:30:16.071443+0000","last_clean":"2024-09-16T10:30:16.071443+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:30:16.071443+0000","last_undegraded":"2024-09-16T10:30:16.071443+0000","last_fullsized":"2024-09-16T10:30:16.071443+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:27:09.622239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:09.622239+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":949,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075917+0000","last_change":"2024-09-16T10:30:15.229737+0000","last_active":"2024-09-16T10:30:16.075917+0000","last_peered":"2024-09-16T10:30:16.075917+0000","last_clean":"2024-09-16T10:30:16.075917+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:30:16.075917+0000","last_undegraded":"2024-09-16T10:30:16.075917+0000","last_fullsized":"2024-09-16T10:30:16.075917+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.229669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.229669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:33.889521+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":672,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.483495+0000","last_change":"2024-09-16T10:29:30.751685+0000","last_active":"2024-09-16T10:30:16.483495+0000","last_peered":"2024-09-16T10:30:16.483495+0000","last_clean":"2024-09-16T10:30:16.483495+0000","last_became_active":"2024-09-16T10:28:28.070293+0000","last_became_peered":"2024-09-16T10:28:28.070293+0000","last_unstale":"2024-09-16T10:30:16.483495+0000","last_undegraded":"2024-09-16T10:30:16.483495+0000","last_fullsized":"2024-09-16T10:30:16.483495+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:29:30.751623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.751623+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:34.435816+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":320,"num_read_kb":315,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":933,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070355+0000","last_change":"2024-09-16T10:30:08.854974+0000","last_active":"2024-09-16T10:30:16.070355+0000","last_peered":"2024-09-16T10:30:16.070355+0000","last_clean":"2024-09-16T10:30:16.070355+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:30:16.070355+0000","last_undegraded":"2024-09-16T10:30:16.070355+0000","last_fullsized":"2024-09-16T10:30:16.070355+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.854924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.854924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:28.369520+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":929,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065561+0000","last_change":"2024-09-16T10:27:19.374485+0000","last_active":"2024-09-16T10:30:16.065561+0000","last_peered":"2024-09-16T10:30:16.065561+0000","last_clean":"2024-09-16T10:30:16.065561+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:30:16.065561+0000","last_undegraded":"2024-09-16T10:30:16.065561+0000","last_fullsized":"2024-09-16T10:30:16.065561+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"492'4186","reported_seq":4228,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.216104+0000","last_change":"2024-09-16T10:29:35.169178+0000","last_active":"2024-09-16T10:30:18.216104+0000","last_peered":"2024-09-16T10:30:18.216104+0000","last_clean":"2024-09-16T10:30:18.216104+0000","last_became_active":"2024-09-16T10:29:35.168851+0000","last_became_peered":"2024-09-16T10:29:35.168851+0000","last_unstale":"2024-09-16T10:30:18.216104+0000","last_undegraded":"2024-09-16T10:30:18.216104+0000","last_fullsized":"2024-09-16T10:30:18.216104+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.747588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274333696,"num_objects":4186,"num_object_clones":0,"num_object_copies":12558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4186,"num_write_kb":267904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"442'30","reported_seq":1157,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064074+0000","last_change":"2024-09-16T10:29:18.202869+0000","last_active":"2024-09-16T10:30:16.064074+0000","last_peered":"2024-09-16T10:30:16.064074+0000","last_clean":"2024-09-16T10:30:16.064074+0000","last_became_active":"2024-09-16T10:28:28.072900+0000","last_became_peered":"2024-09-16T10:28:28.072900+0000","last_unstale":"2024-09-16T10:30:16.064074+0000","last_undegraded":"2024-09-16T10:30:16.064074+0000","last_fullsized":"2024-09-16T10:30:16.064074+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:29:18.202771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:18.202771+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:45.201112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":888,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.077590+0000","last_change":"2024-09-16T10:26:47.493915+0000","last_active":"2024-09-16T10:30:16.077590+0000","last_peered":"2024-09-16T10:30:16.077590+0000","last_clean":"2024-09-16T10:30:16.077590+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:30:16.077590+0000","last_undegraded":"2024-09-16T10:30:16.077590+0000","last_fullsized":"2024-09-16T10:30:16.077590+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:47.493848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:47.493848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":390,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070658+0000","last_change":"2024-09-16T10:28:28.090815+0000","last_active":"2024-09-16T10:30:16.070658+0000","last_peered":"2024-09-16T10:30:16.070658+0000","last_clean":"2024-09-16T10:30:16.070658+0000","last_became_active":"2024-09-16T10:28:28.090031+0000","last_became_peered":"2024-09-16T10:28:28.090031+0000","last_unstale":"2024-09-16T10:30:16.070658+0000","last_undegraded":"2024-09-16T10:30:16.070658+0000","last_fullsized":"2024-09-16T10:30:16.070658+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:01.375714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:01.375714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.945193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":786,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064952+0000","last_change":"2024-09-16T10:28:28.072722+0000","last_active":"2024-09-16T10:30:16.064952+0000","last_peered":"2024-09-16T10:30:16.064952+0000","last_clean":"2024-09-16T10:30:16.064952+0000","last_became_active":"2024-09-16T10:28:28.072192+0000","last_became_peered":"2024-09-16T10:28:28.072192+0000","last_unstale":"2024-09-16T10:30:16.064952+0000","last_undegraded":"2024-09-16T10:30:16.064952+0000","last_fullsized":"2024-09-16T10:30:16.064952+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:35.591323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.591323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.446390+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":913,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070258+0000","last_change":"2024-09-16T10:26:19.084467+0000","last_active":"2024-09-16T10:30:16.070258+0000","last_peered":"2024-09-16T10:30:16.070258+0000","last_clean":"2024-09-16T10:30:16.070258+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:30:16.070258+0000","last_undegraded":"2024-09-16T10:30:16.070258+0000","last_fullsized":"2024-09-16T10:30:16.070258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:19.084409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:19.084409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.011496+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"492'4205","reported_seq":4248,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.572874+0000","last_change":"2024-09-16T10:29:35.171216+0000","last_active":"2024-09-16T10:30:18.572874+0000","last_peered":"2024-09-16T10:30:18.572874+0000","last_clean":"2024-09-16T10:30:18.572874+0000","last_became_active":"2024-09-16T10:29:35.170866+0000","last_became_peered":"2024-09-16T10:29:35.170866+0000","last_unstale":"2024-09-16T10:30:18.572874+0000","last_undegraded":"2024-09-16T10:30:18.572874+0000","last_fullsized":"2024-09-16T10:30:18.572874+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.526803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275578880,"num_objects":4205,"num_object_clones":0,"num_object_copies":12615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4205,"num_write_kb":269120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":810,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063774+0000","last_change":"2024-09-16T10:29:27.703960+0000","last_active":"2024-09-16T10:30:16.063774+0000","last_peered":"2024-09-16T10:30:16.063774+0000","last_clean":"2024-09-16T10:30:16.063774+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:30:16.063774+0000","last_undegraded":"2024-09-16T10:30:16.063774+0000","last_fullsized":"2024-09-16T10:30:16.063774+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:27.703908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.703908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.239882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":853,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063822+0000","last_change":"2024-09-16T10:26:28.556079+0000","last_active":"2024-09-16T10:30:16.063822+0000","last_peered":"2024-09-16T10:30:16.063822+0000","last_clean":"2024-09-16T10:30:16.063822+0000","last_became_active":"2024-09-16T10:26:28.554604+0000","last_became_peered":"2024-09-16T10:26:28.554604+0000","last_unstale":"2024-09-16T10:30:16.063822+0000","last_undegraded":"2024-09-16T10:30:16.063822+0000","last_fullsized":"2024-09-16T10:30:16.063822+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:26:10.386128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.386128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.578412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":730,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064720+0000","last_change":"2024-09-16T10:29:23.474783+0000","last_active":"2024-09-16T10:30:16.064720+0000","last_peered":"2024-09-16T10:30:16.064720+0000","last_clean":"2024-09-16T10:30:16.064720+0000","last_became_active":"2024-09-16T10:28:28.071386+0000","last_became_peered":"2024-09-16T10:28:28.071386+0000","last_unstale":"2024-09-16T10:30:16.064720+0000","last_undegraded":"2024-09-16T10:30:16.064720+0000","last_fullsized":"2024-09-16T10:30:16.064720+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.474733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.474733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:52.069257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":925,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070108+0000","last_change":"2024-09-16T10:29:30.955964+0000","last_active":"2024-09-16T10:30:16.070108+0000","last_peered":"2024-09-16T10:30:16.070108+0000","last_clean":"2024-09-16T10:30:16.070108+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:30:16.070108+0000","last_undegraded":"2024-09-16T10:30:16.070108+0000","last_fullsized":"2024-09-16T10:30:16.070108+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:30.955877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.955877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:51.333008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":860,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074468+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:30:16.074468+0000","last_peered":"2024-09-16T10:30:16.074468+0000","last_clean":"2024-09-16T10:30:16.074468+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:30:16.074468+0000","last_undegraded":"2024-09-16T10:30:16.074468+0000","last_fullsized":"2024-09-16T10:30:16.074468+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.037961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"492'4192","reported_seq":4235,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:19.223626+0000","last_change":"2024-09-16T10:29:35.171033+0000","last_active":"2024-09-16T10:30:19.223626+0000","last_peered":"2024-09-16T10:30:19.223626+0000","last_clean":"2024-09-16T10:30:19.223626+0000","last_became_active":"2024-09-16T10:29:35.170730+0000","last_became_peered":"2024-09-16T10:29:35.170730+0000","last_unstale":"2024-09-16T10:30:19.223626+0000","last_undegraded":"2024-09-16T10:30:19.223626+0000","last_fullsized":"2024-09-16T10:30:19.223626+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:36.397873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274726912,"num_objects":4192,"num_object_clones":0,"num_object_copies":12576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4192,"num_write_kb":268288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":785,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070098+0000","last_change":"2024-09-16T10:30:05.823698+0000","last_active":"2024-09-16T10:30:16.070098+0000","last_peered":"2024-09-16T10:30:16.070098+0000","last_clean":"2024-09-16T10:30:16.070098+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:30:16.070098+0000","last_undegraded":"2024-09-16T10:30:16.070098+0000","last_fullsized":"2024-09-16T10:30:16.070098+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:05.823644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.823644+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:09.445585+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":917,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063864+0000","last_change":"2024-09-16T10:27:21.459062+0000","last_active":"2024-09-16T10:30:16.063864+0000","last_peered":"2024-09-16T10:30:16.063864+0000","last_clean":"2024-09-16T10:30:16.063864+0000","last_became_active":"2024-09-16T10:26:28.553171+0000","last_became_peered":"2024-09-16T10:26:28.553171+0000","last_unstale":"2024-09-16T10:30:16.063864+0000","last_undegraded":"2024-09-16T10:30:16.063864+0000","last_fullsized":"2024-09-16T10:30:16.063864+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:27:21.458969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:21.458969+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.454543+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":924,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062358+0000","last_change":"2024-09-16T10:30:03.609790+0000","last_active":"2024-09-16T10:30:16.062358+0000","last_peered":"2024-09-16T10:30:16.062358+0000","last_clean":"2024-09-16T10:30:16.062358+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:30:16.062358+0000","last_undegraded":"2024-09-16T10:30:16.062358+0000","last_fullsized":"2024-09-16T10:30:16.062358+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.609728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.609728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:06.614571+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":909,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065600+0000","last_change":"2024-09-16T10:29:21.720973+0000","last_active":"2024-09-16T10:30:16.065600+0000","last_peered":"2024-09-16T10:30:16.065600+0000","last_clean":"2024-09-16T10:30:16.065600+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:30:16.065600+0000","last_undegraded":"2024-09-16T10:30:16.065600+0000","last_fullsized":"2024-09-16T10:30:16.065600+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.720910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.720910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.270176+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":450,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.066052+0000","last_change":"2024-09-16T10:25:14.383697+0000","last_active":"2024-09-16T10:30:16.066052+0000","last_peered":"2024-09-16T10:30:16.066052+0000","last_clean":"2024-09-16T10:30:16.066052+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:30:16.066052+0000","last_undegraded":"2024-09-16T10:30:16.066052+0000","last_fullsized":"2024-09-16T10:30:16.066052+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.383621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.383621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.497494+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"492'4342","reported_seq":4385,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.574346+0000","last_change":"2024-09-16T10:29:35.279567+0000","last_active":"2024-09-16T10:30:18.574346+0000","last_peered":"2024-09-16T10:30:18.574346+0000","last_clean":"2024-09-16T10:30:18.574346+0000","last_became_active":"2024-09-16T10:29:35.279253+0000","last_became_peered":"2024-09-16T10:29:35.279253+0000","last_unstale":"2024-09-16T10:30:18.574346+0000","last_undegraded":"2024-09-16T10:30:18.574346+0000","last_fullsized":"2024-09-16T10:30:18.574346+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.137735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284557312,"num_objects":4342,"num_object_clones":0,"num_object_copies":13026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4342,"num_write_kb":277888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":598,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073886+0000","last_change":"2024-09-16T10:30:07.951920+0000","last_active":"2024-09-16T10:30:16.073886+0000","last_peered":"2024-09-16T10:30:16.073886+0000","last_clean":"2024-09-16T10:30:16.073886+0000","last_became_active":"2024-09-16T10:28:28.095730+0000","last_became_peered":"2024-09-16T10:28:28.095730+0000","last_unstale":"2024-09-16T10:30:16.073886+0000","last_undegraded":"2024-09-16T10:30:16.073886+0000","last_fullsized":"2024-09-16T10:30:16.073886+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:07.951845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.951845+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:11.529319+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":833,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074041+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:30:16.074041+0000","last_peered":"2024-09-16T10:30:16.074041+0000","last_clean":"2024-09-16T10:30:16.074041+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:30:16.074041+0000","last_undegraded":"2024-09-16T10:30:16.074041+0000","last_fullsized":"2024-09-16T10:30:16.074041+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.568656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":907,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074301+0000","last_change":"2024-09-16T10:27:12.155297+0000","last_active":"2024-09-16T10:30:16.074301+0000","last_peered":"2024-09-16T10:30:16.074301+0000","last_clean":"2024-09-16T10:30:16.074301+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:30:16.074301+0000","last_undegraded":"2024-09-16T10:30:16.074301+0000","last_fullsized":"2024-09-16T10:30:16.074301+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:12.155186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.155186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"442'12","reported_seq":566,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064164+0000","last_change":"2024-09-16T10:29:21.445504+0000","last_active":"2024-09-16T10:30:16.064164+0000","last_peered":"2024-09-16T10:30:16.064164+0000","last_clean":"2024-09-16T10:30:16.064164+0000","last_became_active":"2024-09-16T10:28:28.072074+0000","last_became_peered":"2024-09-16T10:28:28.072074+0000","last_unstale":"2024-09-16T10:30:16.064164+0000","last_undegraded":"2024-09-16T10:30:16.064164+0000","last_fullsized":"2024-09-16T10:30:16.064164+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:29:21.445439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.445439+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.856000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":716,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.066277+0000","last_change":"2024-09-16T10:27:13.405307+0000","last_active":"2024-09-16T10:30:16.066277+0000","last_peered":"2024-09-16T10:30:16.066277+0000","last_clean":"2024-09-16T10:30:16.066277+0000","last_became_active":"2024-09-16T10:26:28.550959+0000","last_became_peered":"2024-09-16T10:26:28.550959+0000","last_unstale":"2024-09-16T10:30:16.066277+0000","last_undegraded":"2024-09-16T10:30:16.066277+0000","last_fullsized":"2024-09-16T10:30:16.066277+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.405239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.405239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"492'4089","reported_seq":4132,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.570275+0000","last_change":"2024-09-16T10:29:35.280282+0000","last_active":"2024-09-16T10:30:18.570275+0000","last_peered":"2024-09-16T10:30:18.570275+0000","last_clean":"2024-09-16T10:30:18.570275+0000","last_became_active":"2024-09-16T10:29:35.279983+0000","last_became_peered":"2024-09-16T10:29:35.279983+0000","last_unstale":"2024-09-16T10:30:18.570275+0000","last_undegraded":"2024-09-16T10:30:18.570275+0000","last_fullsized":"2024-09-16T10:30:18.570275+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:48.246468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267976704,"num_objects":4089,"num_object_clones":0,"num_object_copies":12267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4089,"num_write_kb":261696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":255,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.059704+0000","last_change":"2024-09-16T10:29:28.900601+0000","last_active":"2024-09-16T10:30:16.059704+0000","last_peered":"2024-09-16T10:30:16.059704+0000","last_clean":"2024-09-16T10:30:16.059704+0000","last_became_active":"2024-09-16T10:28:28.076677+0000","last_became_peered":"2024-09-16T10:28:28.076677+0000","last_unstale":"2024-09-16T10:30:16.059704+0000","last_undegraded":"2024-09-16T10:30:16.059704+0000","last_fullsized":"2024-09-16T10:30:16.059704+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.900551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.900551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:32.135504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":945,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:19.209404+0000","last_change":"2024-09-16T10:30:19.209404+0000","last_active":"2024-09-16T10:30:19.209404+0000","last_peered":"2024-09-16T10:30:19.209404+0000","last_clean":"2024-09-16T10:30:19.209404+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:30:19.209404+0000","last_undegraded":"2024-09-16T10:30:19.209404+0000","last_fullsized":"2024-09-16T10:30:19.209404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:30:19.209335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:19.209335+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:25.345201+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":933,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069923+0000","last_change":"2024-09-16T10:26:44.070285+0000","last_active":"2024-09-16T10:30:16.069923+0000","last_peered":"2024-09-16T10:30:16.069923+0000","last_clean":"2024-09-16T10:30:16.069923+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:30:16.069923+0000","last_undegraded":"2024-09-16T10:30:16.069923+0000","last_fullsized":"2024-09-16T10:30:16.069923+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:44.070218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.070218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:01.788169+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":450,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064663+0000","last_change":"2024-09-16T10:26:22.501028+0000","last_active":"2024-09-16T10:30:16.064663+0000","last_peered":"2024-09-16T10:30:16.064663+0000","last_clean":"2024-09-16T10:30:16.064663+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:30:16.064663+0000","last_undegraded":"2024-09-16T10:30:16.064663+0000","last_fullsized":"2024-09-16T10:30:16.064663+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.500963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.500963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.464209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":960,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070783+0000","last_change":"2024-09-16T10:30:05.174345+0000","last_active":"2024-09-16T10:30:16.070783+0000","last_peered":"2024-09-16T10:30:16.070783+0000","last_clean":"2024-09-16T10:30:16.070783+0000","last_became_active":"2024-09-16T10:28:28.074693+0000","last_became_peered":"2024-09-16T10:28:28.074693+0000","last_unstale":"2024-09-16T10:30:16.070783+0000","last_undegraded":"2024-09-16T10:30:16.070783+0000","last_fullsized":"2024-09-16T10:30:16.070783+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:05.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.174161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:12.678829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"492'4152","reported_seq":4195,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.952845+0000","last_change":"2024-09-16T10:29:35.279583+0000","last_active":"2024-09-16T10:30:17.952845+0000","last_peered":"2024-09-16T10:30:17.952845+0000","last_clean":"2024-09-16T10:30:17.952845+0000","last_became_active":"2024-09-16T10:29:35.279260+0000","last_became_peered":"2024-09-16T10:29:35.279260+0000","last_unstale":"2024-09-16T10:30:17.952845+0000","last_undegraded":"2024-09-16T10:30:17.952845+0000","last_fullsized":"2024-09-16T10:30:17.952845+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.395721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272105472,"num_objects":4152,"num_object_clones":0,"num_object_copies":12456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4152,"num_write_kb":265728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":881,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063630+0000","last_change":"2024-09-16T10:25:40.594525+0000","last_active":"2024-09-16T10:30:16.063630+0000","last_peered":"2024-09-16T10:30:16.063630+0000","last_clean":"2024-09-16T10:30:16.063630+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:30:16.063630+0000","last_undegraded":"2024-09-16T10:30:16.063630+0000","last_fullsized":"2024-09-16T10:30:16.063630+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:25:40.594454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.594454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.350308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":994,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.061947+0000","last_change":"2024-09-16T10:28:28.088838+0000","last_active":"2024-09-16T10:30:16.061947+0000","last_peered":"2024-09-16T10:30:16.061947+0000","last_clean":"2024-09-16T10:30:16.061947+0000","last_became_active":"2024-09-16T10:28:28.088338+0000","last_became_peered":"2024-09-16T10:28:28.088338+0000","last_unstale":"2024-09-16T10:30:16.061947+0000","last_undegraded":"2024-09-16T10:30:16.061947+0000","last_fullsized":"2024-09-16T10:30:16.061947+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:56.539717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":314,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069559+0000","last_change":"2024-09-16T10:28:28.101048+0000","last_active":"2024-09-16T10:30:16.069559+0000","last_peered":"2024-09-16T10:30:16.069559+0000","last_clean":"2024-09-16T10:30:16.069559+0000","last_became_active":"2024-09-16T10:28:28.100755+0000","last_became_peered":"2024-09-16T10:28:28.100755+0000","last_unstale":"2024-09-16T10:30:16.069559+0000","last_undegraded":"2024-09-16T10:30:16.069559+0000","last_fullsized":"2024-09-16T10:30:16.069559+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.031940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.031940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.981262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":450,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063224+0000","last_change":"2024-09-16T10:26:43.199824+0000","last_active":"2024-09-16T10:30:16.063224+0000","last_peered":"2024-09-16T10:30:16.063224+0000","last_clean":"2024-09-16T10:30:16.063224+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:30:16.063224+0000","last_undegraded":"2024-09-16T10:30:16.063224+0000","last_fullsized":"2024-09-16T10:30:16.063224+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:43.199752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:43.199752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:56.394503+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":790,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062817+0000","last_change":"2024-09-16T10:28:28.065678+0000","last_active":"2024-09-16T10:30:16.062817+0000","last_peered":"2024-09-16T10:30:16.062817+0000","last_clean":"2024-09-16T10:30:16.062817+0000","last_became_active":"2024-09-16T10:28:28.065386+0000","last_became_peered":"2024-09-16T10:28:28.065386+0000","last_unstale":"2024-09-16T10:30:16.062817+0000","last_undegraded":"2024-09-16T10:30:16.062817+0000","last_fullsized":"2024-09-16T10:30:16.062817+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:34.550928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:34.550928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.472997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"492'4308","reported_seq":4351,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:20.000199+0000","last_change":"2024-09-16T10:29:35.172946+0000","last_active":"2024-09-16T10:30:20.000199+0000","last_peered":"2024-09-16T10:30:20.000199+0000","last_clean":"2024-09-16T10:30:20.000199+0000","last_became_active":"2024-09-16T10:29:35.172649+0000","last_became_peered":"2024-09-16T10:29:35.172649+0000","last_unstale":"2024-09-16T10:30:20.000199+0000","last_undegraded":"2024-09-16T10:30:20.000199+0000","last_fullsized":"2024-09-16T10:30:20.000199+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.718311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4308,"num_write_kb":275712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":832,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073652+0000","last_change":"2024-09-16T10:26:10.271093+0000","last_active":"2024-09-16T10:30:16.073652+0000","last_peered":"2024-09-16T10:30:16.073652+0000","last_clean":"2024-09-16T10:30:16.073652+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:30:16.073652+0000","last_undegraded":"2024-09-16T10:30:16.073652+0000","last_fullsized":"2024-09-16T10:30:16.073652+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:10.270833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.270833+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:55.805778+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":841,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062740+0000","last_change":"2024-09-16T10:26:42.132944+0000","last_active":"2024-09-16T10:30:16.062740+0000","last_peered":"2024-09-16T10:30:16.062740+0000","last_clean":"2024-09-16T10:30:16.062740+0000","last_became_active":"2024-09-16T10:26:28.553750+0000","last_became_peered":"2024-09-16T10:26:28.553750+0000","last_unstale":"2024-09-16T10:30:16.062740+0000","last_undegraded":"2024-09-16T10:30:16.062740+0000","last_fullsized":"2024-09-16T10:30:16.062740+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:26:42.132853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:42.132853+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":264,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075246+0000","last_change":"2024-09-16T10:27:13.285419+0000","last_active":"2024-09-16T10:30:16.075246+0000","last_peered":"2024-09-16T10:30:16.075246+0000","last_clean":"2024-09-16T10:30:16.075246+0000","last_became_active":"2024-09-16T10:25:41.990483+0000","last_became_peered":"2024-09-16T10:25:41.990483+0000","last_unstale":"2024-09-16T10:30:16.075246+0000","last_undegraded":"2024-09-16T10:30:16.075246+0000","last_fullsized":"2024-09-16T10:30:16.075246+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":450,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062913+0000","last_change":"2024-09-16T10:26:21.479904+0000","last_active":"2024-09-16T10:30:16.062913+0000","last_peered":"2024-09-16T10:30:16.062913+0000","last_clean":"2024-09-16T10:30:16.062913+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:30:16.062913+0000","last_undegraded":"2024-09-16T10:30:16.062913+0000","last_fullsized":"2024-09-16T10:30:16.062913+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.479835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.479835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:45.459449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":922,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073697+0000","last_change":"2024-09-16T10:28:28.095795+0000","last_active":"2024-09-16T10:30:16.073697+0000","last_peered":"2024-09-16T10:30:16.073697+0000","last_clean":"2024-09-16T10:30:16.073697+0000","last_became_active":"2024-09-16T10:28:28.095174+0000","last_became_peered":"2024-09-16T10:28:28.095174+0000","last_unstale":"2024-09-16T10:30:16.073697+0000","last_undegraded":"2024-09-16T10:30:16.073697+0000","last_fullsized":"2024-09-16T10:30:16.073697+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:41.478267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:41.478267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:51.823658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"492'4217","reported_seq":4260,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:18.011982+0000","last_change":"2024-09-16T10:29:35.165138+0000","last_active":"2024-09-16T10:30:18.011982+0000","last_peered":"2024-09-16T10:30:18.011982+0000","last_clean":"2024-09-16T10:30:18.011982+0000","last_became_active":"2024-09-16T10:29:35.164843+0000","last_became_peered":"2024-09-16T10:29:35.164843+0000","last_unstale":"2024-09-16T10:30:18.011982+0000","last_undegraded":"2024-09-16T10:30:18.011982+0000","last_fullsized":"2024-09-16T10:30:18.011982+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.727993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276365312,"num_objects":4217,"num_object_clones":0,"num_object_copies":12651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4217,"num_write_kb":269888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":641,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063686+0000","last_change":"2024-09-16T10:29:29.763437+0000","last_active":"2024-09-16T10:30:16.063686+0000","last_peered":"2024-09-16T10:30:16.063686+0000","last_clean":"2024-09-16T10:30:16.063686+0000","last_became_active":"2024-09-16T10:28:28.071881+0000","last_became_peered":"2024-09-16T10:28:28.071881+0000","last_unstale":"2024-09-16T10:30:16.063686+0000","last_undegraded":"2024-09-16T10:30:16.063686+0000","last_fullsized":"2024-09-16T10:30:16.063686+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:32.911633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":959,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070192+0000","last_change":"2024-09-16T10:28:28.097779+0000","last_active":"2024-09-16T10:30:16.070192+0000","last_peered":"2024-09-16T10:30:16.070192+0000","last_clean":"2024-09-16T10:30:16.070192+0000","last_became_active":"2024-09-16T10:28:28.097478+0000","last_became_peered":"2024-09-16T10:28:28.097478+0000","last_unstale":"2024-09-16T10:30:16.070192+0000","last_undegraded":"2024-09-16T10:30:16.070192+0000","last_fullsized":"2024-09-16T10:30:16.070192+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:26:20.104708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.104708+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.925950+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":253,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071281+0000","last_change":"2024-09-16T10:28:28.081893+0000","last_active":"2024-09-16T10:30:16.071281+0000","last_peered":"2024-09-16T10:30:16.071281+0000","last_clean":"2024-09-16T10:30:16.071281+0000","last_became_active":"2024-09-16T10:28:28.081512+0000","last_became_peered":"2024-09-16T10:28:28.081512+0000","last_unstale":"2024-09-16T10:30:16.071281+0000","last_undegraded":"2024-09-16T10:30:16.071281+0000","last_fullsized":"2024-09-16T10:30:16.071281+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:08.381010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.381010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.124691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"442'11","reported_seq":405,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062513+0000","last_change":"2024-09-16T10:28:28.074072+0000","last_active":"2024-09-16T10:30:16.062513+0000","last_peered":"2024-09-16T10:30:16.062513+0000","last_clean":"2024-09-16T10:30:16.062513+0000","last_became_active":"2024-09-16T10:28:28.073675+0000","last_became_peered":"2024-09-16T10:28:28.073675+0000","last_unstale":"2024-09-16T10:30:16.062513+0000","last_undegraded":"2024-09-16T10:30:16.062513+0000","last_fullsized":"2024-09-16T10:30:16.062513+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'9","last_scrub_stamp":"2024-09-16T10:25:37.549468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:37.549468+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:52.506426+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":929,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069865+0000","last_change":"2024-09-16T10:30:07.792293+0000","last_active":"2024-09-16T10:30:16.069865+0000","last_peered":"2024-09-16T10:30:16.069865+0000","last_clean":"2024-09-16T10:30:16.069865+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:30:16.069865+0000","last_undegraded":"2024-09-16T10:30:16.069865+0000","last_fullsized":"2024-09-16T10:30:16.069865+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.792060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.792060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:34.132214+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":497,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069996+0000","last_change":"2024-09-16T10:29:28.935051+0000","last_active":"2024-09-16T10:30:16.069996+0000","last_peered":"2024-09-16T10:30:16.069996+0000","last_clean":"2024-09-16T10:30:16.069996+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:30:16.069996+0000","last_undegraded":"2024-09-16T10:30:16.069996+0000","last_fullsized":"2024-09-16T10:30:16.069996+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.935001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.935001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:50.004175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":908,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062189+0000","last_change":"2024-09-16T10:26:20.469286+0000","last_active":"2024-09-16T10:30:16.062189+0000","last_peered":"2024-09-16T10:30:16.062189+0000","last_clean":"2024-09-16T10:30:16.062189+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:30:16.062189+0000","last_undegraded":"2024-09-16T10:30:16.062189+0000","last_fullsized":"2024-09-16T10:30:16.062189+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:20.469240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.469240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:57.570970+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":939,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075632+0000","last_change":"2024-09-16T10:29:17.916063+0000","last_active":"2024-09-16T10:30:16.075632+0000","last_peered":"2024-09-16T10:30:16.075632+0000","last_clean":"2024-09-16T10:30:16.075632+0000","last_became_active":"2024-09-16T10:28:28.096974+0000","last_became_peered":"2024-09-16T10:28:28.096974+0000","last_unstale":"2024-09-16T10:30:16.075632+0000","last_undegraded":"2024-09-16T10:30:16.075632+0000","last_fullsized":"2024-09-16T10:30:16.075632+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:29:17.915982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:17.915982+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.961249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":908,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069849+0000","last_change":"2024-09-16T10:26:46.061085+0000","last_active":"2024-09-16T10:30:16.069849+0000","last_peered":"2024-09-16T10:30:16.069849+0000","last_clean":"2024-09-16T10:30:16.069849+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:30:16.069849+0000","last_undegraded":"2024-09-16T10:30:16.069849+0000","last_fullsized":"2024-09-16T10:30:16.069849+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:53.863200+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":735,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.059557+0000","last_change":"2024-09-16T10:28:28.077871+0000","last_active":"2024-09-16T10:30:16.059557+0000","last_peered":"2024-09-16T10:30:16.059557+0000","last_clean":"2024-09-16T10:30:16.059557+0000","last_became_active":"2024-09-16T10:28:28.072905+0000","last_became_peered":"2024-09-16T10:28:28.072905+0000","last_unstale":"2024-09-16T10:30:16.059557+0000","last_undegraded":"2024-09-16T10:30:16.059557+0000","last_fullsized":"2024-09-16T10:30:16.059557+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:11.324847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:11.324847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.018433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":811,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063890+0000","last_change":"2024-09-16T10:24:32.912700+0000","last_active":"2024-09-16T10:30:16.063890+0000","last_peered":"2024-09-16T10:30:16.063890+0000","last_clean":"2024-09-16T10:30:16.063890+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:30:16.063890+0000","last_undegraded":"2024-09-16T10:30:16.063890+0000","last_fullsized":"2024-09-16T10:30:16.063890+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:24:32.912636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:24:32.912636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:42.632169+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":955,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071535+0000","last_change":"2024-09-16T10:30:07.157911+0000","last_active":"2024-09-16T10:30:16.071535+0000","last_peered":"2024-09-16T10:30:16.071535+0000","last_clean":"2024-09-16T10:30:16.071535+0000","last_became_active":"2024-09-16T10:28:28.082315+0000","last_became_peered":"2024-09-16T10:28:28.082315+0000","last_unstale":"2024-09-16T10:30:16.071535+0000","last_undegraded":"2024-09-16T10:30:16.071535+0000","last_fullsized":"2024-09-16T10:30:16.071535+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:07.157828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.157828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:34.427287+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":253,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062818+0000","last_change":"2024-09-16T10:29:24.266598+0000","last_active":"2024-09-16T10:30:16.062818+0000","last_peered":"2024-09-16T10:30:16.062818+0000","last_clean":"2024-09-16T10:30:16.062818+0000","last_became_active":"2024-09-16T10:28:28.071950+0000","last_became_peered":"2024-09-16T10:28:28.071950+0000","last_unstale":"2024-09-16T10:30:16.062818+0000","last_undegraded":"2024-09-16T10:30:16.062818+0000","last_fullsized":"2024-09-16T10:30:16.062818+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:24.266543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:24.266543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:41.996180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":917,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075453+0000","last_change":"2024-09-16T10:28:28.098654+0000","last_active":"2024-09-16T10:30:16.075453+0000","last_peered":"2024-09-16T10:30:16.075453+0000","last_clean":"2024-09-16T10:30:16.075453+0000","last_became_active":"2024-09-16T10:28:28.097898+0000","last_became_peered":"2024-09-16T10:28:28.097898+0000","last_unstale":"2024-09-16T10:30:16.075453+0000","last_undegraded":"2024-09-16T10:30:16.075453+0000","last_fullsized":"2024-09-16T10:30:16.075453+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:21.419085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:21.419085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":461,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065714+0000","last_change":"2024-09-16T10:29:36.541047+0000","last_active":"2024-09-16T10:30:16.065714+0000","last_peered":"2024-09-16T10:30:16.065714+0000","last_clean":"2024-09-16T10:30:16.065714+0000","last_became_active":"2024-09-16T10:28:28.072347+0000","last_became_peered":"2024-09-16T10:28:28.072347+0000","last_unstale":"2024-09-16T10:30:16.065714+0000","last_undegraded":"2024-09-16T10:30:16.065714+0000","last_fullsized":"2024-09-16T10:30:16.065714+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.541001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.541001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.828112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":499,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070047+0000","last_change":"2024-09-16T10:29:34.955215+0000","last_active":"2024-09-16T10:30:16.070047+0000","last_peered":"2024-09-16T10:30:16.070047+0000","last_clean":"2024-09-16T10:30:16.070047+0000","last_became_active":"2024-09-16T10:28:28.096538+0000","last_became_peered":"2024-09-16T10:28:28.096538+0000","last_unstale":"2024-09-16T10:30:16.070047+0000","last_undegraded":"2024-09-16T10:30:16.070047+0000","last_fullsized":"2024-09-16T10:30:16.070047+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:59.442771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":940,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069732+0000","last_change":"2024-09-16T10:29:20.061250+0000","last_active":"2024-09-16T10:30:16.069732+0000","last_peered":"2024-09-16T10:30:16.069732+0000","last_clean":"2024-09-16T10:30:16.069732+0000","last_became_active":"2024-09-16T10:28:28.101644+0000","last_became_peered":"2024-09-16T10:28:28.101644+0000","last_unstale":"2024-09-16T10:30:16.069732+0000","last_undegraded":"2024-09-16T10:30:16.069732+0000","last_fullsized":"2024-09-16T10:30:16.069732+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:29:20.061184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.061184+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:21.959259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":935,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071088+0000","last_change":"2024-09-16T10:29:38.060471+0000","last_active":"2024-09-16T10:30:16.071088+0000","last_peered":"2024-09-16T10:30:16.071088+0000","last_clean":"2024-09-16T10:30:16.071088+0000","last_became_active":"2024-09-16T10:28:28.072930+0000","last_became_peered":"2024-09-16T10:28:28.072930+0000","last_unstale":"2024-09-16T10:30:16.071088+0000","last_undegraded":"2024-09-16T10:30:16.071088+0000","last_fullsized":"2024-09-16T10:30:16.071088+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:38.060395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:38.060395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:05.957375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"442'12","reported_seq":1166,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074417+0000","last_change":"2024-09-16T10:28:28.095653+0000","last_active":"2024-09-16T10:30:16.074417+0000","last_peered":"2024-09-16T10:30:16.074417+0000","last_clean":"2024-09-16T10:30:16.074417+0000","last_became_active":"2024-09-16T10:28:28.094870+0000","last_became_peered":"2024-09-16T10:28:28.094870+0000","last_unstale":"2024-09-16T10:30:16.074417+0000","last_undegraded":"2024-09-16T10:30:16.074417+0000","last_fullsized":"2024-09-16T10:30:16.074417+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10","last_scrub_stamp":"2024-09-16T10:25:16.464663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:16.464663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:39.703651+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":715,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065924+0000","last_change":"2024-09-16T10:26:45.194577+0000","last_active":"2024-09-16T10:30:16.065924+0000","last_peered":"2024-09-16T10:30:16.065924+0000","last_clean":"2024-09-16T10:30:16.065924+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:30:16.065924+0000","last_undegraded":"2024-09-16T10:30:16.065924+0000","last_fullsized":"2024-09-16T10:30:16.065924+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.194454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.194454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:53.451840+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":494,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.075993+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:30:16.075993+0000","last_peered":"2024-09-16T10:30:16.075993+0000","last_clean":"2024-09-16T10:30:16.075993+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:30:16.075993+0000","last_undegraded":"2024-09-16T10:30:16.075993+0000","last_fullsized":"2024-09-16T10:30:16.075993+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:03.173196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":968,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073909+0000","last_change":"2024-09-16T10:28:28.096614+0000","last_active":"2024-09-16T10:30:16.073909+0000","last_peered":"2024-09-16T10:30:16.073909+0000","last_clean":"2024-09-16T10:30:16.073909+0000","last_became_active":"2024-09-16T10:28:28.095319+0000","last_became_peered":"2024-09-16T10:28:28.095319+0000","last_unstale":"2024-09-16T10:30:16.073909+0000","last_undegraded":"2024-09-16T10:30:16.073909+0000","last_fullsized":"2024-09-16T10:30:16.073909+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:26:20.405507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.405507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:58.832029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":823,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.059570+0000","last_change":"2024-09-16T10:29:21.812647+0000","last_active":"2024-09-16T10:30:16.059570+0000","last_peered":"2024-09-16T10:30:16.059570+0000","last_clean":"2024-09-16T10:30:16.059570+0000","last_became_active":"2024-09-16T10:28:28.071944+0000","last_became_peered":"2024-09-16T10:28:28.071944+0000","last_unstale":"2024-09-16T10:30:16.059570+0000","last_undegraded":"2024-09-16T10:30:16.059570+0000","last_fullsized":"2024-09-16T10:30:16.059570+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.812585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.812585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:23.180131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1125,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069373+0000","last_change":"2024-09-16T10:26:11.007271+0000","last_active":"2024-09-16T10:30:16.069373+0000","last_peered":"2024-09-16T10:30:16.069373+0000","last_clean":"2024-09-16T10:30:16.069373+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:30:16.069373+0000","last_undegraded":"2024-09-16T10:30:16.069373+0000","last_fullsized":"2024-09-16T10:30:16.069373+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:26:11.007191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:11.007191+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:52.906529+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":914,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070946+0000","last_change":"2024-09-16T10:29:20.053166+0000","last_active":"2024-09-16T10:30:16.070946+0000","last_peered":"2024-09-16T10:30:16.070946+0000","last_clean":"2024-09-16T10:30:16.070946+0000","last_became_active":"2024-09-16T10:28:28.081510+0000","last_became_peered":"2024-09-16T10:28:28.081510+0000","last_unstale":"2024-09-16T10:30:16.070946+0000","last_undegraded":"2024-09-16T10:30:16.070946+0000","last_fullsized":"2024-09-16T10:30:16.070946+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.053102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.053102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:31.409241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":853,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063056+0000","last_change":"2024-09-16T10:28:02.318109+0000","last_active":"2024-09-16T10:30:16.063056+0000","last_peered":"2024-09-16T10:30:16.063056+0000","last_clean":"2024-09-16T10:30:16.063056+0000","last_became_active":"2024-09-16T10:26:28.557003+0000","last_became_peered":"2024-09-16T10:26:28.557003+0000","last_unstale":"2024-09-16T10:30:16.063056+0000","last_undegraded":"2024-09-16T10:30:16.063056+0000","last_fullsized":"2024-09-16T10:30:16.063056+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:02.318062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:02.318062+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":377,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.065473+0000","last_change":"2024-09-16T10:28:28.105143+0000","last_active":"2024-09-16T10:30:16.065473+0000","last_peered":"2024-09-16T10:30:16.065473+0000","last_clean":"2024-09-16T10:30:16.065473+0000","last_became_active":"2024-09-16T10:28:28.104604+0000","last_became_peered":"2024-09-16T10:28:28.104604+0000","last_unstale":"2024-09-16T10:30:16.065473+0000","last_undegraded":"2024-09-16T10:30:16.065473+0000","last_fullsized":"2024-09-16T10:30:16.065473+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:08.657212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:08.657212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.718457+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":909,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063060+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:30:16.063060+0000","last_peered":"2024-09-16T10:30:16.063060+0000","last_clean":"2024-09-16T10:30:16.063060+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:30:16.063060+0000","last_undegraded":"2024-09-16T10:30:16.063060+0000","last_fullsized":"2024-09-16T10:30:16.063060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:57.827326+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":903,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073714+0000","last_change":"2024-09-16T10:26:14.413104+0000","last_active":"2024-09-16T10:30:16.073714+0000","last_peered":"2024-09-16T10:30:16.073714+0000","last_clean":"2024-09-16T10:30:16.073714+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:30:16.073714+0000","last_undegraded":"2024-09-16T10:30:16.073714+0000","last_fullsized":"2024-09-16T10:30:16.073714+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:14.413019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:14.413019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:57.810624+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":972,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063356+0000","last_change":"2024-09-16T10:29:19.698059+0000","last_active":"2024-09-16T10:30:16.063356+0000","last_peered":"2024-09-16T10:30:16.063356+0000","last_clean":"2024-09-16T10:30:16.063356+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:30:16.063356+0000","last_undegraded":"2024-09-16T10:30:16.063356+0000","last_fullsized":"2024-09-16T10:30:16.063356+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.530423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":321,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074939+0000","last_change":"2024-09-16T10:30:03.845884+0000","last_active":"2024-09-16T10:30:16.074939+0000","last_peered":"2024-09-16T10:30:16.074939+0000","last_clean":"2024-09-16T10:30:16.074939+0000","last_became_active":"2024-09-16T10:28:28.094896+0000","last_became_peered":"2024-09-16T10:28:28.094896+0000","last_unstale":"2024-09-16T10:30:16.074939+0000","last_undegraded":"2024-09-16T10:30:16.074939+0000","last_fullsized":"2024-09-16T10:30:16.074939+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:22.698082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":872,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073782+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:30:16.073782+0000","last_peered":"2024-09-16T10:30:16.073782+0000","last_clean":"2024-09-16T10:30:16.073782+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:30:16.073782+0000","last_undegraded":"2024-09-16T10:30:16.073782+0000","last_fullsized":"2024-09-16T10:30:16.073782+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:53.762247+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":888,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.070288+0000","last_change":"2024-09-16T10:25:18.286464+0000","last_active":"2024-09-16T10:30:16.070288+0000","last_peered":"2024-09-16T10:30:16.070288+0000","last_clean":"2024-09-16T10:30:16.070288+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:30:16.070288+0000","last_undegraded":"2024-09-16T10:30:16.070288+0000","last_fullsized":"2024-09-16T10:30:16.070288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:18.286280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:25:18.286280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:51.918285+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":834,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064836+0000","last_change":"2024-09-16T10:29:35.573887+0000","last_active":"2024-09-16T10:30:16.064836+0000","last_peered":"2024-09-16T10:30:16.064836+0000","last_clean":"2024-09-16T10:30:16.064836+0000","last_became_active":"2024-09-16T10:28:28.071473+0000","last_became_peered":"2024-09-16T10:28:28.071473+0000","last_unstale":"2024-09-16T10:30:16.064836+0000","last_undegraded":"2024-09-16T10:30:16.064836+0000","last_fullsized":"2024-09-16T10:30:16.064836+0000","mapping_epoch":441,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:35.573835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.573835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:43.601451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":937,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073979+0000","last_change":"2024-09-16T10:29:19.861104+0000","last_active":"2024-09-16T10:30:16.073979+0000","last_peered":"2024-09-16T10:30:16.073979+0000","last_clean":"2024-09-16T10:30:16.073979+0000","last_became_active":"2024-09-16T10:28:28.095283+0000","last_became_peered":"2024-09-16T10:28:28.095283+0000","last_unstale":"2024-09-16T10:30:16.073979+0000","last_undegraded":"2024-09-16T10:30:16.073979+0000","last_fullsized":"2024-09-16T10:30:16.073979+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:47.496265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":935,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.069930+0000","last_change":"2024-09-16T10:29:23.002575+0000","last_active":"2024-09-16T10:30:16.069930+0000","last_peered":"2024-09-16T10:30:16.069930+0000","last_clean":"2024-09-16T10:30:16.069930+0000","last_became_active":"2024-09-16T10:28:28.109359+0000","last_became_peered":"2024-09-16T10:28:28.109359+0000","last_unstale":"2024-09-16T10:30:16.069930+0000","last_undegraded":"2024-09-16T10:30:16.069930+0000","last_fullsized":"2024-09-16T10:30:16.069930+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.002518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.002518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:26.887462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":747,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.058843+0000","last_change":"2024-09-16T10:29:19.866083+0000","last_active":"2024-09-16T10:30:16.058843+0000","last_peered":"2024-09-16T10:30:16.058843+0000","last_clean":"2024-09-16T10:30:16.058843+0000","last_became_active":"2024-09-16T10:28:28.091060+0000","last_became_peered":"2024-09-16T10:28:28.091060+0000","last_unstale":"2024-09-16T10:30:16.058843+0000","last_undegraded":"2024-09-16T10:30:16.058843+0000","last_fullsized":"2024-09-16T10:30:16.058843+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.866022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:36.779287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":919,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074203+0000","last_change":"2024-09-16T10:29:34.789497+0000","last_active":"2024-09-16T10:30:16.074203+0000","last_peered":"2024-09-16T10:30:16.074203+0000","last_clean":"2024-09-16T10:30:16.074203+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:30:16.074203+0000","last_undegraded":"2024-09-16T10:30:16.074203+0000","last_fullsized":"2024-09-16T10:30:16.074203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.789441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.789441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.777028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":957,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.073909+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:30:16.073909+0000","last_peered":"2024-09-16T10:30:16.073909+0000","last_clean":"2024-09-16T10:30:16.073909+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:30:16.073909+0000","last_undegraded":"2024-09-16T10:30:16.073909+0000","last_fullsized":"2024-09-16T10:30:16.073909+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:54.772164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":913,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071503+0000","last_change":"2024-09-16T10:27:27.598364+0000","last_active":"2024-09-16T10:30:16.071503+0000","last_peered":"2024-09-16T10:30:16.071503+0000","last_clean":"2024-09-16T10:30:16.071503+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:30:16.071503+0000","last_undegraded":"2024-09-16T10:30:16.071503+0000","last_fullsized":"2024-09-16T10:30:16.071503+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:27.598299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:27.598299+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":927,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.062624+0000","last_change":"2024-09-16T10:29:31.282374+0000","last_active":"2024-09-16T10:30:16.062624+0000","last_peered":"2024-09-16T10:30:16.062624+0000","last_clean":"2024-09-16T10:30:16.062624+0000","last_became_active":"2024-09-16T10:28:28.072759+0000","last_became_peered":"2024-09-16T10:28:28.072759+0000","last_unstale":"2024-09-16T10:30:16.062624+0000","last_undegraded":"2024-09-16T10:30:16.062624+0000","last_fullsized":"2024-09-16T10:30:16.062624+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:31.282323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:31.282323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.735338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":545,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.064100+0000","last_change":"2024-09-16T10:29:32.563307+0000","last_active":"2024-09-16T10:30:16.064100+0000","last_peered":"2024-09-16T10:30:16.064100+0000","last_clean":"2024-09-16T10:30:16.064100+0000","last_became_active":"2024-09-16T10:28:28.066479+0000","last_became_peered":"2024-09-16T10:28:28.066479+0000","last_unstale":"2024-09-16T10:30:16.064100+0000","last_undegraded":"2024-09-16T10:30:16.064100+0000","last_fullsized":"2024-09-16T10:30:16.064100+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:32.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:32.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:49.751962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":912,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063785+0000","last_change":"2024-09-16T10:30:15.832984+0000","last_active":"2024-09-16T10:30:16.063785+0000","last_peered":"2024-09-16T10:30:16.063785+0000","last_clean":"2024-09-16T10:30:16.063785+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:30:16.063785+0000","last_undegraded":"2024-09-16T10:30:16.063785+0000","last_fullsized":"2024-09-16T10:30:16.063785+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.832904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.832904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:34.748880+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":922,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.076432+0000","last_change":"2024-09-16T10:26:13.362416+0000","last_active":"2024-09-16T10:30:16.076432+0000","last_peered":"2024-09-16T10:30:16.076432+0000","last_clean":"2024-09-16T10:30:16.076432+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:30:16.076432+0000","last_undegraded":"2024-09-16T10:30:16.076432+0000","last_fullsized":"2024-09-16T10:30:16.076432+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.362347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.362347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:59.797698+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":960,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.063824+0000","last_change":"2024-09-16T10:28:06.383385+0000","last_active":"2024-09-16T10:30:16.063824+0000","last_peered":"2024-09-16T10:30:16.063824+0000","last_clean":"2024-09-16T10:30:16.063824+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:30:16.063824+0000","last_undegraded":"2024-09-16T10:30:16.063824+0000","last_fullsized":"2024-09-16T10:30:16.063824+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:06.383318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:06.383318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":934,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.071114+0000","last_change":"2024-09-16T10:29:19.094876+0000","last_active":"2024-09-16T10:30:16.071114+0000","last_peered":"2024-09-16T10:30:16.071114+0000","last_clean":"2024-09-16T10:30:16.071114+0000","last_became_active":"2024-09-16T10:28:28.081905+0000","last_became_peered":"2024-09-16T10:28:28.081905+0000","last_unstale":"2024-09-16T10:30:16.071114+0000","last_undegraded":"2024-09-16T10:30:16.071114+0000","last_fullsized":"2024-09-16T10:30:16.071114+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.094826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.094826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:35.628616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4384161792,"num_objects":66897,"num_object_clones":0,"num_object_copies":200691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66897,"num_write_kb":4281408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13244891136,"data_stored":13244891136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66897,"ondisk_log_size":66897,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":320,"num_read_kb":315,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":441,"seq":1894080577560,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1864640,"kb_used_data":1679280,"kb_used_omap":7,"kb_used_meta":185336,"kb_avail":91880512,"statfs":{"total":95995035648,"available":94085644288,"internally_reserved":0,"allocated":1719582720,"data_stored":1718416439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8012,"internal_metadata":189784244},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528528,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2802744,"kb_used_data":2144040,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":90942408,"statfs":{"total":95995035648,"available":93125025792,"internally_reserved":0,"allocated":2195496960,"data_stored":2194338213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674494918},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790166,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698632,"kb_used_data":1071864,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92046520,"statfs":{"total":95995035648,"available":94255636480,"internally_reserved":0,"allocated":1097588736,"data_stored":1096415611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986395,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1664920,"kb_used_data":1047880,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92080232,"statfs":{"total":95995035648,"available":94290157568,"internally_reserved":0,"allocated":1073029120,"data_stored":1071838646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215328,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2618136,"kb_used_data":1865032,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":91127016,"statfs":{"total":95995035648,"available":93314064384,"internally_reserved":0,"allocated":1909792768,"data_stored":1908603677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":771160520},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2074756,"kb_used_data":1592368,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91670396,"statfs":{"total":95995035648,"available":93870485504,"internally_reserved":0,"allocated":1630584832,"data_stored":1629418708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2871788,"kb_used_data":2136724,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":90873364,"statfs":{"total":95995035648,"available":93054324736,"internally_reserved":0,"allocated":2188005376,"data_stored":2186803134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2076104,"kb_used_data":1350776,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":91669048,"statfs":{"total":95995035648,"available":93869105152,"internally_reserved":0,"allocated":1383194624,"data_stored":1382020853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1399980032,"data_stored":1399980032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2192506880,"data_stored":2192506880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1643642880,"data_stored":1643642880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1913389056,"data_stored":1913389056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1086652416,"data_stored":1086652416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1097662464,"data_stored":1097662464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2193883136,"data_stored":2193883136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1717174272,"data_stored":1717174272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:22.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952221+0000", 2024-09-16T10:30:22.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5039999999999996e-06 2024-09-16T10:30:22.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952274+0000", 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3239000000000003e-05 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952423+0000", 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014868300000000001 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.952582+0000", 2024-09-16T10:30:22.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015959200000000001 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.954428+0000", 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018452830000000001 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:03.954923+0000", 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049532799999999996 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:04.051018+0000", 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096095503999999998 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:04.051072+0000", 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4114999999999999e-05 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:04.051117+0000", 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4922999999999997e-05 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:55115 11.5 11:acaebeb9:::benchmark_data_smithi063_176907_object55114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:09.023936+0000", 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.524353459, 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.056486093000000001, 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.633 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 55115 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.023936+0000", 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.023936+0000", 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.023954+0000", 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8683999999999999e-05 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.023999+0000", 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4240999999999999e-05 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.024010+0000", 2024-09-16T10:30:22.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.024017+0000", 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5050000000000001e-06 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.024058+0000", 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0354e-05 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.024151+0000", 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2771999999999999e-05 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.024244+0000", 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3053000000000005e-05 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.027301+0000", 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003057461 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.027313+0000", 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1568e-05 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.080322+0000", 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.053009552000000001 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.080379+0000", 2024-09-16T10:30:22.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6650999999999997e-05 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.080422+0000", 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2901999999999997e-05 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:55117 11.5 11:aa3eeb7d:::benchmark_data_smithi063_176907_object55116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:09.025370+0000", 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.522918971999999, 2024-09-16T10:30:22.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.055170930999999999, 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 55117 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025370+0000", 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025370+0000", 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025371+0000", 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2950000000000001e-06 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025447+0000", 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5223999999999997e-05 2024-09-16T10:30:22.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025456+0000", 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6479999999999998e-06 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025463+0000", 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7150000000000001e-06 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025703+0000", 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239702 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025828+0000", 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125609 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.025948+0000", 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119512 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.027447+0000", 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014994909999999999 2024-09-16T10:30:22.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.027483+0000", 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5623000000000001e-05 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.080468+0000", 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.052984652 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.080509+0000", 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0920999999999997e-05 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:09.080541+0000", 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2539000000000001e-05 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:58012 11.7 11:e9766e3a:::benchmark_data_smithi063_176907_object58011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:11.178195+0000", 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.370093882000001, 2024-09-16T10:30:22.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.052604981000000002, 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 58012 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.178195+0000", 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.178195+0000", 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.178196+0000", 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2300000000000001e-07 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.178267+0000", 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1224999999999997e-05 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.178276+0000", 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6030000000000005e-06 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.178282+0000", 2024-09-16T10:30:22.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4679999999999997e-06 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.180137+0000", 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00185523 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.180265+0000", 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000127298 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.180370+0000", 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105596 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.181604+0000", 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001234197 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.182253+0000", 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064904999999999997 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.230708+0000", 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048454254000000002 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.230756+0000", 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8658000000000001e-05 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.230800+0000", 2024-09-16T10:30:22.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3878999999999998e-05 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:58327 11.7 11:e38428c7:::benchmark_data_smithi063_176907_object58326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:11.407480+0000", 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.140809148000001, 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.074769577000000004, 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.652 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 58327 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407480+0000", 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407480+0000", 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407482+0000", 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.903e-06 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407522+0000", 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0275000000000003e-05 2024-09-16T10:30:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407533+0000", 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0883000000000001e-05 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407540+0000", 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.43e-06 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407624+0000", 2024-09-16T10:30:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3164999999999994e-05 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407746+0000", 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012285799999999999 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.407884+0000", 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013771200000000001 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.410977+0000", 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030928679999999999 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.411288+0000", 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031069399999999998 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.482109+0000", 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070821536000000004 2024-09-16T10:30:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.482196+0000", 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6817999999999994e-05 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.482250+0000", 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3434999999999999e-05 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:58343 11.7 11:ef86d812:::benchmark_data_smithi063_176907_object58342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:11.417613+0000", 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.13067612, 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.068476756, 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 58343 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417613+0000", 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417613+0000", 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417615+0000", 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7269999999999999e-06 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417718+0000", 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010326 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417729+0000", 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1454000000000001e-05 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417737+0000", 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.576e-06 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417788+0000", 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1474999999999998e-05 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.417923+0000", 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134449 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.418044+0000", 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120913 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.419830+0000", 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001786211 2024-09-16T10:30:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.420106+0000", 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027563799999999998 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.485980+0000", 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.065874322999999999 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.486041+0000", 2024-09-16T10:30:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0992000000000002e-05 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.486090+0000", 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8738e-05 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:58347 11.7 11:ef5d5251:::benchmark_data_smithi063_176907_object58346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:11.421224+0000", 2024-09-16T10:30:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.127065062, 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.073382880999999997, 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 58347 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421224+0000", 2024-09-16T10:30:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421224+0000", 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421225+0000", 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4580000000000001e-06 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421297+0000", 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1787999999999998e-05 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421312+0000", 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4275e-05 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421319+0000", 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-16T10:30:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421371+0000", 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2676999999999997e-05 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421517+0000", 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145785 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.421654+0000", 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013725099999999999 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.423864+0000", 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022100380000000001 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.423988+0000", 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012355800000000001 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.494463+0000", 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070474615000000004 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.494559+0000", 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6824999999999995e-05 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:11.494607+0000", 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7472000000000002e-05 2024-09-16T10:30:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:59278 11.7 11:e4f5b985:::benchmark_data_smithi063_176907_object59277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:12.191788+0000", 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.356500839000001, 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.056104781999999999, 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 59278 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191788+0000", 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191788+0000", 2024-09-16T10:30:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191790+0000", 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.052e-06 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191871+0000", 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1057000000000007e-05 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191888+0000", 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6616000000000002e-05 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191896+0000", 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6629999999999993e-06 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191946+0000", 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0562000000000001e-05 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.192069+0000", 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012320199999999999 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.192192+0000", 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012296200000000001 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.196780+0000", 2024-09-16T10:30:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045879709999999997 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.196945+0000", 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016445199999999999 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.247781+0000", 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.050835809000000003 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.247847+0000", 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6115e-05 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.247893+0000", 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6320999999999999e-05 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:59279 11.7 11:e20ebe54:::benchmark_data_smithi063_176907_object59278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:30:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:12.191915+0000", 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.356374109000001, 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.056084672000000002, 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 59279 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191915+0000", 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191915+0000", 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191916+0000", 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5700000000000002e-07 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191947+0000", 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1356999999999998e-05 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191954+0000", 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4919999999999998e-06 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.191959+0000", 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4009999999999996e-06 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.194374+0000", 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024147499999999998 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.194525+0000", 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015133799999999999 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.194661+0000", 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013586699999999999 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.196970+0000", 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023093089999999998 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.197085+0000", 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115223 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.247928+0000", 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.050842492000000003 2024-09-16T10:30:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.247969+0000", 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0794000000000003e-05 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:12.248000+0000", 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0892000000000002e-05 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:68684 11.5 11:ae3cf4e1:::benchmark_data_smithi063_176907_object68683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:20.149908+0000", 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.398380741, 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.053616469999999999, 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 68684 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.149908+0000", 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.149908+0000", 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.149911+0000", 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3329999999999999e-06 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.149940+0000", 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9196e-05 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.149951+0000", 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1199000000000001e-05 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.149958+0000", 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9500000000000004e-06 2024-09-16T10:30:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.151774+0000", 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001816422 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.151919+0000", 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144697 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6,7", 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.152055+0000", 2024-09-16T10:30:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000136038 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.154281+0000", 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022261949999999998 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.154407+0000", 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012587899999999999 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.203393+0000", 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048985450999999999 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.203482+0000", 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8900000000000006e-05 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:20.203525+0000", 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3210000000000001e-05 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:22.691 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:30:22.697 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-09-16T10:30:22.705 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:22.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:22.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:22.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:22.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:22.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:30:22.943 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:22.985 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:22.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:23.086 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:23.148 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:23.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:23.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:23.289 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:23.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:23.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 43 16 72495 72479 105.331 73.5625 0.0129157 0.00948818 2024-09-16T10:30:23.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:23.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:23.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:23.468 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:23.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:23 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:30:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:23 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:30:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:23 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:23 smithi165 ceph-mon[32696]: from='client.46439 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:30:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:23 smithi165 ceph-mon[32696]: osdmap e493: 8 total, 8 up, 8 in 2024-09-16T10:30:23.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:23 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:30:23.560 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:23.581 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:23.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:23.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[29993]: from='client.46439 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[29993]: osdmap e493: 8 total, 8 up, 8 in 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[39470]: from='client.46439 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:30:23.616 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[39470]: osdmap e493: 8 total, 8 up, 8 in 2024-09-16T10:30:23.617 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:23 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:30:23.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:23.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:23.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:23.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:23.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:23.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:23.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:23.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:23.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:24.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:24.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:24.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:24.193 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:24.239 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:24.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:24.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 44 16 73324 73308 104.114 51.8125 0.00364559 0.00959972 2024-09-16T10:30:24.340 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:24.387 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:24.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:24.489 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:24.511 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:24.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:24.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:24.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:24.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:24.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:30:24.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:24.777 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:24.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:24.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:24.917 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:24.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:25.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:24 smithi165 ceph-mon[32696]: pgmap v911: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:30:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:25.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2264001569' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:30:25.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:24 smithi165 ceph-mon[32696]: from='client.26563 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:30:25.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2979147712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:25.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:25.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:25.089 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:25.173 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:25.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[29993]: pgmap v911: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2264001569' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[29993]: from='client.26563 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2979147712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[39470]: pgmap v911: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:25.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2264001569' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:30:25.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[39470]: from='client.26563 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:30:25.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2979147712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:25.209 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:25.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:25.268 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 11 pgp_num to 16 2024-09-16T10:30:25.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:25.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 45 16 74317 74301 103.18 62.0625 0.0152797 0.00968654 2024-09-16T10:30:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:25.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:25.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:25.370 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:30:25.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:25.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:25.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:25.491 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:25.491 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:25.492 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:25.493 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:25.493 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:25.493 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:25.591 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:25.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:25.633 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:25.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:25.677 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:25.677 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:25.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:25.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:25.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:25.865 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:25.909 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:25.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:26.011 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:26.047 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:26.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:26.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:26.186 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:26.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:26.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:26.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 46 16 75578 75562 102.65 78.8125 0.0108971 0.00973687 2024-09-16T10:30:26.329 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:26.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:26.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:26.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:26.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:26.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:26 smithi165 ceph-mon[32696]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:30:26.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:26 smithi165 ceph-mon[32696]: osdmap e494: 8 total, 8 up, 8 in 2024-09-16T10:30:26.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:26 smithi165 ceph-mon[32696]: pgmap v913: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-16T10:30:26.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:30:26.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:26 smithi063 ceph-mon[39470]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:30:26.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:26 smithi063 ceph-mon[39470]: osdmap e494: 8 total, 8 up, 8 in 2024-09-16T10:30:26.545 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:26 smithi063 ceph-mon[39470]: pgmap v913: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-16T10:30:26.546 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:26 smithi063 ceph-mon[29993]: from='client.26563 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:30:26.546 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:26 smithi063 ceph-mon[29993]: osdmap e494: 8 total, 8 up, 8 in 2024-09-16T10:30:26.546 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:26 smithi063 ceph-mon[29993]: pgmap v913: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-16T10:30:26.573 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:26.610 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:26.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:26.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:26.750 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:26.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:26.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:26.874 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:26.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:26.884 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:26.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:27.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:27.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:27.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:27.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:27.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:27.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:27.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:27.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:27.316 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 47 16 76748 76732 102.021 73.125 0.0169215 0.00979642 2024-09-16T10:30:27.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:27.443 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:27.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:27.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:30:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:30:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:30:27.497 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:27.497 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:27.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:27.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:27.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:27.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:27.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:27.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:27.803 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:27.851 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:27.856 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:27.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:27.961 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:28.002 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:28.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:28.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:28.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:28.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.270 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:28.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:28.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 48 16 77855 77839 101.337 69.1875 0.0176822 0.00986318 2024-09-16T10:30:28.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:28.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:28.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:30:28.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:28.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:28.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:28.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[29993]: pgmap v914: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-16T10:30:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:30:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:30:28.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[39470]: pgmap v914: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-16T10:30:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:30:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:30:28.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:28 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:28.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:28 smithi165 ceph-mon[32696]: pgmap v914: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-16T10:30:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:28 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:30:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:28 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:30:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:28 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:28.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:28.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:28.863 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:28.953 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:28.990 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:28.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:29.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:29.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:29.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:29.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:29.257 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:29.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:29.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:30:29 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:30:29.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 49 16 79053 79037 100.797 74.875 0.00758346 0.00991579 2024-09-16T10:30:29.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:29.399 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:29.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:29.483 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:29.483 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:29.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:29.529 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:30:29.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:29.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:29.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:29.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:29.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:29.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:30:29.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:29.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:29.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:29.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:29 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3330009518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:29.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:29 smithi165 ceph-mon[32696]: from='client.26575 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:29.896 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:29.940 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:29.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:29.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:29.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:29 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3330009518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:29 smithi063 ceph-mon[29993]: from='client.26575 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:29.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:29 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3330009518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:29.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:29 smithi063 ceph-mon[39470]: from='client.26575 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:30.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:30.105 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:30.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:30.206 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:30.252 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:30.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:30.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 50 16 80276 80260 100.31 76.4375 0.010474 0.0099639 2024-09-16T10:30:30.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:30:30.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:30.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:30.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:30.480 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:30.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:30.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:30.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:30.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:30 smithi165 ceph-mon[32696]: pgmap v915: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-16T10:30:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:30 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:30 smithi165 ceph-mon[32696]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:30.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:30 smithi165 ceph-mon[32696]: osdmap e495: 8 total, 8 up, 8 in 2024-09-16T10:30:30.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[29993]: pgmap v915: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[29993]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[29993]: osdmap e495: 8 total, 8 up, 8 in 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[39470]: pgmap v915: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-16T10:30:30.672 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:30.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[39470]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:30.673 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:30 smithi063 ceph-mon[39470]: osdmap e495: 8 total, 8 up, 8 in 2024-09-16T10:30:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:30.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:30.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:30.774 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:30.789 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:30.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:30.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:30.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:30.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:31.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:31.111 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:31.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:31.153 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:31.154 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:31.200 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:31.200 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:31.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:31.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:31.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:31.242 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:31.242 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:31.277 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:31.312 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:31.312 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:31.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 51 16 81472 81456 99.8084 74.75 0.0150153 0.0100145 2024-09-16T10:30:31.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:31.347 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:31.347 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:31.374 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:31.374 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:31.382 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:31.382 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:31.384 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:31.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:31.416 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:31.417 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:31.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:31.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:31.628 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:31.669 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:31.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:31.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:31.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:31.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:31.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:31.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:31.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:32.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:32.071 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:32.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:32.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:32.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:32.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:32.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:30:32.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 52 16 82728 82712 99.3985 78.5 0.0125001 0.0100557 2024-09-16T10:30:32.317 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:32 smithi165 ceph-mon[32696]: osdmap e496: 8 total, 8 up, 8 in 2024-09-16T10:30:32.317 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:32 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/482613577' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:32.318 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:32 smithi165 ceph-mon[32696]: pgmap v918: 168 pgs: 168 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 827 B/s rd, 55 MiB/s wr, 886 op/s 2024-09-16T10:30:32.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:32.360 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:32.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:32.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:30:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:32 smithi063 ceph-mon[39470]: osdmap e496: 8 total, 8 up, 8 in 2024-09-16T10:30:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:32 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/482613577' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:32.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:32 smithi063 ceph-mon[39470]: pgmap v918: 168 pgs: 168 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 827 B/s rd, 55 MiB/s wr, 886 op/s 2024-09-16T10:30:32.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:32 smithi063 ceph-mon[29993]: osdmap e496: 8 total, 8 up, 8 in 2024-09-16T10:30:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:32 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/482613577' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:32.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:32 smithi063 ceph-mon[29993]: pgmap v918: 168 pgs: 168 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 827 B/s rd, 55 MiB/s wr, 886 op/s 2024-09-16T10:30:32.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:32.507 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:32.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:32.529 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:30:32.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:30:32.529 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:30:32.530 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:30:32.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:32.650 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:32.679 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:32.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:32.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:32.845 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:32.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:32.946 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:32.950 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:32.998 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:32.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:33.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:33.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:33.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:33.195 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:30:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:30:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:30:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:30:33.196 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:30:33.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:33.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:33.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:33.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 53 16 83674 83658 98.6386 59.125 0.00813394 0.0101331 2024-09-16T10:30:33.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:33.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:33.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:33.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:33.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:33.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:33.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:33.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:33.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:33.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:33.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:33.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:33.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:33.919 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:33.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:34.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:34.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:34.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:34.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:30:34.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:34.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:30:34.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:34.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:34.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:34.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 54 16 84802 84786 98.1174 70.5 0.012372 0.0101865 2024-09-16T10:30:34.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:34.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:34.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:34.474 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:34.474 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:30:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:34.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:34.576 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:34.601 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:34.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:34.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:34.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:34.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:34.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:34 smithi165 ceph-mon[32696]: pgmap v919: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:30:34.835 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:34.865 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:34.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:34.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:34 smithi063 ceph-mon[39470]: pgmap v919: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:30:34.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:34 smithi063 ceph-mon[29993]: pgmap v919: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-16T10:30:34.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:35.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:35.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:35.079 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:30:35.079 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:30:35.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216464 10.11 e443/442 10:8bbe5995:::benchmark_data_smithi063_149043_object84027:head v 443'11994, mlcod=389'6270)", 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.082241526, 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180527202, 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994108+0000", 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1179999999999999e-06 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994115+0000", 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7109999999999998e-06 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994124+0000", 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8120000000000002e-06 2024-09-16T10:30:35.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994131+0000", 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2929999999999997e-06 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996650+0000", 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002519777 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996675+0000", 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4800999999999999e-05 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174595+0000", 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17792013200000001 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174634+0000", 2024-09-16T10:30:35.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8557999999999999e-05 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(10.1 443/442 [PushOp(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head, version: 391'6545, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head@391'6545, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.07959415400001, 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19788486499999999, 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.085 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996756+0000", 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4810000000000001e-06 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996812+0000", 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6508999999999997e-05 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996831+0000", 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9009e-05 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996849+0000", 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7612999999999999e-05 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193170+0000", 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19632173999999999 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193181+0000", 2024-09-16T10:30:35.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0138e-05 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.194639+0000", 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001458375 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216466 10.9 e443/442 10:96e6e3aa:::benchmark_data_smithi063_149043_object84029:head v 443'12076, mlcod=386'6101)", 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.079156678, 2024-09-16T10:30:35.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18400087900000001, 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997193+0000", 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2759999999999999e-06 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997200+0000", 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8950000000000001e-06 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997209+0000", 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2010000000000005e-06 2024-09-16T10:30:35.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997226+0000", 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7059e-05 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002158+0000", 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049319699999999999 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002173+0000", 2024-09-16T10:30:35.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4661000000000001e-05 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181097+0000", 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.178924214 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181192+0000", 2024-09-16T10:30:35.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5602999999999999e-05 2024-09-16T10:30:35.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.094 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.094 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216473 10.9 e443/442 10:9420f438:::benchmark_data_smithi063_149043_object84036:head v 443'12077, mlcod=386'6101)", 2024-09-16T10:30:35.094 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.075526709, 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18039601599999999, 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000823+0000", 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.221e-06 2024-09-16T10:30:35.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000830+0000", 2024-09-16T10:30:35.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1319999999999996e-06 2024-09-16T10:30:35.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000840+0000", 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0378e-05 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000847+0000", 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4710000000000004e-06 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007645+0000", 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067982770000000001 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007677+0000", 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2172999999999998e-05 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181195+0000", 2024-09-16T10:30:35.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17351757300000001 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181218+0000", 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2790999999999999e-05 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216481 10.11 e443/442 10:8cd97c52:::benchmark_data_smithi063_149043_object84044:head v 443'11995, mlcod=389'6270)", 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.070368707, 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18177754400000001, 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.101 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005981+0000", 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.618e-06 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005991+0000", 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4660000000000005e-06 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006004+0000", 2024-09-16T10:30:35.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2964e-05 2024-09-16T10:30:35.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006012+0000", 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5939999999999994e-06 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007635+0000", 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001622706 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007678+0000", 2024-09-16T10:30:35.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3282000000000003e-05 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187680+0000", 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180001892 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187757+0000", 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7021999999999995e-05 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.106 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216483 10.9 e443/442 10:91f62e6a:::benchmark_data_smithi063_149043_object84046:head v 443'12078, mlcod=386'6101)", 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.06922005, 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.185520875, 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007129+0000", 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.198e-06 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007136+0000", 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7780000000000002e-06 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007158+0000", 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2013000000000001e-05 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007166+0000", 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999999e-06 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014305+0000", 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071398319999999996 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014326+0000", 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0767999999999998e-05 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192615+0000", 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17828850800000001 2024-09-16T10:30:35.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192649+0000", 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4366999999999998e-05 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216472 10.8 e443/442 10:167e16b6:::benchmark_data_smithi063_149043_object84035:head v 443'11858, mlcod=389'6215)", 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:30:35.111 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.067563663, 2024-09-16T10:30:35.112 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:35.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.167553278, 2024-09-16T10:30:35.113 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.113 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008786+0000", 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.296e-06 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008820+0000", 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4329999999999998e-05 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008830+0000", 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3360000000000001e-06 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008836+0000", 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3040000000000001e-06 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010938+0000", 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021016780000000001 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010959+0000", 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1454000000000002e-05 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176290+0000", 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16533061800000001 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176338+0000", 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8262e-05 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.118 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216489 10.c e443/442 10:36cb7a7c:::benchmark_data_smithi063_149043_object84052:head v 443'13145, mlcod=443'13143)", 2024-09-16T10:30:35.118 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:30:35.118 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.06369705199999, 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16334574800000001, 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:30:35.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012653+0000", 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.606e-06 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012661+0000", 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0469999999999994e-06 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012673+0000", 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2490000000000001e-05 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012682+0000", 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3450000000000006e-06 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012753+0000", 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1565999999999999e-05 2024-09-16T10:30:35.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012773+0000", 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9942999999999999e-05 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175914+0000", 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16314114399999999 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175997+0000", 2024-09-16T10:30:35.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2607000000000004e-05 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216490 10.10 e443/441 10:0ed1dccc:::benchmark_data_smithi063_149043_object84053:head v 443'12109, mlcod=443'12107)", 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.061675116, 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16562529300000001, 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014675+0000", 2024-09-16T10:30:35.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.593e-06 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014687+0000", 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2071e-05 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014697+0000", 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0148e-05 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014704+0000", 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.613e-06 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174625+0000", 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15992183700000001 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6707e-05 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180235+0000", 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055831880000000002 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180298+0000", 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3136e-05 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216488 10.11 e443/442 10:8a4eba15:::benchmark_data_smithi063_149043_object84051:head v 443'11996, mlcod=389'6270)", 2024-09-16T10:30:35.127 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.06074404100001, 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.177066683, 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015606+0000", 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.404e-06 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015617+0000", 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1769e-05 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015628+0000", 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0489000000000001e-05 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015634+0000", 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2960000000000004e-06 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174659+0000", 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15902495699999999 2024-09-16T10:30:35.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174681+0000", 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1778999999999999e-05 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192652+0000", 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017971311 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192671+0000", 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8678e-05 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216491 10.10 e443/441 10:0e46f635:::benchmark_data_smithi063_149043_object84054:head v 443'12110, mlcod=443'12107)", 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.05905769, 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.163036928, 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.132 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017292+0000", 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.443e-06 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017300+0000", 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174635+0000", 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.157335014 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174657+0000", 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1985e-05 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176330+0000", 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016733029999999999 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176359+0000", 2024-09-16T10:30:35.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9564e-05 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180301+0000", 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00394174 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180328+0000", 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6319e-05 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216495 10.c e443/442 10:33a8a0ac:::benchmark_data_smithi063_149043_object84058:head v 443'13146, mlcod=443'13143)", 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.05776269499999, 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.160016309, 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018587+0000", 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3820000000000001e-06 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018595+0000", 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4759999999999998e-06 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174544+0000", 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15594872200000001 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174566+0000", 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1979000000000001e-05 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174626+0000", 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9899999999999999e-05 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.561e-05 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178550+0000", 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038981969999999999 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178602+0000", 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2043000000000003e-05 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221476 10.9 e446/442 10:900b709d:::benchmark_data_smithi063_149043_object89039:head v 446'12225, mlcod=386'6101)", 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 118.997699498, 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50937916299999997, 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078650+0000", 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.091e-06 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078657+0000", 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078665+0000", 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6619999999999994e-06 2024-09-16T10:30:35.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078671+0000", 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1700000000000002e-06 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078762+0000", 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0346999999999994e-05 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078786+0000", 2024-09-16T10:30:35.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4314000000000001e-05 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587999+0000", 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.509212514 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.588028+0000", 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9326999999999999e-05 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221483 10.c e446/442 10:32ed840b:::benchmark_data_smithi063_149043_object89046:head v 446'13312, mlcod=445'13310)", 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 118.996970412, 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50397308900000004, 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:30:35.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079379+0000", 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.395e-06 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079385+0000", 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5639999999999996e-06 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079392+0000", 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.481e-06 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079398+0000", 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8950000000000003e-06 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079450+0000", 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1786999999999997e-05 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079469+0000", 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8841000000000001e-05 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583301+0000", 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50383221600000005 2024-09-16T10:30:35.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583351+0000", 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9910000000000002e-05 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221482 10.1f e446/441 10:ffb425b8:::benchmark_data_smithi063_149043_object89045:head v 446'13070, mlcod=445'13068)", 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 118.99676295899999, 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50798407899999998, 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.150 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079587+0000", 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.187e-06 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079593+0000", 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0929999999999999e-06 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079600+0000", 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5070000000000003e-06 2024-09-16T10:30:35.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079606+0000", 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7509999999999997e-06 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583349+0000", 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50374310499999997 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583378+0000", 2024-09-16T10:30:35.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8649e-05 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587530+0000", 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041521559999999997 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587569+0000", 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9631000000000001e-05 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:249520 10.1 10:827f4c35:::benchmark_data_smithi063_149043_object117083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.619383780000007, 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11004649, 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 249520 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.155 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456966+0000", 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.689e-06 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456972+0000", 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3410000000000002e-06 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456983+0000", 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1039999999999999e-05 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456988+0000", 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6579999999999999e-06 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.457040+0000", 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2166000000000002e-05 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.457143+0000", 2024-09-16T10:30:35.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102909 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.458594+0000", 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001450914 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.460804+0000", 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022102369999999999 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:35.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.461370+0000", 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056559799999999995 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.566893+0000", 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105523061 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.566950+0000", 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6637e-05 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.567011+0000", 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1240000000000003e-05 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:251245 10.1 10:810fe972:::benchmark_data_smithi063_149043_object118808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.186033369, 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083403509000000001, 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.161 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 251245 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890317+0000", 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.809e-06 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890322+0000", 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5219999999999998e-06 2024-09-16T10:30:35.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890333+0000", 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1069e-05 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890342+0000", 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7590000000000003e-06 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890709+0000", 2024-09-16T10:30:35.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036659099999999999 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890839+0000", 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000130147 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.891118+0000", 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027897099999999998 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:30:35.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.892553+0000", 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014351450000000001 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.892975+0000", 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042191700000000001 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973569+0000", 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080593926999999996 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973658+0000", 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9295000000000005e-05 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973718+0000", 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0356999999999998e-05 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.167 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDelete(10.8 e460,442 10:146751e1:::benchmark_data_smithi063_149043_object32211:head,408'10091)", 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.968104996999998, 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111867522, 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108244+0000", 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2900000000000002e-07 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108248+0000", 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.121e-06 2024-09-16T10:30:35.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219378+0000", 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11112931500000001 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219407+0000", 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9363e-05 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220044+0000", 2024-09-16T10:30:35.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063718900000000003 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220053+0000", 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9199999999999993e-06 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220111+0000", 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7685000000000001e-05 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.171 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:254612 10.22 e460/441 10:460a596d:::benchmark_data_smithi063_149043_object122175:head v 460'12456, mlcod=460'12454)", 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.956988039999999, 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101663158, 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:30:35.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119362+0000", 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4839999999999999e-06 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119368+0000", 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.348e-06 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219363+0000", 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099994825999999995 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219389+0000", 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6383000000000001e-05 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219541+0000", 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151476 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219560+0000", 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9440999999999999e-05 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220981+0000", 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001421095 2024-09-16T10:30:35.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.221023+0000", 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2104999999999998e-05 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:72894 11.5 e493/476 11:ad5dc03e:::benchmark_data_smithi063_176907_object72893:head v 493'4542, mlcod=493'4539)", 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.344239984, 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060735732000000001, 2024-09-16T10:30:35.176 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:30:35.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732109+0000", 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.119e-06 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732131+0000", 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1999000000000001e-05 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732141+0000", 2024-09-16T10:30:35.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4579999999999999e-06 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732147+0000", 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2759999999999997e-06 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732195+0000", 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8288999999999998e-05 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:30:35.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732215+0000", 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9961000000000001e-05 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.792747+0000", 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060531198000000001 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.792844+0000", 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7431999999999998e-05 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.180 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:30:35.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:30:35.181 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:30:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:35.222 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:35.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:35.317 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 55 16 85828 85812 97.4993 64.125 0.00314998 0.0102484 2024-09-16T10:30:35.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:35.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:35.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:35.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:35.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:35.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:35.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:35.587 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:35.608 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:35.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:35.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:35.733 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:35.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:35.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:35 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:30:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:35 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:30:35.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:35 smithi165 ceph-mon[32696]: osdmap e497: 8 total, 8 up, 8 in 2024-09-16T10:30:35.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:35.868 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:35.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:35.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:35 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:30:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:35 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:30:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:35 smithi063 ceph-mon[29993]: osdmap e497: 8 total, 8 up, 8 in 2024-09-16T10:30:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:35 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:30:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:35 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:30:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:35 smithi063 ceph-mon[39470]: osdmap e497: 8 total, 8 up, 8 in 2024-09-16T10:30:35.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:36.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:36.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:36.096 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:30:36.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:36.213 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:36.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:36.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:36.319 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 56 16 86821 86805 96.8656 62.0625 0.0173753 0.0103186 2024-09-16T10:30:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:36.368 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:36.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:36.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:36.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:36.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:36.521 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:36.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:36.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:36.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:36 smithi063 ceph-mon[29993]: pgmap v921: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-16T10:30:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:36 smithi063 ceph-mon[29993]: osdmap e498: 8 total, 8 up, 8 in 2024-09-16T10:30:36.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:36 smithi063 ceph-mon[39470]: pgmap v921: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-16T10:30:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:36 smithi063 ceph-mon[39470]: osdmap e498: 8 total, 8 up, 8 in 2024-09-16T10:30:36.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:36.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:36.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:36.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:36 smithi165 ceph-mon[32696]: pgmap v921: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-16T10:30:36.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:36 smithi165 ceph-mon[32696]: osdmap e498: 8 total, 8 up, 8 in 2024-09-16T10:30:36.855 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:36.910 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:36.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:37.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:37.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:37.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:37.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:37.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:37.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:37.272 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:37.312 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:37.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:37.318 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 57 16 87890 87874 96.3383 66.8125 0.0235654 0.0103739 2024-09-16T10:30:37.327 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:30:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:30:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:30:37.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:37.459 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:37.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:37.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:37.561 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:30:37.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:37 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1401675672' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:37.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:37 smithi063 ceph-mon[29993]: from='client.26587 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:37.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:37 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1401675672' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:37.585 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:37 smithi063 ceph-mon[39470]: from='client.26587 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:37.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:37.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:37.668 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:37.668 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:37.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:37.707 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:37.707 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:37.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:37.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:37.753 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:37.753 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 499, 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:30:37.549346+0000", 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:28:27.035727+0000", 2024-09-16T10:30:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:22:01.866838+0000", 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:30:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 19, 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 11, 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:30:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 21, 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 21, 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 21, 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 21, 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 21, 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:30:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.b", 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 275, 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 274, 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 274, 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "357", 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "275", 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "306", 2024-09-16T10:30:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:30:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:30:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.6699999570846558, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.6699999570846558, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.8999999761581421, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.8999999761581421, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:30:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:30:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:30:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:30:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:30:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:30:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.3299999237060547, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.3299999237060547, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-16T10:30:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:30:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:30:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:30:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:30:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:30:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:30:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:30:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:30:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:30:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:30:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:30:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:30:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:30:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.9700000286102295, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.9700000286102295, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2.25, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2.25, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:30:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:30:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:30:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:30:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:30:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:30:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:30:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:30:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:30:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:30:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.3200000524520874, 2024-09-16T10:30:37.826 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:37.827 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:37.827 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.3200000524520874, 2024-09-16T10:30:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T10:30:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5, 2024-09-16T10:30:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5, 2024-09-16T10:30:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:30:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:30:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:30:37.830 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:30:37.831 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:30:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:30:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T10:30:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 11, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_4", 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:29:33.629083+0000", 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:30:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:30:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "494", 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:30:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:30:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:30:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:30:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:30:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 476, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:30:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:30:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:30:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 476, 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:30:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:30:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 476, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:30:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:30:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 476, 2024-09-16T10:30:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:30:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:30:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 476, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.864 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.864 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:30:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:37.865 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:37.865 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:30:37.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.865 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.866 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:30:37.867 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.868 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 476, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:30:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.873 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.876 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:30:37.876 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:30:37.876 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:30:37.876 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:30:37.876 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0.293609619140625, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 54, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 386, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 441, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 452, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 387, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:30:37.877 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:30:37.878 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:30:37.879 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/1921603083", 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/1921603083", 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1921603083", 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1921603083", 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:29:57.767687+0000", 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:30:34.122190+0000", 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:30:21.344326+0000", 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:29:16.295749+0000", 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:29:57.057220+0000", 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:29:24.503397+0000", 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:29:54.855663+0000", 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:30:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:29:31.576907+0000", 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 386 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [ 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.5", 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: 5 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.0", 2024-09-16T10:30:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.e", 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 7, 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 3 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.10", 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 6, 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.14", 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 1 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.13", 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:30:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [], 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:30:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:30:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:30:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:37.897 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:30:37.898 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:37.899 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:37.910 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:37.911 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:37.911 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:37.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:37.946 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:37.946 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:38.013 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:38.050 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:38.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:38.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:37 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1401675672' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:37 smithi165 ceph-mon[32696]: from='client.26587 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:30:38.152 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:38.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:38.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:38.319 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 58 16 88932 88916 95.7983 65.125 0.0135649 0.0104338 2024-09-16T10:30:38.333 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:38.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:38.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:38.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:38.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:38.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:38.612 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:38.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:38.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:38.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:38.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:38.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:38.876 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:38.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:38.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: pgmap v923: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 800 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-16T10:30:38.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: from='client.26587 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: osdmap e499: 8 total, 8 up, 8 in 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/6141870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1359814251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: pgmap v923: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 800 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:38.902 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: from='client.26587 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:30:38.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: osdmap e499: 8 total, 8 up, 8 in 2024-09-16T10:30:38.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/6141870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:38.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1359814251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:30:38.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:30:38.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:38 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:30:38.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:39.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: pgmap v923: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 800 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: from='client.26587 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: osdmap e499: 8 total, 8 up, 8 in 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/6141870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1359814251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:30:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:38 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:30:39.112 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:39.146 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:39.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:39.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:39.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:39.319 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 59 16 90170 90154 95.486 77.375 0.00470103 0.010467 2024-09-16T10:30:39.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:39.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:39.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:39.536 INFO:tasks.thrashosds.thrasher:j is {'epoch': 499, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:30:37.549346+0000', 'last_up_change': '2024-09-16T10:28:27.035727+0000', 'last_in_change': '2024-09-16T10:22:01.866838+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 275, 'last_epoch_started': 274, 'last_epoch_clean': 274, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '357', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '275', 'last_force_op_resend_preluminous': '306', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6699999570846558, 'score_stable': 1.6699999570846558, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-16T10:29:33.629083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '494', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.293609619140625, 'last_clean_begin': 54, 'last_clean_end': 386, 'up_from': 441, 'up_thru': 452, 'down_at': 387, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1921603083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1921603083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1921603083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1921603083}]}, 'public_addr': '172.21.15.165:6812/1921603083', 'cluster_addr': '172.21.15.165:6813/1921603083', 'heartbeat_back_addr': '172.21.15.165:6815/1921603083', 'heartbeat_front_addr': '172.21.15.165:6814/1921603083', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:29:57.767687+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:30:34.122190+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:30:21.344326+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:29:16.295749+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:29:57.057220+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:29:24.503397+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:29:54.855663+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:29:31.576907+0000', 'dead_epoch': 386}], 'pg_upmap': [{'pgid': '1.5', 'osds': [6, 3, 5]}, {'pgid': '4.0', 'osds': [4, 6, 2]}], 'pg_upmap_items': [{'pgid': '3.e', 'mappings': [{'from': 7, 'to': 3}, {'from': 4, 'to': 5}, {'from': 1, 'to': 0}]}, {'pgid': '3.10', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:30:39.536 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.10 2024-09-16T10:30:39.537 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap-items 3.10 2024-09-16T10:30:39.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:39.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:30:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:30:39.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:39.692 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:39.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:39.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:39.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[29993]: 11.1 scrub starts 2024-09-16T10:30:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[29993]: 4.1b scrub starts 2024-09-16T10:30:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[29993]: 4.1b scrub ok 2024-09-16T10:30:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[39470]: 11.1 scrub starts 2024-09-16T10:30:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[39470]: 4.1b scrub starts 2024-09-16T10:30:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[39470]: 4.1b scrub ok 2024-09-16T10:30:39.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:39.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:39 smithi165 ceph-mon[32696]: 11.1 scrub starts 2024-09-16T10:30:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:39 smithi165 ceph-mon[32696]: 4.1b scrub starts 2024-09-16T10:30:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:39 smithi165 ceph-mon[32696]: 4.1b scrub ok 2024-09-16T10:30:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:40.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:40.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:40.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:40.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:40.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:40.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:40.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:40.319 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:30:40.318775+0000 min lat: 0.00120603 max lat: 0.224582 avg lat: 0.0105093 2024-09-16T10:30:40.319 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:30:40.320 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 60 16 91336 91320 95.109 72.875 0.0057463 0.0105093 2024-09-16T10:30:40.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:40.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:40.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:40.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:40.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:40.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:40.614 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:40.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:40.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:40.713 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:40.713 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:30:40.713 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:30:40.713 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[29993]: pgmap v925: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-16T10:30:40.713 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:30:40.714 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:30:40.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:40.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:30:40.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:30:40.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[39470]: pgmap v925: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-16T10:30:40.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:30:40.714 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:40 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:30:40.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:40.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:40.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:40.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:40.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:40.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:41.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:41.038 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:41.041 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:41.041 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:41.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:40 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:30:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:40 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:30:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:40 smithi165 ceph-mon[32696]: pgmap v925: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-16T10:30:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:40 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:30:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:40 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:30:41.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:41.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:41.158 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:41.202 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:41.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:41.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:41.320 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 61 16 92512 92496 94.7541 73.5 0.00794119 0.010549 2024-09-16T10:30:41.342 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:41.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:41.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:30:41.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:41.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:41.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:41.574 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:41.621 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:41.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:30:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:30:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: 6.15 scrub starts 2024-09-16T10:30:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: 6.15 scrub ok 2024-09-16T10:30:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: 11.1 scrub ok 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: osdmap e500: 8 total, 8 up, 8 in 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: 6.f scrub starts 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[29993]: 6.f scrub ok 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: 6.15 scrub starts 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: 6.15 scrub ok 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: 11.1 scrub ok 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: osdmap e500: 8 total, 8 up, 8 in 2024-09-16T10:30:41.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: 6.f scrub starts 2024-09-16T10:30:41.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:41 smithi063 ceph-mon[39470]: 6.f scrub ok 2024-09-16T10:30:41.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:41.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:41.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:41.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:41.903 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:41.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:42.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:42.043 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:42.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:42.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:30:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:30:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: 6.15 scrub starts 2024-09-16T10:30:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: 6.15 scrub ok 2024-09-16T10:30:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: 11.1 scrub ok 2024-09-16T10:30:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: osdmap e500: 8 total, 8 up, 8 in 2024-09-16T10:30:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: 6.f scrub starts 2024-09-16T10:30:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:41 smithi165 ceph-mon[32696]: 6.f scrub ok 2024-09-16T10:30:42.136 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:30:42.144 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:42.223 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:42.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:42.321 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 62 16 93626 93610 94.3467 69.625 0.0113693 0.0105941 2024-09-16T10:30:42.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:42.374 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:42.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:42.458 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:42.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:42.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:30:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:42.551 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:42.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:42.652 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:42.715 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:42.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:42.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:42.833 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:42 smithi063 ceph-mon[29993]: 6.1c scrub starts 2024-09-16T10:30:42.834 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:42 smithi063 ceph-mon[29993]: 6.1c scrub ok 2024-09-16T10:30:42.834 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:42 smithi063 ceph-mon[29993]: pgmap v927: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 883 op/s 2024-09-16T10:30:42.834 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:42 smithi063 ceph-mon[39470]: 6.1c scrub starts 2024-09-16T10:30:42.834 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:42 smithi063 ceph-mon[39470]: 6.1c scrub ok 2024-09-16T10:30:42.834 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:42 smithi063 ceph-mon[39470]: pgmap v927: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 883 op/s 2024-09-16T10:30:42.860 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:42.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:42.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:42.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:42.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:43.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:42 smithi165 ceph-mon[32696]: 6.1c scrub starts 2024-09-16T10:30:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:42 smithi165 ceph-mon[32696]: 6.1c scrub ok 2024-09-16T10:30:43.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:42 smithi165 ceph-mon[32696]: pgmap v927: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 883 op/s 2024-09-16T10:30:43.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:43.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:43.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:43.193 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:43.223 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:43.280 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:43.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:43.321 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 63 16 94593 94577 93.8084 60.4375 0.0143422 0.0106556 2024-09-16T10:30:43.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:43.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:43.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:43.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:43.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:43.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:43.618 INFO:teuthology.orchestra.run.smithi063.stderr:clear 3.10 pg_upmap_items mapping 2024-09-16T10:30:43.621 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:30:43.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:43.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:43.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:43.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:43.802 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:43.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:43.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: 5.1b scrub starts 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: 5.1b scrub ok 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: 1.f scrub ok 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1158639627' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"} : dispatch 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: from='client.26602 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"} : dispatch 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2444433893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:43.870 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[29993]: from='client.26608 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: 5.1b scrub starts 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: 5.1b scrub ok 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: 1.f scrub ok 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1158639627' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"} : dispatch 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: from='client.26602 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"} : dispatch 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2444433893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:43.871 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:43 smithi063 ceph-mon[39470]: from='client.26608 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:43.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:43.949 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:43.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:44.023 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:44.023 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:44.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:44.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: 5.1b scrub starts 2024-09-16T10:30:44.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: 5.1b scrub ok 2024-09-16T10:30:44.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:30:44.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:30:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:30:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: 1.f scrub ok 2024-09-16T10:30:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1158639627' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"} : dispatch 2024-09-16T10:30:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: from='client.26602 ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"} : dispatch 2024-09-16T10:30:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2444433893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:44.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:43 smithi165 ceph-mon[32696]: from='client.26608 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:30:44.092 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:44.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:44.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:44.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:44.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:44.321 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 64 16 95745 95729 93.4677 72 0.00547917 0.0106938 2024-09-16T10:30:44.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:44.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:44.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:44.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:44.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:44.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:44.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:44.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:44.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:30:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:30:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: pgmap v928: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-16T10:30:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:30:44.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: from='client.26602 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"}]': finished 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: from='client.26608 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: osdmap e501: 8 total, 8 up, 8 in 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: pgmap v928: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: from='client.26602 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"}]': finished 2024-09-16T10:30:44.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: from='client.26608 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:44.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: osdmap e501: 8 total, 8 up, 8 in 2024-09-16T10:30:44.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:30:44.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:44 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:30:44.719 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:44.756 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:44.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:44.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:44.895 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:44.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:44.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:45.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:45.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: pgmap v928: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: from='client.26602 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.10"}]': finished 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: from='client.26608 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:30:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: osdmap e501: 8 total, 8 up, 8 in 2024-09-16T10:30:45.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:30:45.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:44 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:30:45.129 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:45.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:45.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:45.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:45.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:45.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:45.321 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 65 16 96906 96890 93.146 72.5625 0.00632152 0.0107309 2024-09-16T10:30:45.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:30:45.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:45.486 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:45.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:45.587 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:45.636 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:45.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:45.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:45.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:45.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:45.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:45.865 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:45.910 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:45.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:45.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[29993]: 6.7 scrub starts 2024-09-16T10:30:45.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[29993]: 6.7 scrub ok 2024-09-16T10:30:45.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:30:45.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:30:45.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[39470]: 6.7 scrub starts 2024-09-16T10:30:45.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[39470]: 6.7 scrub ok 2024-09-16T10:30:45.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:30:45.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:45 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:30:46.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:46.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:45 smithi165 ceph-mon[32696]: 6.7 scrub starts 2024-09-16T10:30:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:45 smithi165 ceph-mon[32696]: 6.7 scrub ok 2024-09-16T10:30:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:45 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:30:46.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:45 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:30:46.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:46.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:46.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:46.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:46.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:46.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:46.321 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 66 16 98043 98027 92.8113 71.0625 0.0129451 0.0107699 2024-09-16T10:30:46.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:46.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:46.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:46.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:46.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:46.556 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:46.590 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:46.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:46.682 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:46.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:46.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:46.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:46.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:46.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:46.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[29993]: 3.1b scrub starts 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[29993]: 3.1b scrub ok 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[29993]: pgmap v930: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[29993]: osdmap e502: 8 total, 8 up, 8 in 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:30:46.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:30:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[39470]: 3.1b scrub starts 2024-09-16T10:30:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[39470]: 3.1b scrub ok 2024-09-16T10:30:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[39470]: pgmap v930: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:30:46.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:46 smithi063 ceph-mon[39470]: osdmap e502: 8 total, 8 up, 8 in 2024-09-16T10:30:47.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:47.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:46 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:30:47.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:46 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:30:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:46 smithi165 ceph-mon[32696]: 3.1b scrub starts 2024-09-16T10:30:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:46 smithi165 ceph-mon[32696]: 3.1b scrub ok 2024-09-16T10:30:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:46 smithi165 ceph-mon[32696]: pgmap v930: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:30:47.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:46 smithi165 ceph-mon[32696]: osdmap e502: 8 total, 8 up, 8 in 2024-09-16T10:30:47.077 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:47.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:47.178 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:47.225 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:47.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:47.288 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:30:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:30:47.321 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 67 16 99270 99254 92.5706 76.6875 0.0187395 0.0107978 2024-09-16T10:30:47.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:47.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:47.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:47.386 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:47.386 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:47.431 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:47.431 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:47.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:47.466 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:47.481 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:47.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:47.501 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:47.502 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:47.533 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:47.533 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:47.565 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:47.565 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:47.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:47.600 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:47.600 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:47.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:47.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:47.637 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:47.637 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:47.716 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:47.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:47.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:47.912 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:47.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:47.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[29993]: 1.4 scrub ok 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[29993]: 1.7 scrub ok 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4262097515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:30:47.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[39470]: 1.4 scrub ok 2024-09-16T10:30:47.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:30:47.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[39470]: 1.7 scrub ok 2024-09-16T10:30:47.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4262097515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:48.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:48.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:48.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:48.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:47 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:30:48.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:47 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:30:48.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:47 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:30:48.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:47 smithi165 ceph-mon[32696]: 1.4 scrub ok 2024-09-16T10:30:48.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:47 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:30:48.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:47 smithi165 ceph-mon[32696]: 1.7 scrub ok 2024-09-16T10:30:48.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4262097515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:48.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:48.186 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:48.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:48.199 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:30:48.200 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:30:48.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:30:48.200 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:30:48.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.200 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9:dc100d6c:::benchmark_data_smithi063_124717_object60003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.201 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:30:48.201 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31227024100002, 2024-09-16T10:30:48.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1260944720000001, 2024-09-16T10:30:48.201 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.201 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.201 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:30:48.202 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.202 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.202 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.202 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:30:48.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.203 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:30:48.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.204 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881599+0000", 2024-09-16T10:30:48.204 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881601+0000", 2024-09-16T10:30:48.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.841e-06 2024-09-16T10:30:48.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881621+0000", 2024-09-16T10:30:48.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9976e-05 2024-09-16T10:30:48.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881629+0000", 2024-09-16T10:30:48.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2120000000000003e-06 2024-09-16T10:30:48.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881636+0000", 2024-09-16T10:30:48.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.472e-06 2024-09-16T10:30:48.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881700+0000", 2024-09-16T10:30:48.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4152000000000005e-05 2024-09-16T10:30:48.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:30:48.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881707+0000", 2024-09-16T10:30:48.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3100000000000003e-06 2024-09-16T10:30:48.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.210 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.210 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882746+0000", 2024-09-16T10:30:48.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001038574 2024-09-16T10:30:48.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882755+0000", 2024-09-16T10:30:48.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0480000000000004e-06 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007615+0000", 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.124860408 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.007694+0000", 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8479000000000005e-05 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9:d2f9f966:::benchmark_data_smithi063_124717_object60060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31222311699997, 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1652373410000001, 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.214 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881646+0000", 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881647+0000", 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7599999999999997e-07 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881671+0000", 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4284e-05 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881677+0000", 2024-09-16T10:30:48.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3619999999999997e-06 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881681+0000", 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1980000000000003e-06 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881840+0000", 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015907000000000001 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting_for_map not empty", 2024-09-16T10:30:48.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881846+0000", 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6250000000000004e-06 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883410+0000", 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001564085 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883417+0000", 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4749999999999996e-06 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.218 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008717+0000", 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.125299746 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008819+0000", 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102545 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.008934+0000", 2024-09-16T10:30:48.219 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000114923 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044911+0000", 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035976248000000002 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044965+0000", 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4756000000000002e-05 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045041+0000", 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5828000000000007e-05 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045130+0000", 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9221000000000005e-05 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045181+0000", 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0691e-05 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046884+0000", 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001702708 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9:de195d7b:::benchmark_data_smithi063_124717_object60066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31217404799997, 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1671491920000001, 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.222 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881696+0000", 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5400000000000001e-07 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881718+0000", 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1948999999999999e-05 2024-09-16T10:30:48.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881724+0000", 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7080000000000002e-06 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881728+0000", 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.946e-06 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883442+0000", 2024-09-16T10:30:48.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00171441 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883449+0000", 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4309999999999999e-06 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010618+0000", 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1271692849999999 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010714+0000", 2024-09-16T10:30:48.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6600000000000003e-05 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.010840+0000", 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012543600000000001 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.044920+0000", 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034079854 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.227 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.045083+0000", 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000163688 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047061+0000", 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019780269999999998 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047155+0000", 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3529999999999994e-05 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047228+0000", 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2822000000000002e-05 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048845+0000", 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016169520000000001 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9:d28fbaa4:::benchmark_data_smithi063_124717_object60144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31211735400001, 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.171973098, 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881752+0000", 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881753+0000", 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8699999999999995e-07 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881772+0000", 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9159000000000001e-05 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881779+0000", 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.362e-06 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881784+0000", 2024-09-16T10:30:48.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2580000000000003e-06 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883478+0000", 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016937180000000001 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883485+0000", 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3000000000000004e-06 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.233 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012549+0000", 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.129063785 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012705+0000", 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015551399999999999 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.012901+0000", 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019668299999999999 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046970+0000", 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.034068909000000001 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.047098+0000", 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012766899999999999 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.235 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.048912+0000", 2024-09-16T10:30:48.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001813797 2024-09-16T10:30:48.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.236 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051278+0000", 2024-09-16T10:30:48.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023664480000000002 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051344+0000", 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5415999999999998e-05 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053725+0000", 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002381593 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9:dfbc90fc:::benchmark_data_smithi063_124717_object60211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31207432799999, 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1744191719999999, 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.238 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881795+0000", 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881796+0000", 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3499999999999996e-07 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881881+0000", 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5458999999999996e-05 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881889+0000", 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9060000000000002e-06 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881896+0000", 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7830000000000001e-06 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883507+0000", 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016115249999999999 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883514+0000", 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1530000000000002e-06 2024-09-16T10:30:48.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.014943+0000", 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1314289719999999 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015122+0000", 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000178982 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.015285+0000", 2024-09-16T10:30:48.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016376499999999999 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.046978+0000", 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031692310000000001 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.051202+0000", 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042246860000000001 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053812+0000", 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026092720000000002 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053866+0000", 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4876000000000001e-05 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.053932+0000", 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5314000000000005e-05 2024-09-16T10:30:48.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056214+0000", 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002282634 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9:dcccd893:::benchmark_data_smithi063_124717_object60240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.311965085, 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2232440419999999, 2024-09-16T10:30:48.245 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:30:48.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881904+0000", 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881905+0000", 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2900000000000003e-07 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881928+0000", 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2747000000000001e-05 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881935+0000", 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3150000000000003e-06 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881941+0000", 2024-09-16T10:30:48.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1639999999999998e-06 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883538+0000", 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015963939999999999 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883543+0000", 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6540000000000004e-06 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056263+0000", 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.172719834 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056408+0000", 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014513800000000001 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.056584+0000", 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017557299999999999 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064153+0000", 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0075692820000000001 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068713+0000", 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045601620000000004 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068767+0000", 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3881000000000001e-05 2024-09-16T10:30:48.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101726+0000", 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032958901999999998 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101782+0000", 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5609999999999998e-05 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105149+0000", 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0033666569999999999 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9:dbba527e:::benchmark_data_smithi063_124717_object60254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31192076999997, 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2264516890000001, 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.253 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.254 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881949+0000", 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1399999999999997e-07 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881971+0000", 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1461999999999999e-05 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881979+0000", 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.259e-06 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881985+0000", 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7320000000000003e-06 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883562+0000", 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001577171 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883568+0000", 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7910000000000001e-06 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059019+0000", 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1754510929999999 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059188+0000", 2024-09-16T10:30:48.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016928700000000001 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.059360+0000", 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017208100000000001 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068663+0000", 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0093023089999999999 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068831+0000", 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016804999999999999 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082474+0000", 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013643597 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105299+0000", 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.022824816000000001 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105363+0000", 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4088e-05 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108400+0000", 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003037339 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.260 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9:ddb5c386:::benchmark_data_smithi063_124717_object60260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.311877573, 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2293183249999999, 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881992+0000", 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.881993+0000", 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9999999999999997e-07 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882015+0000", 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1946e-05 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882021+0000", 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9430000000000003e-06 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882028+0000", 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7599999999999997e-06 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883589+0000", 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001560875 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883595+0000", 2024-09-16T10:30:48.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0719999999999996e-06 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061597+0000", 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1780015660000001 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.061781+0000", 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000183909 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.265 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.062032+0000", 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025166899999999999 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068729+0000", 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0066966940000000004 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068881+0000", 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000151965 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.101683+0000", 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032802338 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108512+0000", 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0068289229999999998 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108574+0000", 2024-09-16T10:30:48.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1630000000000005e-05 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.111310+0000", 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027364350000000002 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60273 9.b 9:d6970cdd:::benchmark_data_smithi063_124717_object60272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:30:48.268 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31183378499998, 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2328018119999999, 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60273 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:30:48.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882036+0000", 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3499999999999996e-07 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882058+0000", 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1236000000000001e-05 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882064+0000", 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5230000000000001e-06 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.271 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882070+0000", 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4670000000000003e-06 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883625+0000", 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015556820000000001 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883632+0000", 2024-09-16T10:30:48.272 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6610000000000003e-06 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064190+0000", 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1805575690000001 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064363+0000", 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000173124 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.273 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.064567+0000", 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020472000000000001 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068929+0000", 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0043613879999999999 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076738+0000", 2024-09-16T10:30:48.274 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0078089300000000004 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105256+0000", 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.028517793 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112019+0000", 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067630110000000002 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.275 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.112083+0000", 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4938999999999996e-05 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114838+0000", 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027541340000000001 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.276 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60317 9.b 9:d4f738e0:::benchmark_data_smithi063_124717_object60316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31179304699998, 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.235504929, 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60317 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.277 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.278 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882077+0000", 2024-09-16T10:30:48.279 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5799999999999999e-07 2024-09-16T10:30:48.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882098+0000", 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0758e-05 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882104+0000", 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3239999999999999e-06 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.280 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882110+0000", 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5879999999999997e-06 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883724+0000", 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016140670000000001 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883731+0000", 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.068967+0000", 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1852353390000001 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069080+0000", 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113442 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.069174+0000", 2024-09-16T10:30:48.282 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3623999999999997e-05 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079562+0000", 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010388029 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082423+0000", 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028609740000000001 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.283 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114907+0000", 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032484065999999999 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114978+0000", 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0919000000000003e-05 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.115036+0000", 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7878999999999999e-05 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.284 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117581+0000", 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025458019999999998 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60326 9.b 9:d0ff592b:::benchmark_data_smithi063_124717_object60325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31174658899999, 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2387679700000001, 2024-09-16T10:30:48.285 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60326 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.286 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882123+0000", 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882124+0000", 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.99e-07 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882145+0000", 2024-09-16T10:30:48.287 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1650000000000001e-05 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882151+0000", 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2110000000000003e-06 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882158+0000", 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6669999999999999e-06 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883760+0000", 2024-09-16T10:30:48.288 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001601781 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883766+0000", 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4019999999999999e-06 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071354+0000", 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1875881050000001 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.289 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071501+0000", 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014622300000000001 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.071673+0000", 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000172518 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082372+0000", 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010698874000000001 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082515+0000", 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000143409 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117647+0000", 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.035131837999999999 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117792+0000", 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014443299999999999 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117866+0000", 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4274000000000001e-05 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120891+0000", 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030249859999999999 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.292 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60333 9.b 9:d83577cf:::benchmark_data_smithi063_124717_object60332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31170430600002, 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2414839360000001, 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60333 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.293 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882165+0000", 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882166+0000", 2024-09-16T10:30:48.294 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0100000000000005e-07 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882188+0000", 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2252999999999999e-05 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882221+0000", 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2459999999999998e-05 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.295 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882227+0000", 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.015e-06 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883790+0000", 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015635759999999999 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883797+0000", 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.567e-06 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076778+0000", 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1929816879999999 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.076959+0000", 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000180607 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.297 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.077150+0000", 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019081600000000001 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082436+0000", 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0052863240000000002 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.082555+0000", 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119305 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.298 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120947+0000", 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.038391740000000001 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.120989+0000", 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2107999999999997e-05 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.299 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.121049+0000", 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9888e-05 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123649+0000", 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025999880000000001 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60362 9.b 9:d2c32f51:::benchmark_data_smithi063_124717_object60361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.300 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31163318500001, 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.244097462, 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60362 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:30:48.301 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882236+0000", 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882237+0000", 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0399999999999995e-07 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.302 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882261+0000", 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4269000000000001e-05 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882268+0000", 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8619999999999997e-06 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882274+0000", 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1829999999999999e-06 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883834+0000", 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015597930000000001 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883840+0000", 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.251e-06 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079597+0000", 2024-09-16T10:30:48.304 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1957566589999999 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.079755+0000", 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015786399999999999 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.080192+0000", 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00043717099999999998 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.305 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.105209+0000", 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.025016842000000001 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.114928+0000", 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0097189919999999992 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123709+0000", 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0087810240000000001 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123762+0000", 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2567e-05 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.123828+0000", 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6799999999999997e-05 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126334+0000", 2024-09-16T10:30:48.307 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025054809999999999 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60366 9.b 9:d4ec99e6:::benchmark_data_smithi063_124717_object60365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.311587686, 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.246752562, 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.308 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60366 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:30:48.309 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.882282+0000", 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7899999999999998e-07 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883494+0000", 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012115959999999999 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.310 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883505+0000", 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.111e-05 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883513+0000", 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4320000000000002e-06 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883866+0000", 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00035366599999999998 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883873+0000", 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2910000000000004e-06 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099038+0000", 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2151655379999999 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.312 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099212+0000", 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017424000000000001 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.099404+0000", 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000191366 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.108466+0000", 2024-09-16T10:30:48.313 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0090625159999999996 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.117602+0000", 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0091353030000000009 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126413+0000", 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0088110150000000002 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.314 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126572+0000", 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015944599999999999 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.126628+0000", 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5819999999999997e-05 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129034+0000", 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024066439999999999 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60423 9.b 9:db831613:::benchmark_data_smithi063_124717_object60422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.31032323300002, 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2882874870000001, 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60423 2024-09-16T10:30:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883546+0000", 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883548+0000", 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.206e-06 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883603+0000", 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5036e-05 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883614+0000", 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1352000000000001e-05 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883624+0000", 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0519999999999999e-05 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883899+0000", 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027465699999999998 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.883905+0000", 2024-09-16T10:30:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3119999999999999e-06 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129084+0000", 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.245178672 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129268+0000", 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018391999999999999 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.129423+0000", 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015478000000000001 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136532+0000", 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071096270000000003 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136582+0000", 2024-09-16T10:30:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9962e-05 2024-09-16T10:30:48.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:48.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.323 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.152752+0000", 2024-09-16T10:30:48.323 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016169444000000002 2024-09-16T10:30:48.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.323 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168726+0000", 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015974559999999999 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168790+0000", 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3474999999999998e-05 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.171834+0000", 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030439640000000001 2024-09-16T10:30:48.324 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60452 9.b 9:d848878f:::benchmark_data_smithi063_124717_object60451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 492.25135476100002, 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.233084683, 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.325 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60452 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942515+0000", 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942516+0000", 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5969999999999999e-06 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942536+0000", 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9970999999999999e-05 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942548+0000", 2024-09-16T10:30:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2057e-05 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942556+0000", 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0350000000000001e-06 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942605+0000", 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8770999999999998e-05 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.942611+0000", 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1040000000000003e-06 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131546+0000", 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.188934811 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131704+0000", 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015802399999999999 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,6", 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.131894+0000", 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00018940499999999999 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136598+0000", 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047045409999999996 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.136646+0000", 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8204999999999997e-05 2024-09-16T10:30:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.168552+0000", 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.031905880999999997 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172759+0000", 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042071809999999999 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.172950+0000", 2024-09-16T10:30:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019102799999999999 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:37.175599+0000", 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026490720000000001 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:88281 10.c 10:356f8e40:::benchmark_data_smithi063_149043_object88280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:30:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 260.65658254800002, 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.020604785, 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 88281 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.334 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537289+0000", 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3520000000000001e-06 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537316+0000", 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7038000000000001e-05 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.336 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537335+0000", 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8247e-05 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537343+0000", 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7239999999999998e-06 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543029+0000", 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056852450000000002 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.337 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543036+0000", 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2749999999999998e-06 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.559029+0000", 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015993265 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.559039+0000", 2024-09-16T10:30:48.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0185e-05 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.556377+0000", 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99733790600000005 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.557892+0000", 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015145479999999999 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:88285 10.c 10:348a9cd0:::benchmark_data_smithi063_149043_object88284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 260.65651358600002, 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0211189380000001, 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 88285 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.340 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537357+0000", 2024-09-16T10:30:48.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9500000000000002e-07 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537424+0000", 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7220999999999999e-05 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537459+0000", 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5497999999999998e-05 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537467+0000", 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6450000000000005e-06 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543085+0000", 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056183730000000003 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting_for_map not empty", 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543092+0000", 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2899999999999999e-06 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.562892+0000", 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.019800512999999999 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.562904+0000", 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1368e-05 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.558397+0000", 2024-09-16T10:30:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99549375600000001 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.558475+0000", 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7578999999999997e-05 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:98186 10.c 10:30a2e491:::benchmark_data_smithi063_149043_object98185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 251.22580963799999, 2024-09-16T10:30:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0526488009999999, 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 98186 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968062+0000", 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.928e-06 2024-09-16T10:30:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968086+0000", 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3671000000000001e-05 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968096+0000", 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0839000000000001e-05 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968104+0000", 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8890000000000002e-06 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.974858+0000", 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067535399999999997 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.975006+0000", 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014837499999999999 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.975142+0000", 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013551899999999999 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.986786+0000", 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011644408 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.992120+0000", 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0053334860000000001 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.012439+0000", 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.020319265999999999 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.012449+0000", 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0223e-05 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020605+0000", 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0081562230000001 2024-09-16T10:30:48.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020709+0000", 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103434 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:98194 10.c 10:3f81be78:::benchmark_data_smithi063_149043_object98193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 251.21580921200001, 2024-09-16T10:30:48.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.050357231, 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 98194 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978063+0000", 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4260000000000002e-06 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978090+0000", 2024-09-16T10:30:48.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7475e-05 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978103+0000", 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2544000000000001e-05 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978110+0000", 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0060000000000003e-06 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980486+0000", 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023762700000000002 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980605+0000", 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011847 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980747+0000", 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014267800000000001 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.992126+0000", 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011379142 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.015722+0000", 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.023596004 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.015734+0000", 2024-09-16T10:30:48.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2058999999999999e-05 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020985+0000", 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005250813 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.021617+0000", 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00063140899999999999 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:30:48.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.028349+0000", 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.006732839 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.028418+0000", 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8096000000000007e-05 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:30:48.359 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:30:48.360 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 68 16 100364 100348 92.2147 68.375 0.0134682 0.0108378 2024-09-16T10:30:48.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:48.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:48.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:48.534 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:48.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:48.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:48.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:48.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:48.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:30:48.764 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:48.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:48.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:48.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:48.925 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:48 smithi063 ceph-mon[29993]: pgmap v932: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 745 B/s rd, 51 MiB/s wr, 818 op/s 2024-09-16T10:30:48.925 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:48 smithi063 ceph-mon[39470]: pgmap v932: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 745 B/s rd, 51 MiB/s wr, 818 op/s 2024-09-16T10:30:48.978 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:48.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:49.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:48 smithi165 ceph-mon[32696]: pgmap v932: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 745 B/s rd, 51 MiB/s wr, 818 op/s 2024-09-16T10:30:49.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:49.125 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:49.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:49.227 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:49.272 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:49.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:49.322 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 69 16 101488 101472 91.8964 70.25 0.00549535 0.0108773 2024-09-16T10:30:49.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:49.417 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:49.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:49.519 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:49.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:30:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:30:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:49.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:49.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:49.694 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:49.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:49.796 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:49.833 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:49.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:49.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:49.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[39470]: 3.1e scrub starts 2024-09-16T10:30:49.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[39470]: 3.1e scrub ok 2024-09-16T10:30:49.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[39470]: 1.13 scrub ok 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[39470]: 3.d scrub starts 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[39470]: 3.d scrub ok 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[29993]: 3.1e scrub starts 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[29993]: 3.1e scrub ok 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[29993]: 1.13 scrub ok 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[29993]: 3.d scrub starts 2024-09-16T10:30:49.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:49 smithi063 ceph-mon[29993]: 3.d scrub ok 2024-09-16T10:30:49.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:49.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:50.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:49 smithi165 ceph-mon[32696]: 3.1e scrub starts 2024-09-16T10:30:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:49 smithi165 ceph-mon[32696]: 3.1e scrub ok 2024-09-16T10:30:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:49 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:30:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:49 smithi165 ceph-mon[32696]: 1.13 scrub ok 2024-09-16T10:30:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:49 smithi165 ceph-mon[32696]: 3.d scrub starts 2024-09-16T10:30:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:49 smithi165 ceph-mon[32696]: 3.d scrub ok 2024-09-16T10:30:50.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:50.111 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:50.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:50.213 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:50.258 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:50.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:50.322 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 70 16 102590 102574 91.5674 68.875 0.0255238 0.010916 2024-09-16T10:30:50.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:50.399 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:50.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:50.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:50.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:50.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:50.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:50.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:50.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:50.804 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:50.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:50.906 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:50.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:30:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:50.941 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:50.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:50.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:50.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:30:50.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:30:50.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: pgmap v933: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: 6.8 scrub starts 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: 6.8 scrub ok 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: pgmap v933: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:30:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: 6.8 scrub starts 2024-09-16T10:30:50.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: 6.8 scrub ok 2024-09-16T10:30:50.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:30:50.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:50 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:30:51.041 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:51.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:30:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:30:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:30:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: pgmap v933: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-16T10:30:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: 6.8 scrub starts 2024-09-16T10:30:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: 6.8 scrub ok 2024-09-16T10:30:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:30:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:50 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:30:51.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:51.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:30:51.119 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:30:51.119 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T10:30:51.119 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-16T10:30:51.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-16T10:30:51.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:51.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:51.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:51.322 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 71 16 103535 103519 91.1094 59.0625 0.01601 0.0109704 2024-09-16T10:30:51.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:51.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:51.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:51.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:51.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:51.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:51.743 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:51.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:51.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:51.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 3.14 scrub starts 2024-09-16T10:30:51.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 4.17 scrub starts 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 3.14 scrub ok 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 4.17 scrub ok 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 3.14 scrub starts 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 4.17 scrub starts 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 3.14 scrub ok 2024-09-16T10:30:51.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 4.17 scrub ok 2024-09-16T10:30:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:30:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:30:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:30:51.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:51 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:30:51.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:51.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:51.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 3.14 scrub starts 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 4.17 scrub starts 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 3.14 scrub ok 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 4.17 scrub ok 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:30:52.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:51 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:30:52.100 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:52.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:52.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:52.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:52.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:52.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:52.322 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 72 16 104465 104449 90.6513 58.125 0.0125701 0.0110262 2024-09-16T10:30:52.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:52.399 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:52.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:52.500 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:52.537 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:52.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:52.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:52.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:52.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:52.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:52.798 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:52.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:52.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:52.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[29993]: pgmap v934: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 988 op/s 2024-09-16T10:30:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[29993]: 5.4 scrub starts 2024-09-16T10:30:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[29993]: 5.4 scrub ok 2024-09-16T10:30:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[29993]: 5.11 scrub starts 2024-09-16T10:30:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[29993]: 5.11 scrub ok 2024-09-16T10:30:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[39470]: pgmap v934: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 988 op/s 2024-09-16T10:30:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[39470]: 5.4 scrub starts 2024-09-16T10:30:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[39470]: 5.4 scrub ok 2024-09-16T10:30:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[39470]: 5.11 scrub starts 2024-09-16T10:30:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:52 smithi063 ceph-mon[39470]: 5.11 scrub ok 2024-09-16T10:30:53.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:53.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:52 smithi165 ceph-mon[32696]: pgmap v934: 168 pgs: 168 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 988 op/s 2024-09-16T10:30:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:52 smithi165 ceph-mon[32696]: 5.4 scrub starts 2024-09-16T10:30:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:52 smithi165 ceph-mon[32696]: 5.4 scrub ok 2024-09-16T10:30:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:52 smithi165 ceph-mon[32696]: 5.11 scrub starts 2024-09-16T10:30:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:52 smithi165 ceph-mon[32696]: 5.11 scrub ok 2024-09-16T10:30:53.077 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:53.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:53.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:53.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:53.322 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 73 16 105602 105586 90.3829 71.0625 0.00402888 0.0110587 2024-09-16T10:30:53.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:53.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:53.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:53.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:53.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:53.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:53.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:53.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.854 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:53.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:53.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[29993]: 3.8 scrub starts 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[29993]: 3.8 scrub ok 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[29993]: 6.10 scrub starts 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[29993]: 6.10 scrub ok 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[29993]: 5.8 scrub starts 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[29993]: 5.8 scrub ok 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[39470]: 3.8 scrub starts 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[39470]: 3.8 scrub ok 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[39470]: 6.10 scrub starts 2024-09-16T10:30:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[39470]: 6.10 scrub ok 2024-09-16T10:30:53.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[39470]: 5.8 scrub starts 2024-09-16T10:30:53.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:53 smithi063 ceph-mon[39470]: 5.8 scrub ok 2024-09-16T10:30:53.956 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:53.990 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:53.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:53 smithi165 ceph-mon[32696]: 3.8 scrub starts 2024-09-16T10:30:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:53 smithi165 ceph-mon[32696]: 3.8 scrub ok 2024-09-16T10:30:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:53 smithi165 ceph-mon[32696]: 6.10 scrub starts 2024-09-16T10:30:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:53 smithi165 ceph-mon[32696]: 6.10 scrub ok 2024-09-16T10:30:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:53 smithi165 ceph-mon[32696]: 5.8 scrub starts 2024-09-16T10:30:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:53 smithi165 ceph-mon[32696]: 5.8 scrub ok 2024-09-16T10:30:54.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:54.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:54.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:54.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:54.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:54.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:54.322 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 74 16 106506 106490 89.925 56.5 0.0135472 0.0111157 2024-09-16T10:30:54.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:54.391 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:54.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:54.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:54.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:54.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:54.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:54.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:54.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:54.754 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:54.789 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:54.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:54.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:54.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:54.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: 6.b scrub starts 2024-09-16T10:30:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: 6.b scrub ok 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: 5.5 scrub starts 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: 5.5 scrub ok 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: pgmap v935: 168 pgs: 168 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: 5.9 scrub starts 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: 5.9 scrub ok 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: 6.b scrub starts 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: 6.b scrub ok 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: 5.5 scrub starts 2024-09-16T10:30:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: 5.5 scrub ok 2024-09-16T10:30:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: pgmap v935: 168 pgs: 168 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-16T10:30:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: 5.9 scrub starts 2024-09-16T10:30:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: 5.9 scrub ok 2024-09-16T10:30:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:54.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:55.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:55.027 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:30:55.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:55.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: 6.b scrub starts 2024-09-16T10:30:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: 6.b scrub ok 2024-09-16T10:30:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: 5.5 scrub starts 2024-09-16T10:30:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: 5.5 scrub ok 2024-09-16T10:30:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: pgmap v935: 168 pgs: 168 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-16T10:30:55.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: 5.9 scrub starts 2024-09-16T10:30:55.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: 5.9 scrub ok 2024-09-16T10:30:55.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:30:55.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:30:55.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:55.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:55.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:55.191 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:55.243 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:55.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:55.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:55.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:55.325 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 75 16 107641 107625 89.6698 70.9375 0.0411615 0.0111472 2024-09-16T10:30:55.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:55.462 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:55.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:55.563 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:55.618 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:55.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:55.719 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:55.755 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:55.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:55.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:55.899 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:55.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:55.908 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[39470]: 5.1e scrub starts 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[39470]: 5.1e scrub ok 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[39470]: 4.5 scrub starts 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[39470]: 4.5 scrub ok 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[39470]: 1.1 scrub ok 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[29993]: 5.1e scrub starts 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[29993]: 5.1e scrub ok 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[29993]: 4.5 scrub starts 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[29993]: 4.5 scrub ok 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:30:55.909 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:55 smithi063 ceph-mon[29993]: 1.1 scrub ok 2024-09-16T10:30:56.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:56.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:56.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:56.043 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:56.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:55 smithi165 ceph-mon[32696]: 5.1e scrub starts 2024-09-16T10:30:56.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:55 smithi165 ceph-mon[32696]: 5.1e scrub ok 2024-09-16T10:30:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:55 smithi165 ceph-mon[32696]: 4.5 scrub starts 2024-09-16T10:30:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:55 smithi165 ceph-mon[32696]: 4.5 scrub ok 2024-09-16T10:30:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:55 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:30:56.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:55 smithi165 ceph-mon[32696]: 1.1 scrub ok 2024-09-16T10:30:56.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:56.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:56.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:56.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:30:56.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:56.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:30:56.325 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 76 16 108686 108670 89.3478 65.3125 0.0183904 0.0111862 2024-09-16T10:30:56.384 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:30:56.384 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:30:56.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:56.419 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:30:56.420 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:30:56.450 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:56.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:56.469 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:30:56.469 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:30:56.505 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:30:56.505 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:30:56.539 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:30:56.540 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:30:56.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:56.578 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:30:56.579 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:30:56.600 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:30:56.603 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:56.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:56.620 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:30:56.620 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:30:56.662 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:30:56.663 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:30:56.704 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:56.752 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:56.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:56.854 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:56.900 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:56.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: pgmap v936: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: 4.1c scrub starts 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: 4.1c scrub ok 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: 11.3 scrub starts 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: 4.8 scrub starts 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: 4.8 scrub ok 2024-09-16T10:30:56.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3398354520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3772087888' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[29993]: from='client.26617 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: pgmap v936: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: 4.1c scrub starts 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: 4.1c scrub ok 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: 11.3 scrub starts 2024-09-16T10:30:56.983 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: 4.8 scrub starts 2024-09-16T10:30:56.984 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: 4.8 scrub ok 2024-09-16T10:30:56.984 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3398354520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:56.984 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3772087888' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:30:56.984 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:56 smithi063 ceph-mon[39470]: from='client.26617 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:30:57.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:57.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: pgmap v936: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: 4.1c scrub starts 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: 4.1c scrub ok 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: 11.3 scrub starts 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: 4.8 scrub starts 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: 4.8 scrub ok 2024-09-16T10:30:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3398354520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:30:57.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3772087888' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:30:57.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:56 smithi165 ceph-mon[32696]: from='client.26617 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:30:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T10:30:57.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:57.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:57.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:57.293 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:57.325 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 77 16 109598 109582 88.9277 57 0.0112491 0.0112329 2024-09-16T10:30:57.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:57.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:57.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:30:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:30:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:30:57.528 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:57.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:57.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:57.798 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:57.840 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:57.942 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:57.981 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:57.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:58.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:58.100 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:30:58.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:58.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:58.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[39470]: 5.15 scrub starts 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[39470]: 5.15 scrub ok 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[39470]: from='client.26617 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[39470]: osdmap e503: 8 total, 8 up, 8 in 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2044218995' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[39470]: pgmap v938: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[39470]: 11.3 scrub ok 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[29993]: 5.15 scrub starts 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[29993]: 5.15 scrub ok 2024-09-16T10:30:58.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[29993]: from='client.26617 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:30:58.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[29993]: osdmap e503: 8 total, 8 up, 8 in 2024-09-16T10:30:58.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2044218995' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:58.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[29993]: pgmap v938: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-16T10:30:58.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:58 smithi063 ceph-mon[29993]: 11.3 scrub ok 2024-09-16T10:30:58.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:58.253 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:58.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:58.326 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 78 16 110838 110822 88.7806 77.5 0.0086832 0.0112592 2024-09-16T10:30:58.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:30:58.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:58.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:30:58.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:58.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:58.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:58.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:58 smithi165 ceph-mon[32696]: 5.15 scrub starts 2024-09-16T10:30:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:58 smithi165 ceph-mon[32696]: 5.15 scrub ok 2024-09-16T10:30:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:58 smithi165 ceph-mon[32696]: from='client.26617 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:30:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:58 smithi165 ceph-mon[32696]: osdmap e503: 8 total, 8 up, 8 in 2024-09-16T10:30:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:58 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2044218995' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:30:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:58 smithi165 ceph-mon[32696]: pgmap v938: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-16T10:30:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:58 smithi165 ceph-mon[32696]: 11.3 scrub ok 2024-09-16T10:30:58.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:30:58.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:58.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:30:58.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:58.784 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:58.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:58.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:30:58.886 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:58.923 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:58.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:59.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:59.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:59.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:59.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:30:59.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:30:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:59.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:30:59.225 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:59 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:59.225 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2044218995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:30:59.225 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:30:59 smithi063 ceph-mon[29993]: osdmap e504: 8 total, 8 up, 8 in 2024-09-16T10:30:59.225 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:59 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:59.225 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2044218995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:30:59.226 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:30:59 smithi063 ceph-mon[39470]: osdmap e504: 8 total, 8 up, 8 in 2024-09-16T10:30:59.289 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:30:59.289 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:30:59.293 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:59.326 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 79 16 112030 112014 88.5993 74.5 0.0304546 0.0112811 2024-09-16T10:30:59.335 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:59.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:59.437 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:30:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:59.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:30:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:59 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:30:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2044218995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:30:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:30:59 smithi165 ceph-mon[32696]: osdmap e504: 8 total, 8 up, 8 in 2024-09-16T10:30:59.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:30:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:30:59.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:30:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:59.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:30:59.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:30:59.762 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:59.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:30:59.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:30:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:30:59.903 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:30:59.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:00.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:00.041 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:00.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:00.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:00.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:00.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:00.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:00.326 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:00.327 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:2024-09-16T10:31:00.326071+0000 min lat: 0.00120603 max lat: 0.224764 avg lat: 0.0112961 2024-09-16T10:31:00.327 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:31:00.327 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 80 16 113306 113290 88.4886 79.75 0.0078553 0.0112961 2024-09-16T10:31:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:00.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:00.381 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:31:00.417 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:00.417 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:00 smithi063 ceph-mon[29993]: pgmap v940: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:31:00.417 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:00.417 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:00 smithi063 ceph-mon[39470]: pgmap v940: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:31:00.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:00.470 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:00.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:00.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:00 smithi165 ceph-mon[32696]: pgmap v940: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-16T10:31:00.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:00.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:00.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:00.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:31:00.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:00.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:00.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:00.826 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:00.876 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:00.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:00.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:00.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:01.009 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:31:01.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:01.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:01.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:01.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:01.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:01.327 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 81 16 114313 114297 88.1729 62.9375 0.017719 0.0113368 2024-09-16T10:31:01.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:01.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:01.506 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:01.530 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:01.560 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:31:01.560 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:31:01.600 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:31:01.600 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:31:01.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:01.644 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:31:01.644 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:31:01.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:01.675 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:01.678 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:31:01.679 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:31:01.714 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:01.717 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:31:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:31:01.753 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:31:01.753 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:31:01.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:01.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:01.790 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:31:01.790 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:31:01.825 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:31:01.826 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:31:01.887 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:01.931 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:01.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:02.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:02.070 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:02.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:02.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:02.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:02.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:02.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:02.327 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 82 16 115356 115340 87.8922 65.1875 0.0090089 0.011372 2024-09-16T10:31:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:02.349 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:02.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:02.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:31:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:31:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[29993]: pgmap v941: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s 2024-09-16T10:31:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2706755908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:31:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:31:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[39470]: pgmap v941: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s 2024-09-16T10:31:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2706755908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:02.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:02.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:02 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:31:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:02 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:31:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:02 smithi165 ceph-mon[32696]: pgmap v941: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s 2024-09-16T10:31:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2706755908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:02.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:02.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:02.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:02.690 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:31:02.690 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:31:02.690 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:31:02.690 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:31:02.690 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:31:02.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:02.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:02.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:02.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:02.882 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:02.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:02.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:03.030 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:03.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:03.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:03.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:03.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:03.327 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 83 16 116568 116552 87.7458 75.75 0.0287858 0.0113912 2024-09-16T10:31:03.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:03.413 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:03 smithi063 ceph-mon[29993]: osdmap e505: 8 total, 8 up, 8 in 2024-09-16T10:31:03.414 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:03 smithi063 ceph-mon[39470]: osdmap e505: 8 total, 8 up, 8 in 2024-09-16T10:31:03.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:03.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:03.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:03 smithi165 ceph-mon[32696]: osdmap e505: 8 total, 8 up, 8 in 2024-09-16T10:31:03.562 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:03.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:03.706 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:03.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:03.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:03.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:03.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:03.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:03.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:04.069 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:04.092 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:04.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:04.110 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:31:04.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:04.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:04.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:04.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:04.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:04.327 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 84 16 117664 117648 87.5167 68.5 0.00670274 0.011422 2024-09-16T10:31:04.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:04.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:04.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[29993]: pgmap v943: 168 pgs: 168 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.6 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2232822581' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[29993]: from='client.26635 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[39470]: pgmap v943: 168 pgs: 168 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.6 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:31:04.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2232822581' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:04.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:04 smithi063 ceph-mon[39470]: from='client.26635 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:04.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:04.505 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:04.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:04.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:04 smithi165 ceph-mon[32696]: pgmap v943: 168 pgs: 168 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.6 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-16T10:31:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:04 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:31:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:04 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:31:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:04 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2232822581' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:04 smithi165 ceph-mon[32696]: from='client.26635 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:04.607 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:04.642 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:04.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:04.691 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:31:04.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:31:04.691 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-16T10:31:04.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-16T10:31:04.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:04.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:04.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:04.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:04.916 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:04.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:05.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:05.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:05.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.198 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:05.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:05.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:05.328 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 85 16 118711 118695 87.2561 65.4375 0.0141212 0.0114557 2024-09-16T10:31:05.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:05.382 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:05 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:05.383 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:05 smithi063 ceph-mon[29993]: from='client.26635 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:05.383 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:05 smithi063 ceph-mon[29993]: osdmap e506: 8 total, 8 up, 8 in 2024-09-16T10:31:05.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:05 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:05.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:05 smithi063 ceph-mon[39470]: from='client.26635 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:05.383 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:05 smithi063 ceph-mon[39470]: osdmap e506: 8 total, 8 up, 8 in 2024-09-16T10:31:05.432 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:05.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:05.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:05 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:05 smithi165 ceph-mon[32696]: from='client.26635 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:05 smithi165 ceph-mon[32696]: osdmap e506: 8 total, 8 up, 8 in 2024-09-16T10:31:05.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:05.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:05.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:05.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:05.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:31:05.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:05.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:05.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:05.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:05.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:06.090 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:06.136 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:06.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:06.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:06.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:06.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:06.328 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 86 16 119925 119909 87.1237 75.875 0.00581121 0.0114729 2024-09-16T10:31:06.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:06.404 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:06.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:06.428 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:06 smithi063 ceph-mon[29993]: pgmap v945: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-16T10:31:06.428 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:06 smithi063 ceph-mon[39470]: pgmap v945: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-16T10:31:06.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:06.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:06 smithi165 ceph-mon[32696]: pgmap v945: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-16T10:31:06.588 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:06.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:06.608 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:06.622 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:06.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:06.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:06.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:06.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:06.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:06.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:06.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:07.021 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:07.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:07.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:07.164 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:07.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:07.248 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:31:07.265 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:07.328 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 87 16 121032 121016 86.9175 69.1875 0.0393289 0.0115003 2024-09-16T10:31:07.330 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:07.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:07.336 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:31:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:31:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:31:07.336 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:46191 11.e 11:785d30f5:::benchmark_data_smithi063_176907_object46190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:03.062127+0000", 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 64.269020857000001, 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12686272700000001, 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.337 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 46191 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.062127+0000", 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.338 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.062127+0000", 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.062129+0000", 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8789999999999999e-06 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.062153+0000", 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4349999999999999e-05 2024-09-16T10:31:07.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.062162+0000", 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4009999999999997e-06 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.062167+0000", 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6350000000000003e-06 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.340 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.074455+0000", 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012287859 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.074583+0000", 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012769600000000001 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.074722+0000", 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000139336 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.187311+0000", 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.112588494 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.187383+0000", 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2081000000000003e-05 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.188883+0000", 2024-09-16T10:31:07.342 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015003779999999999 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.188942+0000", 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8838999999999997e-05 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.188990+0000", 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7778999999999998e-05 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:46202 11.e 11:7c0b8c7d:::benchmark_data_smithi063_176907_object46201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:03.068842+0000", 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 64.262306026999994, 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.126959445, 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 46202 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.068842+0000", 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.068842+0000", 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.068843+0000", 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.06e-06 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.068873+0000", 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9739000000000001e-05 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.068882+0000", 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1540000000000008e-06 2024-09-16T10:31:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.068888+0000", 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4309999999999999e-06 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.189389+0000", 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.120500869 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.189563+0000", 2024-09-16T10:31:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000174437 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.189730+0000", 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000166785 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.195669+0000", 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0059382879999999999 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.195680+0000", 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1484000000000001e-05 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.195707+0000", 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7013999999999999e-05 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.195760+0000", 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3244e-05 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:03.195801+0000", 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0939999999999998e-05 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:58816 11.0 11:07ed086b:::benchmark_data_smithi063_176907_object58815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:11.773674+0000", 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 55.557474092, 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12888253499999999, 2024-09-16T10:31:07.350 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 58816 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773674+0000", 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773674+0000", 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773676+0000", 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8610000000000001e-06 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.352 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773700+0000", 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4267999999999999e-05 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773712+0000", 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1742e-05 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773719+0000", 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0679999999999999e-06 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773808+0000", 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9812e-05 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.773997+0000", 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000188524 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.774123+0000", 2024-09-16T10:31:07.354 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125668 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.776843+0000", 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027201669999999999 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.778300+0000", 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014576089999999999 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.355 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.902179+0000", 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12387860000000001 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.902509+0000", 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000329508 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:11.902556+0000", 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7707999999999998e-05 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59843 11.0 11:0fb7d6ac:::benchmark_data_smithi063_176907_object59842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.653547+0000", 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 54.677600495, 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21896771900000001, 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.357 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59843 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653547+0000", 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653547+0000", 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.358 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653549+0000", 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9209999999999999e-06 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653577+0000", 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7968000000000001e-05 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653588+0000", 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1105e-05 2024-09-16T10:31:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653596+0000", 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6290000000000001e-06 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653731+0000", 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013499399999999999 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653890+0000", 2024-09-16T10:31:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015866200000000001 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.654025+0000", 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013570800000000001 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.657295+0000", 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032695929999999999 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.661282+0000", 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0039872370000000002 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.872409+0000", 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21112699200000001 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.872470+0000", 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1330999999999999e-05 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.872515+0000", 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4579000000000001e-05 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59851 11.0 11:0dd86e01:::benchmark_data_smithi063_176907_object59850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.658329+0000", 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 54.672819134999997, 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21565917000000001, 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59851 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658329+0000", 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658329+0000", 2024-09-16T10:31:07.365 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.365 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.365 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658329+0000", 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0400000000000005e-07 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658362+0000", 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2267999999999997e-05 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658377+0000", 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5712000000000001e-05 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.366 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658383+0000", 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7849999999999997e-06 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658440+0000", 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6541e-05 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658752+0000", 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031261700000000001 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.367 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.658898+0000", 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014559100000000001 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.661487+0000", 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025888640000000002 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.662254+0000", 2024-09-16T10:31:07.368 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00076732600000000003 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.873903+0000", 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21164845299999999 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.873951+0000", 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8309000000000001e-05 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.873988+0000", 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.6900000000000002e-05 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59860 11.0 11:0fa1e0f7:::benchmark_data_smithi063_176907_object59859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.665539+0000", 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 54.665608878999997, 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21336379599999999, 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.370 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59860 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665539+0000", 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.371 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665539+0000", 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665540+0000", 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4300000000000001e-07 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665558+0000", 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8051e-05 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665569+0000", 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1019000000000001e-05 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665574+0000", 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8239999999999999e-06 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665615+0000", 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0787000000000001e-05 2024-09-16T10:31:07.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665714+0000", 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9704000000000006e-05 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.665832+0000", 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000118158 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.667391+0000", 2024-09-16T10:31:07.374 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015584559999999999 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.668501+0000", 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0011098029999999999 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.878785+0000", 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21028444900000001 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.375 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.878852+0000", 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6692000000000005e-05 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.878903+0000", 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0909999999999999e-05 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59936 11.f 11:fd696c59:::benchmark_data_smithi063_176907_object59935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:07.376 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.927610+0000", 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 54.403538085999998, 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18809806400000001, 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59936 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.377 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.927610+0000", 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.927610+0000", 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.927612+0000", 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.04e-06 2024-09-16T10:31:07.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.927665+0000", 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3316999999999998e-05 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.927677+0000", 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2213e-05 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.927685+0000", 2024-09-16T10:31:07.379 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3849999999999996e-06 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.932457+0000", 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047724519999999999 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.932571+0000", 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011356 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.932693+0000", 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012258200000000001 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.935250+0000", 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025571539999999999 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.381 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.935973+0000", 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00072295200000000001 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.115602+0000", 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.17962819699999999 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.115661+0000", 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9834999999999999e-05 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.115708+0000", 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6377e-05 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59961 11.c 11:3bea3dc9:::benchmark_data_smithi063_176907_object59960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.942130+0000", 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 54.389018212000003, 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18619851400000001, 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.383 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59961 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942130+0000", 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.384 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942130+0000", 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942131+0000", 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4169999999999999e-06 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942184+0000", 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3004999999999998e-05 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942197+0000", 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2855e-05 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942204+0000", 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0450000000000003e-06 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942263+0000", 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9048000000000001e-05 2024-09-16T10:31:07.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942391+0000", 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012772200000000001 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.942521+0000", 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013027000000000001 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.944695+0000", 2024-09-16T10:31:07.387 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0021741379999999999 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.945173+0000", 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00047749199999999999 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.128215+0000", 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18304257600000001 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.128282+0000", 2024-09-16T10:31:07.388 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7225999999999996e-05 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.128328+0000", 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5720000000000003e-05 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59966 11.f 11:f167c173:::benchmark_data_smithi063_176907_object59965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.946385+0000", 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 54.384763151999998, 2024-09-16T10:31:07.389 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.17667075700000001, 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59966 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946385+0000", 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946385+0000", 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946386+0000", 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6649999999999999e-06 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946436+0000", 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9765999999999998e-05 2024-09-16T10:31:07.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946449+0000", 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2971e-05 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946456+0000", 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1930000000000004e-06 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946511+0000", 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5053000000000003e-05 2024-09-16T10:31:07.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.393 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.393 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946650+0000", 2024-09-16T10:31:07.393 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013892599999999999 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.946755+0000", 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000104802 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.948614+0000", 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018593559999999999 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.394 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.948975+0000", 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00036046899999999998 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.122957+0000", 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.17398217199999999 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.123011+0000", 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3724999999999998e-05 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.395 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:13.123055+0000", 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4659000000000001e-05 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:84456 11.e 11:757eb2d0:::benchmark_data_smithi063_176907_object84455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:33.910960+0000", 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 33.420187902000002, 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12959254200000001, 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.396 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 84456 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.910960+0000", 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.397 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.910960+0000", 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.910962+0000", 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0810000000000001e-06 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.910987+0000", 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5479000000000001e-05 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.911000+0000", 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2639999999999999e-05 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.911010+0000", 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5089999999999999e-06 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.911093+0000", 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3708999999999994e-05 2024-09-16T10:31:07.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.911218+0000", 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012511099999999999 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.911380+0000", 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016136300000000001 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.400 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914235+0000", 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002855261 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040177+0000", 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12594170199999999 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040457+0000", 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000280446 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.401 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040507+0000", 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9942999999999999e-05 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040552+0000", 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5297999999999999e-05 2024-09-16T10:31:07.402 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:84461 11.0 11:01a8009a:::benchmark_data_smithi063_176907_object84460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:33.914106+0000", 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 33.417041466000001, 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12972478800000001, 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.404 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 84461 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914106+0000", 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914106+0000", 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914108+0000", 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7099999999999999e-06 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914130+0000", 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1957999999999999e-05 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.406 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914141+0000", 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1042e-05 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914148+0000", 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8240000000000002e-06 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914252+0000", 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010443899999999999 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.407 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914395+0000", 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014316599999999999 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914543+0000", 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014721999999999999 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.043343+0000", 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.128800413 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.043405+0000", 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1422999999999998e-05 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.043454+0000", 2024-09-16T10:31:07.409 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9879999999999997e-05 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.043509+0000", 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4197999999999998e-05 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.043831+0000", 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000322515 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:84462 11.0 11:0eb1a853:::benchmark_data_smithi063_176907_object84461:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:33.914168+0000", 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 33.416980109000001, 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.134728601, 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 84462 2024-09-16T10:31:07.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914168+0000", 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914168+0000", 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.412 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914168+0000", 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7199999999999998e-07 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914197+0000", 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8416999999999999e-05 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914203+0000", 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.63e-06 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.914209+0000", 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2639999999999999e-06 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040421+0000", 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12621241799999999 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040593+0000", 2024-09-16T10:31:07.414 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017200599999999999 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040759+0000", 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016637600000000001 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.046970+0000", 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0062109419999999997 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.415 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.046986+0000", 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5311999999999998e-05 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.048793+0000", 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00180747 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.048854+0000", 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0954e-05 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.048896+0000", 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.214e-05 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:84465 11.0 11:0eccde6e:::benchmark_data_smithi063_176907_object84464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:33.915898+0000", 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 33.415249387999999, 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.135113073, 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 84465 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.418 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.915898+0000", 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.915898+0000", 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.915900+0000", 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5960000000000001e-06 2024-09-16T10:31:07.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.915922+0000", 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1911000000000001e-05 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.915931+0000", 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2210000000000003e-06 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.915940+0000", 2024-09-16T10:31:07.420 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1609999999999992e-06 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.043865+0000", 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12792431300000001 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.043993+0000", 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012833000000000001 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.421 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.044132+0000", 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013891199999999999 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.046990+0000", 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002858247 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.047865+0000", 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087534400000000001 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.050923+0000", 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030572199999999998 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.050972+0000", 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9208e-05 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.051011+0000", 2024-09-16T10:31:07.423 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9610000000000002e-05 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:84470 11.7 e496/476 11:e0ee36d1:::benchmark_data_smithi063_176907_object84469:head v 496'5307, mlcod=496'5305)", 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:33.916635+0000", 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 33.414512707, 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13374472600000001, 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.424 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.916635+0000", 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.916635+0000", 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.425 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.916636+0000", 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.257e-06 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:33.916667+0000", 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0963999999999997e-05 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040204+0000", 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.123536465 2024-09-16T10:31:07.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040211+0000", 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9140000000000002e-06 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040267+0000", 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6496000000000003e-05 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.427 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.040291+0000", 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3493999999999999e-05 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.050305+0000", 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010014504 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:34.050380+0000", 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4632e-05 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.428 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:86128 11.0 11:0b4b9eb7:::benchmark_data_smithi063_176907_object86127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e497)", 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:35.564285+0000", 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 31.766863098999998, 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13410030000000001, 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 86128 2024-09-16T10:31:07.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564285+0000", 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564285+0000", 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.430 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564287+0000", 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8610000000000001e-06 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564311+0000", 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.421e-05 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564328+0000", 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7305000000000001e-05 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.431 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564336+0000", 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4279999999999999e-06 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564391+0000", 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5430000000000003e-05 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.432 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564515+0000", 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000123638 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.564644+0000", 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012917899999999999 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.567811+0000", 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031675150000000001 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.433 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.569752+0000", 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019410250000000001 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.698287+0000", 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12853466999999999 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.698343+0000", 2024-09-16T10:31:07.434 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6091000000000002e-05 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:35.698385+0000", 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1947999999999999e-05 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:102776 11.f 11:f845c471:::benchmark_data_smithi063_176907_object102775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:50.472568+0000", 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 16.858579593000002, 2024-09-16T10:31:07.435 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12804253800000001, 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 102776 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.472568+0000", 2024-09-16T10:31:07.436 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.472568+0000", 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.472570+0000", 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5939999999999999e-06 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.437 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.438 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.472594+0000", 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4128999999999999e-05 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.472603+0000", 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1330000000000005e-06 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.472610+0000", 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0219999999999999e-06 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.472936+0000", 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00032575399999999998 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.440 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.473080+0000", 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000144633 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.473213+0000", 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000132788 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.484069+0000", 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010855946999999999 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.441 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.484241+0000", 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017134800000000001 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.600507+0000", 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.116266573 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.600560+0000", 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2972e-05 2024-09-16T10:31:07.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:50.600611+0000", 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0645e-05 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:104218 11.f 11:f234ec3a:::benchmark_data_smithi063_176907_object104217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:51.941480+0000", 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 15.389667754, 2024-09-16T10:31:07.443 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.189488248, 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 104218 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941480+0000", 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941480+0000", 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941482+0000", 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6840000000000001e-06 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941506+0000", 2024-09-16T10:31:07.445 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.393e-05 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941521+0000", 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5194999999999999e-05 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941528+0000", 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2649999999999999e-06 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.446 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941576+0000", 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7457000000000003e-05 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941746+0000", 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017079000000000001 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.941968+0000", 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000221979 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944655+0000", 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002686328 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.945868+0000", 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012133140000000001 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.130877+0000", 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18500868100000001 2024-09-16T10:31:07.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.130928+0000", 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1714e-05 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.130968+0000", 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9910999999999998e-05 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:104220 11.c 11:319ce8dd:::benchmark_data_smithi063_176907_object104219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:51.944693+0000", 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 15.386454871, 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18558983700000001, 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 104220 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.450 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944693+0000", 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944693+0000", 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944695+0000", 2024-09-16T10:31:07.451 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7239999999999999e-06 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944718+0000", 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3394e-05 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944731+0000", 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2482e-05 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.452 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944736+0000", 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5629999999999999e-06 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944788+0000", 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2191000000000002e-05 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.944953+0000", 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000165107 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.945102+0000", 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014887599999999999 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.947647+0000", 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025450199999999998 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.947886+0000", 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000239201 2024-09-16T10:31:07.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.130123+0000", 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.18223636300000001 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.130235+0000", 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111648 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.455 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.130283+0000", 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8267999999999999e-05 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:105940 11.0 11:0e84f1d0:::benchmark_data_smithi063_176907_object105939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:53.625898+0000", 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 13.705249434000001, 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21353418699999999, 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.456 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 105940 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.625898+0000", 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.625898+0000", 2024-09-16T10:31:07.457 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.625900+0000", 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6199999999999999e-06 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.625921+0000", 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.139e-05 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.458 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.625932+0000", 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0426e-05 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.625939+0000", 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0659999999999998e-06 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.626818+0000", 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087920700000000001 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.626930+0000", 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111518 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.627022+0000", 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2077000000000006e-05 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.709331+0000", 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.082309630999999994 2024-09-16T10:31:07.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.709606+0000", 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027451800000000002 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.838694+0000", 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.12908816100000001 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.461 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.839386+0000", 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069159699999999996 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.839433+0000", 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6975999999999999e-05 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:105959 11.0 11:093fd787:::benchmark_data_smithi063_176907_object105958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-16T10:31:07.462 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:53.717348+0000", 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 13.613799395000001, 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.138234198, 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 105959 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:07.463 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717348+0000", 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717348+0000", 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717350+0000", 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.291e-06 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717372+0000", 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2339000000000001e-05 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717382+0000", 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9809999999999998e-06 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717390+0000", 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8979999999999996e-06 2024-09-16T10:31:07.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717443+0000", 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3498000000000002e-05 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717563+0000", 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119367 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 3,7", 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.717707+0000", 2024-09-16T10:31:07.466 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014433100000000001 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.720275+0000", 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025676480000000001 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.720666+0000", 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000391219 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.467 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.855484+0000", 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.13481775200000001 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.855546+0000", 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1792999999999996e-05 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:53.855583+0000", 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7081e-05 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.468 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:07.469 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:07.469 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:31:07.531 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:07.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:07.633 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:07.667 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:07.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:07.769 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:07.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:07.902 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:07.943 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:07.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:08.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:08.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:08.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:08.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:08.211 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:08.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:08.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:08.313 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:08.329 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 88 16 122086 122070 86.6774 65.875 0.00686525 0.0115323 2024-09-16T10:31:08.353 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:08.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:08.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:08.488 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:31:08.495 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:08.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:08.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:08.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:08.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:08.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:08.785 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:08.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:08.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:08 smithi165 ceph-mon[32696]: pgmap v946: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-16T10:31:08.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:08.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:08.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:08.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:08 smithi063 ceph-mon[29993]: pgmap v946: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-16T10:31:08.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:08 smithi063 ceph-mon[39470]: pgmap v946: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-16T10:31:09.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:09.059 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:09.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:09.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:09.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:09.191 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:09.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:09.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:09.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:09.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:09.329 INFO:tasks.radosbench.radosbench.0.smithi063.stdout: 89 16 123146 123130 86.4478 66.25 0.00974672 0.0115631 2024-09-16T10:31:09.425 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:09.463 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:09.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:09.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:31:09 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:31:09.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:09.606 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:09.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:09.708 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:09.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:09.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:09.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:09.877 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:09.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:09.979 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:10.022 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:10.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:10.153 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:10.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:10.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total time run: 90.008 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Total writes made: 124273 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Write size: 65536 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Object size: 65536 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Bandwidth (MB/sec): 86.293 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Bandwidth: 30.6231 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max bandwidth (MB/sec): 203.125 2024-09-16T10:31:10.330 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min bandwidth (MB/sec): 51.8125 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average IOPS: 1380 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev IOPS: 489.97 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max IOPS: 3250 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min IOPS: 829 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Average Latency(s): 0.011584 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Stddev Latency(s): 0.0110832 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Max latency(s): 0.224764 2024-09-16T10:31:10.331 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Min latency(s): 0.00120603 2024-09-16T10:31:10.334 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Cleaning up (deleting benchmark objects) 2024-09-16T10:31:10.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:10.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:10.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:10.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:10.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:10.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[29993]: pgmap v947: 168 pgs: 168 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-16T10:31:10.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:10.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:10.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:10.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:10 smithi063 ceph-mon[39470]: pgmap v947: 168 pgs: 168 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-16T10:31:10.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:10.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:10.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:10.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:10 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:10 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:10 smithi165 ceph-mon[32696]: pgmap v947: 168 pgs: 168 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-16T10:31:10.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:10.674 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:10.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:10.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:10.811 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:10.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:10.951 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:10.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:11.053 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:11.075 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:11.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:11.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:11.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:11.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:11.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:11.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:11.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:11.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:11.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:11.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:11.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:11.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:11.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:11.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:31:11.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:11.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:11.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:12.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:12.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:12.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:12.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:12.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:12.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:12.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:12.377 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:12.401 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:12.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:12.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:12 smithi063 ceph-mon[29993]: pgmap v948: 168 pgs: 168 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-16T10:31:12.456 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:12 smithi063 ceph-mon[39470]: pgmap v948: 168 pgs: 168 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-16T10:31:12.457 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:12.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:12 smithi165 ceph-mon[32696]: pgmap v948: 168 pgs: 168 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-16T10:31:12.553 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:12.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:12.655 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:12.694 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:12.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:12.790 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:12 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:31:12.514+0000 7f17f27f5640 -1 mon.a@0(leader).osd e506 definitely_dead 0 2024-09-16T10:31:12.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:12.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:12.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:12.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:12.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:12.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:13.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:13.094 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:13.118 INFO:teuthology.orchestra.run.smithi063.stderr:marked out osd.7. 2024-09-16T10:31:13.196 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:13.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:13.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:13.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:13.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:13.448 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:31:13.448 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:31:13.448 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2060510468' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-16T10:31:13.448 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[29993]: from='client.46520 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-16T10:31:13.448 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[29993]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-16T10:31:13.448 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:31:13.449 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:31:13.449 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2060510468' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-16T10:31:13.449 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[39470]: from='client.46520 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-16T10:31:13.449 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:13 smithi063 ceph-mon[39470]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-16T10:31:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.467 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:13.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:13 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:31:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:13 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:31:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2060510468' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-16T10:31:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:13 smithi165 ceph-mon[32696]: from='client.46520 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-16T10:31:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:13 smithi165 ceph-mon[32696]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-16T10:31:13.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:13.590 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:13.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:13.715 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:13.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:13.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:13.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:13.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:13.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:14.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:14.094 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:14.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:14.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:14.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:14.340 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:14.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:14.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[29993]: from='client.46520 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-16T10:31:14.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[29993]: osdmap e507: 8 total, 8 up, 7 in 2024-09-16T10:31:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[29993]: pgmap v950: 168 pgs: 1 unknown, 21 peering, 146 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.6 KiB/s rd, 54 MiB/s wr, 1.06k op/s; 2.8 KiB/s, 0 objects/s recovering 2024-09-16T10:31:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[29993]: osdmap e508: 8 total, 8 up, 7 in 2024-09-16T10:31:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[39470]: from='client.46520 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-16T10:31:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[39470]: osdmap e507: 8 total, 8 up, 7 in 2024-09-16T10:31:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[39470]: pgmap v950: 168 pgs: 1 unknown, 21 peering, 146 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.6 KiB/s rd, 54 MiB/s wr, 1.06k op/s; 2.8 KiB/s, 0 objects/s recovering 2024-09-16T10:31:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:14 smithi063 ceph-mon[39470]: osdmap e508: 8 total, 8 up, 7 in 2024-09-16T10:31:14.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:14.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:14 smithi165 ceph-mon[32696]: from='client.46520 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-16T10:31:14.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:14 smithi165 ceph-mon[32696]: osdmap e507: 8 total, 8 up, 7 in 2024-09-16T10:31:14.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:14 smithi165 ceph-mon[32696]: pgmap v950: 168 pgs: 1 unknown, 21 peering, 146 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.6 KiB/s rd, 54 MiB/s wr, 1.06k op/s; 2.8 KiB/s, 0 objects/s recovering 2024-09-16T10:31:14.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:14 smithi165 ceph-mon[32696]: osdmap e508: 8 total, 8 up, 7 in 2024-09-16T10:31:14.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:14.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:14.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:14.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:14.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:14.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:14.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:14.960 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:14.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:15.062 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:15.085 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:15.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:15.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:15.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T10:31:15.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[29993]: osdmap e509: 8 total, 8 up, 7 in 2024-09-16T10:31:15.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[29993]: 1.e starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:15.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[29993]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T10:31:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[39470]: osdmap e509: 8 total, 8 up, 7 in 2024-09-16T10:31:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[39470]: 1.e starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:15.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:15 smithi063 ceph-mon[39470]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:15.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:15.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:15.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:15.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:15.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:15.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:15.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:15.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:15 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T10:31:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:15 smithi165 ceph-mon[32696]: osdmap e509: 8 total, 8 up, 7 in 2024-09-16T10:31:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:15 smithi165 ceph-mon[32696]: 1.e starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:15 smithi165 ceph-mon[32696]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:15.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:15.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:15.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127376 11.2 11:45bdf2af:::benchmark_data_smithi063_176907_object3100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.285590+0000", 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.19951291099999999, 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.19953499999999999, 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127376 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285590+0000", 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285590+0000", 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285593+0000", 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4059999999999999e-06 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285598+0000", 2024-09-16T10:31:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3310000000000003e-06 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285612+0000", 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3451999999999999e-05 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285619+0000", 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2219999999999996e-06 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285702+0000", 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2832999999999998e-05 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285832+0000", 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013025499999999999 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285906+0000", 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4453e-05 2024-09-16T10:31:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.288461+0000", 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002554635 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.288821+0000", 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00036031799999999999 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127352 11.2 11:4562636d:::benchmark_data_smithi063_176907_object3076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.242850+0000", 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.24225302800000001, 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.24236980699999999, 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127352 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242850+0000", 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242850+0000", 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242852+0000", 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0899999999999999e-06 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242859+0000", 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5810000000000002e-06 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.244847+0000", 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001987862 2024-09-16T10:31:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.244863+0000", 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6030000000000001e-05 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.244934+0000", 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1384999999999995e-05 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.245066+0000", 2024-09-16T10:31:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131478 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.245541+0000", 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00047533900000000002 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.256992+0000", 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011450638000000001 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259422+0000", 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024297960000000001 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127357 11.2 11:48f6d68a:::benchmark_data_smithi063_176907_object3081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.259888+0000", 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.225215153, 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22538177000000001, 2024-09-16T10:31:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127357 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259888+0000", 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259888+0000", 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259890+0000", 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8199999999999999e-06 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259895+0000", 2024-09-16T10:31:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1159999999999998e-06 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259911+0000", 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6313000000000001e-05 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259920+0000", 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6559999999999998e-06 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259978+0000", 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7766999999999998e-05 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.260103+0000", 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125308 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.260226+0000", 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000122903 2024-09-16T10:31:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.265746+0000", 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0055199480000000002 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.267454+0000", 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017075199999999999 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127358 11.2 11:49a0a50e:::benchmark_data_smithi063_176907_object3082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.262940+0000", 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.222163839, 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.22236983900000001, 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127358 2024-09-16T10:31:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262940+0000", 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262940+0000", 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262942+0000", 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0540000000000002e-06 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262947+0000", 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8189999999999997e-06 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262958+0000", 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0879e-05 2024-09-16T10:31:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262988+0000", 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9958999999999999e-05 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.263042+0000", 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3380000000000001e-05 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.263167+0000", 2024-09-16T10:31:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125029 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.263274+0000", 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010689 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.265829+0000", 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025558 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.267649+0000", 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001819928 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 4 2024-09-16T10:31:15.694 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:31:15.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:15.739 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:15.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:15.840 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:15.877 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:15.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:15.979 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:16.025 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:16.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:16.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:16.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:16.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 506'8037 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 506'7798 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 1.14 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 506'7971 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 506'7924 2024-09-16T10:31:16.331 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 506'8020 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 506'8162 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: pgmap v953: 168 pgs: 3 remapped+peering, 1 unknown, 35 peering, 129 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.8 KiB/s rd, 24 MiB/s wr, 895 op/s; 4.5 KiB/s, 3 objects/s recovering 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[29993]: osdmap e510: 8 total, 8 up, 7 in 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 506'8037 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 506'7798 2024-09-16T10:31:16.332 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 1.14 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 506'7971 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 506'7924 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 506'8020 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 506'8162 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: pgmap v953: 168 pgs: 3 remapped+peering, 1 unknown, 35 peering, 129 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.8 KiB/s rd, 24 MiB/s wr, 895 op/s; 4.5 KiB/s, 3 objects/s recovering 2024-09-16T10:31:16.333 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:16 smithi063 ceph-mon[39470]: osdmap e510: 8 total, 8 up, 7 in 2024-09-16T10:31:16.397 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:16.434 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:16.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 506'8037 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 506'7798 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 1.14 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 506'7971 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 1.10 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:31:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 506'7924 2024-09-16T10:31:16.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 506'8020 2024-09-16T10:31:16.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 506'8162 2024-09-16T10:31:16.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:31:16.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: pgmap v953: 168 pgs: 3 remapped+peering, 1 unknown, 35 peering, 129 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.8 KiB/s rd, 24 MiB/s wr, 895 op/s; 4.5 KiB/s, 3 objects/s recovering 2024-09-16T10:31:16.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:16 smithi165 ceph-mon[32696]: osdmap e510: 8 total, 8 up, 7 in 2024-09-16T10:31:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.577 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:16.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:16.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:16.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:16.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:16.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:16.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:16.985 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:16.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:17.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:17.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:17.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:17.219 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:17.255 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:17.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:17.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:17.419 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:17.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:17.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:31:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:31:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:31:17.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:17.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:17.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:17.614 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:31:17.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:17.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:17.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:17.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:17.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:17.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:17.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:17.929 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:31:17.930 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:31:17.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:17.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:17.979 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:31:17.980 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:31:18.022 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:31:18.022 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:31:18.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:18.062 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:31:18.062 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:31:18.094 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:31:18.094 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:31:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:18.116 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:18.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:18.119 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:31:18.134 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:31:18.134 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:31:18.144 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[29993]: osdmap e511: 8 total, 8 up, 7 in 2024-09-16T10:31:18.145 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2443991591' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:31:18.145 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[29993]: from='client.46523 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:31:18.145 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[29993]: pgmap v956: 168 pgs: 3 remapped+peering, 1 unknown, 35 peering, 129 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 255 B/s rd, 7.7 MiB/s wr, 407 op/s; 136 B/s, 5 objects/s recovering 2024-09-16T10:31:18.145 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/454739931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:18.145 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[39470]: osdmap e511: 8 total, 8 up, 7 in 2024-09-16T10:31:18.145 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2443991591' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:31:18.145 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[39470]: from='client.46523 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:31:18.146 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[39470]: pgmap v956: 168 pgs: 3 remapped+peering, 1 unknown, 35 peering, 129 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 255 B/s rd, 7.7 MiB/s wr, 407 op/s; 136 B/s, 5 objects/s recovering 2024-09-16T10:31:18.146 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:18 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/454739931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:18.150 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:18.174 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:31:18.174 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:31:18.214 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:31:18.214 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:31:18.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:18.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:18.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:18.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:18.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:18.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:18.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:18.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:18.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:18 smithi165 ceph-mon[32696]: osdmap e511: 8 total, 8 up, 7 in 2024-09-16T10:31:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2443991591' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:31:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:18 smithi165 ceph-mon[32696]: from='client.46523 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:31:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:18 smithi165 ceph-mon[32696]: pgmap v956: 168 pgs: 3 remapped+peering, 1 unknown, 35 peering, 129 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 255 B/s rd, 7.7 MiB/s wr, 407 op/s; 136 B/s, 5 objects/s recovering 2024-09-16T10:31:18.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:18 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/454739931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:18.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:18.662 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:18.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:18.764 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:18.803 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:18.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:18.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:18.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:18.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:19.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:19.082 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:19.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:19.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:19.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:19.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:19.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:19.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:19.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:19.398 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:31:19.399 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:19.399 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[29993]: from='client.46523 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:31:19.399 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[29993]: osdmap e512: 8 total, 8 up, 7 in 2024-09-16T10:31:19.399 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:31:19.399 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:19.399 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[39470]: from='client.46523 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:31:19.399 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:19 smithi063 ceph-mon[39470]: osdmap e512: 8 total, 8 up, 7 in 2024-09-16T10:31:19.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:19.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:19.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:19 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:31:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:19 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:19 smithi165 ceph-mon[32696]: from='client.46523 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:31:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:19 smithi165 ceph-mon[32696]: osdmap e512: 8 total, 8 up, 7 in 2024-09-16T10:31:19.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:31:19 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:31:19.591 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:19.628 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:19.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:19.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:19.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:19.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:19.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:19.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:19.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:19.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:20.039 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:20.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:20.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:20.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:20.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:20.373 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:31:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:31:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:20.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[29993]: pgmap v958: 168 pgs: 4 active+remapped+backfilling, 2 remapped+peering, 4 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 689 op/s; 42249/355248 objects misplaced (11.893%); 33 MiB/s, 538 objects/s recovering 2024-09-16T10:31:20.375 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:31:20.375 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:31:20.375 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:20.375 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:20 smithi063 ceph-mon[39470]: pgmap v958: 168 pgs: 4 active+remapped+backfilling, 2 remapped+peering, 4 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 689 op/s; 42249/355248 objects misplaced (11.893%); 33 MiB/s, 538 objects/s recovering 2024-09-16T10:31:20.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:20.437 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:31:20.437 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:31:20.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:20.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:20.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:20 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:31:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:20 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:31:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:20 smithi165 ceph-mon[32696]: pgmap v958: 168 pgs: 4 active+remapped+backfilling, 2 remapped+peering, 4 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 689 op/s; 42249/355248 objects misplaced (11.893%); 33 MiB/s, 538 objects/s recovering 2024-09-16T10:31:20.586 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:20.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:20.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:20.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:20.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:20.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:20.998 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:20.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:21.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:21.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:21.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:21.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:21.276 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:21.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:21.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:21.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:31:21.418 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:21.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:21.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:21 smithi063 ceph-mon[39470]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:31:21.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:21 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-16T10:31:21.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:21 smithi063 ceph-mon[29993]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:31:21.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:21 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-16T10:31:21.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:21.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:21.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:21.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:21 smithi165 ceph-mon[32696]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:31:21.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:21 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-16T10:31:21.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:21.697 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:21.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:21.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:21.829 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 2] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:31:21.829 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:31:21.829 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T10:31:21.829 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T10:31:21.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:21.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:21.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:21.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:31:21.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:22.019 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:22.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:22.120 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:22.183 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:22.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:22.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:22.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:22.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:22.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:22.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:22.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:22.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:22 smithi063 ceph-mon[29993]: pgmap v959: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 557 op/s; 42249/354234 objects misplaced (11.927%); 24 MiB/s, 391 objects/s recovering 2024-09-16T10:31:22.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:22 smithi063 ceph-mon[39470]: pgmap v959: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 557 op/s; 42249/354234 objects misplaced (11.927%); 24 MiB/s, 391 objects/s recovering 2024-09-16T10:31:22.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:22 smithi165 ceph-mon[32696]: pgmap v959: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 557 op/s; 42249/354234 objects misplaced (11.927%); 24 MiB/s, 391 objects/s recovering 2024-09-16T10:31:22.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:22.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:22.688 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:22.730 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:22.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:22.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:22.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:22.975 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.016 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:23.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:23.118 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:23.153 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:23.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:23.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:23.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:23.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:23.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:23.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:23.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:23.667 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:23.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:23.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:23.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:23 smithi165 ceph-mon[32696]: 5.6 scrub starts 2024-09-16T10:31:23.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:23 smithi165 ceph-mon[32696]: 5.6 scrub ok 2024-09-16T10:31:23.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:23.926 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:23 smithi063 ceph-mon[29993]: 5.6 scrub starts 2024-09-16T10:31:23.926 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:23 smithi063 ceph-mon[29993]: 5.6 scrub ok 2024-09-16T10:31:23.926 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:23 smithi063 ceph-mon[39470]: 5.6 scrub starts 2024-09-16T10:31:23.926 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:23 smithi063 ceph-mon[39470]: 5.6 scrub ok 2024-09-16T10:31:23.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:23.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:24.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:24.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:24.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:24.110 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:31:24.170 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:24.237 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:24.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:24.264 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:24.271 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:24.283 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:24.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:24.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:24.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:24.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:24.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:24.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:24.599 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:24.668 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:24.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:24.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:24 smithi165 ceph-mon[32696]: pgmap v960: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 916 op/s; 39180/343986 objects misplaced (11.390%); 34 MiB/s, 553 objects/s recovering 2024-09-16T10:31:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:24.809 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:24.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:24.816 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:24 smithi063 ceph-mon[29993]: pgmap v960: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 916 op/s; 39180/343986 objects misplaced (11.390%); 34 MiB/s, 553 objects/s recovering 2024-09-16T10:31:24.816 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:24.816 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:24.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:24 smithi063 ceph-mon[39470]: pgmap v960: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 916 op/s; 39180/343986 objects misplaced (11.390%); 34 MiB/s, 553 objects/s recovering 2024-09-16T10:31:24.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:24.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:24.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:24.950 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:24.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:25.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:25.096 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:25.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:25.197 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:25.243 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:25.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:25.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:25.386 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:25.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:25.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:31:25.418 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:31:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 512, 2024-09-16T10:31:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "7cb51a9a-7413-11ef-bceb-c7b262605968", 2024-09-16T10:31:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-09-16T10:08:21.081928+0000", 2024-09-16T10:31:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-09-16T10:31:18.088794+0000", 2024-09-16T10:31:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "last_up_change": "2024-09-16T10:28:27.035727+0000", 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "last_in_change": "2024-09-16T10:31:13.105778+0000", 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_num": 5804032, 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_set": [ 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "nodeep-scrub", 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "pglog_hardlimit", 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "purged_snapdirs", 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_deletes", 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "sortbitwise" 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_version": 19, 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T10:31:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_snapshot": "", 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_max": 11, 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd": 8, 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "require_min_compat_client": "luminous", 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "min_compat_client": "luminous", 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "require_osd_release": "squid", 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_crimson": false, 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "pools": [ 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 1, 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".mgr", 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:11:23.055843+0000", 2024-09-16T10:31:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:31:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 21, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 21, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 21, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 21, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 21, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "1.b", 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 275, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 274, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 274, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "73'79", 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "73'79" 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "357", 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "275", 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "306", 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:31:25.423 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:31:25.424 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:31:25.425 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_max": 32, 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_min": 1 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": {} 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:31:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.6699999570846558, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.6699999570846558, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 2, 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "datapool", 2024-09-16T10:31:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:34.734896+0000", 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 8193, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:31:25.428 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 3, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 3, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 3, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 3, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 3, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "61", 2024-09-16T10:31:25.429 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 2, 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 61, 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:31:25.430 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.431 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd": {} 2024-09-16T10:31:25.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 2.3299999237060547, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 2.3299999237060547, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 0.86000001430511475, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 2, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 2, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.433 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 3, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": ".rgw.root", 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:35.550974+0000", 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:31:25.434 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:31:25.435 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "60", 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:31:25.436 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:31:25.437 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:31:25.440 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:31:25.440 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:31:25.440 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:31:25.440 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:31:25.440 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:31:25.440 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:31:25.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:31:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:31:25.442 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 4, 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.log", 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:37.166765+0000", 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:31:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:31:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "62", 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:31:25.446 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:31:25.447 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:31:25.448 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.9700000286102295, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.9700000286102295, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 5, 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.control", 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:39.104673+0000", 2024-09-16T10:31:25.449 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:31:25.450 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "64", 2024-09-16T10:31:25.451 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:31:25.452 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:31:25.453 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:31:25.454 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:31:25.455 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:31:25.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.455 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:31:25.455 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:31:25.455 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.5299999713897705, 2024-09-16T10:31:25.455 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.5299999713897705, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 6, 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "default.rgw.meta", 2024-09-16T10:31:25.456 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:13:41.175068+0000", 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:31:25.457 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 32, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 32, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 32, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 32, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 32, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:31:25.458 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "66", 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:31:25.459 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:31:25.460 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:31:25.461 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "options": { 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_bias": 4 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "rgw": {} 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:31:25.462 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "pool": 11, 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": "unique_pool_4", 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "create_time": "2024-09-16T10:29:33.629083+0000", 2024-09-16T10:31:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "flags_names": "hashpspool", 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "type": 1, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 3, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": 2, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_rule": 0, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_count": 0, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_target": 0, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "is_stretch_pool": false, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "object_hash": 2, 2024-09-16T10:31:25.464 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscale_mode": "off", 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": 16, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num": 16, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_placement_num_target": 16, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_target": 16, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num_pending": 16, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "last_pg_merge_meta": { 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "source_pgid": "0.0", 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "ready_epoch": 0, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_started": 0, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "last_epoch_clean": 0, 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "source_version": "0'0", 2024-09-16T10:31:25.465 INFO:teuthology.orchestra.run.smithi063.stdout: "target_version": "0'0" 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_change": "494", 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend": "0", 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "auid": 0, 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_mode": "selfmanaged", 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_seq": 0, 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_epoch": 0, 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_snaps": [], 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps": "[]", 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_bytes": 0, 2024-09-16T10:31:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: "quota_max_objects": 0, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "tiers": [], 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_of": -1, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "read_tier": -1, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "write_tier": -1, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_mode": "none", 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_bytes": 0, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "target_max_objects": 0, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_flush_age": 0, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "cache_min_evict_age": 0, 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profile": "", 2024-09-16T10:31:25.467 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_params": { 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "none" 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_period": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_count": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "use_gmt_hitset": true, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "min_read_recency_for_promote": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "min_write_recency_for_promote": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "hit_set_search_last_n": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "grade_table": [], 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "stripe_width": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "expected_num_objects": 0, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "fast_read": false, 2024-09-16T10:31:25.468 INFO:teuthology.orchestra.run.smithi063.stdout: "options": {}, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "application_metadata": { 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "rados": {} 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "read_balance": { 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "score_type": "Fair distribution", 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "score_acting": 1.75, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "score_stable": 1.75, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "optimal_score": 1, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_acting": 1.75, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "raw_score_stable": 1.75, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity_weighted": 0.78670120239257812, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-16T10:31:25.469 INFO:teuthology.orchestra.run.smithi063.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "c4234e56-a2f2-467f-9616-190a52181d42", 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:31:25.470 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 8, 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 510, 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:31:25.471 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6802", 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.472 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6804", 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.473 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6803", 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2031112888 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6801/2031112888", 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6802/2031112888", 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6804/2031112888", 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6803/2031112888", 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "6fe9a4c9-c351-4223-8aab-6405152696c6", 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 13, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 510, 2024-09-16T10:31:25.475 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6805", 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.476 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6806", 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.477 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6808", 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6807", 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3717451957 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.478 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6805/3717451957", 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6806/3717451957", 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6808/3717451957", 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6807/3717451957", 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.479 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8", 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 18, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 510, 2024-09-16T10:31:25.480 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6809", 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.481 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6810", 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6812", 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.482 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6811", 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 196353360 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6809/196353360", 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6810/196353360", 2024-09-16T10:31:25.483 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6812/196353360", 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6811/196353360", 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "355833dd-7cc5-4f59-b036-ed8939892ac4", 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:31:25.484 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 24, 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 510, 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6813", 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.485 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6814", 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.486 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6816", 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6815", 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 249840957 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6813/249840957", 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.63:6814/249840957", 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.63:6816/249840957", 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.63:6815/249840957", 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "da2fbe06-9462-44ec-9ffb-c5563ccee4b7", 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.488 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 31, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 362, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6800", 2024-09-16T10:31:25.489 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6801", 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.490 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6803", 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6802", 2024-09-16T10:31:25.491 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2373552307 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6800/2373552307", 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6801/2373552307", 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6803/2373552307", 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6802/2373552307", 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.492 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "b866ad81-ae05-4c8e-9e12-822440fcb713", 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 37, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 508, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.494 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.494 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.494 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6804", 2024-09-16T10:31:25.494 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:31:25.494 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:31:25.495 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6805", 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.496 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6807", 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6806", 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 783450148 2024-09-16T10:31:25.497 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6804/783450148", 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6805/783450148", 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6807/783450148", 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6806/783450148", 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:31:25.498 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "ce05cb1a-e5c6-4ea9-a2c0-0039c874affa", 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 1, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 1, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 1, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 0, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 0, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 45, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 508, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 0, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.499 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6808", 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6809", 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.500 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6811", 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6810", 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 424252741 2024-09-16T10:31:25.501 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6808/424252741", 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6809/424252741", 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6811/424252741", 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6810/424252741", 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.502 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "uuid": "1b464a8a-ad32-47ab-82fb-3097077ebc48", 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "up": 1, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "in": 0, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_affinity": 0.293609619140625, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_begin": 54, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "last_clean_end": 386, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "up_from": 441, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "up_thru": 452, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "down_at": 387, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "lost_at": 0, 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-09-16T10:31:25.503 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6812", 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addrs": { 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6813", 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.504 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addrs": { 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6815", 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addrs": { 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.505 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.165:6814", 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 1921603083 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.165:6812/1921603083", 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_addr": "172.21.15.165:6813/1921603083", 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1921603083", 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1921603083", 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "state": [ 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "exists", 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "up" 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:25.506 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_xinfo": [ 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 0, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:31:03.665201+0000", 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 1, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.507 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:30:34.122190+0000", 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 2, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:30:21.344326+0000", 2024-09-16T10:31:25.508 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 3, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:30:39.651493+0000", 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 4, 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.509 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:31:12.264556+0000", 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 5, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:30:43.361323+0000", 2024-09-16T10:31:25.510 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 6, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 0, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:31:01.283209+0000", 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 0 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": 7, 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "down_stamp": "0.000000", 2024-09-16T10:31:25.511 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_probability": 0, 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "laggy_interval": 0, 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "features": 4540719139924082687, 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "old_weight": 65536, 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "last_purged_snaps_scrub": "2024-09-16T10:30:39.611904+0000", 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "dead_epoch": 386 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap": [ 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.5", 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.512 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: 5 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.0", 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.513 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_items": [ 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.e", 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:31:25.514 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "3.14", 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 1 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.e", 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:31:25.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.13", 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 2 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "4.1d", 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 1, 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 5 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "5.1", 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 4, 2024-09-16T10:31:25.517 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 6 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "6.19", 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: "mappings": [ 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: "from": 3, 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: "to": 0 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:25.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_upmap_primaries": [], 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_temp": [ 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.7", 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "1.10", 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: 5, 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:31:25.519 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "11.0", 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: 3, 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "11.3", 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.520 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "11.4", 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: 2, 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "11.5", 2024-09-16T10:31:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: 5, 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: 6, 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "11.e", 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: 4, 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: "pgid": "11.f", 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "osds": [ 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: 2, 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: 7 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "primary_temp": [], 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "blocklist": { 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/56975596": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/455268617": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/3928610681": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/955884223": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1387667925": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:31:25.523 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/670090510": "2024-09-17T10:08:51.022048+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1037308168": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2411639026": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/812143427": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/274451570": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/1121825432": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:6800/2945123439": "2024-09-17T10:14:06.991973+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2625337613": "2024-09-17T10:09:14.602270+0000", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "172.21.15.63:0/2993184887": "2024-09-17T10:14:06.991973+0000" 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "range_blocklist": {}, 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "erasure_code_profiles": { 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "default": { 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-failure-domain": "osd", 2024-09-16T10:31:25.524 INFO:teuthology.orchestra.run.smithi063.stdout: "k": "2", 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "m": "1", 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "plugin": "jerasure", 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "technique": "reed_sol_van" 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_snaps_queue": [], 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "new_removed_snaps": [], 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "new_purged_snaps": [], 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_node_flags": {}, 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "device_class_flags": {}, 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": { 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_enabled": false, 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_bucket_count": 0, 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "degraded_stretch_mode": 0, 2024-09-16T10:31:25.525 INFO:teuthology.orchestra.run.smithi063.stdout: "recovering_stretch_mode": 0, 2024-09-16T10:31:25.526 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode_bucket": 0 2024-09-16T10:31:25.526 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:25.526 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:31:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:25.551 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:25.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:25.559 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:31:25.560 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:31:25.609 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:31:25.609 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:31:25.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/966478988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:25.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2156438033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:31:25.610 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/966478988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:25.610 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2156438033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:31:25.642 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:31:25.643 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:31:25.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:25.679 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:31:25.679 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:31:25.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:25.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:25.713 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:31:25.713 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:31:25.746 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:31:25.747 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:31:25.780 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:31:25.781 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:31:25.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/966478988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2156438033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-16T10:31:25.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:25.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:25.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:25.946 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:25.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:26.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:26.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:26.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:26.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:26.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:26.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:26.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:26.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:26.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:26.452 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:26.496 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:26.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:26.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:26.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:26.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:26.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:26.776 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:26.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:26.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:26 smithi165 ceph-mon[32696]: pgmap v961: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 36901/338352 objects misplaced (10.906%); 43 MiB/s, 693 objects/s recovering 2024-09-16T10:31:26.868 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:26 smithi063 ceph-mon[29993]: pgmap v961: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 36901/338352 objects misplaced (10.906%); 43 MiB/s, 693 objects/s recovering 2024-09-16T10:31:26.868 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:26 smithi063 ceph-mon[39470]: pgmap v961: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 36901/338352 objects misplaced (10.906%); 43 MiB/s, 693 objects/s recovering 2024-09-16T10:31:26.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:26.921 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:26.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:27.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:27.069 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:27.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:27.140 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:31:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:31:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:31:27.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:27.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:27.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:27.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:27.344 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:27.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:27.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:27.486 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:27.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:27.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:27.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:27.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:27.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:27.759 INFO:tasks.thrashosds.thrasher:j is {'epoch': 512, 'fsid': '7cb51a9a-7413-11ef-bceb-c7b262605968', 'created': '2024-09-16T10:08:21.081928+0000', 'modified': '2024-09-16T10:31:18.088794+0000', 'last_up_change': '2024-09-16T10:28:27.035727+0000', 'last_in_change': '2024-09-16T10:31:13.105778+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:11:23.055843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 275, 'last_epoch_started': 274, 'last_epoch_clean': 274, 'source_version': "73'79", 'target_version': "73'79"}, 'last_change': '357', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '275', 'last_force_op_resend_preluminous': '306', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6699999570846558, 'score_stable': 1.6699999570846558, 'optimal_score': 1, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-16T10:13:34.734896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-16T10:13:35.550974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-16T10:13:37.166765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-16T10:13:39.104673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-16T10:13:41.175068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-16T10:29:33.629083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '494', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.7867012023925781, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.7900000214576721}}], 'osds': [{'osd': 0, 'uuid': 'c4234e56-a2f2-467f-9616-190a52181d42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 510, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6801', 'nonce': 2031112888}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6802', 'nonce': 2031112888}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6804', 'nonce': 2031112888}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6803', 'nonce': 2031112888}]}, 'public_addr': '172.21.15.63:6801/2031112888', 'cluster_addr': '172.21.15.63:6802/2031112888', 'heartbeat_back_addr': '172.21.15.63:6804/2031112888', 'heartbeat_front_addr': '172.21.15.63:6803/2031112888', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '6fe9a4c9-c351-4223-8aab-6405152696c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 510, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6805', 'nonce': 3717451957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6806', 'nonce': 3717451957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6808', 'nonce': 3717451957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6807', 'nonce': 3717451957}]}, 'public_addr': '172.21.15.63:6805/3717451957', 'cluster_addr': '172.21.15.63:6806/3717451957', 'heartbeat_back_addr': '172.21.15.63:6808/3717451957', 'heartbeat_front_addr': '172.21.15.63:6807/3717451957', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 510, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6809', 'nonce': 196353360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6810', 'nonce': 196353360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6812', 'nonce': 196353360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6811', 'nonce': 196353360}]}, 'public_addr': '172.21.15.63:6809/196353360', 'cluster_addr': '172.21.15.63:6810/196353360', 'heartbeat_back_addr': '172.21.15.63:6812/196353360', 'heartbeat_front_addr': '172.21.15.63:6811/196353360', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '355833dd-7cc5-4f59-b036-ed8939892ac4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 24, 'up_thru': 510, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6813', 'nonce': 249840957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6814', 'nonce': 249840957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6816', 'nonce': 249840957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.63:6815', 'nonce': 249840957}]}, 'public_addr': '172.21.15.63:6813/249840957', 'cluster_addr': '172.21.15.63:6814/249840957', 'heartbeat_back_addr': '172.21.15.63:6816/249840957', 'heartbeat_front_addr': '172.21.15.63:6815/249840957', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'da2fbe06-9462-44ec-9ffb-c5563ccee4b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 2373552307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2373552307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2373552307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 2373552307}]}, 'public_addr': '172.21.15.165:6800/2373552307', 'cluster_addr': '172.21.15.165:6801/2373552307', 'heartbeat_back_addr': '172.21.15.165:6803/2373552307', 'heartbeat_front_addr': '172.21.15.165:6802/2373552307', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b866ad81-ae05-4c8e-9e12-822440fcb713', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 508, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 783450148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 783450148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 783450148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 783450148}]}, 'public_addr': '172.21.15.165:6804/783450148', 'cluster_addr': '172.21.15.165:6805/783450148', 'heartbeat_back_addr': '172.21.15.165:6807/783450148', 'heartbeat_front_addr': '172.21.15.165:6806/783450148', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce05cb1a-e5c6-4ea9-a2c0-0039c874affa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 508, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 424252741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 424252741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 424252741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 424252741}]}, 'public_addr': '172.21.15.165:6808/424252741', 'cluster_addr': '172.21.15.165:6809/424252741', 'heartbeat_back_addr': '172.21.15.165:6811/424252741', 'heartbeat_front_addr': '172.21.15.165:6810/424252741', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b464a8a-ad32-47ab-82fb-3097077ebc48', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.293609619140625, 'last_clean_begin': 54, 'last_clean_end': 386, 'up_from': 441, 'up_thru': 452, 'down_at': 387, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1921603083}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1921603083}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1921603083}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1921603083}]}, 'public_addr': '172.21.15.165:6812/1921603083', 'cluster_addr': '172.21.15.165:6813/1921603083', 'heartbeat_back_addr': '172.21.15.165:6815/1921603083', 'heartbeat_front_addr': '172.21.15.165:6814/1921603083', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:31:03.665201+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:30:34.122190+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:30:21.344326+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:30:39.651493+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:31:12.264556+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:30:43.361323+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T10:31:01.283209+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T10:30:39.611904+0000', 'dead_epoch': 386}], 'pg_upmap': [{'pgid': '1.5', 'osds': [6, 3, 5]}, {'pgid': '4.0', 'osds': [4, 6, 2]}], 'pg_upmap_items': [{'pgid': '3.e', 'mappings': [{'from': 4, 'to': 5}, {'from': 1, 'to': 0}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '6.19', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.7', 'osds': [6, 3, 7]}, {'pgid': '1.10', 'osds': [5, 6, 7]}, {'pgid': '11.0', 'osds': [1, 3, 7]}, {'pgid': '11.3', 'osds': [6, 1, 7]}, {'pgid': '11.4', 'osds': [6, 2, 7]}, {'pgid': '11.5', 'osds': [5, 6, 7]}, {'pgid': '11.e', 'osds': [1, 4, 7]}, {'pgid': '11.f', 'osds': [1, 2, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.63:0/56975596': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/455268617': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/3928610681': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/955884223': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/1387667925': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:6800/670090510': '2024-09-17T10:08:51.022048+0000', '172.21.15.63:0/1037308168': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2411639026': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/812143427': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/274451570': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/1121825432': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:6800/2945123439': '2024-09-17T10:14:06.991973+0000', '172.21.15.63:0/2625337613': '2024-09-17T10:09:14.602270+0000', '172.21.15.63:0/2993184887': '2024-09-17T10:14:06.991973+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T10:31:27.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:31:27.776 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:27.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:27.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:27.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:27.922 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:27.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:28.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:28.068 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:28.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:28.169 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:28.213 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:28.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:28.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:28.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:28.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:28.449 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:28.493 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:28.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:28.552 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:28.560 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:31:28.594 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:28.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:28.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:28.743 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:28.803 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:28.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:28.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[29993]: pgmap v962: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s; 36901/338352 objects misplaced (10.906%); 36 MiB/s, 581 objects/s recovering 2024-09-16T10:31:28.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2403800819' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:28.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[29993]: from='client.46544 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:28.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[29993]: 5.d scrub starts 2024-09-16T10:31:28.902 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[29993]: 5.d scrub ok 2024-09-16T10:31:28.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[39470]: pgmap v962: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s; 36901/338352 objects misplaced (10.906%); 36 MiB/s, 581 objects/s recovering 2024-09-16T10:31:28.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2403800819' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:28.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[39470]: from='client.46544 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:28.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[39470]: 5.d scrub starts 2024-09-16T10:31:28.903 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:28 smithi063 ceph-mon[39470]: 5.d scrub ok 2024-09-16T10:31:28.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:28.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:28.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:29.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:28 smithi165 ceph-mon[32696]: pgmap v962: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 864 op/s; 36901/338352 objects misplaced (10.906%); 36 MiB/s, 581 objects/s recovering 2024-09-16T10:31:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:28 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2403800819' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:28 smithi165 ceph-mon[32696]: from='client.46544 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:28 smithi165 ceph-mon[32696]: 5.d scrub starts 2024-09-16T10:31:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:28 smithi165 ceph-mon[32696]: 5.d scrub ok 2024-09-16T10:31:29.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:29.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:29.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:29.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:29.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:29.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:29.369 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:29.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:29.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:29.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:29.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:29.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:31:29 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:31:29.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:29.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:29.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:29.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:29.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:29.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:29.786 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:31:29.786 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[29993]: from='client.46544 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[29993]: osdmap e513: 8 total, 8 up, 7 in 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[39470]: from='client.46544 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[39470]: osdmap e513: 8 total, 8 up, 7 in 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:29.787 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:29 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:29.859 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60004 9.b 9.36b0083b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.39112639999996, 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41720589800000002, 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.860 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60004 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463114+0000", 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463115+0000", 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7430000000000001e-06 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463140+0000", 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4477e-05 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463151+0000", 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0891e-05 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.463161+0000", 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0020000000000001e-05 2024-09-16T10:31:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464130+0000", 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00096871499999999998 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.464137+0000", 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1749999999999999e-06 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.864 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880306+0000", 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.41616966999999999 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880320+0000", 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3207000000000001e-05 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60061 9.b 9.669f9f4b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.865 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.34967678600003, 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37579170000000001, 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60061 2024-09-16T10:31:29.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504563+0000", 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.867 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504565+0000", 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9400000000000001e-06 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504589+0000", 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3971000000000001e-05 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.868 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504602+0000", 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2539e-05 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504610+0000", 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4460000000000001e-06 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504674+0000", 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3895999999999999e-05 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.504682+0000", 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.0260000000000007e-06 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880348+0000", 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37566548300000002 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880355+0000", 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3989999999999998e-06 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60067 9.b 9.deba987b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.34351585800005, 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36966423799999998, 2024-09-16T10:31:29.871 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60067 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510724+0000", 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510727+0000", 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8370000000000001e-06 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510777+0000", 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9957999999999998e-05 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510794+0000", 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7356000000000001e-05 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510820+0000", 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6071000000000002e-05 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510892+0000", 2024-09-16T10:31:29.874 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1520999999999995e-05 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.510901+0000", 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5920000000000001e-06 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880380+0000", 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36947966100000001 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880388+0000", 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.242e-06 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60145 9.b 9.255df14b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.29670502199997, 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32288046799999998, 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.876 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60145 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.877 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557535+0000", 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557537+0000", 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9080000000000002e-06 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.878 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557562+0000", 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5131e-05 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557575+0000", 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3039e-05 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557582+0000", 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0110000000000003e-06 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.879 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557855+0000", 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027304500000000003 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.557867+0000", 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1507e-05 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880409+0000", 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.32254223399999998 2024-09-16T10:31:29.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880416+0000", 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5930000000000002e-06 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60212 9.b 9.3f093dfb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.25253044299996, 2024-09-16T10:31:29.881 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27873467899999999, 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60212 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.882 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601710+0000", 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.883 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601712+0000", 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4609999999999998e-06 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601740+0000", 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8008999999999999e-05 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601759+0000", 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8427000000000002e-05 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601767+0000", 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5539999999999998e-06 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601927+0000", 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015967299999999999 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.601935+0000", 2024-09-16T10:31:29.885 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8040000000000002e-06 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880436+0000", 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27850148000000002 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880444+0000", 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2709999999999992e-06 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60241 9.b 9.c91b333b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.22721549699997, 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25344918100000002, 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60241 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.887 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627025+0000", 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627028+0000", 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.948e-06 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627081+0000", 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3631999999999997e-05 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627100+0000", 2024-09-16T10:31:29.889 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9191999999999999e-05 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627110+0000", 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6409999999999997e-06 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627207+0000", 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6762999999999995e-05 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.627215+0000", 2024-09-16T10:31:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2260000000000005e-06 2024-09-16T10:31:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880467+0000", 2024-09-16T10:31:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.25325206 2024-09-16T10:31:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880474+0000", 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7190000000000004e-06 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60255 9.b 9.7e4a5ddb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.21894220900003, 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.245202226, 2024-09-16T10:31:29.893 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60255 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.894 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635298+0000", 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635300+0000", 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2620000000000001e-06 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.895 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635352+0000", 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2079000000000001e-05 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635365+0000", 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3081000000000001e-05 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.896 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635373+0000", 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6909999999999997e-06 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635435+0000", 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2271000000000001e-05 2024-09-16T10:31:29.897 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.635445+0000", 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6099999999999995e-06 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880493+0000", 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.24504827800000001 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.898 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880500+0000", 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9539999999999998e-06 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:60261 9.b 9.61c3adbb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 534.21217965100004, 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23846684600000001, 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.899 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 60261 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.900 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642060+0000", 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642062+0000", 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0269999999999998e-06 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642104+0000", 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1473000000000001e-05 2024-09-16T10:31:29.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642120+0000", 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6235e-05 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642127+0000", 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2470000000000002e-06 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.902 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642184+0000", 2024-09-16T10:31:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6725999999999999e-05 2024-09-16T10:31:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.642192+0000", 2024-09-16T10:31:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1200000000000002e-06 2024-09-16T10:31:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880520+0000", 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.23832803799999999 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.904 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:35.880527+0000", 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9800000000000001e-06 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14514.0:2964 5.d 5:b4812045:::notify.4:head [watch ping cookie 93976561472000 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:37.320904+0000", 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 292.53333632900001, 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68289200299999997, 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.905 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14514", 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 2964 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320904+0000", 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.906 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320904+0000", 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320907+0000", 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2569999999999999e-06 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320919+0000", 2024-09-16T10:31:29.907 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2444e-05 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320945+0000", 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5389e-05 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.320956+0000", 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0919e-05 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.908 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.909 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321365+0000", 2024-09-16T10:31:29.909 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000409326 2024-09-16T10:31:29.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.909 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.909 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321381+0000", 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6087999999999999e-05 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.003572+0000", 2024-09-16T10:31:29.910 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68219090000000004 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.003725+0000", 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000153392 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.003796+0000", 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0288000000000002e-05 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.911 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14514.0:2972 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 93976561472000 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:37.321458+0000", 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 292.53278209199999, 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68540705099999999, 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.912 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14514", 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 2972 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321458+0000", 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321458+0000", 2024-09-16T10:31:29.913 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321459+0000", 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2589999999999999e-06 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321464+0000", 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4209999999999997e-06 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321476+0000", 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2704999999999999e-05 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321484+0000", 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.447e-06 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321550+0000", 2024-09-16T10:31:29.915 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6458000000000006e-05 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.321559+0000", 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0140000000000004e-06 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.004206+0000", 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.68264650900000001 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.916 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.004285+0000", 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8716999999999999e-05 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.006865+0000", 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025805210000000001 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.917 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59829 11.2 11:45cc2119:::benchmark_data_smithi063_176907_object59828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.647584+0000", 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 77.206655683999998, 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.214262705, 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59829 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.918 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.647584+0000", 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.647584+0000", 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.919 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.647587+0000", 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1500000000000002e-06 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.647612+0000", 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4995999999999999e-05 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.647629+0000", 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7469000000000002e-05 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.647637+0000", 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2800000000000003e-06 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.648543+0000", 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00090531599999999998 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.648877+0000", 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00033436799999999998 2024-09-16T10:31:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.649141+0000", 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026372199999999998 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.657150+0000", 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0080091390000000002 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.657161+0000", 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1504e-05 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.861679+0000", 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.204518002 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.861786+0000", 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010713300000000001 2024-09-16T10:31:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.861847+0000", 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0625999999999999e-05 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59836 11.2 11:4e68f5eb:::benchmark_data_smithi063_176907_object59835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.650865+0000", 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 77.203374797999999, 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.211404387, 2024-09-16T10:31:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59836 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.650865+0000", 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.650865+0000", 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.650867+0000", 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.807e-06 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.650892+0000", 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4697000000000001e-05 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.650903+0000", 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1236000000000001e-05 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.650911+0000", 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6820000000000003e-06 2024-09-16T10:31:29.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653058+0000", 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0021471300000000001 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653367+0000", 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030880399999999998 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.928 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.653497+0000", 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013071900000000001 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.659506+0000", 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0060090789999999996 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.659746+0000", 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000239208 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.861890+0000", 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.20214464800000001 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.862116+0000", 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00022558499999999999 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.930 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.862270+0000", 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015379200000000001 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:59839 11.2 11:4781d219:::benchmark_data_smithi063_176907_object59838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:12.651247+0000", 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 77.202993339000002, 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21432679299999999, 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 59839 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.651247+0000", 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.932 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.651247+0000", 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.651248+0000", 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.39e-06 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.651270+0000", 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1844999999999998e-05 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.651358+0000", 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7635999999999999e-05 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.651391+0000", 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3370000000000001e-05 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.934 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.657190+0000", 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0057989890000000001 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.657313+0000", 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012279399999999999 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.657440+0000", 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000127145 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.935 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.659760+0000", 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023204609999999998 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.660637+0000", 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087612599999999997 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.865434+0000", 2024-09-16T10:31:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.20479734699999999 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.865529+0000", 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5155000000000006e-05 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:12.865574+0000", 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4535e-05 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.937 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:104183 11.2 11:4e6d85d5:::benchmark_data_smithi063_176907_object104182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:51.911400+0000", 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 37.942839919000001, 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.215152502, 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.938 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 104183 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.911400+0000", 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.911400+0000", 2024-09-16T10:31:29.939 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.911402+0000", 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9939999999999999e-06 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.911424+0000", 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1375000000000001e-05 2024-09-16T10:31:29.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.911433+0000", 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.8730000000000002e-06 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.911440+0000", 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2990000000000002e-06 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.941 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.912971+0000", 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015310600000000001 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913063+0000", 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2493999999999999e-05 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913153+0000", 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9845000000000005e-05 2024-09-16T10:31:29.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.923606+0000", 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010453318 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.933969+0000", 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010362303999999999 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.943 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.126391+0000", 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.19242241700000001 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.126499+0000", 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000107624 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.126553+0000", 2024-09-16T10:31:29.944 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3899e-05 2024-09-16T10:31:29.945 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.945 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:104187 11.2 11:4b1cbddf:::benchmark_data_smithi063_176907_object104186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e502)", 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:30:51.913413+0000", 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 37.940826981000001, 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.21335494699999999, 2024-09-16T10:31:29.946 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 104187 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.947 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913413+0000", 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.948 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913413+0000", 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913415+0000", 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5200000000000001e-06 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913437+0000", 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2424000000000001e-05 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913447+0000", 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9629999999999993e-06 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.950 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.913454+0000", 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5810000000000002e-06 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.916590+0000", 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031359550000000002 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.916715+0000", 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012579900000000001 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.951 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.916869+0000", 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015327599999999999 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:51.927008+0000", 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010139336000000001 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.126598+0000", 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.19958957499999999 2024-09-16T10:31:29.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.126636+0000", 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8282999999999999e-05 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.126729+0000", 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3040000000000004e-05 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.953 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:30:52.126768+0000", 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9195e-05 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14514.0:3542 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 93976561472000 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:13.112531+0000", 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 16.741709051000001, 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0481043489999999, 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.954 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14514", 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 3542 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112531+0000", 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112531+0000", 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112533+0000", 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1689999999999999e-06 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112540+0000", 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.827e-06 2024-09-16T10:31:29.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.957 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.957 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.957 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112557+0000", 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6880000000000001e-05 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112566+0000", 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6619999999999994e-06 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112633+0000", 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7818999999999996e-05 2024-09-16T10:31:29.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112640+0000", 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5719999999999999e-06 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.126977+0000", 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.014337333000000001 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:29.960 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.126989+0000", 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1579e-05 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.139433+0000", 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0124442899999999 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.139762+0000", 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000329267 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.155002+0000", 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.01524004 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.155091+0000", 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8868000000000004e-05 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.160635+0000", 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0055440430000000002 2024-09-16T10:31:29.962 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127352 11.2 11:4562636d:::benchmark_data_smithi063_176907_object3076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.242850+0000", 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 14.611389784, 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.28510269500000002, 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.963 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127352 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242850+0000", 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.964 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242850+0000", 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242852+0000", 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0899999999999999e-06 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.242859+0000", 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5810000000000002e-06 2024-09-16T10:31:29.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.244847+0000", 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001987862 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.244863+0000", 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6030000000000001e-05 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.966 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.244934+0000", 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1384999999999995e-05 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.245066+0000", 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000131478 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.245541+0000", 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00047533900000000002 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.967 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.256992+0000", 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011450638000000001 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259422+0000", 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024297960000000001 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.527859+0000", 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26843726600000001 2024-09-16T10:31:29.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.527911+0000", 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2000999999999999e-05 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.527953+0000", 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2228999999999998e-05 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127357 11.2 11:48f6d68a:::benchmark_data_smithi063_176907_object3081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.259888+0000", 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 14.594351909, 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26817089100000002, 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.970 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127357 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259888+0000", 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259888+0000", 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259890+0000", 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8199999999999999e-06 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259895+0000", 2024-09-16T10:31:29.972 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1159999999999998e-06 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259911+0000", 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6313000000000001e-05 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259920+0000", 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.6559999999999998e-06 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.973 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.259978+0000", 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7766999999999998e-05 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.260103+0000", 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125308 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.260226+0000", 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000122903 2024-09-16T10:31:29.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.265746+0000", 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0055199480000000002 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.267454+0000", 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017075199999999999 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.975 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.527991+0000", 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.260537146 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.528029+0000", 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8300000000000003e-05 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.976 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.528059+0000", 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0094e-05 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127358 11.2 11:49a0a50e:::benchmark_data_smithi063_176907_object3082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.262940+0000", 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 14.591300595, 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26796584400000001, 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.977 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127358 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.978 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262940+0000", 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262940+0000", 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262942+0000", 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0540000000000002e-06 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262947+0000", 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8189999999999997e-06 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262958+0000", 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0879e-05 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.262988+0000", 2024-09-16T10:31:29.980 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.9958999999999999e-05 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.263042+0000", 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3380000000000001e-05 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.263167+0000", 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125029 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.263274+0000", 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010689 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.265829+0000", 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025558 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.267649+0000", 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001819928 2024-09-16T10:31:29.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.530810+0000", 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26316040200000002 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.530862+0000", 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2547999999999999e-05 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.983 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.530905+0000", 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3155999999999998e-05 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:127376 11.2 11:45bdf2af:::benchmark_data_smithi063_176907_object3100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.285590+0000", 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 14.568649667000001, 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.247654125, 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:29.984 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 127376 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285590+0000", 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.985 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285590+0000", 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285593+0000", 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4059999999999999e-06 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285598+0000", 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3310000000000003e-06 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:29.986 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285612+0000", 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3451999999999999e-05 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285619+0000", 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2219999999999996e-06 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285702+0000", 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2832999999999998e-05 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:29.987 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285832+0000", 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013025499999999999 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3", 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.285906+0000", 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4453e-05 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.288461+0000", 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002554635 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.988 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.288821+0000", 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00036031799999999999 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.533150+0000", 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.24432884199999999 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.533204+0000", 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4280000000000002e-05 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:29.989 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.533245+0000", 2024-09-16T10:31:29.990 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0098000000000001e-05 2024-09-16T10:31:29.990 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.990 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.990 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.990 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:29.990 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:29.990 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:31:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:29.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:29.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:29 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:31:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:29 smithi165 ceph-mon[32696]: from='client.46544 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:29 smithi165 ceph-mon[32696]: osdmap e513: 8 total, 8 up, 7 in 2024-09-16T10:31:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:29 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:31:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:29 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:29 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:30.092 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:30.132 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:30.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:30.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:30.277 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:30.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:30.379 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:30.415 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:30.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:30.516 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:30.532 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:31:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:30.560 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:30.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:30.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:30.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:30.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:30.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:30.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:30.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:30.926 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:30.933 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:30.934 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[29993]: pgmap v964: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 31803/324147 objects misplaced (9.811%); 44 MiB/s, 699 objects/s recovering 2024-09-16T10:31:30.934 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:31:30.934 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:31:30.934 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:30.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:30.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[39470]: pgmap v964: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 31803/324147 objects misplaced (9.811%); 44 MiB/s, 699 objects/s recovering 2024-09-16T10:31:30.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:31:30.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:31:30.935 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:30 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:30.970 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:30.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:31.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:30 smithi165 ceph-mon[32696]: pgmap v964: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 31803/324147 objects misplaced (9.811%); 44 MiB/s, 699 objects/s recovering 2024-09-16T10:31:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:30 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:31:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:30 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:31:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:30 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:31.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:31.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:31.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:31.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:31.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:31.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:31.301 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:31:31.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:31.392 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:31.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:31.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:31.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:31.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:31.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:31.672 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:31.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:31.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:31 smithi063 ceph-mon[39470]: 3.9 scrub starts 2024-09-16T10:31:31.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:31 smithi063 ceph-mon[39470]: 3.9 scrub ok 2024-09-16T10:31:31.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:31 smithi063 ceph-mon[29993]: 3.9 scrub starts 2024-09-16T10:31:31.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:31 smithi063 ceph-mon[29993]: 3.9 scrub ok 2024-09-16T10:31:31.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:31.815 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:31.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:31.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:31.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:31.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:32.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:31 smithi165 ceph-mon[32696]: 3.9 scrub starts 2024-09-16T10:31:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:31 smithi165 ceph-mon[32696]: 3.9 scrub ok 2024-09-16T10:31:32.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:32.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:32.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:32.217 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:32.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:32.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:32.364 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:32.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:32.465 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:32.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:32.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:32.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:32.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:32.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:32.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:32.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:32.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:32.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:32.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:32.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:32.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:32 smithi063 ceph-mon[39470]: pgmap v965: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 31803/323247 objects misplaced (9.839%); 44 MiB/s, 699 objects/s recovering 2024-09-16T10:31:32.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:32 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:31:32.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:32 smithi063 ceph-mon[39470]: 1.14 scrub ok 2024-09-16T10:31:32.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:32 smithi063 ceph-mon[29993]: pgmap v965: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 31803/323247 objects misplaced (9.839%); 44 MiB/s, 699 objects/s recovering 2024-09-16T10:31:32.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:32 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:31:32.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:32 smithi063 ceph-mon[29993]: 1.14 scrub ok 2024-09-16T10:31:33.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:33.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:33.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:32 smithi165 ceph-mon[32696]: pgmap v965: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 31803/323247 objects misplaced (9.839%); 44 MiB/s, 699 objects/s recovering 2024-09-16T10:31:33.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:32 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:31:33.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:32 smithi165 ceph-mon[32696]: 1.14 scrub ok 2024-09-16T10:31:33.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:33.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:33.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:33.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:33.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:33.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:33.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:33.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:33.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:33.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:33.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:33.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:33.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:33.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:33.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:34.075 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:34.116 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:34.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:34.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:34.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:34.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:34.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:34.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:34.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:34.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:34.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:34.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:34.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:34.638 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:31:34.643 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:31:34.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:34.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:34.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:34.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:34 smithi063 ceph-mon[29993]: pgmap v966: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 28989/313482 objects misplaced (9.247%); 44 MiB/s, 700 objects/s recovering 2024-09-16T10:31:34.761 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:34 smithi063 ceph-mon[39470]: pgmap v966: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 28989/313482 objects misplaced (9.247%); 44 MiB/s, 700 objects/s recovering 2024-09-16T10:31:34.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:34.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:34.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:34.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:34.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:34.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:34.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:35.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:35.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:34 smithi165 ceph-mon[32696]: pgmap v966: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 28989/313482 objects misplaced (9.247%); 44 MiB/s, 700 objects/s recovering 2024-09-16T10:31:35.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:35.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:35.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:35.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:35.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:35.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:35.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:35.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:35.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:35.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:35.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:35.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:35.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:35.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:35.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:31:35.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:35.732 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:35.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:35.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:35.893 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:35.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:35 smithi063 ceph-mon[29993]: from='client.46550 v1:172.21.15.63:0/4004031688' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:31:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:35 smithi063 ceph-mon[29993]: 3.13 scrub starts 2024-09-16T10:31:35.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:35 smithi063 ceph-mon[29993]: 3.13 scrub ok 2024-09-16T10:31:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:35 smithi063 ceph-mon[39470]: from='client.46550 v1:172.21.15.63:0/4004031688' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:31:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:35 smithi063 ceph-mon[39470]: 3.13 scrub starts 2024-09-16T10:31:35.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:35 smithi063 ceph-mon[39470]: 3.13 scrub ok 2024-09-16T10:31:35.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:36.015 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:36.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:35 smithi165 ceph-mon[32696]: from='client.46550 v1:172.21.15.63:0/4004031688' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:31:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:35 smithi165 ceph-mon[32696]: 3.13 scrub starts 2024-09-16T10:31:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:35 smithi165 ceph-mon[32696]: 3.13 scrub ok 2024-09-16T10:31:36.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:36.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:36.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:36.279 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:36.381 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:36.419 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:36.521 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.547 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:36.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:36.648 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.693 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:36.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:36.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:36.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:36.919 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:36.947 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:36.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:36.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:36 smithi063 ceph-mon[29993]: pgmap v967: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 26988/307596 objects misplaced (8.774%); 43 MiB/s, 686 objects/s recovering 2024-09-16T10:31:36.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:36 smithi063 ceph-mon[39470]: pgmap v967: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 26988/307596 objects misplaced (8.774%); 43 MiB/s, 686 objects/s recovering 2024-09-16T10:31:37.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:37.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:36 smithi165 ceph-mon[32696]: pgmap v967: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 26988/307596 objects misplaced (8.774%); 43 MiB/s, 686 objects/s recovering 2024-09-16T10:31:37.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:37.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:37.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:37.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:37.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:37.330 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:37.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:37.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:37.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:31:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:31:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:31:37.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:37.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:37.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:37.579 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:37.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:37.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:37.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:37.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:37.813 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:37.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:37.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:37 smithi063 ceph-mon[29993]: osdmap e514: 8 total, 8 up, 7 in 2024-09-16T10:31:37.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:37 smithi063 ceph-mon[39470]: osdmap e514: 8 total, 8 up, 7 in 2024-09-16T10:31:37.959 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:38.000 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:38.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:37 smithi165 ceph-mon[32696]: osdmap e514: 8 total, 8 up, 7 in 2024-09-16T10:31:38.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:38.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:38.231 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:38.258 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:38.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:38.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:38.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:38.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:38.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:38.649 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:38.751 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:38.788 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:38.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:38.932 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:38.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:38.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[39470]: pgmap v969: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 795 op/s; 26988/307596 objects misplaced (8.774%); 35 MiB/s, 567 objects/s recovering 2024-09-16T10:31:38.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[39470]: osdmap e515: 8 total, 8 up, 7 in 2024-09-16T10:31:38.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:31:38.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:31:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[29993]: pgmap v969: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 795 op/s; 26988/307596 objects misplaced (8.774%); 35 MiB/s, 567 objects/s recovering 2024-09-16T10:31:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[29993]: osdmap e515: 8 total, 8 up, 7 in 2024-09-16T10:31:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:31:38.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:38 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:31:39.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:39.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:38 smithi165 ceph-mon[32696]: pgmap v969: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 795 op/s; 26988/307596 objects misplaced (8.774%); 35 MiB/s, 567 objects/s recovering 2024-09-16T10:31:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:38 smithi165 ceph-mon[32696]: osdmap e515: 8 total, 8 up, 7 in 2024-09-16T10:31:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:38 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:31:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:38 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:31:39.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:39.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:39.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:39.296 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:39.322 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:39.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:39.467 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:39.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:31:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:31:39.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:39.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:39.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:39.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:39.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:39.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:39.940 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:39.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[39470]: osdmap e516: 8 total, 8 up, 7 in 2024-09-16T10:31:39.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[39470]: 6.14 scrub starts 2024-09-16T10:31:39.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[39470]: 6.14 scrub ok 2024-09-16T10:31:39.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:39.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[29993]: osdmap e516: 8 total, 8 up, 7 in 2024-09-16T10:31:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[29993]: 6.14 scrub starts 2024-09-16T10:31:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[29993]: 6.14 scrub ok 2024-09-16T10:31:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:39.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:39.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:40.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:39 smithi165 ceph-mon[32696]: osdmap e516: 8 total, 8 up, 7 in 2024-09-16T10:31:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:39 smithi165 ceph-mon[32696]: 6.14 scrub starts 2024-09-16T10:31:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:39 smithi165 ceph-mon[32696]: 6.14 scrub ok 2024-09-16T10:31:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:39 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:40.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:40.122 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:40.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:40.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:40.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:40.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:40.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:40.376 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":966,"stamp":"2024-09-16T10:31:33.488469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6835634950,"num_objects":104494,"num_object_clones":0,"num_object_copies":313482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28989,"num_objects_unfound":0,"num_objects_dirty":104494,"num_whiteouts":0,"num_read":1525,"num_read_kb":1282,"num_write":144980,"num_write_kb":7953592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11235,"num_bytes_recovered":724074000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67472,"ondisk_log_size":67472,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23443336,"kb_used_data":18309016,"kb_used_omap":10,"kb_used_meta":5134133,"kb_avail":632772728,"statfs":{"total":671965249536,"available":647959273472,"internally_reserved":0,"allocated":18748432384,"data_stored":18745156398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11161,"internal_metadata":5257352295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,10,7,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-666370048,"num_objects":-10168,"num_object_clones":0,"num_object_copies":-30504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10191,"num_objects_unfound":0,"num_objects_dirty":-10168,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7008,"num_bytes_recovered":459276288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004728"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":981,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576005+0000","last_change":"2024-09-16T10:29:36.039702+0000","last_active":"2024-09-16T10:31:28.576005+0000","last_peered":"2024-09-16T10:31:28.576005+0000","last_clean":"2024-09-16T10:31:28.576005+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:31:28.576005+0000","last_undegraded":"2024-09-16T10:31:28.576005+0000","last_fullsized":"2024-09-16T10:31:28.576005+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.039652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.039652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.261361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":927,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566951+0000","last_change":"2024-09-16T10:30:16.596737+0000","last_active":"2024-09-16T10:31:28.566951+0000","last_peered":"2024-09-16T10:31:28.566951+0000","last_clean":"2024-09-16T10:31:28.566951+0000","last_became_active":"2024-09-16T10:26:28.557096+0000","last_became_peered":"2024-09-16T10:26:28.557096+0000","last_unstale":"2024-09-16T10:31:28.566951+0000","last_undegraded":"2024-09-16T10:31:28.566951+0000","last_fullsized":"2024-09-16T10:31:28.566951+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:16.596655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.596655+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:38.661213+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":985,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.577902+0000","last_change":"2024-09-16T10:31:14.141815+0000","last_active":"2024-09-16T10:31:28.577902+0000","last_peered":"2024-09-16T10:31:28.577902+0000","last_clean":"2024-09-16T10:31:28.577902+0000","last_became_active":"2024-09-16T10:31:14.141497+0000","last_became_peered":"2024-09-16T10:31:14.141497+0000","last_unstale":"2024-09-16T10:31:28.577902+0000","last_undegraded":"2024-09-16T10:31:28.577902+0000","last_fullsized":"2024-09-16T10:31:28.577902+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.872906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.872906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:31.850644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":291,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569319+0000","last_change":"2024-09-16T10:30:17.879751+0000","last_active":"2024-09-16T10:31:28.569319+0000","last_peered":"2024-09-16T10:31:28.569319+0000","last_clean":"2024-09-16T10:31:28.569319+0000","last_became_active":"2024-09-16T10:26:28.551847+0000","last_became_peered":"2024-09-16T10:26:28.551847+0000","last_unstale":"2024-09-16T10:31:28.569319+0000","last_undegraded":"2024-09-16T10:31:28.569319+0000","last_fullsized":"2024-09-16T10:31:28.569319+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:17.879610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:17.879610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:38.628891+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":963,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575742+0000","last_change":"2024-09-16T10:31:14.148991+0000","last_active":"2024-09-16T10:31:28.575742+0000","last_peered":"2024-09-16T10:31:28.575742+0000","last_clean":"2024-09-16T10:31:28.575742+0000","last_became_active":"2024-09-16T10:31:14.148273+0000","last_became_peered":"2024-09-16T10:31:14.148273+0000","last_unstale":"2024-09-16T10:31:28.575742+0000","last_undegraded":"2024-09-16T10:31:28.575742+0000","last_fullsized":"2024-09-16T10:31:28.575742+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":988,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575636+0000","last_change":"2024-09-16T10:30:48.140925+0000","last_active":"2024-09-16T10:31:28.575636+0000","last_peered":"2024-09-16T10:31:28.575636+0000","last_clean":"2024-09-16T10:31:28.575636+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:31:28.575636+0000","last_undegraded":"2024-09-16T10:31:28.575636+0000","last_fullsized":"2024-09-16T10:31:28.575636+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:51.960827+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":1058,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576558+0000","last_change":"2024-09-16T10:30:15.998707+0000","last_active":"2024-09-16T10:31:28.576558+0000","last_peered":"2024-09-16T10:31:28.576558+0000","last_clean":"2024-09-16T10:31:28.576558+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:31:28.576558+0000","last_undegraded":"2024-09-16T10:31:28.576558+0000","last_fullsized":"2024-09-16T10:31:28.576558+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:15.998614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.998614+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:33.187618+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":291,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567113+0000","last_change":"2024-09-16T10:30:07.591239+0000","last_active":"2024-09-16T10:31:28.567113+0000","last_peered":"2024-09-16T10:31:28.567113+0000","last_clean":"2024-09-16T10:31:28.567113+0000","last_became_active":"2024-09-16T10:26:28.556509+0000","last_became_peered":"2024-09-16T10:26:28.556509+0000","last_unstale":"2024-09-16T10:31:28.567113+0000","last_undegraded":"2024-09-16T10:31:28.567113+0000","last_fullsized":"2024-09-16T10:31:28.567113+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.591170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.591170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":991,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569840+0000","last_change":"2024-09-16T10:30:04.838255+0000","last_active":"2024-09-16T10:31:28.569840+0000","last_peered":"2024-09-16T10:31:28.569840+0000","last_clean":"2024-09-16T10:31:28.569840+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:31:28.569840+0000","last_undegraded":"2024-09-16T10:31:28.569840+0000","last_fullsized":"2024-09-16T10:31:28.569840+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.838182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.838182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":885,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.574973+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:31:28.574973+0000","last_peered":"2024-09-16T10:31:28.574973+0000","last_clean":"2024-09-16T10:31:28.574973+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:31:28.574973+0000","last_undegraded":"2024-09-16T10:31:28.574973+0000","last_fullsized":"2024-09-16T10:31:28.574973+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":985,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569142+0000","last_change":"2024-09-16T10:31:14.144852+0000","last_active":"2024-09-16T10:31:28.569142+0000","last_peered":"2024-09-16T10:31:28.569142+0000","last_clean":"2024-09-16T10:31:28.569142+0000","last_became_active":"2024-09-16T10:31:14.144519+0000","last_became_peered":"2024-09-16T10:31:14.144519+0000","last_unstale":"2024-09-16T10:31:28.569142+0000","last_undegraded":"2024-09-16T10:31:28.569142+0000","last_fullsized":"2024-09-16T10:31:28.569142+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:41.762023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:41.762023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:56.611809+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":972,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572042+0000","last_change":"2024-09-16T10:31:14.144630+0000","last_active":"2024-09-16T10:31:28.572042+0000","last_peered":"2024-09-16T10:31:28.572042+0000","last_clean":"2024-09-16T10:31:28.572042+0000","last_became_active":"2024-09-16T10:31:14.142022+0000","last_became_peered":"2024-09-16T10:31:14.142022+0000","last_unstale":"2024-09-16T10:31:28.572042+0000","last_undegraded":"2024-09-16T10:31:28.572042+0000","last_fullsized":"2024-09-16T10:31:28.572042+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:08.436243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.436243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":943,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569316+0000","last_change":"2024-09-16T10:30:13.941093+0000","last_active":"2024-09-16T10:31:28.569316+0000","last_peered":"2024-09-16T10:31:28.569316+0000","last_clean":"2024-09-16T10:31:28.569316+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:31:28.569316+0000","last_undegraded":"2024-09-16T10:31:28.569316+0000","last_fullsized":"2024-09-16T10:31:28.569316+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:13.941032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:13.941032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":873,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576210+0000","last_change":"2024-09-16T10:30:39.082533+0000","last_active":"2024-09-16T10:31:28.576210+0000","last_peered":"2024-09-16T10:31:28.576210+0000","last_clean":"2024-09-16T10:31:28.576210+0000","last_became_active":"2024-09-16T10:26:28.556933+0000","last_became_peered":"2024-09-16T10:26:28.556933+0000","last_unstale":"2024-09-16T10:31:28.576210+0000","last_undegraded":"2024-09-16T10:31:28.576210+0000","last_fullsized":"2024-09-16T10:31:28.576210+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:30:39.082450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.082450+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:45.258681+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":668,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.581033+0000","last_change":"2024-09-16T10:31:14.143481+0000","last_active":"2024-09-16T10:31:28.581033+0000","last_peered":"2024-09-16T10:31:28.581033+0000","last_clean":"2024-09-16T10:31:28.581033+0000","last_became_active":"2024-09-16T10:31:14.143134+0000","last_became_peered":"2024-09-16T10:31:14.143134+0000","last_unstale":"2024-09-16T10:31:28.581033+0000","last_undegraded":"2024-09-16T10:31:28.581033+0000","last_fullsized":"2024-09-16T10:31:28.581033+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.995349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.995349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:55.073140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":978,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568992+0000","last_change":"2024-09-16T10:30:45.741431+0000","last_active":"2024-09-16T10:31:28.568992+0000","last_peered":"2024-09-16T10:31:28.568992+0000","last_clean":"2024-09-16T10:31:28.568992+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:31:28.568992+0000","last_undegraded":"2024-09-16T10:31:28.568992+0000","last_fullsized":"2024-09-16T10:31:28.568992+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.741362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.741362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.595620+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":283,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566295+0000","last_change":"2024-09-16T10:29:16.166206+0000","last_active":"2024-09-16T10:31:28.566295+0000","last_peered":"2024-09-16T10:31:28.566295+0000","last_clean":"2024-09-16T10:31:28.566295+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:31:28.566295+0000","last_undegraded":"2024-09-16T10:31:28.566295+0000","last_fullsized":"2024-09-16T10:31:28.566295+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:16.166144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:16.166144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":976,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578517+0000","last_change":"2024-09-16T10:31:14.144948+0000","last_active":"2024-09-16T10:31:28.578517+0000","last_peered":"2024-09-16T10:31:28.578517+0000","last_clean":"2024-09-16T10:31:28.578517+0000","last_became_active":"2024-09-16T10:31:14.144533+0000","last_became_peered":"2024-09-16T10:31:14.144533+0000","last_unstale":"2024-09-16T10:31:28.578517+0000","last_undegraded":"2024-09-16T10:31:28.578517+0000","last_fullsized":"2024-09-16T10:31:28.578517+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.446934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.446934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.164812+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":534,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568546+0000","last_change":"2024-09-16T10:30:55.602908+0000","last_active":"2024-09-16T10:31:28.568546+0000","last_peered":"2024-09-16T10:31:28.568546+0000","last_clean":"2024-09-16T10:31:28.568546+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:31:28.568546+0000","last_undegraded":"2024-09-16T10:31:28.568546+0000","last_fullsized":"2024-09-16T10:31:28.568546+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.599903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.599903+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:18.999085+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":959,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.580616+0000","last_change":"2024-09-16T10:30:09.918436+0000","last_active":"2024-09-16T10:31:28.580616+0000","last_peered":"2024-09-16T10:31:28.580616+0000","last_clean":"2024-09-16T10:31:28.580616+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:31:28.580616+0000","last_undegraded":"2024-09-16T10:31:28.580616+0000","last_fullsized":"2024-09-16T10:31:28.580616+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:09.918371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:09.918371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":911,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575831+0000","last_change":"2024-09-16T10:27:18.318288+0000","last_active":"2024-09-16T10:31:28.575831+0000","last_peered":"2024-09-16T10:31:28.575831+0000","last_clean":"2024-09-16T10:31:28.575831+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:31:28.575831+0000","last_undegraded":"2024-09-16T10:31:28.575831+0000","last_fullsized":"2024-09-16T10:31:28.575831+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:18.318214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:18.318214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":747,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.581576+0000","last_change":"2024-09-16T10:26:28.551407+0000","last_active":"2024-09-16T10:31:28.581576+0000","last_peered":"2024-09-16T10:31:28.581576+0000","last_clean":"2024-09-16T10:31:28.581576+0000","last_became_active":"2024-09-16T10:26:28.551058+0000","last_became_peered":"2024-09-16T10:26:28.551058+0000","last_unstale":"2024-09-16T10:31:28.581576+0000","last_undegraded":"2024-09-16T10:31:28.581576+0000","last_fullsized":"2024-09-16T10:31:28.581576+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:59.137209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":1028,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578023+0000","last_change":"2024-09-16T10:30:38.072547+0000","last_active":"2024-09-16T10:31:28.578023+0000","last_peered":"2024-09-16T10:31:28.578023+0000","last_clean":"2024-09-16T10:31:28.578023+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:31:28.578023+0000","last_undegraded":"2024-09-16T10:31:28.578023+0000","last_fullsized":"2024-09-16T10:31:28.578023+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:30:38.072460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:38.072460+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:58.801909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":755,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576844+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:31:28.576844+0000","last_peered":"2024-09-16T10:31:28.576844+0000","last_clean":"2024-09-16T10:31:28.576844+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:31:28.576844+0000","last_undegraded":"2024-09-16T10:31:28.576844+0000","last_fullsized":"2024-09-16T10:31:28.576844+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:14.190419+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":388,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569121+0000","last_change":"2024-09-16T10:31:14.169546+0000","last_active":"2024-09-16T10:31:28.569121+0000","last_peered":"2024-09-16T10:31:28.569121+0000","last_clean":"2024-09-16T10:31:28.569121+0000","last_became_active":"2024-09-16T10:31:14.151297+0000","last_became_peered":"2024-09-16T10:31:14.151297+0000","last_unstale":"2024-09-16T10:31:28.569121+0000","last_undegraded":"2024-09-16T10:31:28.569121+0000","last_fullsized":"2024-09-16T10:31:28.569121+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:40.788922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.788922+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:02.804769+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":974,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578077+0000","last_change":"2024-09-16T10:30:12.928363+0000","last_active":"2024-09-16T10:31:28.578077+0000","last_peered":"2024-09-16T10:31:28.578077+0000","last_clean":"2024-09-16T10:31:28.578077+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:31:28.578077+0000","last_undegraded":"2024-09-16T10:31:28.578077+0000","last_fullsized":"2024-09-16T10:31:28.578077+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:30:12.928274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.928274+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":1023,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575281+0000","last_change":"2024-09-16T10:31:14.187902+0000","last_active":"2024-09-16T10:31:28.575281+0000","last_peered":"2024-09-16T10:31:28.575281+0000","last_clean":"2024-09-16T10:31:28.575281+0000","last_became_active":"2024-09-16T10:31:14.156596+0000","last_became_peered":"2024-09-16T10:31:14.156596+0000","last_unstale":"2024-09-16T10:31:28.575281+0000","last_undegraded":"2024-09-16T10:31:28.575281+0000","last_fullsized":"2024-09-16T10:31:28.575281+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:29.106770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.106770+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"508'11","reported_seq":1209,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566600+0000","last_change":"2024-09-16T10:31:14.167033+0000","last_active":"2024-09-16T10:31:28.566600+0000","last_peered":"2024-09-16T10:31:28.566600+0000","last_clean":"2024-09-16T10:31:28.566600+0000","last_became_active":"2024-09-16T10:31:14.137776+0000","last_became_peered":"2024-09-16T10:31:14.137776+0000","last_unstale":"2024-09-16T10:31:28.566600+0000","last_undegraded":"2024-09-16T10:31:28.566600+0000","last_fullsized":"2024-09-16T10:31:28.566600+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"513'9122","reported_seq":13895,"reported_epoch":513,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T10:31:33.260479+0000","last_change":"2024-09-16T10:31:15.251325+0000","last_active":"2024-09-16T10:31:33.260479+0000","last_peered":"2024-09-16T10:31:33.260479+0000","last_clean":"2024-09-16T10:31:13.052611+0000","last_became_active":"2024-09-16T10:31:15.218914+0000","last_became_peered":"2024-09-16T10:31:15.218914+0000","last_unstale":"2024-09-16T10:31:33.260479+0000","last_undegraded":"2024-09-16T10:31:33.260479+0000","last_fullsized":"2024-09-16T10:31:33.260479+0000","mapping_epoch":508,"log_start":"497'5413","ondisk_log_start":"497'5413","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421003264,"num_objects":6424,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4286,"num_objects_unfound":0,"num_objects_dirty":6424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":497472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2339,"num_bytes_recovered":153223168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":6424}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":813,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568884+0000","last_change":"2024-09-16T10:29:21.013569+0000","last_active":"2024-09-16T10:31:28.568884+0000","last_peered":"2024-09-16T10:31:28.568884+0000","last_clean":"2024-09-16T10:31:28.568884+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:31:28.568884+0000","last_undegraded":"2024-09-16T10:31:28.568884+0000","last_fullsized":"2024-09-16T10:31:28.568884+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:21.013503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.013503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:11.999269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":998,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578533+0000","last_change":"2024-09-16T10:27:12.407528+0000","last_active":"2024-09-16T10:31:28.578533+0000","last_peered":"2024-09-16T10:31:28.578533+0000","last_clean":"2024-09-16T10:31:28.578533+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:31:28.578533+0000","last_undegraded":"2024-09-16T10:31:28.578533+0000","last_fullsized":"2024-09-16T10:31:28.578533+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:27:12.407441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.407441+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":992,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578514+0000","last_change":"2024-09-16T10:31:14.159286+0000","last_active":"2024-09-16T10:31:28.578514+0000","last_peered":"2024-09-16T10:31:28.578514+0000","last_clean":"2024-09-16T10:31:28.578514+0000","last_became_active":"2024-09-16T10:31:14.158830+0000","last_became_peered":"2024-09-16T10:31:14.158830+0000","last_unstale":"2024-09-16T10:31:28.578514+0000","last_undegraded":"2024-09-16T10:31:28.578514+0000","last_fullsized":"2024-09-16T10:31:28.578514+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.213821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.213821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:57.158091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":378,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569520+0000","last_change":"2024-09-16T10:30:16.857737+0000","last_active":"2024-09-16T10:31:28.569520+0000","last_peered":"2024-09-16T10:31:28.569520+0000","last_clean":"2024-09-16T10:31:28.569520+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:31:28.569520+0000","last_undegraded":"2024-09-16T10:31:28.569520+0000","last_fullsized":"2024-09-16T10:31:28.569520+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:30:16.857676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.857676+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:33.191176+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":963,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569206+0000","last_change":"2024-09-16T10:30:14.925349+0000","last_active":"2024-09-16T10:31:28.569206+0000","last_peered":"2024-09-16T10:31:28.569206+0000","last_clean":"2024-09-16T10:31:28.569206+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:31:28.569206+0000","last_undegraded":"2024-09-16T10:31:28.569206+0000","last_fullsized":"2024-09-16T10:31:28.569206+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:14.925284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:14.925284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:39.491806+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"513'9244","reported_seq":12636,"reported_epoch":513,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T10:31:33.272234+0000","last_change":"2024-09-16T10:31:15.263335+0000","last_active":"2024-09-16T10:31:33.272234+0000","last_peered":"2024-09-16T10:31:33.272234+0000","last_clean":"2024-09-16T10:31:13.116495+0000","last_became_active":"2024-09-16T10:31:15.229467+0000","last_became_peered":"2024-09-16T10:31:15.229467+0000","last_unstale":"2024-09-16T10:31:33.272234+0000","last_undegraded":"2024-09-16T10:31:33.272234+0000","last_fullsized":"2024-09-16T10:31:33.272234+0000","mapping_epoch":508,"log_start":"498'5562","ondisk_log_start":"498'5562","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422838272,"num_objects":6452,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4943,"num_objects_unfound":0,"num_objects_dirty":6452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9244,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1650,"num_bytes_recovered":108134400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":6452}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":395,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567866+0000","last_change":"2024-09-16T10:30:18.655233+0000","last_active":"2024-09-16T10:31:28.567866+0000","last_peered":"2024-09-16T10:31:28.567866+0000","last_clean":"2024-09-16T10:31:28.567866+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:31:28.567866+0000","last_undegraded":"2024-09-16T10:31:28.567866+0000","last_fullsized":"2024-09-16T10:31:28.567866+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:18.655172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:18.655172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:47.426763+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":348,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566622+0000","last_change":"2024-09-16T10:26:44.445756+0000","last_active":"2024-09-16T10:31:28.566622+0000","last_peered":"2024-09-16T10:31:28.566622+0000","last_clean":"2024-09-16T10:31:28.566622+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:31:28.566622+0000","last_undegraded":"2024-09-16T10:31:28.566622+0000","last_fullsized":"2024-09-16T10:31:28.566622+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:26:44.445661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.445661+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":957,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576304+0000","last_change":"2024-09-16T10:31:14.146422+0000","last_active":"2024-09-16T10:31:28.576304+0000","last_peered":"2024-09-16T10:31:28.576304+0000","last_clean":"2024-09-16T10:31:28.576304+0000","last_became_active":"2024-09-16T10:31:14.146107+0000","last_became_peered":"2024-09-16T10:31:14.146107+0000","last_unstale":"2024-09-16T10:31:28.576304+0000","last_undegraded":"2024-09-16T10:31:28.576304+0000","last_fullsized":"2024-09-16T10:31:28.576304+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:55.254993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.254993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:58.755978+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":715,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566576+0000","last_change":"2024-09-16T10:31:14.164915+0000","last_active":"2024-09-16T10:31:28.566576+0000","last_peered":"2024-09-16T10:31:28.566576+0000","last_clean":"2024-09-16T10:31:28.566576+0000","last_became_active":"2024-09-16T10:31:14.164506+0000","last_became_peered":"2024-09-16T10:31:14.164506+0000","last_unstale":"2024-09-16T10:31:28.566576+0000","last_undegraded":"2024-09-16T10:31:28.566576+0000","last_fullsized":"2024-09-16T10:31:28.566576+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.521586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.521586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":767,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.574879+0000","last_change":"2024-09-16T10:27:40.298188+0000","last_active":"2024-09-16T10:31:28.574879+0000","last_peered":"2024-09-16T10:31:28.574879+0000","last_clean":"2024-09-16T10:31:28.574879+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:31:28.574879+0000","last_undegraded":"2024-09-16T10:31:28.574879+0000","last_fullsized":"2024-09-16T10:31:28.574879+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:32.026659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"513'9262","reported_seq":9362,"reported_epoch":513,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:31:33.259782+0000","last_change":"2024-09-16T10:31:15.266974+0000","last_active":"2024-09-16T10:31:33.259782+0000","last_peered":"2024-09-16T10:31:33.259782+0000","last_clean":"2024-09-16T10:31:13.099472+0000","last_became_active":"2024-09-16T10:31:15.212663+0000","last_became_peered":"2024-09-16T10:31:15.212663+0000","last_unstale":"2024-09-16T10:31:33.259782+0000","last_undegraded":"2024-09-16T10:31:33.259782+0000","last_fullsized":"2024-09-16T10:31:33.259782+0000","mapping_epoch":508,"log_start":"499'5579","ondisk_log_start":"499'5579","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423231488,"num_objects":6458,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6458,"num_objects_unfound":0,"num_objects_dirty":6458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":6458}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":892,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.577079+0000","last_change":"2024-09-16T10:31:17.119206+0000","last_active":"2024-09-16T10:31:28.577079+0000","last_peered":"2024-09-16T10:31:28.577079+0000","last_clean":"2024-09-16T10:31:28.577079+0000","last_became_active":"2024-09-16T10:31:17.118842+0000","last_became_peered":"2024-09-16T10:31:17.118842+0000","last_unstale":"2024-09-16T10:31:28.577079+0000","last_undegraded":"2024-09-16T10:31:28.577079+0000","last_fullsized":"2024-09-16T10:31:28.577079+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.198923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.198923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:11.115767+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":912,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569030+0000","last_change":"2024-09-16T10:30:12.893400+0000","last_active":"2024-09-16T10:31:28.569030+0000","last_peered":"2024-09-16T10:31:28.569030+0000","last_clean":"2024-09-16T10:31:28.569030+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:31:28.569030+0000","last_undegraded":"2024-09-16T10:31:28.569030+0000","last_fullsized":"2024-09-16T10:31:28.569030+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:30:12.893340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.893340+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:28.993798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":960,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576091+0000","last_change":"2024-09-16T10:27:44.624946+0000","last_active":"2024-09-16T10:31:28.576091+0000","last_peered":"2024-09-16T10:31:28.576091+0000","last_clean":"2024-09-16T10:31:28.576091+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:31:28.576091+0000","last_undegraded":"2024-09-16T10:31:28.576091+0000","last_fullsized":"2024-09-16T10:31:28.576091+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":1185,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575108+0000","last_change":"2024-09-16T10:27:24.281038+0000","last_active":"2024-09-16T10:31:28.575108+0000","last_peered":"2024-09-16T10:31:28.575108+0000","last_clean":"2024-09-16T10:31:28.575108+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:31:28.575108+0000","last_undegraded":"2024-09-16T10:31:28.575108+0000","last_fullsized":"2024-09-16T10:31:28.575108+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:27:24.280960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:24.280960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":825,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566988+0000","last_change":"2024-09-16T10:31:14.128649+0000","last_active":"2024-09-16T10:31:28.566988+0000","last_peered":"2024-09-16T10:31:28.566988+0000","last_clean":"2024-09-16T10:31:28.566988+0000","last_became_active":"2024-09-16T10:31:14.128195+0000","last_became_peered":"2024-09-16T10:31:14.128195+0000","last_unstale":"2024-09-16T10:31:28.566988+0000","last_undegraded":"2024-09-16T10:31:28.566988+0000","last_fullsized":"2024-09-16T10:31:28.566988+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.267597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.267597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"513'8641","reported_seq":8726,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.305101+0000","last_change":"2024-09-16T10:29:35.170854+0000","last_active":"2024-09-16T10:31:29.305101+0000","last_peered":"2024-09-16T10:31:29.305101+0000","last_clean":"2024-09-16T10:31:29.305101+0000","last_became_active":"2024-09-16T10:29:35.170400+0000","last_became_peered":"2024-09-16T10:29:35.170400+0000","last_unstale":"2024-09-16T10:31:29.305101+0000","last_undegraded":"2024-09-16T10:31:29.305101+0000","last_fullsized":"2024-09-16T10:31:29.305101+0000","mapping_epoch":476,"log_start":"490'3568","ondisk_log_start":"490'3568","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429522968,"num_objects":6555,"num_object_clones":0,"num_object_copies":19665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8641,"num_write_kb":486209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":932,"reported_epoch":513,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:31:28.566186+0000","last_change":"2024-09-16T10:31:15.128788+0000","last_active":"2024-09-16T10:31:28.566186+0000","last_peered":"2024-09-16T10:31:28.566186+0000","last_clean":"2024-09-16T10:31:08.647700+0000","last_became_active":"2024-09-16T10:31:15.127316+0000","last_became_peered":"2024-09-16T10:31:15.127316+0000","last_unstale":"2024-09-16T10:31:28.566186+0000","last_undegraded":"2024-09-16T10:31:28.566186+0000","last_fullsized":"2024-09-16T10:31:28.566186+0000","mapping_epoch":508,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.663450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.663450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,7],"avail_no_missing":["6","3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":1002,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.581409+0000","last_change":"2024-09-16T10:29:25.663951+0000","last_active":"2024-09-16T10:31:28.581409+0000","last_peered":"2024-09-16T10:31:28.581409+0000","last_clean":"2024-09-16T10:31:28.581409+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:31:28.581409+0000","last_undegraded":"2024-09-16T10:31:28.581409+0000","last_fullsized":"2024-09-16T10:31:28.581409+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:12.241564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":982,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569935+0000","last_change":"2024-09-16T10:29:27.891418+0000","last_active":"2024-09-16T10:31:28.569935+0000","last_peered":"2024-09-16T10:31:28.569935+0000","last_clean":"2024-09-16T10:31:28.569935+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:31:28.569935+0000","last_undegraded":"2024-09-16T10:31:28.569935+0000","last_fullsized":"2024-09-16T10:31:28.569935+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.891362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.891362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1174,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578003+0000","last_change":"2024-09-16T10:30:08.183366+0000","last_active":"2024-09-16T10:31:28.578003+0000","last_peered":"2024-09-16T10:31:28.578003+0000","last_clean":"2024-09-16T10:31:28.578003+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:31:28.578003+0000","last_undegraded":"2024-09-16T10:31:28.578003+0000","last_fullsized":"2024-09-16T10:31:28.578003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:08.182949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.182949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:30.974057+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":977,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572184+0000","last_change":"2024-09-16T10:30:12.258492+0000","last_active":"2024-09-16T10:31:28.572184+0000","last_peered":"2024-09-16T10:31:28.572184+0000","last_clean":"2024-09-16T10:31:28.572184+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:31:28.572184+0000","last_undegraded":"2024-09-16T10:31:28.572184+0000","last_fullsized":"2024-09-16T10:31:28.572184+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:12.258426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.258426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:29.743926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"513'9052","reported_seq":9138,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.232325+0000","last_change":"2024-09-16T10:29:35.168446+0000","last_active":"2024-09-16T10:31:33.232325+0000","last_peered":"2024-09-16T10:31:33.232325+0000","last_clean":"2024-09-16T10:31:33.232325+0000","last_became_active":"2024-09-16T10:29:35.168175+0000","last_became_peered":"2024-09-16T10:29:35.168175+0000","last_unstale":"2024-09-16T10:31:33.232325+0000","last_undegraded":"2024-09-16T10:31:33.232325+0000","last_fullsized":"2024-09-16T10:31:33.232325+0000","mapping_epoch":476,"log_start":"497'5343","ondisk_log_start":"497'5343","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.127276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417595392,"num_objects":6372,"num_object_clones":0,"num_object_copies":19116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9052,"num_write_kb":493568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":919,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566116+0000","last_change":"2024-09-16T10:30:11.662236+0000","last_active":"2024-09-16T10:31:28.566116+0000","last_peered":"2024-09-16T10:31:28.566116+0000","last_clean":"2024-09-16T10:31:28.566116+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:31:28.566116+0000","last_undegraded":"2024-09-16T10:31:28.566116+0000","last_fullsized":"2024-09-16T10:31:28.566116+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:11.662185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.662185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":1050,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578174+0000","last_change":"2024-09-16T10:31:14.184131+0000","last_active":"2024-09-16T10:31:28.578174+0000","last_peered":"2024-09-16T10:31:28.578174+0000","last_clean":"2024-09-16T10:31:28.578174+0000","last_became_active":"2024-09-16T10:31:14.160400+0000","last_became_peered":"2024-09-16T10:31:14.160400+0000","last_unstale":"2024-09-16T10:31:28.578174+0000","last_undegraded":"2024-09-16T10:31:28.578174+0000","last_fullsized":"2024-09-16T10:31:28.578174+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:29:27.111715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.111715+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":982,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.579645+0000","last_change":"2024-09-16T10:29:26.686298+0000","last_active":"2024-09-16T10:31:28.579645+0000","last_peered":"2024-09-16T10:31:28.579645+0000","last_clean":"2024-09-16T10:31:28.579645+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:31:28.579645+0000","last_undegraded":"2024-09-16T10:31:28.579645+0000","last_fullsized":"2024-09-16T10:31:28.579645+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:15.187317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":975,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.565862+0000","last_change":"2024-09-16T10:30:50.670023+0000","last_active":"2024-09-16T10:31:28.565862+0000","last_peered":"2024-09-16T10:31:28.565862+0000","last_clean":"2024-09-16T10:31:28.565862+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:31:28.565862+0000","last_undegraded":"2024-09-16T10:31:28.565862+0000","last_fullsized":"2024-09-16T10:31:28.565862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.669938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.669938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:00.286307+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":962,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.577780+0000","last_change":"2024-09-16T10:27:41.587855+0000","last_active":"2024-09-16T10:31:28.577780+0000","last_peered":"2024-09-16T10:31:28.577780+0000","last_clean":"2024-09-16T10:31:28.577780+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:31:28.577780+0000","last_undegraded":"2024-09-16T10:31:28.577780+0000","last_fullsized":"2024-09-16T10:31:28.577780+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:34.201280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"513'9135","reported_seq":9221,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.215438+0000","last_change":"2024-09-16T10:29:35.171242+0000","last_active":"2024-09-16T10:31:33.215438+0000","last_peered":"2024-09-16T10:31:33.215438+0000","last_clean":"2024-09-16T10:31:33.215438+0000","last_became_active":"2024-09-16T10:29:35.171007+0000","last_became_peered":"2024-09-16T10:29:35.171007+0000","last_unstale":"2024-09-16T10:31:33.215438+0000","last_undegraded":"2024-09-16T10:31:33.215438+0000","last_fullsized":"2024-09-16T10:31:33.215438+0000","mapping_epoch":476,"log_start":"496'5235","ondisk_log_start":"496'5235","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417529856,"num_objects":6371,"num_object_clones":0,"num_object_copies":19113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9135,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":865,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568889+0000","last_change":"2024-09-16T10:30:54.802360+0000","last_active":"2024-09-16T10:31:28.568889+0000","last_peered":"2024-09-16T10:31:28.568889+0000","last_clean":"2024-09-16T10:31:28.568889+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:31:28.568889+0000","last_undegraded":"2024-09-16T10:31:28.568889+0000","last_fullsized":"2024-09-16T10:31:28.568889+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:54.802277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.802277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:03.521355+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":1320,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.579433+0000","last_change":"2024-09-16T10:27:14.409646+0000","last_active":"2024-09-16T10:31:28.579433+0000","last_peered":"2024-09-16T10:31:28.579433+0000","last_clean":"2024-09-16T10:31:28.579433+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:31:28.579433+0000","last_undegraded":"2024-09-16T10:31:28.579433+0000","last_fullsized":"2024-09-16T10:31:28.579433+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:27:14.409552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:14.409552+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":283,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578420+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:31:28.578420+0000","last_peered":"2024-09-16T10:31:28.578420+0000","last_clean":"2024-09-16T10:31:28.578420+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:31:28.578420+0000","last_undegraded":"2024-09-16T10:31:28.578420+0000","last_fullsized":"2024-09-16T10:31:28.578420+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":986,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569607+0000","last_change":"2024-09-16T10:30:11.895955+0000","last_active":"2024-09-16T10:31:28.569607+0000","last_peered":"2024-09-16T10:31:28.569607+0000","last_clean":"2024-09-16T10:31:28.569607+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:31:28.569607+0000","last_undegraded":"2024-09-16T10:31:28.569607+0000","last_fullsized":"2024-09-16T10:31:28.569607+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:30:11.895746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.895746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:41.723636+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":951,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572992+0000","last_change":"2024-09-16T10:30:53.380048+0000","last_active":"2024-09-16T10:31:28.572992+0000","last_peered":"2024-09-16T10:31:28.572992+0000","last_clean":"2024-09-16T10:31:28.572992+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:31:28.572992+0000","last_undegraded":"2024-09-16T10:31:28.572992+0000","last_fullsized":"2024-09-16T10:31:28.572992+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.379980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.379980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:57.338825+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":988,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575385+0000","last_change":"2024-09-16T10:31:14.143712+0000","last_active":"2024-09-16T10:31:28.575385+0000","last_peered":"2024-09-16T10:31:28.575385+0000","last_clean":"2024-09-16T10:31:28.575385+0000","last_became_active":"2024-09-16T10:31:14.143349+0000","last_became_peered":"2024-09-16T10:31:14.143349+0000","last_unstale":"2024-09-16T10:31:28.575385+0000","last_undegraded":"2024-09-16T10:31:28.575385+0000","last_fullsized":"2024-09-16T10:31:28.575385+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.877913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.877913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"513'9194","reported_seq":9280,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.280971+0000","last_change":"2024-09-16T10:29:35.167788+0000","last_active":"2024-09-16T10:31:33.280971+0000","last_peered":"2024-09-16T10:31:33.280971+0000","last_clean":"2024-09-16T10:31:33.280971+0000","last_became_active":"2024-09-16T10:29:35.167515+0000","last_became_peered":"2024-09-16T10:29:35.167515+0000","last_unstale":"2024-09-16T10:31:33.280971+0000","last_undegraded":"2024-09-16T10:31:33.280971+0000","last_fullsized":"2024-09-16T10:31:33.280971+0000","mapping_epoch":476,"log_start":"496'5284","ondisk_log_start":"496'5284","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:35.081064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425852928,"num_objects":6498,"num_object_clones":0,"num_object_copies":19494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9194,"num_write_kb":502144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":458,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.573387+0000","last_change":"2024-09-16T10:31:17.108542+0000","last_active":"2024-09-16T10:31:28.573387+0000","last_peered":"2024-09-16T10:31:28.573387+0000","last_clean":"2024-09-16T10:31:28.573387+0000","last_became_active":"2024-09-16T10:31:17.108022+0000","last_became_peered":"2024-09-16T10:31:17.108022+0000","last_unstale":"2024-09-16T10:31:28.573387+0000","last_undegraded":"2024-09-16T10:31:28.573387+0000","last_fullsized":"2024-09-16T10:31:28.573387+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:49.434225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.434225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:57.929492+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":981,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578626+0000","last_change":"2024-09-16T10:30:05.864251+0000","last_active":"2024-09-16T10:31:28.578626+0000","last_peered":"2024-09-16T10:31:28.578626+0000","last_clean":"2024-09-16T10:31:28.578626+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:31:28.578626+0000","last_undegraded":"2024-09-16T10:31:28.578626+0000","last_fullsized":"2024-09-16T10:31:28.578626+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:30:05.864159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.864159+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":966,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.580250+0000","last_change":"2024-09-16T10:29:20.691013+0000","last_active":"2024-09-16T10:31:28.580250+0000","last_peered":"2024-09-16T10:31:28.580250+0000","last_clean":"2024-09-16T10:31:28.580250+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:31:28.580250+0000","last_undegraded":"2024-09-16T10:31:28.580250+0000","last_fullsized":"2024-09-16T10:31:28.580250+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":497,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.565185+0000","last_change":"2024-09-16T10:26:18.549126+0000","last_active":"2024-09-16T10:31:28.565185+0000","last_peered":"2024-09-16T10:31:28.565185+0000","last_clean":"2024-09-16T10:31:28.565185+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:31:28.565185+0000","last_undegraded":"2024-09-16T10:31:28.565185+0000","last_fullsized":"2024-09-16T10:31:28.565185+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:26:18.549005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:26:18.549005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.749930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":519,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568503+0000","last_change":"2024-09-16T10:31:22.862857+0000","last_active":"2024-09-16T10:31:28.568503+0000","last_peered":"2024-09-16T10:31:28.568503+0000","last_clean":"2024-09-16T10:31:28.568503+0000","last_became_active":"2024-09-16T10:31:14.142132+0000","last_became_peered":"2024-09-16T10:31:14.142132+0000","last_unstale":"2024-09-16T10:31:28.568503+0000","last_undegraded":"2024-09-16T10:31:28.568503+0000","last_fullsized":"2024-09-16T10:31:28.568503+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:22.862677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:22.862677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:23.084617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":373,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567662+0000","last_change":"2024-09-16T10:31:14.164501+0000","last_active":"2024-09-16T10:31:28.567662+0000","last_peered":"2024-09-16T10:31:28.567662+0000","last_clean":"2024-09-16T10:31:28.567662+0000","last_became_active":"2024-09-16T10:31:14.164133+0000","last_became_peered":"2024-09-16T10:31:14.164133+0000","last_unstale":"2024-09-16T10:31:28.567662+0000","last_undegraded":"2024-09-16T10:31:28.567662+0000","last_fullsized":"2024-09-16T10:31:28.567662+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.706111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.706111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:09.595073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"513'9069","reported_seq":9155,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.300905+0000","last_change":"2024-09-16T10:29:35.171912+0000","last_active":"2024-09-16T10:31:33.300905+0000","last_peered":"2024-09-16T10:31:33.300905+0000","last_clean":"2024-09-16T10:31:33.300905+0000","last_became_active":"2024-09-16T10:29:35.171449+0000","last_became_peered":"2024-09-16T10:29:35.171449+0000","last_unstale":"2024-09-16T10:31:33.300905+0000","last_undegraded":"2024-09-16T10:31:33.300905+0000","last_fullsized":"2024-09-16T10:31:33.300905+0000","mapping_epoch":476,"log_start":"496'5172","ondisk_log_start":"496'5172","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418185216,"num_objects":6381,"num_object_clones":0,"num_object_copies":19143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9069,"num_write_kb":494400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":853,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.581248+0000","last_change":"2024-09-16T10:30:16.866299+0000","last_active":"2024-09-16T10:31:28.581248+0000","last_peered":"2024-09-16T10:31:28.581248+0000","last_clean":"2024-09-16T10:31:28.581248+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:31:28.581248+0000","last_undegraded":"2024-09-16T10:31:28.581248+0000","last_fullsized":"2024-09-16T10:31:28.581248+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:16.862456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.862456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:41.111359+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":1065,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575390+0000","last_change":"2024-09-16T10:30:20.268496+0000","last_active":"2024-09-16T10:31:28.575390+0000","last_peered":"2024-09-16T10:31:28.575390+0000","last_clean":"2024-09-16T10:31:28.575390+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:31:28.575390+0000","last_undegraded":"2024-09-16T10:31:28.575390+0000","last_fullsized":"2024-09-16T10:31:28.575390+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:30:20.268438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.268438+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:49.275236+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":991,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572202+0000","last_change":"2024-09-16T10:30:15.229737+0000","last_active":"2024-09-16T10:31:28.572202+0000","last_peered":"2024-09-16T10:31:28.572202+0000","last_clean":"2024-09-16T10:31:28.572202+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:31:28.572202+0000","last_undegraded":"2024-09-16T10:31:28.572202+0000","last_fullsized":"2024-09-16T10:31:28.572202+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.229669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.229669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:33.889521+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":809,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.231117+0000","last_change":"2024-09-16T10:31:14.176415+0000","last_active":"2024-09-16T10:31:32.231117+0000","last_peered":"2024-09-16T10:31:32.231117+0000","last_clean":"2024-09-16T10:31:32.231117+0000","last_became_active":"2024-09-16T10:31:14.148770+0000","last_became_peered":"2024-09-16T10:31:14.148770+0000","last_unstale":"2024-09-16T10:31:32.231117+0000","last_undegraded":"2024-09-16T10:31:32.231117+0000","last_fullsized":"2024-09-16T10:31:32.231117+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:30:39.226399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.226399+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:46.941234+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":390,"num_read_kb":385,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":976,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569875+0000","last_change":"2024-09-16T10:30:08.854974+0000","last_active":"2024-09-16T10:31:28.569875+0000","last_peered":"2024-09-16T10:31:28.569875+0000","last_clean":"2024-09-16T10:31:28.569875+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:31:28.569875+0000","last_undegraded":"2024-09-16T10:31:28.569875+0000","last_fullsized":"2024-09-16T10:31:28.569875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.854924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.854924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":972,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.579053+0000","last_change":"2024-09-16T10:27:19.374485+0000","last_active":"2024-09-16T10:31:28.579053+0000","last_peered":"2024-09-16T10:31:28.579053+0000","last_clean":"2024-09-16T10:31:28.579053+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:31:28.579053+0000","last_undegraded":"2024-09-16T10:31:28.579053+0000","last_fullsized":"2024-09-16T10:31:28.579053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"513'9152","reported_seq":9238,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.290800+0000","last_change":"2024-09-16T10:29:35.169178+0000","last_active":"2024-09-16T10:31:33.290800+0000","last_peered":"2024-09-16T10:31:33.290800+0000","last_clean":"2024-09-16T10:31:33.290800+0000","last_became_active":"2024-09-16T10:29:35.168851+0000","last_became_peered":"2024-09-16T10:29:35.168851+0000","last_unstale":"2024-09-16T10:31:33.290800+0000","last_undegraded":"2024-09-16T10:31:33.290800+0000","last_fullsized":"2024-09-16T10:31:33.290800+0000","mapping_epoch":476,"log_start":"496'5214","ondisk_log_start":"496'5214","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417202176,"num_objects":6366,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":496576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"508'31","reported_seq":1248,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567745+0000","last_change":"2024-09-16T10:31:14.178826+0000","last_active":"2024-09-16T10:31:28.567745+0000","last_peered":"2024-09-16T10:31:28.567745+0000","last_clean":"2024-09-16T10:31:28.567745+0000","last_became_active":"2024-09-16T10:31:14.137441+0000","last_became_peered":"2024-09-16T10:31:14.137441+0000","last_unstale":"2024-09-16T10:31:28.567745+0000","last_undegraded":"2024-09-16T10:31:28.567745+0000","last_fullsized":"2024-09-16T10:31:28.567745+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:30:54.737043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.737043+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:11.390039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":931,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575210+0000","last_change":"2024-09-16T10:26:47.493915+0000","last_active":"2024-09-16T10:31:28.575210+0000","last_peered":"2024-09-16T10:31:28.575210+0000","last_clean":"2024-09-16T10:31:28.575210+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:31:28.575210+0000","last_undegraded":"2024-09-16T10:31:28.575210+0000","last_fullsized":"2024-09-16T10:31:28.575210+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:47.493848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:47.493848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":458,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572372+0000","last_change":"2024-09-16T10:31:17.118674+0000","last_active":"2024-09-16T10:31:28.572372+0000","last_peered":"2024-09-16T10:31:28.572372+0000","last_clean":"2024-09-16T10:31:28.572372+0000","last_became_active":"2024-09-16T10:31:17.118308+0000","last_became_peered":"2024-09-16T10:31:17.118308+0000","last_unstale":"2024-09-16T10:31:28.572372+0000","last_undegraded":"2024-09-16T10:31:28.572372+0000","last_fullsized":"2024-09-16T10:31:28.572372+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:50.410561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.410561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:12.776555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":844,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567986+0000","last_change":"2024-09-16T10:31:14.146544+0000","last_active":"2024-09-16T10:31:28.567986+0000","last_peered":"2024-09-16T10:31:28.567986+0000","last_clean":"2024-09-16T10:31:28.567986+0000","last_became_active":"2024-09-16T10:31:14.145333+0000","last_became_peered":"2024-09-16T10:31:14.145333+0000","last_unstale":"2024-09-16T10:31:28.567986+0000","last_undegraded":"2024-09-16T10:31:28.567986+0000","last_fullsized":"2024-09-16T10:31:28.567986+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.613780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.613780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:01.567578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":964,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569772+0000","last_change":"2024-09-16T10:30:43.705706+0000","last_active":"2024-09-16T10:31:28.569772+0000","last_peered":"2024-09-16T10:31:28.569772+0000","last_clean":"2024-09-16T10:31:28.569772+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:31:28.569772+0000","last_undegraded":"2024-09-16T10:31:28.569772+0000","last_fullsized":"2024-09-16T10:31:28.569772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.705640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.705640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:44.810616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"513'8791","reported_seq":8943,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.620969+0000","last_change":"2024-09-16T10:30:40.160552+0000","last_active":"2024-09-16T10:31:28.620969+0000","last_peered":"2024-09-16T10:31:28.620969+0000","last_clean":"2024-09-16T10:31:28.620969+0000","last_became_active":"2024-09-16T10:29:35.170866+0000","last_became_peered":"2024-09-16T10:29:35.170866+0000","last_unstale":"2024-09-16T10:31:28.620969+0000","last_undegraded":"2024-09-16T10:31:28.620969+0000","last_fullsized":"2024-09-16T10:31:28.620969+0000","mapping_epoch":476,"log_start":"492'4434","ondisk_log_start":"492'4434","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'5743","last_scrub_stamp":"2024-09-16T10:30:40.160478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.160478+0000","objects_scrubbed":5687,"log_size":4357,"log_dups_size":0,"ondisk_log_size":4357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.130078+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444661760,"num_objects":6785,"num_object_clones":0,"num_object_copies":20355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":498432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":852,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.579715+0000","last_change":"2024-09-16T10:29:27.703960+0000","last_active":"2024-09-16T10:31:28.579715+0000","last_peered":"2024-09-16T10:31:28.579715+0000","last_clean":"2024-09-16T10:31:28.579715+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:31:28.579715+0000","last_undegraded":"2024-09-16T10:31:28.579715+0000","last_fullsized":"2024-09-16T10:31:28.579715+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:27.703908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.703908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":896,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566477+0000","last_change":"2024-09-16T10:26:28.556079+0000","last_active":"2024-09-16T10:31:28.566477+0000","last_peered":"2024-09-16T10:31:28.566477+0000","last_clean":"2024-09-16T10:31:28.566477+0000","last_became_active":"2024-09-16T10:26:28.554604+0000","last_became_peered":"2024-09-16T10:26:28.554604+0000","last_unstale":"2024-09-16T10:31:28.566477+0000","last_undegraded":"2024-09-16T10:31:28.566477+0000","last_fullsized":"2024-09-16T10:31:28.566477+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:26:10.386128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.386128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.928740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":780,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.564882+0000","last_change":"2024-09-16T10:31:14.135558+0000","last_active":"2024-09-16T10:31:28.564882+0000","last_peered":"2024-09-16T10:31:28.564882+0000","last_clean":"2024-09-16T10:31:28.564882+0000","last_became_active":"2024-09-16T10:31:14.134560+0000","last_became_peered":"2024-09-16T10:31:14.134560+0000","last_unstale":"2024-09-16T10:31:28.564882+0000","last_undegraded":"2024-09-16T10:31:28.564882+0000","last_fullsized":"2024-09-16T10:31:28.564882+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.474733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.474733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":967,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569629+0000","last_change":"2024-09-16T10:29:30.955964+0000","last_active":"2024-09-16T10:31:28.569629+0000","last_peered":"2024-09-16T10:31:28.569629+0000","last_clean":"2024-09-16T10:31:28.569629+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:31:28.569629+0000","last_undegraded":"2024-09-16T10:31:28.569629+0000","last_fullsized":"2024-09-16T10:31:28.569629+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:30.955877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.955877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":902,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.577000+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:31:28.577000+0000","last_peered":"2024-09-16T10:31:28.577000+0000","last_clean":"2024-09-16T10:31:28.577000+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:31:28.577000+0000","last_undegraded":"2024-09-16T10:31:28.577000+0000","last_fullsized":"2024-09-16T10:31:28.577000+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"513'8841","reported_seq":8927,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.306021+0000","last_change":"2024-09-16T10:29:35.171033+0000","last_active":"2024-09-16T10:31:29.306021+0000","last_peered":"2024-09-16T10:31:29.306021+0000","last_clean":"2024-09-16T10:31:29.306021+0000","last_became_active":"2024-09-16T10:29:35.170730+0000","last_became_peered":"2024-09-16T10:29:35.170730+0000","last_unstale":"2024-09-16T10:31:29.306021+0000","last_undegraded":"2024-09-16T10:31:29.306021+0000","last_fullsized":"2024-09-16T10:31:29.306021+0000","mapping_epoch":476,"log_start":"490'3827","ondisk_log_start":"490'3827","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":19887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8841,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":828,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568500+0000","last_change":"2024-09-16T10:30:05.823698+0000","last_active":"2024-09-16T10:31:28.568500+0000","last_peered":"2024-09-16T10:31:28.568500+0000","last_clean":"2024-09-16T10:31:28.568500+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:31:28.568500+0000","last_undegraded":"2024-09-16T10:31:28.568500+0000","last_fullsized":"2024-09-16T10:31:28.568500+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:05.823644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.823644+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":968,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569138+0000","last_change":"2024-09-16T10:31:18.771352+0000","last_active":"2024-09-16T10:31:28.569138+0000","last_peered":"2024-09-16T10:31:28.569138+0000","last_clean":"2024-09-16T10:31:28.569138+0000","last_became_active":"2024-09-16T10:26:28.553171+0000","last_became_peered":"2024-09-16T10:26:28.553171+0000","last_unstale":"2024-09-16T10:31:28.569138+0000","last_undegraded":"2024-09-16T10:31:28.569138+0000","last_fullsized":"2024-09-16T10:31:28.569138+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:31:18.771268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:31:18.771268+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:26.557862+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":967,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566254+0000","last_change":"2024-09-16T10:30:03.609790+0000","last_active":"2024-09-16T10:31:28.566254+0000","last_peered":"2024-09-16T10:31:28.566254+0000","last_clean":"2024-09-16T10:31:28.566254+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:31:28.566254+0000","last_undegraded":"2024-09-16T10:31:28.566254+0000","last_fullsized":"2024-09-16T10:31:28.566254+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.609728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.609728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":951,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.579172+0000","last_change":"2024-09-16T10:29:21.720973+0000","last_active":"2024-09-16T10:31:28.579172+0000","last_peered":"2024-09-16T10:31:28.579172+0000","last_clean":"2024-09-16T10:31:28.579172+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:31:28.579172+0000","last_undegraded":"2024-09-16T10:31:28.579172+0000","last_fullsized":"2024-09-16T10:31:28.579172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.720910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.720910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.090647+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":501,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567281+0000","last_change":"2024-09-16T10:30:40.613125+0000","last_active":"2024-09-16T10:31:28.567281+0000","last_peered":"2024-09-16T10:31:28.567281+0000","last_clean":"2024-09-16T10:31:28.567281+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:31:28.567281+0000","last_undegraded":"2024-09-16T10:31:28.567281+0000","last_fullsized":"2024-09-16T10:31:28.567281+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:40.612878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.612878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:00.522715+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"513'9020","reported_seq":18427,"reported_epoch":513,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T10:31:28.658541+0000","last_change":"2024-09-16T10:31:15.249974+0000","last_active":"2024-09-16T10:31:28.658541+0000","last_peered":"2024-09-16T10:31:28.658541+0000","last_clean":"2024-09-16T10:31:13.106101+0000","last_became_active":"2024-09-16T10:31:15.213983+0000","last_became_peered":"2024-09-16T10:31:15.213983+0000","last_unstale":"2024-09-16T10:31:28.658541+0000","last_undegraded":"2024-09-16T10:31:28.658541+0000","last_fullsized":"2024-09-16T10:31:28.658541+0000","mapping_epoch":508,"log_start":"492'4615","ondisk_log_start":"492'4615","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'7025","last_scrub_stamp":"2024-09-16T10:30:57.490342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:30:57.490342+0000","objects_scrubbed":6969,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452329472,"num_objects":6902,"num_object_clones":0,"num_object_copies":20706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2555,"num_objects_unfound":0,"num_objects_dirty":6902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":509504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4616,"num_bytes_recovered":302383104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"1,6,7","objects":6902}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":647,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575961+0000","last_change":"2024-09-16T10:31:14.142529+0000","last_active":"2024-09-16T10:31:28.575961+0000","last_peered":"2024-09-16T10:31:28.575961+0000","last_clean":"2024-09-16T10:31:28.575961+0000","last_became_active":"2024-09-16T10:31:14.140156+0000","last_became_peered":"2024-09-16T10:31:14.140156+0000","last_unstale":"2024-09-16T10:31:28.575961+0000","last_undegraded":"2024-09-16T10:31:28.575961+0000","last_fullsized":"2024-09-16T10:31:28.575961+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:07.951845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.951845+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":876,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576655+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:31:28.576655+0000","last_peered":"2024-09-16T10:31:28.576655+0000","last_clean":"2024-09-16T10:31:28.576655+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:31:28.576655+0000","last_undegraded":"2024-09-16T10:31:28.576655+0000","last_fullsized":"2024-09-16T10:31:28.576655+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":950,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576439+0000","last_change":"2024-09-16T10:27:12.155297+0000","last_active":"2024-09-16T10:31:28.576439+0000","last_peered":"2024-09-16T10:31:28.576439+0000","last_clean":"2024-09-16T10:31:28.576439+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:31:28.576439+0000","last_undegraded":"2024-09-16T10:31:28.576439+0000","last_fullsized":"2024-09-16T10:31:28.576439+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:12.155186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.155186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"508'13","reported_seq":643,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.571848+0000","last_change":"2024-09-16T10:31:27.828520+0000","last_active":"2024-09-16T10:31:28.571848+0000","last_peered":"2024-09-16T10:31:28.571848+0000","last_clean":"2024-09-16T10:31:28.571848+0000","last_became_active":"2024-09-16T10:31:14.138957+0000","last_became_peered":"2024-09-16T10:31:14.138957+0000","last_unstale":"2024-09-16T10:31:28.571848+0000","last_undegraded":"2024-09-16T10:31:28.571848+0000","last_fullsized":"2024-09-16T10:31:28.571848+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'13","last_scrub_stamp":"2024-09-16T10:31:27.828439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:27.828439+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:43.708396+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":767,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.579636+0000","last_change":"2024-09-16T10:30:22.007418+0000","last_active":"2024-09-16T10:31:28.579636+0000","last_peered":"2024-09-16T10:31:28.579636+0000","last_clean":"2024-09-16T10:31:28.579636+0000","last_became_active":"2024-09-16T10:26:28.550959+0000","last_became_peered":"2024-09-16T10:26:28.550959+0000","last_unstale":"2024-09-16T10:31:28.579636+0000","last_undegraded":"2024-09-16T10:31:28.579636+0000","last_fullsized":"2024-09-16T10:31:28.579636+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:22.007341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:22.007341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:45.689451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"513'8647","reported_seq":8745,"reported_epoch":513,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:31:28.659333+0000","last_change":"2024-09-16T10:31:15.259363+0000","last_active":"2024-09-16T10:31:28.659333+0000","last_peered":"2024-09-16T10:31:28.659333+0000","last_clean":"2024-09-16T10:31:13.061686+0000","last_became_active":"2024-09-16T10:31:15.212576+0000","last_became_peered":"2024-09-16T10:31:15.212576+0000","last_unstale":"2024-09-16T10:31:28.659333+0000","last_undegraded":"2024-09-16T10:31:28.659333+0000","last_fullsized":"2024-09-16T10:31:28.659333+0000","mapping_epoch":508,"log_start":"492'4251","ondisk_log_start":"492'4251","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431554560,"num_objects":6585,"num_object_clones":0,"num_object_copies":19755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6585,"num_objects_unfound":0,"num_objects_dirty":6585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":487424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[{"shards":"2,6,7","objects":6585}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":229,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575169+0000","last_change":"2024-09-16T10:31:18.127633+0000","last_active":"2024-09-16T10:31:28.575169+0000","last_peered":"2024-09-16T10:31:28.575169+0000","last_clean":"2024-09-16T10:31:28.575169+0000","last_became_active":"2024-09-16T10:31:17.119269+0000","last_became_peered":"2024-09-16T10:31:17.119269+0000","last_unstale":"2024-09-16T10:31:28.575169+0000","last_undegraded":"2024-09-16T10:31:28.575169+0000","last_fullsized":"2024-09-16T10:31:28.575169+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.900551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.900551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:31:47.148100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":987,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578096+0000","last_change":"2024-09-16T10:30:19.209404+0000","last_active":"2024-09-16T10:31:28.578096+0000","last_peered":"2024-09-16T10:31:28.578096+0000","last_clean":"2024-09-16T10:31:28.578096+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:31:28.578096+0000","last_undegraded":"2024-09-16T10:31:28.578096+0000","last_fullsized":"2024-09-16T10:31:28.578096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:30:19.209335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:19.209335+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":975,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569517+0000","last_change":"2024-09-16T10:26:44.070285+0000","last_active":"2024-09-16T10:31:28.569517+0000","last_peered":"2024-09-16T10:31:28.569517+0000","last_clean":"2024-09-16T10:31:28.569517+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:31:28.569517+0000","last_undegraded":"2024-09-16T10:31:28.569517+0000","last_fullsized":"2024-09-16T10:31:28.569517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:44.070218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.070218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.953782+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":493,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567544+0000","last_change":"2024-09-16T10:26:22.501028+0000","last_active":"2024-09-16T10:31:28.567544+0000","last_peered":"2024-09-16T10:31:28.567544+0000","last_clean":"2024-09-16T10:31:28.567544+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:31:28.567544+0000","last_undegraded":"2024-09-16T10:31:28.567544+0000","last_fullsized":"2024-09-16T10:31:28.567544+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.500963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.500963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:09.764402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":1014,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572092+0000","last_change":"2024-09-16T10:31:14.161646+0000","last_active":"2024-09-16T10:31:28.572092+0000","last_peered":"2024-09-16T10:31:28.572092+0000","last_clean":"2024-09-16T10:31:28.572092+0000","last_became_active":"2024-09-16T10:31:14.152023+0000","last_became_peered":"2024-09-16T10:31:14.152023+0000","last_unstale":"2024-09-16T10:31:28.572092+0000","last_undegraded":"2024-09-16T10:31:28.572092+0000","last_fullsized":"2024-09-16T10:31:28.572092+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:05.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.174161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:32.660246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"513'9054","reported_seq":14022,"reported_epoch":513,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T10:31:33.042830+0000","last_change":"2024-09-16T10:31:15.234664+0000","last_active":"2024-09-16T10:31:33.042830+0000","last_peered":"2024-09-16T10:31:33.042830+0000","last_clean":"2024-09-16T10:31:13.067588+0000","last_became_active":"2024-09-16T10:31:15.215811+0000","last_became_peered":"2024-09-16T10:31:15.215811+0000","last_unstale":"2024-09-16T10:31:33.042830+0000","last_undegraded":"2024-09-16T10:31:33.042830+0000","last_fullsized":"2024-09-16T10:31:33.042830+0000","mapping_epoch":508,"log_start":"494'4799","ondisk_log_start":"494'4799","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4255,"log_dups_size":0,"ondisk_log_size":4255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420741120,"num_objects":6420,"num_object_clones":0,"num_object_copies":19260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4162,"num_objects_unfound":0,"num_objects_dirty":6420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":495168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2438,"num_bytes_recovered":159776768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":6420}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":931,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566810+0000","last_change":"2024-09-16T10:30:42.550612+0000","last_active":"2024-09-16T10:31:28.566810+0000","last_peered":"2024-09-16T10:31:28.566810+0000","last_clean":"2024-09-16T10:31:28.566810+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:31:28.566810+0000","last_undegraded":"2024-09-16T10:31:28.566810+0000","last_fullsized":"2024-09-16T10:31:28.566810+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:42.550567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.550567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:56.031040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":1064,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566006+0000","last_change":"2024-09-16T10:31:14.185001+0000","last_active":"2024-09-16T10:31:28.566006+0000","last_peered":"2024-09-16T10:31:28.566006+0000","last_clean":"2024-09-16T10:31:28.566006+0000","last_became_active":"2024-09-16T10:31:14.141161+0000","last_became_peered":"2024-09-16T10:31:14.141161+0000","last_unstale":"2024-09-16T10:31:28.566006+0000","last_undegraded":"2024-09-16T10:31:28.566006+0000","last_fullsized":"2024-09-16T10:31:28.566006+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":371,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569565+0000","last_change":"2024-09-16T10:31:14.148058+0000","last_active":"2024-09-16T10:31:28.569565+0000","last_peered":"2024-09-16T10:31:28.569565+0000","last_clean":"2024-09-16T10:31:28.569565+0000","last_became_active":"2024-09-16T10:31:14.147606+0000","last_became_peered":"2024-09-16T10:31:14.147606+0000","last_unstale":"2024-09-16T10:31:28.569565+0000","last_undegraded":"2024-09-16T10:31:28.569565+0000","last_fullsized":"2024-09-16T10:31:28.569565+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.745214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.745214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:51.075041+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":493,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566873+0000","last_change":"2024-09-16T10:26:43.199824+0000","last_active":"2024-09-16T10:31:28.566873+0000","last_peered":"2024-09-16T10:31:28.566873+0000","last_clean":"2024-09-16T10:31:28.566873+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:31:28.566873+0000","last_undegraded":"2024-09-16T10:31:28.566873+0000","last_fullsized":"2024-09-16T10:31:28.566873+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:43.199752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:43.199752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:17.774662+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":847,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.564575+0000","last_change":"2024-09-16T10:31:14.148903+0000","last_active":"2024-09-16T10:31:28.564575+0000","last_peered":"2024-09-16T10:31:28.564575+0000","last_clean":"2024-09-16T10:31:28.564575+0000","last_became_active":"2024-09-16T10:31:14.148572+0000","last_became_peered":"2024-09-16T10:31:14.148572+0000","last_unstale":"2024-09-16T10:31:28.564575+0000","last_undegraded":"2024-09-16T10:31:28.564575+0000","last_fullsized":"2024-09-16T10:31:28.564575+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.626720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.626720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:15.346115+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"513'9011","reported_seq":9096,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:30.089742+0000","last_change":"2024-09-16T10:29:35.172946+0000","last_active":"2024-09-16T10:31:30.089742+0000","last_peered":"2024-09-16T10:31:30.089742+0000","last_clean":"2024-09-16T10:31:30.089742+0000","last_became_active":"2024-09-16T10:29:35.172649+0000","last_became_peered":"2024-09-16T10:29:35.172649+0000","last_unstale":"2024-09-16T10:31:30.089742+0000","last_undegraded":"2024-09-16T10:31:30.089742+0000","last_fullsized":"2024-09-16T10:31:30.089742+0000","mapping_epoch":476,"log_start":"494'4723","ondisk_log_start":"494'4723","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:04.288167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438632448,"num_objects":6693,"num_object_clones":0,"num_object_copies":20079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9011,"num_write_kb":502528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":874,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576381+0000","last_change":"2024-09-16T10:26:10.271093+0000","last_active":"2024-09-16T10:31:28.576381+0000","last_peered":"2024-09-16T10:31:28.576381+0000","last_clean":"2024-09-16T10:31:28.576381+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:31:28.576381+0000","last_undegraded":"2024-09-16T10:31:28.576381+0000","last_fullsized":"2024-09-16T10:31:28.576381+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:10.270833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.270833+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:04.211104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":892,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.577989+0000","last_change":"2024-09-16T10:30:20.961973+0000","last_active":"2024-09-16T10:31:28.577989+0000","last_peered":"2024-09-16T10:31:28.577989+0000","last_clean":"2024-09-16T10:31:28.577989+0000","last_became_active":"2024-09-16T10:26:28.553750+0000","last_became_peered":"2024-09-16T10:26:28.553750+0000","last_unstale":"2024-09-16T10:31:28.577989+0000","last_undegraded":"2024-09-16T10:31:28.577989+0000","last_fullsized":"2024-09-16T10:31:28.577989+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:30:20.961894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.961894+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:47.242426+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569301+0000","last_change":"2024-09-16T10:31:14.141250+0000","last_active":"2024-09-16T10:31:28.569301+0000","last_peered":"2024-09-16T10:31:28.569301+0000","last_clean":"2024-09-16T10:31:28.569301+0000","last_became_active":"2024-09-16T10:31:14.140960+0000","last_became_peered":"2024-09-16T10:31:14.140960+0000","last_unstale":"2024-09-16T10:31:28.569301+0000","last_undegraded":"2024-09-16T10:31:28.569301+0000","last_fullsized":"2024-09-16T10:31:28.569301+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":501,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.564673+0000","last_change":"2024-09-16T10:30:52.624617+0000","last_active":"2024-09-16T10:31:28.564673+0000","last_peered":"2024-09-16T10:31:28.564673+0000","last_clean":"2024-09-16T10:31:28.564673+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:31:28.564673+0000","last_undegraded":"2024-09-16T10:31:28.564673+0000","last_fullsized":"2024-09-16T10:31:28.564673+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.624569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.624569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:59.763599+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":979,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575958+0000","last_change":"2024-09-16T10:31:14.149294+0000","last_active":"2024-09-16T10:31:28.575958+0000","last_peered":"2024-09-16T10:31:28.575958+0000","last_clean":"2024-09-16T10:31:28.575958+0000","last_became_active":"2024-09-16T10:31:14.148667+0000","last_became_peered":"2024-09-16T10:31:14.148667+0000","last_unstale":"2024-09-16T10:31:28.575958+0000","last_undegraded":"2024-09-16T10:31:28.575958+0000","last_fullsized":"2024-09-16T10:31:28.575958+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.230039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.230039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:14.325776+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"513'9018","reported_seq":9104,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.018212+0000","last_change":"2024-09-16T10:29:35.165138+0000","last_active":"2024-09-16T10:31:33.018212+0000","last_peered":"2024-09-16T10:31:33.018212+0000","last_clean":"2024-09-16T10:31:33.018212+0000","last_became_active":"2024-09-16T10:29:35.164843+0000","last_became_peered":"2024-09-16T10:29:35.164843+0000","last_unstale":"2024-09-16T10:31:33.018212+0000","last_undegraded":"2024-09-16T10:31:33.018212+0000","last_fullsized":"2024-09-16T10:31:33.018212+0000","mapping_epoch":476,"log_start":"494'4781","ondisk_log_start":"494'4781","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419692544,"num_objects":6404,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":701,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.580277+0000","last_change":"2024-09-16T10:31:17.114223+0000","last_active":"2024-09-16T10:31:28.580277+0000","last_peered":"2024-09-16T10:31:28.580277+0000","last_clean":"2024-09-16T10:31:28.580277+0000","last_became_active":"2024-09-16T10:31:17.113692+0000","last_became_peered":"2024-09-16T10:31:17.113692+0000","last_unstale":"2024-09-16T10:31:28.580277+0000","last_undegraded":"2024-09-16T10:31:28.580277+0000","last_fullsized":"2024-09-16T10:31:28.580277+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":1033,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568759+0000","last_change":"2024-09-16T10:31:14.162215+0000","last_active":"2024-09-16T10:31:28.568759+0000","last_peered":"2024-09-16T10:31:28.568759+0000","last_clean":"2024-09-16T10:31:28.568759+0000","last_became_active":"2024-09-16T10:31:14.136662+0000","last_became_peered":"2024-09-16T10:31:14.136662+0000","last_unstale":"2024-09-16T10:31:28.568759+0000","last_undegraded":"2024-09-16T10:31:28.568759+0000","last_fullsized":"2024-09-16T10:31:28.568759+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.825936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.825936+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:17.528711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":309,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575867+0000","last_change":"2024-09-16T10:31:14.162556+0000","last_active":"2024-09-16T10:31:28.575867+0000","last_peered":"2024-09-16T10:31:28.575867+0000","last_clean":"2024-09-16T10:31:28.575867+0000","last_became_active":"2024-09-16T10:31:14.149571+0000","last_became_peered":"2024-09-16T10:31:14.149571+0000","last_unstale":"2024-09-16T10:31:28.575867+0000","last_undegraded":"2024-09-16T10:31:28.575867+0000","last_fullsized":"2024-09-16T10:31:28.575867+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:08.381010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.381010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"508'12","reported_seq":481,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566866+0000","last_change":"2024-09-16T10:31:14.166453+0000","last_active":"2024-09-16T10:31:28.566866+0000","last_peered":"2024-09-16T10:31:28.566866+0000","last_clean":"2024-09-16T10:31:28.566866+0000","last_became_active":"2024-09-16T10:31:14.131452+0000","last_became_peered":"2024-09-16T10:31:14.131452+0000","last_unstale":"2024-09-16T10:31:28.566866+0000","last_undegraded":"2024-09-16T10:31:28.566866+0000","last_fullsized":"2024-09-16T10:31:28.566866+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'11","last_scrub_stamp":"2024-09-16T10:30:53.694883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.694883+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:02.163955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":972,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568997+0000","last_change":"2024-09-16T10:30:07.792293+0000","last_active":"2024-09-16T10:31:28.568997+0000","last_peered":"2024-09-16T10:31:28.568997+0000","last_clean":"2024-09-16T10:31:28.568997+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:31:28.568997+0000","last_undegraded":"2024-09-16T10:31:28.568997+0000","last_fullsized":"2024-09-16T10:31:28.568997+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.792060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.792060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:34.132214+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":539,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569198+0000","last_change":"2024-09-16T10:29:28.935051+0000","last_active":"2024-09-16T10:31:28.569198+0000","last_peered":"2024-09-16T10:31:28.569198+0000","last_clean":"2024-09-16T10:31:28.569198+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:31:28.569198+0000","last_undegraded":"2024-09-16T10:31:28.569198+0000","last_fullsized":"2024-09-16T10:31:28.569198+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.935001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.935001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":950,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567236+0000","last_change":"2024-09-16T10:26:20.469286+0000","last_active":"2024-09-16T10:31:28.567236+0000","last_peered":"2024-09-16T10:31:28.567236+0000","last_clean":"2024-09-16T10:31:28.567236+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:31:28.567236+0000","last_undegraded":"2024-09-16T10:31:28.567236+0000","last_fullsized":"2024-09-16T10:31:28.567236+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:20.469240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.469240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:03.848934+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":1005,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576115+0000","last_change":"2024-09-16T10:31:14.188538+0000","last_active":"2024-09-16T10:31:28.576115+0000","last_peered":"2024-09-16T10:31:28.576115+0000","last_clean":"2024-09-16T10:31:28.576115+0000","last_became_active":"2024-09-16T10:31:14.154165+0000","last_became_peered":"2024-09-16T10:31:14.154165+0000","last_unstale":"2024-09-16T10:31:28.576115+0000","last_undegraded":"2024-09-16T10:31:28.576115+0000","last_fullsized":"2024-09-16T10:31:28.576115+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:30:50.226333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.226333+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:57.518866+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":951,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569619+0000","last_change":"2024-09-16T10:26:46.061085+0000","last_active":"2024-09-16T10:31:28.569619+0000","last_peered":"2024-09-16T10:31:28.569619+0000","last_clean":"2024-09-16T10:31:28.569619+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:31:28.569619+0000","last_undegraded":"2024-09-16T10:31:28.569619+0000","last_fullsized":"2024-09-16T10:31:28.569619+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:58.797381+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":648,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567603+0000","last_change":"2024-09-16T10:31:14.137702+0000","last_active":"2024-09-16T10:31:28.567603+0000","last_peered":"2024-09-16T10:31:28.567603+0000","last_clean":"2024-09-16T10:31:28.567603+0000","last_became_active":"2024-09-16T10:31:14.136703+0000","last_became_peered":"2024-09-16T10:31:14.136703+0000","last_unstale":"2024-09-16T10:31:28.567603+0000","last_undegraded":"2024-09-16T10:31:28.567603+0000","last_fullsized":"2024-09-16T10:31:28.567603+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:39.865342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.865342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:47.881133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":861,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.581327+0000","last_change":"2024-09-16T10:30:48.209120+0000","last_active":"2024-09-16T10:31:28.581327+0000","last_peered":"2024-09-16T10:31:28.581327+0000","last_clean":"2024-09-16T10:31:28.581327+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:31:28.581327+0000","last_undegraded":"2024-09-16T10:31:28.581327+0000","last_fullsized":"2024-09-16T10:31:28.581327+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:48.209053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.209053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.123048+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":1015,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575989+0000","last_change":"2024-09-16T10:31:14.165639+0000","last_active":"2024-09-16T10:31:28.575989+0000","last_peered":"2024-09-16T10:31:28.575989+0000","last_clean":"2024-09-16T10:31:28.575989+0000","last_became_active":"2024-09-16T10:31:14.151744+0000","last_became_peered":"2024-09-16T10:31:14.151744+0000","last_unstale":"2024-09-16T10:31:28.575989+0000","last_undegraded":"2024-09-16T10:31:28.575989+0000","last_fullsized":"2024-09-16T10:31:28.575989+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:07.157828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.157828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":311,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567405+0000","last_change":"2024-09-16T10:31:14.137225+0000","last_active":"2024-09-16T10:31:28.567405+0000","last_peered":"2024-09-16T10:31:28.567405+0000","last_clean":"2024-09-16T10:31:28.567405+0000","last_became_active":"2024-09-16T10:31:14.132504+0000","last_became_peered":"2024-09-16T10:31:14.132504+0000","last_unstale":"2024-09-16T10:31:28.567405+0000","last_undegraded":"2024-09-16T10:31:28.567405+0000","last_fullsized":"2024-09-16T10:31:28.567405+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:44.598219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:44.598219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.031728+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":975,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575915+0000","last_change":"2024-09-16T10:31:14.154512+0000","last_active":"2024-09-16T10:31:28.575915+0000","last_peered":"2024-09-16T10:31:28.575915+0000","last_clean":"2024-09-16T10:31:28.575915+0000","last_became_active":"2024-09-16T10:31:14.153698+0000","last_became_peered":"2024-09-16T10:31:14.153698+0000","last_unstale":"2024-09-16T10:31:28.575915+0000","last_undegraded":"2024-09-16T10:31:28.575915+0000","last_fullsized":"2024-09-16T10:31:28.575915+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:20.058297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.058297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":511,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.568641+0000","last_change":"2024-09-16T10:31:14.140684+0000","last_active":"2024-09-16T10:31:28.568641+0000","last_peered":"2024-09-16T10:31:28.568641+0000","last_clean":"2024-09-16T10:31:28.568641+0000","last_became_active":"2024-09-16T10:31:14.140004+0000","last_became_peered":"2024-09-16T10:31:14.140004+0000","last_unstale":"2024-09-16T10:31:28.568641+0000","last_undegraded":"2024-09-16T10:31:28.568641+0000","last_fullsized":"2024-09-16T10:31:28.568641+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.541001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.541001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":548,"reported_epoch":513,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:31:28.568475+0000","last_change":"2024-09-16T10:31:15.129956+0000","last_active":"2024-09-16T10:31:28.568475+0000","last_peered":"2024-09-16T10:31:28.568475+0000","last_clean":"2024-09-16T10:31:04.129384+0000","last_became_active":"2024-09-16T10:31:15.129229+0000","last_became_peered":"2024-09-16T10:31:15.129229+0000","last_unstale":"2024-09-16T10:31:28.568475+0000","last_undegraded":"2024-09-16T10:31:28.568475+0000","last_fullsized":"2024-09-16T10:31:28.568475+0000","mapping_epoch":508,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,7],"avail_no_missing":["5","4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":1000,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569716+0000","last_change":"2024-09-16T10:31:14.174635+0000","last_active":"2024-09-16T10:31:28.569716+0000","last_peered":"2024-09-16T10:31:28.569716+0000","last_clean":"2024-09-16T10:31:28.569716+0000","last_became_active":"2024-09-16T10:31:14.149019+0000","last_became_peered":"2024-09-16T10:31:14.149019+0000","last_unstale":"2024-09-16T10:31:28.569716+0000","last_undegraded":"2024-09-16T10:31:28.569716+0000","last_fullsized":"2024-09-16T10:31:28.569716+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:29:20.061184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.061184+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":984,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.573635+0000","last_change":"2024-09-16T10:31:14.154526+0000","last_active":"2024-09-16T10:31:28.573635+0000","last_peered":"2024-09-16T10:31:28.573635+0000","last_clean":"2024-09-16T10:31:28.573635+0000","last_became_active":"2024-09-16T10:31:14.154145+0000","last_became_peered":"2024-09-16T10:31:14.154145+0000","last_unstale":"2024-09-16T10:31:28.573635+0000","last_undegraded":"2024-09-16T10:31:28.573635+0000","last_fullsized":"2024-09-16T10:31:28.573635+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:38.060395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:38.060395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"508'13","reported_seq":1243,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576958+0000","last_change":"2024-09-16T10:31:14.168449+0000","last_active":"2024-09-16T10:31:28.576958+0000","last_peered":"2024-09-16T10:31:28.576958+0000","last_clean":"2024-09-16T10:31:28.576958+0000","last_became_active":"2024-09-16T10:31:14.150871+0000","last_became_peered":"2024-09-16T10:31:14.150871+0000","last_unstale":"2024-09-16T10:31:28.576958+0000","last_undegraded":"2024-09-16T10:31:28.576958+0000","last_fullsized":"2024-09-16T10:31:28.576958+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:30:42.170419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.170419+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:54.536874+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":758,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567137+0000","last_change":"2024-09-16T10:26:45.194577+0000","last_active":"2024-09-16T10:31:28.567137+0000","last_peered":"2024-09-16T10:31:28.567137+0000","last_clean":"2024-09-16T10:31:28.567137+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:31:28.567137+0000","last_undegraded":"2024-09-16T10:31:28.567137+0000","last_fullsized":"2024-09-16T10:31:28.567137+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.194454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.194454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:10.783089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":536,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572297+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:31:28.572297+0000","last_peered":"2024-09-16T10:31:28.572297+0000","last_clean":"2024-09-16T10:31:28.572297+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:31:28.572297+0000","last_undegraded":"2024-09-16T10:31:28.572297+0000","last_fullsized":"2024-09-16T10:31:28.572297+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:03.251453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":1029,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576566+0000","last_change":"2024-09-16T10:31:14.173911+0000","last_active":"2024-09-16T10:31:28.576566+0000","last_peered":"2024-09-16T10:31:28.576566+0000","last_clean":"2024-09-16T10:31:28.576566+0000","last_became_active":"2024-09-16T10:31:14.147045+0000","last_became_peered":"2024-09-16T10:31:14.147045+0000","last_unstale":"2024-09-16T10:31:28.576566+0000","last_undegraded":"2024-09-16T10:31:28.576566+0000","last_fullsized":"2024-09-16T10:31:28.576566+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:26:20.405507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.405507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":728,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566780+0000","last_change":"2024-09-16T10:31:14.132986+0000","last_active":"2024-09-16T10:31:28.566780+0000","last_peered":"2024-09-16T10:31:28.566780+0000","last_clean":"2024-09-16T10:31:28.566780+0000","last_became_active":"2024-09-16T10:31:14.132063+0000","last_became_peered":"2024-09-16T10:31:14.132063+0000","last_unstale":"2024-09-16T10:31:28.566780+0000","last_undegraded":"2024-09-16T10:31:28.566780+0000","last_fullsized":"2024-09-16T10:31:28.566780+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.812585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.812585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1191,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569356+0000","last_change":"2024-09-16T10:30:56.801060+0000","last_active":"2024-09-16T10:31:28.569356+0000","last_peered":"2024-09-16T10:31:28.569356+0000","last_clean":"2024-09-16T10:31:28.569356+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:31:28.569356+0000","last_undegraded":"2024-09-16T10:31:28.569356+0000","last_fullsized":"2024-09-16T10:31:28.569356+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:56.800991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:56.800991+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:58.275908+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":964,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572226+0000","last_change":"2024-09-16T10:31:14.150959+0000","last_active":"2024-09-16T10:31:28.572226+0000","last_peered":"2024-09-16T10:31:28.572226+0000","last_clean":"2024-09-16T10:31:28.572226+0000","last_became_active":"2024-09-16T10:31:14.150453+0000","last_became_peered":"2024-09-16T10:31:14.150453+0000","last_unstale":"2024-09-16T10:31:28.572226+0000","last_undegraded":"2024-09-16T10:31:28.572226+0000","last_fullsized":"2024-09-16T10:31:28.572226+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.053102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.053102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":896,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567124+0000","last_change":"2024-09-16T10:28:02.318109+0000","last_active":"2024-09-16T10:31:28.567124+0000","last_peered":"2024-09-16T10:31:28.567124+0000","last_clean":"2024-09-16T10:31:28.567124+0000","last_became_active":"2024-09-16T10:26:28.557003+0000","last_became_peered":"2024-09-16T10:26:28.557003+0000","last_unstale":"2024-09-16T10:31:28.567124+0000","last_undegraded":"2024-09-16T10:31:28.567124+0000","last_fullsized":"2024-09-16T10:31:28.567124+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:02.318062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:02.318062+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":435,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578872+0000","last_change":"2024-09-16T10:31:14.147632+0000","last_active":"2024-09-16T10:31:28.578872+0000","last_peered":"2024-09-16T10:31:28.578872+0000","last_clean":"2024-09-16T10:31:28.578872+0000","last_became_active":"2024-09-16T10:31:14.144517+0000","last_became_peered":"2024-09-16T10:31:14.144517+0000","last_unstale":"2024-09-16T10:31:28.578872+0000","last_undegraded":"2024-09-16T10:31:28.578872+0000","last_fullsized":"2024-09-16T10:31:28.578872+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.216239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.216239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:57.607858+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":951,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.581132+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:31:28.581132+0000","last_peered":"2024-09-16T10:31:28.581132+0000","last_clean":"2024-09-16T10:31:28.581132+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:31:28.581132+0000","last_undegraded":"2024-09-16T10:31:28.581132+0000","last_fullsized":"2024-09-16T10:31:28.581132+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:00.130482+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":945,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575793+0000","last_change":"2024-09-16T10:26:14.413104+0000","last_active":"2024-09-16T10:31:28.575793+0000","last_peered":"2024-09-16T10:31:28.575793+0000","last_clean":"2024-09-16T10:31:28.575793+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:31:28.575793+0000","last_undegraded":"2024-09-16T10:31:28.575793+0000","last_fullsized":"2024-09-16T10:31:28.575793+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:14.413019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:14.413019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:10.254762+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":1014,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.581463+0000","last_change":"2024-09-16T10:29:19.698059+0000","last_active":"2024-09-16T10:31:28.581463+0000","last_peered":"2024-09-16T10:31:28.581463+0000","last_clean":"2024-09-16T10:31:28.581463+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:31:28.581463+0000","last_undegraded":"2024-09-16T10:31:28.581463+0000","last_fullsized":"2024-09-16T10:31:28.581463+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:35.234941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":370,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.578211+0000","last_change":"2024-09-16T10:31:14.144963+0000","last_active":"2024-09-16T10:31:28.578211+0000","last_peered":"2024-09-16T10:31:28.578211+0000","last_clean":"2024-09-16T10:31:28.578211+0000","last_became_active":"2024-09-16T10:31:14.144634+0000","last_became_peered":"2024-09-16T10:31:14.144634+0000","last_unstale":"2024-09-16T10:31:28.578211+0000","last_undegraded":"2024-09-16T10:31:28.578211+0000","last_fullsized":"2024-09-16T10:31:28.578211+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":914,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576054+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:31:28.576054+0000","last_peered":"2024-09-16T10:31:28.576054+0000","last_clean":"2024-09-16T10:31:28.576054+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:31:28.576054+0000","last_undegraded":"2024-09-16T10:31:28.576054+0000","last_fullsized":"2024-09-16T10:31:28.576054+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:59.273522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":938,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.571890+0000","last_change":"2024-09-16T10:30:52.389337+0000","last_active":"2024-09-16T10:31:28.571890+0000","last_peered":"2024-09-16T10:31:28.571890+0000","last_clean":"2024-09-16T10:31:28.571890+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:31:28.571890+0000","last_undegraded":"2024-09-16T10:31:28.571890+0000","last_fullsized":"2024-09-16T10:31:28.571890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.389262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.389262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.738887+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":893,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567772+0000","last_change":"2024-09-16T10:31:17.119468+0000","last_active":"2024-09-16T10:31:28.567772+0000","last_peered":"2024-09-16T10:31:28.567772+0000","last_clean":"2024-09-16T10:31:28.567772+0000","last_became_active":"2024-09-16T10:31:17.118844+0000","last_became_peered":"2024-09-16T10:31:17.118844+0000","last_unstale":"2024-09-16T10:31:28.567772+0000","last_undegraded":"2024-09-16T10:31:28.567772+0000","last_fullsized":"2024-09-16T10:31:28.567772+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:35.573835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.573835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":998,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576059+0000","last_change":"2024-09-16T10:31:14.175984+0000","last_active":"2024-09-16T10:31:28.576059+0000","last_peered":"2024-09-16T10:31:28.576059+0000","last_clean":"2024-09-16T10:31:28.576059+0000","last_became_active":"2024-09-16T10:31:14.154997+0000","last_became_peered":"2024-09-16T10:31:14.154997+0000","last_unstale":"2024-09-16T10:31:28.576059+0000","last_undegraded":"2024-09-16T10:31:28.576059+0000","last_fullsized":"2024-09-16T10:31:28.576059+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":985,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.569674+0000","last_change":"2024-09-16T10:31:14.147736+0000","last_active":"2024-09-16T10:31:28.569674+0000","last_peered":"2024-09-16T10:31:28.569674+0000","last_clean":"2024-09-16T10:31:28.569674+0000","last_became_active":"2024-09-16T10:31:14.147407+0000","last_became_peered":"2024-09-16T10:31:14.147407+0000","last_unstale":"2024-09-16T10:31:28.569674+0000","last_undegraded":"2024-09-16T10:31:28.569674+0000","last_fullsized":"2024-09-16T10:31:28.569674+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.002518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.002518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":652,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566741+0000","last_change":"2024-09-16T10:31:14.132749+0000","last_active":"2024-09-16T10:31:28.566741+0000","last_peered":"2024-09-16T10:31:28.566741+0000","last_clean":"2024-09-16T10:31:28.566741+0000","last_became_active":"2024-09-16T10:31:14.132459+0000","last_became_peered":"2024-09-16T10:31:14.132459+0000","last_unstale":"2024-09-16T10:31:28.566741+0000","last_undegraded":"2024-09-16T10:31:28.566741+0000","last_fullsized":"2024-09-16T10:31:28.566741+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.866022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":969,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576311+0000","last_change":"2024-09-16T10:30:45.211710+0000","last_active":"2024-09-16T10:31:28.576311+0000","last_peered":"2024-09-16T10:31:28.576311+0000","last_clean":"2024-09-16T10:31:28.576311+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:31:28.576311+0000","last_undegraded":"2024-09-16T10:31:28.576311+0000","last_fullsized":"2024-09-16T10:31:28.576311+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.211638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.211638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:48.077792+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":999,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575168+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:31:28.575168+0000","last_peered":"2024-09-16T10:31:28.575168+0000","last_clean":"2024-09-16T10:31:28.575168+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:31:28.575168+0000","last_undegraded":"2024-09-16T10:31:28.575168+0000","last_fullsized":"2024-09-16T10:31:28.575168+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:02.288615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":964,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.576087+0000","last_change":"2024-09-16T10:30:21.275187+0000","last_active":"2024-09-16T10:31:28.576087+0000","last_peered":"2024-09-16T10:31:28.576087+0000","last_clean":"2024-09-16T10:31:28.576087+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:31:28.576087+0000","last_undegraded":"2024-09-16T10:31:28.576087+0000","last_fullsized":"2024-09-16T10:31:28.576087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:30:21.275103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:21.275103+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:46.046883+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":984,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.567220+0000","last_change":"2024-09-16T10:31:14.139140+0000","last_active":"2024-09-16T10:31:28.567220+0000","last_peered":"2024-09-16T10:31:28.567220+0000","last_clean":"2024-09-16T10:31:28.567220+0000","last_became_active":"2024-09-16T10:31:14.138772+0000","last_became_peered":"2024-09-16T10:31:14.138772+0000","last_unstale":"2024-09-16T10:31:28.567220+0000","last_undegraded":"2024-09-16T10:31:28.567220+0000","last_fullsized":"2024-09-16T10:31:28.567220+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.649610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.649610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.803476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":594,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566580+0000","last_change":"2024-09-16T10:31:14.152414+0000","last_active":"2024-09-16T10:31:28.566580+0000","last_peered":"2024-09-16T10:31:28.566580+0000","last_clean":"2024-09-16T10:31:28.566580+0000","last_became_active":"2024-09-16T10:31:14.152053+0000","last_became_peered":"2024-09-16T10:31:14.152053+0000","last_unstale":"2024-09-16T10:31:28.566580+0000","last_undegraded":"2024-09-16T10:31:28.566580+0000","last_fullsized":"2024-09-16T10:31:28.566580+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:32.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:32.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":955,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.580377+0000","last_change":"2024-09-16T10:30:15.832984+0000","last_active":"2024-09-16T10:31:28.580377+0000","last_peered":"2024-09-16T10:31:28.580377+0000","last_clean":"2024-09-16T10:31:28.580377+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:31:28.580377+0000","last_undegraded":"2024-09-16T10:31:28.580377+0000","last_fullsized":"2024-09-16T10:31:28.580377+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.832904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.832904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:34.748880+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":965,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.575354+0000","last_change":"2024-09-16T10:26:13.362416+0000","last_active":"2024-09-16T10:31:28.575354+0000","last_peered":"2024-09-16T10:31:28.575354+0000","last_clean":"2024-09-16T10:31:28.575354+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:31:28.575354+0000","last_undegraded":"2024-09-16T10:31:28.575354+0000","last_fullsized":"2024-09-16T10:31:28.575354+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.362347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.362347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:13.236069+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":1010,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.566905+0000","last_change":"2024-09-16T10:30:20.544197+0000","last_active":"2024-09-16T10:31:28.566905+0000","last_peered":"2024-09-16T10:31:28.566905+0000","last_clean":"2024-09-16T10:31:28.566905+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:31:28.566905+0000","last_undegraded":"2024-09-16T10:31:28.566905+0000","last_fullsized":"2024-09-16T10:31:28.566905+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:30:20.544147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.544147+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:32.129391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":991,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.572382+0000","last_change":"2024-09-16T10:31:14.142887+0000","last_active":"2024-09-16T10:31:28.572382+0000","last_peered":"2024-09-16T10:31:28.572382+0000","last_clean":"2024-09-16T10:31:28.572382+0000","last_became_active":"2024-09-16T10:31:14.141591+0000","last_became_peered":"2024-09-16T10:31:14.141591+0000","last_unstale":"2024-09-16T10:31:28.572382+0000","last_undegraded":"2024-09-16T10:31:28.572382+0000","last_fullsized":"2024-09-16T10:31:28.572382+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:54.399132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.399132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:13.946810+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6835011608,"num_objects":104295,"num_object_clones":0,"num_object_copies":312885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28989,"num_objects_unfound":0,"num_objects_dirty":104295,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144253,"num_write_kb":7953473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11046,"num_bytes_recovered":723517440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21278830592,"data_stored":21278818376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66453,"ondisk_log_size":66453,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":3400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":390,"num_read_kb":385,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":552211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":1894080577574,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528542,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4280232,"kb_used_data":3563224,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":89464920,"statfs":{"total":95995035648,"available":91612078080,"internally_reserved":0,"allocated":3648741376,"data_stored":3648241780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790181,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2743632,"kb_used_data":2000256,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":91001520,"statfs":{"total":95995035648,"available":93185556480,"internally_reserved":0,"allocated":2048262144,"data_stored":2047810715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986410,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2294112,"kb_used_data":1645136,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91451040,"statfs":{"total":95995035648,"available":93645864960,"internally_reserved":0,"allocated":1684619264,"data_stored":1684150400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215343,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3851300,"kb_used_data":3026004,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":89893852,"statfs":{"total":95995035648,"available":92051304448,"internally_reserved":0,"allocated":3098628096,"data_stored":3098157190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":845085128},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411572,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3114184,"kb_used_data":2559608,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":90630968,"statfs":{"total":95995035648,"available":92806111232,"internally_reserved":0,"allocated":2621038592,"data_stored":2620594078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575096,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4130808,"kb_used_data":3301288,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":89614344,"statfs":{"total":95995035648,"available":91765088256,"internally_reserved":0,"allocated":3380518912,"data_stored":3380026280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738621,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3029068,"kb_used_data":2213500,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":90716084,"statfs":{"total":95995035648,"available":92893270016,"internally_reserved":0,"allocated":2266624000,"data_stored":2266175955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2257649664,"data_stored":2257649664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3377725440,"data_stored":3377725440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2606829568,"data_stored":2606825496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3092185088,"data_stored":3092185088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1672089600,"data_stored":1672085528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2046296064,"data_stored":2046296064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3631026176,"data_stored":3631022104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2595028992,"data_stored":2595028992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:40.384 INFO:tasks.thrashosds.thrasher:Setting 6.0 to [4, 7, 5, 1, 2, 0] 2024-09-16T10:31:40.384 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.0', '4', '7', '5', '1', '2', '0'] 2024-09-16T10:31:40.384 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 6.0 4 7 5 1 2 0 2024-09-16T10:31:40.393 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:40.425 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:40.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:40.460 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:40.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:40.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:40.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:40.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:40.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:40.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:40.787 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:40.835 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:40.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:40.895 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:40.895 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:31:40.895 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:31:40.895 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[29993]: pgmap v972: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 22418/295284 objects misplaced (7.592%); 53 MiB/s, 853 objects/s recovering 2024-09-16T10:31:40.895 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[29993]: osdmap e517: 8 total, 8 up, 7 in 2024-09-16T10:31:40.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:40.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:31:40.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:31:40.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[39470]: pgmap v972: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 22418/295284 objects misplaced (7.592%); 53 MiB/s, 853 objects/s recovering 2024-09-16T10:31:40.896 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:40 smithi063 ceph-mon[39470]: osdmap e517: 8 total, 8 up, 7 in 2024-09-16T10:31:40.936 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:40.982 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:40.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:40.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:40 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:31:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:40 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:31:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:40 smithi165 ceph-mon[32696]: pgmap v972: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 22418/295284 objects misplaced (7.592%); 53 MiB/s, 853 objects/s recovering 2024-09-16T10:31:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:40 smithi165 ceph-mon[32696]: osdmap e517: 8 total, 8 up, 7 in 2024-09-16T10:31:41.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:41.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:41.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:41.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:41.266 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:41.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:41.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:41.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:41.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:41.438 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:31:41.439 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:31:41.439 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:31:41.439 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:31:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:31:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:31:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:31:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:31:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:31:41.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:41.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:41.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:41.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:41.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:41.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:41.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:41.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:41.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:41.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:41.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:41.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:42.014 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:42.062 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:42.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:42.163 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:42.204 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:42.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:42.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:42.345 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:42.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:42.422 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:31:42.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:31:42.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/106998442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:42.423 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[29993]: pgmap v974: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s; 22418/294510 objects misplaced (7.612%); 45 MiB/s, 723 objects/s recovering 2024-09-16T10:31:42.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:31:42.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:31:42.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/106998442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:42.423 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:42 smithi063 ceph-mon[39470]: pgmap v974: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s; 22418/294510 objects misplaced (7.612%); 45 MiB/s, 723 objects/s recovering 2024-09-16T10:31:42.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:42.498 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:42.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:42 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:31:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:42 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:31:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:42 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/106998442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:42.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:42 smithi165 ceph-mon[32696]: pgmap v974: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 895 op/s; 22418/294510 objects misplaced (7.612%); 45 MiB/s, 723 objects/s recovering 2024-09-16T10:31:42.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:42.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:42.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:42.693 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:42.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:42.786 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:42.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:42.887 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:42.939 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:42.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:43.043 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:43.111 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:43.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:43.129 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:31:43.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:43.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:43.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:43.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:43.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:43.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:43.402 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:43 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2072980338' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:31:43.402 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:43 smithi063 ceph-mon[39470]: from='client.46562 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:31:43.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:43 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2072980338' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:31:43.402 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:43 smithi063 ceph-mon[29993]: from='client.46562 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:31:43.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:43.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:43.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:43.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:43 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2072980338' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:31:43.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:43 smithi165 ceph-mon[32696]: from='client.46562 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:31:43.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:43.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:43.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:43.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:43.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:43.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:43.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:43.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:43.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:43.981 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:44.020 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:44.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:44.161 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:44.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:44.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:44.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:44.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[29993]: 11.d scrub starts 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[29993]: from='client.46562 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[29993]: osdmap e518: 8 total, 8 up, 7 in 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[29993]: pgmap v976: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.29k op/s; 19604/284937 objects misplaced (6.880%); 59 MiB/s, 938 objects/s recovering 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[39470]: 11.d scrub starts 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[39470]: from='client.46562 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[39470]: osdmap e518: 8 total, 8 up, 7 in 2024-09-16T10:31:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:44 smithi063 ceph-mon[39470]: pgmap v976: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.29k op/s; 19604/284937 objects misplaced (6.880%); 59 MiB/s, 938 objects/s recovering 2024-09-16T10:31:44.480 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:31:44.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:44.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:44.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:44 smithi165 ceph-mon[32696]: 11.d scrub starts 2024-09-16T10:31:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:44 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:44 smithi165 ceph-mon[32696]: from='client.46562 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:31:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:44 smithi165 ceph-mon[32696]: osdmap e518: 8 total, 8 up, 7 in 2024-09-16T10:31:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:44 smithi165 ceph-mon[32696]: pgmap v976: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.29k op/s; 19604/284937 objects misplaced (6.880%); 59 MiB/s, 938 objects/s recovering 2024-09-16T10:31:44.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:44.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:44.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:44.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:44.898 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:44.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:45.050 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:45.088 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:45.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:45.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:45.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:45.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:45.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:45.364 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:45.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:45.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:45.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:45.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:45.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:45.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:45.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:45.749 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:45.756 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:31:45.756 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:31:45.807 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:45.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:45.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:45.945 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:45.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:46.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:46.093 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:46.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:46.195 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:46.239 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:46.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:46.340 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:46.383 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:46.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:46.485 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:46.530 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:46.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:46.558 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:46.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:31:46.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:46.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:46.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:46.758 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:46.811 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:46.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:46 smithi063 ceph-mon[29993]: pgmap v977: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 747 B/s rd, 0 B/s wr, 969 op/s; 19308/279441 objects misplaced (6.910%); 26 MiB/s, 414 objects/s recovering 2024-09-16T10:31:46.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:46 smithi063 ceph-mon[39470]: pgmap v977: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 747 B/s rd, 0 B/s wr, 969 op/s; 19308/279441 objects misplaced (6.910%); 26 MiB/s, 414 objects/s recovering 2024-09-16T10:31:46.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:46.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:46.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:47.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:47.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:46 smithi165 ceph-mon[32696]: pgmap v977: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 747 B/s rd, 0 B/s wr, 969 op/s; 19308/279441 objects misplaced (6.910%); 26 MiB/s, 414 objects/s recovering 2024-09-16T10:31:47.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:47.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:47.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:47.181 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:31:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:31:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:31:47.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:47.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:47.296 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:47.305 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:47.352 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:47.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:47.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:47.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:47.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:47.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:47.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:47.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:47.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:31:47.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:47.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:47.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:47.850 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:47.894 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:47.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:47.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:48.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:48.191 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:48.234 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:48.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:48.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:48.372 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:48.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:48.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:48.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:48.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:48.583 INFO:teuthology.orchestra.run.smithi063.stderr:set 6.0 pg_upmap_items mapping to [4->7,5->1,2->0] 2024-09-16T10:31:48.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:48.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:48.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:48.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:48.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:48.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:48.839 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:31:48.839 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:31:48.839 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:31:48.839 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:31:48.839 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:31:48.840 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:31:48.841 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:31:48.841 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:31:48.841 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:31:48.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:31:48.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[29993]: pgmap v978: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 660 op/s; 19308/279441 objects misplaced (6.910%); 16 MiB/s, 248 objects/s recovering 2024-09-16T10:31:48.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1677025838' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]} : dispatch 2024-09-16T10:31:48.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[29993]: from='client.46568 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]} : dispatch 2024-09-16T10:31:48.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/629839047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:48.842 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:31:48.842 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[39470]: pgmap v978: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 660 op/s; 19308/279441 objects misplaced (6.910%); 16 MiB/s, 248 objects/s recovering 2024-09-16T10:31:48.842 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1677025838' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]} : dispatch 2024-09-16T10:31:48.842 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[39470]: from='client.46568 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]} : dispatch 2024-09-16T10:31:48.842 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/629839047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:48.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:48.893 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:48.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:48.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:48.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:49.029 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:49.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:48 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:31:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:48 smithi165 ceph-mon[32696]: pgmap v978: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 660 op/s; 19308/279441 objects misplaced (6.910%); 16 MiB/s, 248 objects/s recovering 2024-09-16T10:31:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1677025838' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]} : dispatch 2024-09-16T10:31:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:48 smithi165 ceph-mon[32696]: from='client.46568 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]} : dispatch 2024-09-16T10:31:49.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/629839047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:31:49.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:49.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:49.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:49.300 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:49.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.440 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:49.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:49.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:49.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:31:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:31:49.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:49.670 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:49.712 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:49.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:49.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:49.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:49.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:49 smithi063 ceph-mon[29993]: from='client.46568 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]}]': finished 2024-09-16T10:31:49.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:49 smithi063 ceph-mon[29993]: osdmap e519: 8 total, 8 up, 7 in 2024-09-16T10:31:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:49 smithi063 ceph-mon[39470]: from='client.46568 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]}]': finished 2024-09-16T10:31:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:49 smithi063 ceph-mon[39470]: osdmap e519: 8 total, 8 up, 7 in 2024-09-16T10:31:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:49.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:49.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:50.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:49 smithi165 ceph-mon[32696]: from='client.46568 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.0", "id": [4, 7, 5, 1, 2, 0]}]': finished 2024-09-16T10:31:50.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:49 smithi165 ceph-mon[32696]: osdmap e519: 8 total, 8 up, 7 in 2024-09-16T10:31:50.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:50.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:50.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:50.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:50.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:50.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:50.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:50.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:50.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:50.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:50.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:50.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:50.597 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:50.636 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:31:50.637 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:31:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:50.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:50.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:50.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:50.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:50.874 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:50.922 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:50.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:50.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:50.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:50 smithi063 ceph-mon[29993]: pgmap v980: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 16400/265614 objects misplaced (6.174%); 31 MiB/s, 493 objects/s recovering 2024-09-16T10:31:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:50.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:50 smithi063 ceph-mon[39470]: pgmap v980: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 16400/265614 objects misplaced (6.174%); 31 MiB/s, 493 objects/s recovering 2024-09-16T10:31:51.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:51.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:51.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:51.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:31:51.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:50 smithi165 ceph-mon[32696]: pgmap v980: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 16400/265614 objects misplaced (6.174%); 31 MiB/s, 493 objects/s recovering 2024-09-16T10:31:51.151 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:51.189 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:51.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:51.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:51.328 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:51.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:51.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:51.464 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:51.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:51.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:51.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:51.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:51.694 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:51.735 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:51.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:51.836 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:51.860 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:51.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:51.961 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:52.001 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:52.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:52.136 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:52.238 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:52.275 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:52.377 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:52.417 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:52.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:52.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:52.650 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:52.677 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:52.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:52 smithi063 ceph-mon[29993]: pgmap v981: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 16400/264657 objects misplaced (6.197%); 23 MiB/s, 364 objects/s recovering 2024-09-16T10:31:52.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:52 smithi063 ceph-mon[39470]: pgmap v981: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 16400/264657 objects misplaced (6.197%); 23 MiB/s, 364 objects/s recovering 2024-09-16T10:31:52.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:52.801 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:52.903 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:52.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:53.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:53.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:52 smithi165 ceph-mon[32696]: pgmap v981: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 16400/264657 objects misplaced (6.197%); 23 MiB/s, 364 objects/s recovering 2024-09-16T10:31:53.059 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:53.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:53.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:53.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:53.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:53.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:53.326 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:53.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:53.428 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:53.468 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:53.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:53.570 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:53.605 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:53.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:53.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:53.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:53.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:53.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:53.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:53.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:53.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:54.006 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:54.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:54.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:54.242 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:54.287 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:54.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:54.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:54.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:54.553 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:54.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:54.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:54.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:54.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:54.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:54.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:54.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:54 smithi063 ceph-mon[29993]: pgmap v982: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 974 op/s; 13845/255711 objects misplaced (5.414%); 25 MiB/s, 400 objects/s recovering 2024-09-16T10:31:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:54 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:54 smithi063 ceph-mon[39470]: pgmap v982: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 974 op/s; 13845/255711 objects misplaced (5.414%); 25 MiB/s, 400 objects/s recovering 2024-09-16T10:31:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:54 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:55.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:54 smithi165 ceph-mon[32696]: pgmap v982: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 974 op/s; 13845/255711 objects misplaced (5.414%); 25 MiB/s, 400 objects/s recovering 2024-09-16T10:31:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:54 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:31:55.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:31:55.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:55.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:55.213 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:55.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:55.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:55.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:55.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:55.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:55.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:55.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:55.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:55.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:55.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:55.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:55.985 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:55.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:56.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:31:56.060 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:56.087 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:56.130 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:56.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:56.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:56.231 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:56.259 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:56.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:56.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:56.417 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:56.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:56.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:56.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:56.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:56.614 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:31:56.645 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:56.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:56 smithi063 ceph-mon[39470]: pgmap v983: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s; 13540/250323 objects misplaced (5.409%); 25 MiB/s, 400 objects/s recovering 2024-09-16T10:31:56.648 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:56 smithi063 ceph-mon[39470]: osdmap e520: 8 total, 8 up, 7 in 2024-09-16T10:31:56.674 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:56.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:56.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:56.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:56.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:56.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:56.907 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:56 smithi063 ceph-mon[29993]: pgmap v983: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s; 13540/250323 objects misplaced (5.409%); 25 MiB/s, 400 objects/s recovering 2024-09-16T10:31:56.907 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:56 smithi063 ceph-mon[29993]: osdmap e520: 8 total, 8 up, 7 in 2024-09-16T10:31:56.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:56.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:57.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:57.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:57.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:56 smithi165 ceph-mon[32696]: pgmap v983: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s; 13540/250323 objects misplaced (5.409%); 25 MiB/s, 400 objects/s recovering 2024-09-16T10:31:57.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:56 smithi165 ceph-mon[32696]: osdmap e520: 8 total, 8 up, 7 in 2024-09-16T10:31:57.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:57.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:57.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:31:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:31:57.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:57.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:57.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:57.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:57.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:57.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:57.605 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:31:57.640 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:31:57.644 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:57.677 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:57.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:57.801 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:57.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:57.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:57.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[29993]: osdmap e521: 8 total, 8 up, 7 in 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1526940459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[29993]: from='client.46580 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[29993]: from='client.46580 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[29993]: osdmap e522: 8 total, 8 up, 7 in 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[39470]: osdmap e521: 8 total, 8 up, 7 in 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1526940459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:57.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[39470]: from='client.46580 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:57.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:57.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[39470]: from='client.46580 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:57.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:57 smithi063 ceph-mon[39470]: osdmap e522: 8 total, 8 up, 7 in 2024-09-16T10:31:58.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:58.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:57 smithi165 ceph-mon[32696]: osdmap e521: 8 total, 8 up, 7 in 2024-09-16T10:31:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:57 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1526940459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:57 smithi165 ceph-mon[32696]: from='client.46580 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:31:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:57 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:31:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:57 smithi165 ceph-mon[32696]: from='client.46580 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:31:58.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:57 smithi165 ceph-mon[32696]: osdmap e522: 8 total, 8 up, 7 in 2024-09-16T10:31:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:58.062 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:58.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4108 9.2 9:4f3b5670:::benchmark_data_smithi063_124717_object4107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.903454+0000", 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.62910533100001, 2024-09-16T10:31:58.129 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.000099557, 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4108 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903454+0000", 2024-09-16T10:31:58.130 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903454+0000", 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903455+0000", 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.356e-06 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.131 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903478+0000", 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2731999999999999e-05 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903488+0000", 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0638999999999999e-05 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903496+0000", 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2629999999999997e-06 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903549+0000", 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3103999999999998e-05 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.903558+0000", 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5349999999999993e-06 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.901312+0000", 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.997753899 2024-09-16T10:31:58.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.901469+0000", 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015720600000000001 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.901640+0000", 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017031699999999999 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903362+0000", 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017224199999999999 2024-09-16T10:31:58.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903381+0000", 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8668000000000002e-05 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903444+0000", 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3017999999999999e-05 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.135 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903505+0000", 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1208000000000001e-05 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903553+0000", 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8192000000000002e-05 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4116 9.2 9:4536b227:::benchmark_data_smithi063_124717_object4115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.136 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.905723+0000", 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.62683592300004, 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99950095800000005, 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4116 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.137 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905723+0000", 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905723+0000", 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905724+0000", 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9699999999999994e-07 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.138 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905749+0000", 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.5154999999999999e-05 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905759+0000", 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6139999999999998e-06 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905766+0000", 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4329999999999998e-06 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.139 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905839+0000", 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3011999999999998e-05 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.905849+0000", 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.5349999999999993e-06 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.902620+0000", 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99677087200000003 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.140 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.902752+0000", 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013278899999999999 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.902922+0000", 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000169996 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905057+0000", 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0021347720000000001 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.141 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905120+0000", 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2590999999999998e-05 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905134+0000", 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3942e-05 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905182+0000", 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8699000000000002e-05 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.142 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905224+0000", 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1551000000000002e-05 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4127 9.2 9:4fce5579:::benchmark_data_smithi063_124717_object4126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.909448+0000", 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.623110924, 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99610295500000001, 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.143 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4127 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909448+0000", 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909448+0000", 2024-09-16T10:31:58.144 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909449+0000", 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.156e-06 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909477+0000", 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7736999999999999e-05 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.145 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909486+0000", 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0450000000000006e-06 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909492+0000", 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7740000000000001e-06 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.146 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909614+0000", 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000122077 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.909623+0000", 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1390000000000001e-06 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903602+0000", 2024-09-16T10:31:58.147 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99397868300000003 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903723+0000", 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121278 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.903920+0000", 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000197586 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.148 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905237+0000", 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001316291 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905278+0000", 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1561999999999999e-05 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905355+0000", 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6755999999999995e-05 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905410+0000", 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5198000000000002e-05 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905551+0000", 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014067300000000001 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.150 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4173 9.2 9:48ddd8d1:::benchmark_data_smithi063_124717_object4172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.924332+0000", 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.60822722099999, 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.98297970700000004, 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4173 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.151 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924332+0000", 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924332+0000", 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924333+0000", 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.624e-06 2024-09-16T10:31:58.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924356+0000", 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3155000000000001e-05 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924367+0000", 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0560000000000001e-05 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924374+0000", 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0929999999999997e-06 2024-09-16T10:31:58.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924435+0000", 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1192000000000006e-05 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.924446+0000", 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1032999999999999e-05 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.904397+0000", 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97995030599999999 2024-09-16T10:31:58.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.904498+0000", 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000101488 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.904635+0000", 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013644899999999999 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.155 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905295+0000", 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00066009 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907059+0000", 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017646669999999999 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907221+0000", 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016126899999999999 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.156 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907270+0000", 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9811999999999997e-05 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907311+0000", 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0969e-05 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4200 9.2 9:433bfede:::benchmark_data_smithi063_124717_object4199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.157 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.931343+0000", 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.601216389, 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97620721099999996, 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4200 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.158 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931343+0000", 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931343+0000", 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931344+0000", 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7090000000000001e-06 2024-09-16T10:31:58.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931368+0000", 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3689999999999998e-05 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931378+0000", 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9289999999999993e-06 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.160 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931386+0000", 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8359999999999992e-06 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931493+0000", 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000107453 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.931502+0000", 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8349999999999993e-06 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.161 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905594+0000", 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.97409171800000005 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905741+0000", 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000147429 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.905851+0000", 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011026 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.162 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907329+0000", 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014775350000000001 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907379+0000", 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0488000000000002e-05 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907461+0000", 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1686000000000004e-05 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.163 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907510+0000", 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9320000000000002e-05 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907550+0000", 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9323000000000003e-05 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.164 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4225 9.2 9:4ec3d5f5:::benchmark_data_smithi063_124717_object4224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.939134+0000", 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.59342451099997, 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.970788706, 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4225 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.165 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939134+0000", 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939134+0000", 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939136+0000", 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3799999999999999e-06 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.166 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939160+0000", 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4354000000000001e-05 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939169+0000", 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2629999999999992e-06 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939177+0000", 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2960000000000001e-06 2024-09-16T10:31:58.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939251+0000", 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.394e-05 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.939262+0000", 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1351e-05 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.906274+0000", 2024-09-16T10:31:58.168 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96701167600000004 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.906399+0000", 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012484 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.906535+0000", 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000136148 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907562+0000", 2024-09-16T10:31:58.169 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001027575 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909783+0000", 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022207220000000001 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909836+0000", 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2769999999999998e-05 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.170 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909885+0000", 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9212000000000002e-05 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909923+0000", 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8179000000000002e-05 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4254 9.2 9:42392fd0:::benchmark_data_smithi063_124717_object4253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.948542+0000", 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.58401646899995, 2024-09-16T10:31:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.96154861800000002, 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4254 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.172 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948542+0000", 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948542+0000", 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948544+0000", 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.398e-06 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.173 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948568+0000", 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3935e-05 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948577+0000", 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7960000000000001e-06 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948584+0000", 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2189999999999998e-06 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.174 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948710+0000", 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000126444 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.948720+0000", 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9340000000000001e-06 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907698+0000", 2024-09-16T10:31:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.95897798199999995 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907844+0000", 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014559100000000001 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.907997+0000", 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015361699999999999 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909932+0000", 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019350719999999999 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909967+0000", 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.4382999999999999e-05 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910006+0000", 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8677000000000003e-05 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910050+0000", 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4124e-05 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910091+0000", 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1446000000000003e-05 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4301 9.2 9:4a35399d:::benchmark_data_smithi063_124717_object4300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.178 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.964920+0000", 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.56763884899999, 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.949930108, 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4301 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.179 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964920+0000", 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964920+0000", 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964922+0000", 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6500000000000001e-06 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964945+0000", 2024-09-16T10:31:58.180 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3027000000000002e-05 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964959+0000", 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4579999999999999e-05 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.964968+0000", 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3930000000000008e-06 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.965093+0000", 2024-09-16T10:31:58.181 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000125265 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.965106+0000", 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3213e-05 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.908907+0000", 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94380035100000004 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909214+0000", 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00030766700000000002 2024-09-16T10:31:58.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.909372+0000", 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015809999999999999 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911971+0000", 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025982969999999998 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914587+0000", 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026161579999999999 2024-09-16T10:31:58.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914742+0000", 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000155119 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914796+0000", 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4169000000000003e-05 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914850+0000", 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4119000000000001e-05 2024-09-16T10:31:58.184 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4315 9.2 9:47b319fd:::benchmark_data_smithi063_124717_object4314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.968925+0000", 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.56363361199999, 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94607211800000002, 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.185 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4315 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.968925+0000", 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.968925+0000", 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.186 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.968927+0000", 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3599999999999999e-06 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969031+0000", 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103829 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.187 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969047+0000", 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6626e-05 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969055+0000", 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2700000000000004e-06 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969477+0000", 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00042144699999999998 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.969489+0000", 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2435e-05 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.189 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910125+0000", 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94063537699999999 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910236+0000", 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111457 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.190 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910371+0000", 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013445199999999999 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914644+0000", 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042737440000000003 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914665+0000", 2024-09-16T10:31:58.191 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.086e-05 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914905+0000", 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000239478 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914959+0000", 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3859000000000001e-05 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914997+0000", 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8924000000000002e-05 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4318 9.2 9:4dd0366d:::benchmark_data_smithi063_124717_object4317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.970502+0000", 2024-09-16T10:31:58.193 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.56205716500006, 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94779669700000002, 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4318 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.194 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970502+0000", 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970502+0000", 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970503+0000", 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.514e-06 2024-09-16T10:31:58.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970526+0000", 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2463000000000002e-05 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970536+0000", 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0229e-05 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970543+0000", 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4449999999999999e-06 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970611+0000", 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7148000000000002e-05 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.970621+0000", 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0356999999999999e-05 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910752+0000", 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94013116200000002 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.910872+0000", 2024-09-16T10:31:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120359 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911003+0000", 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013043899999999999 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914672+0000", 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0036691419999999998 2024-09-16T10:31:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917195+0000", 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0025227549999999998 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918197+0000", 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001002544 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918251+0000", 2024-09-16T10:31:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3609999999999997e-05 2024-09-16T10:31:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918298+0000", 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7530000000000001e-05 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4321 9.2 9:4f79c027:::benchmark_data_smithi063_124717_object4320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.971410+0000", 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.56114915600006, 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.947013997, 2024-09-16T10:31:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4321 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971410+0000", 2024-09-16T10:31:58.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971410+0000", 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971411+0000", 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.528e-06 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971434+0000", 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2580999999999999e-05 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971448+0000", 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4082e-05 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971457+0000", 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0969999999999995e-06 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971523+0000", 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6204000000000005e-05 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.971533+0000", 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.91e-06 2024-09-16T10:31:58.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911382+0000", 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93984913299999995 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911500+0000", 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117928 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.911624+0000", 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000124067 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914678+0000", 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030535549999999999 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917292+0000", 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002613804 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918346+0000", 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00105437 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918388+0000", 2024-09-16T10:31:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2024000000000003e-05 2024-09-16T10:31:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918424+0000", 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5713999999999997e-05 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4331 9.2 9:4a8271f9:::benchmark_data_smithi063_124717_object4330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.975610+0000", 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.55694878899999, 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.94326413600000003, 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4331 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975610+0000", 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975610+0000", 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975612+0000", 2024-09-16T10:31:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.807e-06 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975634+0000", 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.232e-05 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975645+0000", 2024-09-16T10:31:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0861e-05 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975650+0000", 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0520000000000001e-06 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975706+0000", 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5744e-05 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.975712+0000", 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1670000000000004e-06 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912011+0000", 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93629880399999998 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912132+0000", 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120968 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912264+0000", 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013233599999999999 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914684+0000", 2024-09-16T10:31:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00241939 2024-09-16T10:31:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917385+0000", 2024-09-16T10:31:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027012970000000001 2024-09-16T10:31:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918768+0000", 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013833090000000001 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918833+0000", 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5068999999999997e-05 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918874+0000", 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1012e-05 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4346 9.2 9:45bbf4f4:::benchmark_data_smithi063_124717_object4345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.981605+0000", 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.55095410299998, 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.937441992, 2024-09-16T10:31:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4346 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981605+0000", 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981605+0000", 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981606+0000", 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6619999999999999e-06 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981630+0000", 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3337e-05 2024-09-16T10:31:58.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981639+0000", 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.516e-06 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981647+0000", 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2699999999999999e-06 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981743+0000", 2024-09-16T10:31:58.224 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6416999999999995e-05 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.981753+0000", 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0362e-05 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912607+0000", 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93085387799999997 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.225 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912708+0000", 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010064 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.912853+0000", 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014459799999999999 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914690+0000", 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001837588 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.226 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917480+0000", 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027903310000000001 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.918961+0000", 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014803699999999999 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919008+0000", 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7469999999999998e-05 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919047+0000", 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8553000000000002e-05 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4360 9.2 9:4df79284:::benchmark_data_smithi063_124717_object4359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.989021+0000", 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.54353772599995, 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.93028860800000002, 2024-09-16T10:31:58.228 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4360 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989021+0000", 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989021+0000", 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989023+0000", 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.875e-06 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989050+0000", 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.6750999999999998e-05 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989062+0000", 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2133999999999999e-05 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989071+0000", 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1710000000000008e-06 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989139+0000", 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7929000000000006e-05 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.989146+0000", 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8159999999999996e-06 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913223+0000", 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92407729800000005 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913347+0000", 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000124194 2024-09-16T10:31:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913480+0000", 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013293100000000001 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914696+0000", 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0012155709999999999 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917578+0000", 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028823519999999999 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919215+0000", 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016364369999999999 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919268+0000", 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3423e-05 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919310+0000", 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1726e-05 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.45596.0:4366 9.2 9:4b5f36f4:::benchmark_data_smithi063_124717_object4365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:22:04.992754+0000", 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 592.53980480200005, 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92669086099999998, 2024-09-16T10:31:58.235 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.45596", 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 4366 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992754+0000", 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992754+0000", 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992755+0000", 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1999999999999999e-06 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992775+0000", 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9598000000000001e-05 2024-09-16T10:31:58.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992786+0000", 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1532000000000001e-05 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992794+0000", 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8469999999999996e-06 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992850+0000", 2024-09-16T10:31:58.238 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5696999999999999e-05 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:04.992857+0000", 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3849999999999996e-06 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913839+0000", 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.92098122199999999 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.239 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.913950+0000", 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111778 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2", 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.914103+0000", 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000152492 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.915901+0000", 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001797739 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.240 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.917675+0000", 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00177462 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919363+0000", 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0016879569999999999 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919405+0000", 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1628999999999997e-05 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:22:05.919445+0000", 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0164999999999999e-05 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14514.0:3543 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 93976561469696 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:13.112937+0000", 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 44.419622148000002, 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.062285218, 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.242 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14514", 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 3543 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112937+0000", 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.243 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112937+0000", 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112939+0000", 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2680000000000001e-06 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112946+0000", 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0430000000000002e-06 2024-09-16T10:31:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112963+0000", 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.734e-05 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112972+0000", 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7770000000000008e-06 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.113039+0000", 2024-09-16T10:31:58.245 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7142000000000002e-05 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.113046+0000", 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3069999999999999e-06 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.145459+0000", 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032413260999999999 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.145468+0000", 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8959999999999992e-06 2024-09-16T10:31:58.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.151255+0000", 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0057870840000001 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.151570+0000", 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031488799999999998 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.158957+0000", 2024-09-16T10:31:58.247 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0073872319999999997 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.159066+0000", 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108924 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.175222+0000", 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016156055999999999 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:127346 11.1 e509/476 11:852ef2a9:::benchmark_data_smithi063_176907_object3070:head v 509'7982, mlcod=507'7980)", 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.235619+0000", 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 42.296940022999998, 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.28887845699999998, 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235619+0000", 2024-09-16T10:31:58.249 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235619+0000", 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235620+0000", 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1659999999999999e-06 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235627+0000", 2024-09-16T10:31:58.250 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4790000000000001e-06 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235635+0000", 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.901e-06 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235642+0000", 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2870000000000001e-06 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.251 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235857+0000", 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000215639 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235876+0000", 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9069999999999999e-05 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.524434+0000", 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.28855762800000001 2024-09-16T10:31:58.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.524497+0000", 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3287000000000006e-05 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:151673 11.b 11:d5f0724c:::benchmark_data_smithi063_176907_object27397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:39.540122+0000", 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.992436992999998, 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.29731222099999999, 2024-09-16T10:31:58.253 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 151673 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540122+0000", 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540122+0000", 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540124+0000", 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2359999999999999e-06 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540130+0000", 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8510000000000004e-06 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540141+0000", 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.093e-05 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540148+0000", 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4390000000000003e-06 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540833+0000", 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00068436799999999998 2024-09-16T10:31:58.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540953+0000", 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120685 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.541071+0000", 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117136 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.257 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.543941+0000", 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028703090000000001 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.547190+0000", 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032495639999999999 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.837327+0000", 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.29013633300000002 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.837393+0000", 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6080000000000004e-05 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.837434+0000", 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1289999999999999e-05 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:151682 11.b 11:d3c875f7:::benchmark_data_smithi063_176907_object27406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-16T10:31:58.259 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:39.544624+0000", 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 17.987934485, 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.301245704, 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 151682 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544624+0000", 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544624+0000", 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544626+0000", 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8810000000000001e-06 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544631+0000", 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0760000000000002e-06 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544642+0000", 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0413e-05 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544649+0000", 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.216e-06 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544752+0000", 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102822 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544889+0000", 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013679999999999999 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544995+0000", 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106654 2024-09-16T10:31:58.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.547247+0000", 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002251435 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.547566+0000", 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031900400000000001 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.845719+0000", 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.298153059 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.845764+0000", 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4996000000000001e-05 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.845870+0000", 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010634800000000001 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:164823 11.9 11:9c734485:::benchmark_data_smithi063_176907_object40547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e519)", 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:52.893527+0000", 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 4.6390317020000005, 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.286719524, 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 164823 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.266 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893527+0000", 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893527+0000", 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893529+0000", 2024-09-16T10:31:58.267 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.308e-06 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893534+0000", 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5079999999999996e-06 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893542+0000", 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2400000000000007e-06 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:31:58.268 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893549+0000", 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9890000000000003e-06 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893610+0000", 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0234e-05 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893715+0000", 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010527999999999999 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893826+0000", 2024-09-16T10:31:58.269 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011135499999999999 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.897595+0000", 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037689350000000002 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.903457+0000", 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058624089999999998 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:31:58.270 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:53.180165+0000", 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27670773599999998 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:53.180211+0000", 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5881000000000003e-05 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:53.180247+0000", 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5648999999999997e-05 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:31:58.272 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:31:58.272 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:31:58.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:58.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:58.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:58.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:31:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:58.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:31:58.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:31:58.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:58.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:31:58.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:31:58.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:58.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:31:58.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:58.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:58.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:58.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:58.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:31:58 smithi063 ceph-mon[29993]: pgmap v986: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s; 13540/250323 objects misplaced (5.409%); 16 MiB/s, 254 objects/s recovering 2024-09-16T10:31:58.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:31:58 smithi063 ceph-mon[39470]: pgmap v986: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s; 13540/250323 objects misplaced (5.409%); 16 MiB/s, 254 objects/s recovering 2024-09-16T10:31:58.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:58.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:59.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:31:58 smithi165 ceph-mon[32696]: pgmap v986: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s; 13540/250323 objects misplaced (5.409%); 16 MiB/s, 254 objects/s recovering 2024-09-16T10:31:59.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:59.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:59.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:59.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:59.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:59.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:59.318 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:31:59.364 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:59.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:31:59.466 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:31:59.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:59.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:31:59.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:31:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:31:59.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:31:59.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:59.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:31:59.754 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:31:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:31:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:59.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:31:59.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:31:59.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:31:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:31:59.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:31:59.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:32:00.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:00.063 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:00.078 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:00.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:00.110 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 2] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:32:00.110 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:32:00.111 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-09-16T10:32:00.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:32:00.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:00.230 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:00.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:00.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:00.362 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:00.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:00.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:00.466 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:00.515 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:00.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:00.617 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:00.669 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:00.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:00.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:00.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[29993]: pgmap v988: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 880 op/s; 10756/239871 objects misplaced (4.484%); 23 MiB/s, 369 objects/s recovering 2024-09-16T10:32:00.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:32:00.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:32:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[39470]: pgmap v988: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 880 op/s; 10756/239871 objects misplaced (4.484%); 23 MiB/s, 369 objects/s recovering 2024-09-16T10:32:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:32:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:00 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:32:00.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:00.787 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:32:00.788 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:32:00.800 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:00 smithi165 ceph-mon[32696]: pgmap v988: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 880 op/s; 10756/239871 objects misplaced (4.484%); 23 MiB/s, 369 objects/s recovering 2024-09-16T10:32:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:00 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:32:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:00 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:32:00.824 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:00.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:00.826 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:32:00.826 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:32:00.857 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:32:00.857 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:32:00.891 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:32:00.891 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:32:00.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:00.928 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:32:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:32:00.960 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:32:00.960 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:32:00.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:00.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:00.994 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:32:00.994 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:32:01.028 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:32:01.028 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:32:01.030 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:32:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:32:01.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:01.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:01.208 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:01.246 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:01.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:01.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:01.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:01.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:01.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:01.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:32:01.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:01.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:01.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:01.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:01.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:01.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:01.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:01.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:01 smithi165 ceph-mon[32696]: osdmap e523: 8 total, 8 up, 7 in 2024-09-16T10:32:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/82186932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:01.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:01.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:01.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:01.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:01 smithi063 ceph-mon[29993]: osdmap e523: 8 total, 8 up, 7 in 2024-09-16T10:32:01.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/82186932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:01.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:01 smithi063 ceph-mon[39470]: osdmap e523: 8 total, 8 up, 7 in 2024-09-16T10:32:01.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/82186932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:01.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:32:02.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:02.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:02.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:02.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:02.193 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:02.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:02.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:02.328 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:02.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:02.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:02.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:02.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:02.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:02.567 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:02.601 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:02.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:02.703 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:02.740 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:02.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:02.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:02 smithi165 ceph-mon[32696]: pgmap v990: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 635 op/s; 10756/239148 objects misplaced (4.498%); 24 MiB/s, 377 objects/s recovering 2024-09-16T10:32:02.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:02 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:32:02.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:02 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:32:02.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:02.882 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:02.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:02.912 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:32:02.912 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:32:02.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:02 smithi063 ceph-mon[29993]: pgmap v990: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 635 op/s; 10756/239148 objects misplaced (4.498%); 24 MiB/s, 377 objects/s recovering 2024-09-16T10:32:02.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:02 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:32:02.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:02 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:32:02.951 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:02 smithi063 ceph-mon[39470]: pgmap v990: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 872 B/s rd, 0 B/s wr, 635 op/s; 10756/239148 objects misplaced (4.498%); 24 MiB/s, 377 objects/s recovering 2024-09-16T10:32:02.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:02 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:32:02.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:02 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:32:02.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:03.023 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:03.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:03.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:03.263 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:03.300 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:03.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:03.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:03.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:03.535 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.577 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:03.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:03.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:32:03.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:03.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:03.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:03.859 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:03.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:03.978 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:03.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:03.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:04.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:04.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:04.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:04.216 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:04.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:04.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:04.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:04.285 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:04 smithi165 ceph-mon[32696]: osdmap e524: 8 total, 8 up, 7 in 2024-09-16T10:32:04.286 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:04 smithi165 ceph-mon[32696]: pgmap v992: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 9903/228330 objects misplaced (4.337%); 28 MiB/s, 450 objects/s recovering 2024-09-16T10:32:04.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:04.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:04.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:04.428 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:32:04.429 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:04 smithi063 ceph-mon[39470]: osdmap e524: 8 total, 8 up, 7 in 2024-09-16T10:32:04.429 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:04 smithi063 ceph-mon[39470]: pgmap v992: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 9903/228330 objects misplaced (4.337%); 28 MiB/s, 450 objects/s recovering 2024-09-16T10:32:04.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:04 smithi063 ceph-mon[29993]: osdmap e524: 8 total, 8 up, 7 in 2024-09-16T10:32:04.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:04 smithi063 ceph-mon[29993]: pgmap v992: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 9903/228330 objects misplaced (4.337%); 28 MiB/s, 450 objects/s recovering 2024-09-16T10:32:04.436 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:32:04.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:04.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:04.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:04.621 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:32:04.621 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:32:04.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:04.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:04.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:04.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:04.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:04.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:04.885 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:04.933 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:04.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:05.035 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:05.069 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:05.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:05.171 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:05.212 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:05.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:05.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:05 smithi165 ceph-mon[32696]: from='client.16761 v1:172.21.15.63:0/211300323' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:32:05.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:32:05.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:05.328 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":992,"stamp":"2024-09-16T10:32:03.498810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4975461126,"num_objects":76110,"num_object_clones":0,"num_object_copies":228330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9903,"num_objects_unfound":0,"num_objects_dirty":76110,"num_whiteouts":0,"num_read":1555,"num_read_kb":1312,"num_write":173364,"num_write_kb":7953592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25050,"num_bytes_recovered":1629453840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67756,"ondisk_log_size":67756,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19449636,"kb_used_data":14085844,"kb_used_omap":10,"kb_used_meta":5363189,"kb_avail":636766428,"statfs":{"total":671965249536,"available":652048822272,"internally_reserved":0,"allocated":14423904256,"data_stored":14421117717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11161,"internal_metadata":5491905639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,9,7,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-480444416,"num_objects":-7331,"num_object_clones":0,"num_object_copies":-21993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3637,"num_objects_unfound":0,"num_objects_dirty":-7331,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2705,"num_bytes_recovered":177274880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001773"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":999,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.643558+0000","last_change":"2024-09-16T10:29:36.039702+0000","last_active":"2024-09-16T10:31:57.643558+0000","last_peered":"2024-09-16T10:31:57.643558+0000","last_clean":"2024-09-16T10:31:57.643558+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:31:57.643558+0000","last_undegraded":"2024-09-16T10:31:57.643558+0000","last_fullsized":"2024-09-16T10:31:57.643558+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:36.039652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:36.039652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.261361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":945,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637654+0000","last_change":"2024-09-16T10:30:16.596737+0000","last_active":"2024-09-16T10:31:57.637654+0000","last_peered":"2024-09-16T10:31:57.637654+0000","last_clean":"2024-09-16T10:31:57.637654+0000","last_became_active":"2024-09-16T10:26:28.557096+0000","last_became_peered":"2024-09-16T10:26:28.557096+0000","last_unstale":"2024-09-16T10:31:57.637654+0000","last_undegraded":"2024-09-16T10:31:57.637654+0000","last_fullsized":"2024-09-16T10:31:57.637654+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:16.596655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.596655+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1007,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.269576+0000","last_change":"2024-09-16T10:31:14.141815+0000","last_active":"2024-09-16T10:32:03.269576+0000","last_peered":"2024-09-16T10:32:03.269576+0000","last_clean":"2024-09-16T10:32:03.269576+0000","last_became_active":"2024-09-16T10:31:14.141497+0000","last_became_peered":"2024-09-16T10:31:14.141497+0000","last_unstale":"2024-09-16T10:32:03.269576+0000","last_undegraded":"2024-09-16T10:32:03.269576+0000","last_fullsized":"2024-09-16T10:32:03.269576+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.872906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.872906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":311,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668758+0000","last_change":"2024-09-16T10:30:17.879751+0000","last_active":"2024-09-16T10:32:00.668758+0000","last_peered":"2024-09-16T10:32:00.668758+0000","last_clean":"2024-09-16T10:32:00.668758+0000","last_became_active":"2024-09-16T10:26:28.551847+0000","last_became_peered":"2024-09-16T10:26:28.551847+0000","last_unstale":"2024-09-16T10:32:00.668758+0000","last_undegraded":"2024-09-16T10:32:00.668758+0000","last_fullsized":"2024-09-16T10:32:00.668758+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:17.879610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:17.879610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":985,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200236+0000","last_change":"2024-09-16T10:31:14.148991+0000","last_active":"2024-09-16T10:32:03.200236+0000","last_peered":"2024-09-16T10:32:03.200236+0000","last_clean":"2024-09-16T10:32:03.200236+0000","last_became_active":"2024-09-16T10:31:14.148273+0000","last_became_peered":"2024-09-16T10:31:14.148273+0000","last_unstale":"2024-09-16T10:32:03.200236+0000","last_undegraded":"2024-09-16T10:32:03.200236+0000","last_fullsized":"2024-09-16T10:32:03.200236+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1011,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.203073+0000","last_change":"2024-09-16T10:30:48.140925+0000","last_active":"2024-09-16T10:32:03.203073+0000","last_peered":"2024-09-16T10:32:03.203073+0000","last_clean":"2024-09-16T10:32:03.203073+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:32:03.203073+0000","last_undegraded":"2024-09-16T10:32:03.203073+0000","last_fullsized":"2024-09-16T10:32:03.203073+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":1080,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.203008+0000","last_change":"2024-09-16T10:30:15.998707+0000","last_active":"2024-09-16T10:32:03.203008+0000","last_peered":"2024-09-16T10:32:03.203008+0000","last_clean":"2024-09-16T10:32:03.203008+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:32:03.203008+0000","last_undegraded":"2024-09-16T10:32:03.203008+0000","last_fullsized":"2024-09-16T10:32:03.203008+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:15.998614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.998614+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":309,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637824+0000","last_change":"2024-09-16T10:30:07.591239+0000","last_active":"2024-09-16T10:31:57.637824+0000","last_peered":"2024-09-16T10:31:57.637824+0000","last_clean":"2024-09-16T10:31:57.637824+0000","last_became_active":"2024-09-16T10:26:28.556509+0000","last_became_peered":"2024-09-16T10:26:28.556509+0000","last_unstale":"2024-09-16T10:31:57.637824+0000","last_undegraded":"2024-09-16T10:31:57.637824+0000","last_fullsized":"2024-09-16T10:31:57.637824+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.591170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.591170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1011,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669662+0000","last_change":"2024-09-16T10:30:04.838255+0000","last_active":"2024-09-16T10:32:00.669662+0000","last_peered":"2024-09-16T10:32:00.669662+0000","last_clean":"2024-09-16T10:32:00.669662+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:32:00.669662+0000","last_undegraded":"2024-09-16T10:32:00.669662+0000","last_fullsized":"2024-09-16T10:32:00.669662+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.838182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.838182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":907,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200373+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:32:03.200373+0000","last_peered":"2024-09-16T10:32:03.200373+0000","last_clean":"2024-09-16T10:32:03.200373+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:32:03.200373+0000","last_undegraded":"2024-09-16T10:32:03.200373+0000","last_fullsized":"2024-09-16T10:32:03.200373+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1005,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668984+0000","last_change":"2024-09-16T10:31:14.144852+0000","last_active":"2024-09-16T10:32:00.668984+0000","last_peered":"2024-09-16T10:32:00.668984+0000","last_clean":"2024-09-16T10:32:00.668984+0000","last_became_active":"2024-09-16T10:31:14.144519+0000","last_became_peered":"2024-09-16T10:31:14.144519+0000","last_unstale":"2024-09-16T10:32:00.668984+0000","last_undegraded":"2024-09-16T10:32:00.668984+0000","last_fullsized":"2024-09-16T10:32:00.668984+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:41.762023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:41.762023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":990,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.640510+0000","last_change":"2024-09-16T10:31:14.144630+0000","last_active":"2024-09-16T10:31:57.640510+0000","last_peered":"2024-09-16T10:31:57.640510+0000","last_clean":"2024-09-16T10:31:57.640510+0000","last_became_active":"2024-09-16T10:31:14.142022+0000","last_became_peered":"2024-09-16T10:31:14.142022+0000","last_unstale":"2024-09-16T10:31:57.640510+0000","last_undegraded":"2024-09-16T10:31:57.640510+0000","last_fullsized":"2024-09-16T10:31:57.640510+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:08.436243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.436243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":963,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669217+0000","last_change":"2024-09-16T10:30:13.941093+0000","last_active":"2024-09-16T10:32:00.669217+0000","last_peered":"2024-09-16T10:32:00.669217+0000","last_clean":"2024-09-16T10:32:00.669217+0000","last_became_active":"2024-09-16T10:13:37.092585+0000","last_became_peered":"2024-09-16T10:13:37.092585+0000","last_unstale":"2024-09-16T10:32:00.669217+0000","last_undegraded":"2024-09-16T10:32:00.669217+0000","last_fullsized":"2024-09-16T10:32:00.669217+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:13.941032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:13.941032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":895,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200742+0000","last_change":"2024-09-16T10:30:39.082533+0000","last_active":"2024-09-16T10:32:03.200742+0000","last_peered":"2024-09-16T10:32:03.200742+0000","last_clean":"2024-09-16T10:32:03.200742+0000","last_became_active":"2024-09-16T10:26:28.556933+0000","last_became_peered":"2024-09-16T10:26:28.556933+0000","last_unstale":"2024-09-16T10:32:03.200742+0000","last_undegraded":"2024-09-16T10:32:03.200742+0000","last_fullsized":"2024-09-16T10:32:03.200742+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:30:39.082450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.082450+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":690,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.197737+0000","last_change":"2024-09-16T10:31:14.143481+0000","last_active":"2024-09-16T10:32:03.197737+0000","last_peered":"2024-09-16T10:32:03.197737+0000","last_clean":"2024-09-16T10:32:03.197737+0000","last_became_active":"2024-09-16T10:31:14.143134+0000","last_became_peered":"2024-09-16T10:31:14.143134+0000","last_unstale":"2024-09-16T10:32:03.197737+0000","last_undegraded":"2024-09-16T10:32:03.197737+0000","last_fullsized":"2024-09-16T10:32:03.197737+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.995349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.995349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":998,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668656+0000","last_change":"2024-09-16T10:30:45.741431+0000","last_active":"2024-09-16T10:32:00.668656+0000","last_peered":"2024-09-16T10:32:00.668656+0000","last_clean":"2024-09-16T10:32:00.668656+0000","last_became_active":"2024-09-16T10:21:10.295734+0000","last_became_peered":"2024-09-16T10:21:10.295734+0000","last_unstale":"2024-09-16T10:32:00.668656+0000","last_undegraded":"2024-09-16T10:32:00.668656+0000","last_fullsized":"2024-09-16T10:32:00.668656+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.741362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.741362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.595620+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":301,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638086+0000","last_change":"2024-09-16T10:29:16.166206+0000","last_active":"2024-09-16T10:31:57.638086+0000","last_peered":"2024-09-16T10:31:57.638086+0000","last_clean":"2024-09-16T10:31:57.638086+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:31:57.638086+0000","last_undegraded":"2024-09-16T10:31:57.638086+0000","last_fullsized":"2024-09-16T10:31:57.638086+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:16.166144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:16.166144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":994,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641688+0000","last_change":"2024-09-16T10:31:14.144948+0000","last_active":"2024-09-16T10:31:57.641688+0000","last_peered":"2024-09-16T10:31:57.641688+0000","last_clean":"2024-09-16T10:31:57.641688+0000","last_became_active":"2024-09-16T10:31:14.144533+0000","last_became_peered":"2024-09-16T10:31:14.144533+0000","last_unstale":"2024-09-16T10:31:57.641688+0000","last_undegraded":"2024-09-16T10:31:57.641688+0000","last_fullsized":"2024-09-16T10:31:57.641688+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.446934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.446934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.164812+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":552,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.639566+0000","last_change":"2024-09-16T10:30:55.602908+0000","last_active":"2024-09-16T10:31:57.639566+0000","last_peered":"2024-09-16T10:31:57.639566+0000","last_clean":"2024-09-16T10:31:57.639566+0000","last_became_active":"2024-09-16T10:22:02.910537+0000","last_became_peered":"2024-09-16T10:22:02.910537+0000","last_unstale":"2024-09-16T10:31:57.639566+0000","last_undegraded":"2024-09-16T10:31:57.639566+0000","last_fullsized":"2024-09-16T10:31:57.639566+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.599903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.599903+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:18.999085+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":981,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.270993+0000","last_change":"2024-09-16T10:30:09.918436+0000","last_active":"2024-09-16T10:32:03.270993+0000","last_peered":"2024-09-16T10:32:03.270993+0000","last_clean":"2024-09-16T10:32:03.270993+0000","last_became_active":"2024-09-16T10:13:41.120011+0000","last_became_peered":"2024-09-16T10:13:41.120011+0000","last_unstale":"2024-09-16T10:32:03.270993+0000","last_undegraded":"2024-09-16T10:32:03.270993+0000","last_fullsized":"2024-09-16T10:32:03.270993+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:09.918371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:09.918371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":933,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202141+0000","last_change":"2024-09-16T10:27:18.318288+0000","last_active":"2024-09-16T10:32:03.202141+0000","last_peered":"2024-09-16T10:32:03.202141+0000","last_clean":"2024-09-16T10:32:03.202141+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:32:03.202141+0000","last_undegraded":"2024-09-16T10:32:03.202141+0000","last_fullsized":"2024-09-16T10:32:03.202141+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:18.318214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:18.318214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":769,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.269445+0000","last_change":"2024-09-16T10:26:28.551407+0000","last_active":"2024-09-16T10:32:03.269445+0000","last_peered":"2024-09-16T10:32:03.269445+0000","last_clean":"2024-09-16T10:32:03.269445+0000","last_became_active":"2024-09-16T10:26:28.551058+0000","last_became_peered":"2024-09-16T10:26:28.551058+0000","last_unstale":"2024-09-16T10:32:03.269445+0000","last_undegraded":"2024-09-16T10:32:03.269445+0000","last_fullsized":"2024-09-16T10:32:03.269445+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.407589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.407589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":1050,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200821+0000","last_change":"2024-09-16T10:30:38.072547+0000","last_active":"2024-09-16T10:32:03.200821+0000","last_peered":"2024-09-16T10:32:03.200821+0000","last_clean":"2024-09-16T10:32:03.200821+0000","last_became_active":"2024-09-16T10:15:08.926588+0000","last_became_peered":"2024-09-16T10:15:08.926588+0000","last_unstale":"2024-09-16T10:32:03.200821+0000","last_undegraded":"2024-09-16T10:32:03.200821+0000","last_fullsized":"2024-09-16T10:32:03.200821+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:30:38.072460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:38.072460+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":777,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201506+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:32:03.201506+0000","last_peered":"2024-09-16T10:32:03.201506+0000","last_clean":"2024-09-16T10:32:03.201506+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:32:03.201506+0000","last_undegraded":"2024-09-16T10:32:03.201506+0000","last_fullsized":"2024-09-16T10:32:03.201506+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:14.190419+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":408,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669053+0000","last_change":"2024-09-16T10:31:14.169546+0000","last_active":"2024-09-16T10:32:00.669053+0000","last_peered":"2024-09-16T10:32:00.669053+0000","last_clean":"2024-09-16T10:32:00.669053+0000","last_became_active":"2024-09-16T10:31:14.151297+0000","last_became_peered":"2024-09-16T10:31:14.151297+0000","last_unstale":"2024-09-16T10:32:00.669053+0000","last_undegraded":"2024-09-16T10:32:00.669053+0000","last_fullsized":"2024-09-16T10:32:00.669053+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:40.788922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.788922+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:02.804769+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":996,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.269665+0000","last_change":"2024-09-16T10:30:12.928363+0000","last_active":"2024-09-16T10:32:03.269665+0000","last_peered":"2024-09-16T10:32:03.269665+0000","last_clean":"2024-09-16T10:32:03.269665+0000","last_became_active":"2024-09-16T10:13:37.082273+0000","last_became_peered":"2024-09-16T10:13:37.082273+0000","last_unstale":"2024-09-16T10:32:03.269665+0000","last_undegraded":"2024-09-16T10:32:03.269665+0000","last_fullsized":"2024-09-16T10:32:03.269665+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:30:12.928274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.928274+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":1042,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.642485+0000","last_change":"2024-09-16T10:31:14.187902+0000","last_active":"2024-09-16T10:31:57.642485+0000","last_peered":"2024-09-16T10:31:57.642485+0000","last_clean":"2024-09-16T10:31:57.642485+0000","last_became_active":"2024-09-16T10:31:14.156596+0000","last_became_peered":"2024-09-16T10:31:14.156596+0000","last_unstale":"2024-09-16T10:31:57.642485+0000","last_undegraded":"2024-09-16T10:31:57.642485+0000","last_fullsized":"2024-09-16T10:31:57.642485+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:29.106770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.106770+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"508'11","reported_seq":1233,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638276+0000","last_change":"2024-09-16T10:31:14.167033+0000","last_active":"2024-09-16T10:31:57.638276+0000","last_peered":"2024-09-16T10:31:57.638276+0000","last_clean":"2024-09-16T10:31:57.638276+0000","last_became_active":"2024-09-16T10:31:14.137776+0000","last_became_peered":"2024-09-16T10:31:14.137776+0000","last_unstale":"2024-09-16T10:31:57.638276+0000","last_undegraded":"2024-09-16T10:31:57.638276+0000","last_fullsized":"2024-09-16T10:31:57.638276+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"524'10865","reported_seq":23272,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.367619+0000","last_change":"2024-09-16T10:31:57.668557+0000","last_active":"2024-09-16T10:32:03.367619+0000","last_peered":"2024-09-16T10:32:03.367619+0000","last_clean":"2024-09-16T10:32:03.367619+0000","last_became_active":"2024-09-16T10:31:57.642759+0000","last_became_peered":"2024-09-16T10:31:57.642759+0000","last_unstale":"2024-09-16T10:32:03.367619+0000","last_undegraded":"2024-09-16T10:32:03.367619+0000","last_fullsized":"2024-09-16T10:32:03.367619+0000","mapping_epoch":521,"log_start":"504'7113","ondisk_log_start":"504'7113","created":476,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306774016,"num_objects":4681,"num_object_clones":0,"num_object_copies":14043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10865,"num_write_kb":497472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6139,"num_bytes_recovered":402259968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":833,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668755+0000","last_change":"2024-09-16T10:29:21.013569+0000","last_active":"2024-09-16T10:32:00.668755+0000","last_peered":"2024-09-16T10:32:00.668755+0000","last_clean":"2024-09-16T10:32:00.668755+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:32:00.668755+0000","last_undegraded":"2024-09-16T10:32:00.668755+0000","last_fullsized":"2024-09-16T10:32:00.668755+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:21.013503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.013503+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:11.999269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":1021,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.274462+0000","last_change":"2024-09-16T10:27:12.407528+0000","last_active":"2024-09-16T10:32:03.274462+0000","last_peered":"2024-09-16T10:32:03.274462+0000","last_clean":"2024-09-16T10:32:03.274462+0000","last_became_active":"2024-09-16T10:13:39.096449+0000","last_became_peered":"2024-09-16T10:13:39.096449+0000","last_unstale":"2024-09-16T10:32:03.274462+0000","last_undegraded":"2024-09-16T10:32:03.274462+0000","last_fullsized":"2024-09-16T10:32:03.274462+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:27:12.407441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.407441+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1014,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201746+0000","last_change":"2024-09-16T10:31:14.159286+0000","last_active":"2024-09-16T10:32:03.201746+0000","last_peered":"2024-09-16T10:32:03.201746+0000","last_clean":"2024-09-16T10:32:03.201746+0000","last_became_active":"2024-09-16T10:31:14.158830+0000","last_became_peered":"2024-09-16T10:31:14.158830+0000","last_unstale":"2024-09-16T10:32:03.201746+0000","last_undegraded":"2024-09-16T10:32:03.201746+0000","last_fullsized":"2024-09-16T10:32:03.201746+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.213821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.213821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":404,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669564+0000","last_change":"2024-09-16T10:30:16.857737+0000","last_active":"2024-09-16T10:32:00.669564+0000","last_peered":"2024-09-16T10:32:00.669564+0000","last_clean":"2024-09-16T10:32:00.669564+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:32:00.669564+0000","last_undegraded":"2024-09-16T10:32:00.669564+0000","last_fullsized":"2024-09-16T10:32:00.669564+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:30:16.857676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.857676+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":984,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669224+0000","last_change":"2024-09-16T10:30:14.925349+0000","last_active":"2024-09-16T10:32:00.669224+0000","last_peered":"2024-09-16T10:32:00.669224+0000","last_clean":"2024-09-16T10:32:00.669224+0000","last_became_active":"2024-09-16T10:13:43.114323+0000","last_became_peered":"2024-09-16T10:13:43.114323+0000","last_unstale":"2024-09-16T10:32:00.669224+0000","last_undegraded":"2024-09-16T10:32:00.669224+0000","last_fullsized":"2024-09-16T10:32:00.669224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:14.925284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:14.925284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"524'11043","reported_seq":22068,"reported_epoch":524,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T10:32:03.380092+0000","last_change":"2024-09-16T10:31:15.263335+0000","last_active":"2024-09-16T10:32:03.380092+0000","last_peered":"2024-09-16T10:32:03.380092+0000","last_clean":"2024-09-16T10:31:13.116495+0000","last_became_active":"2024-09-16T10:31:15.229467+0000","last_became_peered":"2024-09-16T10:31:15.229467+0000","last_unstale":"2024-09-16T10:32:03.380092+0000","last_undegraded":"2024-09-16T10:32:03.380092+0000","last_fullsized":"2024-09-16T10:32:03.380092+0000","mapping_epoch":508,"log_start":"505'7362","ondisk_log_start":"505'7362","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3681,"log_dups_size":0,"ondisk_log_size":3681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304939008,"num_objects":4653,"num_object_clones":0,"num_object_copies":13959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":329,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5455,"num_bytes_recovered":357498880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":4653}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":413,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638507+0000","last_change":"2024-09-16T10:30:18.655233+0000","last_active":"2024-09-16T10:31:57.638507+0000","last_peered":"2024-09-16T10:31:57.638507+0000","last_clean":"2024-09-16T10:31:57.638507+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:31:57.638507+0000","last_undegraded":"2024-09-16T10:31:57.638507+0000","last_fullsized":"2024-09-16T10:31:57.638507+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:18.655172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:18.655172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":366,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637381+0000","last_change":"2024-09-16T10:26:44.445756+0000","last_active":"2024-09-16T10:31:57.637381+0000","last_peered":"2024-09-16T10:31:57.637381+0000","last_clean":"2024-09-16T10:31:57.637381+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:31:57.637381+0000","last_undegraded":"2024-09-16T10:31:57.637381+0000","last_fullsized":"2024-09-16T10:31:57.637381+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:26:44.445661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.445661+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":979,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202621+0000","last_change":"2024-09-16T10:31:14.146422+0000","last_active":"2024-09-16T10:32:03.202621+0000","last_peered":"2024-09-16T10:32:03.202621+0000","last_clean":"2024-09-16T10:32:03.202621+0000","last_became_active":"2024-09-16T10:31:14.146107+0000","last_became_peered":"2024-09-16T10:31:14.146107+0000","last_unstale":"2024-09-16T10:32:03.202621+0000","last_undegraded":"2024-09-16T10:32:03.202621+0000","last_fullsized":"2024-09-16T10:32:03.202621+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:55.254993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.254993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":734,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637393+0000","last_change":"2024-09-16T10:31:14.164915+0000","last_active":"2024-09-16T10:31:57.637393+0000","last_peered":"2024-09-16T10:31:57.637393+0000","last_clean":"2024-09-16T10:31:57.637393+0000","last_became_active":"2024-09-16T10:31:14.164506+0000","last_became_peered":"2024-09-16T10:31:14.164506+0000","last_unstale":"2024-09-16T10:31:57.637393+0000","last_undegraded":"2024-09-16T10:31:57.637393+0000","last_fullsized":"2024-09-16T10:31:57.637393+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:24.521586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:24.521586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":789,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200366+0000","last_change":"2024-09-16T10:27:40.298188+0000","last_active":"2024-09-16T10:32:03.200366+0000","last_peered":"2024-09-16T10:32:03.200366+0000","last_clean":"2024-09-16T10:32:03.200366+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:32:03.200366+0000","last_undegraded":"2024-09-16T10:32:03.200366+0000","last_fullsized":"2024-09-16T10:32:03.200366+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"524'11033","reported_seq":11160,"reported_epoch":524,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:32:03.329744+0000","last_change":"2024-09-16T10:31:45.272896+0000","last_active":"2024-09-16T10:32:03.329744+0000","last_peered":"2024-09-16T10:32:03.329744+0000","last_clean":"2024-09-16T10:31:13.099472+0000","last_became_active":"2024-09-16T10:31:15.212663+0000","last_became_peered":"2024-09-16T10:31:15.212663+0000","last_unstale":"2024-09-16T10:32:03.329744+0000","last_undegraded":"2024-09-16T10:32:03.329744+0000","last_fullsized":"2024-09-16T10:32:03.329744+0000","mapping_epoch":508,"log_start":"505'7279","ondisk_log_start":"505'7279","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307167232,"num_objects":4687,"num_object_clones":0,"num_object_copies":14061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4687,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":4687}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":914,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201798+0000","last_change":"2024-09-16T10:31:17.119206+0000","last_active":"2024-09-16T10:32:03.201798+0000","last_peered":"2024-09-16T10:32:03.201798+0000","last_clean":"2024-09-16T10:32:03.201798+0000","last_became_active":"2024-09-16T10:31:17.118842+0000","last_became_peered":"2024-09-16T10:31:17.118842+0000","last_unstale":"2024-09-16T10:32:03.201798+0000","last_undegraded":"2024-09-16T10:32:03.201798+0000","last_fullsized":"2024-09-16T10:32:03.201798+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.198923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.198923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:11.115767+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":932,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668699+0000","last_change":"2024-09-16T10:30:12.893400+0000","last_active":"2024-09-16T10:32:00.668699+0000","last_peered":"2024-09-16T10:32:00.668699+0000","last_clean":"2024-09-16T10:32:00.668699+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:32:00.668699+0000","last_undegraded":"2024-09-16T10:32:00.668699+0000","last_fullsized":"2024-09-16T10:32:00.668699+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:30:12.893340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.893340+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":978,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.642607+0000","last_change":"2024-09-16T10:27:44.624946+0000","last_active":"2024-09-16T10:31:57.642607+0000","last_peered":"2024-09-16T10:31:57.642607+0000","last_clean":"2024-09-16T10:31:57.642607+0000","last_became_active":"2024-09-16T10:13:37.095293+0000","last_became_peered":"2024-09-16T10:13:37.095293+0000","last_unstale":"2024-09-16T10:31:57.642607+0000","last_undegraded":"2024-09-16T10:31:57.642607+0000","last_fullsized":"2024-09-16T10:31:57.642607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":1214,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201371+0000","last_change":"2024-09-16T10:27:24.281038+0000","last_active":"2024-09-16T10:32:03.201371+0000","last_peered":"2024-09-16T10:32:03.201371+0000","last_clean":"2024-09-16T10:32:03.201371+0000","last_became_active":"2024-09-16T10:13:41.112454+0000","last_became_peered":"2024-09-16T10:13:41.112454+0000","last_unstale":"2024-09-16T10:32:03.201371+0000","last_undegraded":"2024-09-16T10:32:03.201371+0000","last_fullsized":"2024-09-16T10:32:03.201371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:27:24.280960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:27:24.280960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":843,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637984+0000","last_change":"2024-09-16T10:31:14.128649+0000","last_active":"2024-09-16T10:31:57.637984+0000","last_peered":"2024-09-16T10:31:57.637984+0000","last_clean":"2024-09-16T10:31:57.637984+0000","last_became_active":"2024-09-16T10:31:14.128195+0000","last_became_peered":"2024-09-16T10:31:14.128195+0000","last_unstale":"2024-09-16T10:31:57.637984+0000","last_undegraded":"2024-09-16T10:31:57.637984+0000","last_fullsized":"2024-09-16T10:31:57.637984+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.267597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.267597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"522'10323","reported_seq":10444,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:59.345053+0000","last_change":"2024-09-16T10:31:43.158588+0000","last_active":"2024-09-16T10:31:59.345053+0000","last_peered":"2024-09-16T10:31:59.345053+0000","last_clean":"2024-09-16T10:31:59.345053+0000","last_became_active":"2024-09-16T10:29:35.170400+0000","last_became_peered":"2024-09-16T10:29:35.170400+0000","last_unstale":"2024-09-16T10:31:59.345053+0000","last_undegraded":"2024-09-16T10:31:59.345053+0000","last_fullsized":"2024-09-16T10:31:59.345053+0000","mapping_epoch":476,"log_start":"498'5268","ondisk_log_start":"498'5268","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":1199,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:13.157612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319291416,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10323,"num_write_kb":486209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":954,"reported_epoch":522,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:31:57.638132+0000","last_change":"2024-09-16T10:31:45.129569+0000","last_active":"2024-09-16T10:31:57.638132+0000","last_peered":"2024-09-16T10:31:57.638132+0000","last_clean":"2024-09-16T10:31:08.647700+0000","last_became_active":"2024-09-16T10:31:15.127316+0000","last_became_peered":"2024-09-16T10:31:15.127316+0000","last_unstale":"2024-09-16T10:31:57.638132+0000","last_undegraded":"2024-09-16T10:31:57.638132+0000","last_fullsized":"2024-09-16T10:31:57.638132+0000","mapping_epoch":508,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.663450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.663450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,7],"avail_no_missing":["6","3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":1024,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.198021+0000","last_change":"2024-09-16T10:29:25.663951+0000","last_active":"2024-09-16T10:32:03.198021+0000","last_peered":"2024-09-16T10:32:03.198021+0000","last_clean":"2024-09-16T10:32:03.198021+0000","last_became_active":"2024-09-16T10:13:39.095438+0000","last_became_peered":"2024-09-16T10:13:39.095438+0000","last_unstale":"2024-09-16T10:32:03.198021+0000","last_undegraded":"2024-09-16T10:32:03.198021+0000","last_fullsized":"2024-09-16T10:32:03.198021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:12.241564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1002,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669756+0000","last_change":"2024-09-16T10:29:27.891418+0000","last_active":"2024-09-16T10:32:00.669756+0000","last_peered":"2024-09-16T10:32:00.669756+0000","last_clean":"2024-09-16T10:32:00.669756+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:32:00.669756+0000","last_undegraded":"2024-09-16T10:32:00.669756+0000","last_fullsized":"2024-09-16T10:32:00.669756+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:27.891362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.891362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1198,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641429+0000","last_change":"2024-09-16T10:30:08.183366+0000","last_active":"2024-09-16T10:31:57.641429+0000","last_peered":"2024-09-16T10:31:57.641429+0000","last_clean":"2024-09-16T10:31:57.641429+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:31:57.641429+0000","last_undegraded":"2024-09-16T10:31:57.641429+0000","last_fullsized":"2024-09-16T10:31:57.641429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:08.182949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.182949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":996,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.640671+0000","last_change":"2024-09-16T10:30:12.258492+0000","last_active":"2024-09-16T10:31:57.640671+0000","last_peered":"2024-09-16T10:31:57.640671+0000","last_clean":"2024-09-16T10:31:57.640671+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:31:57.640671+0000","last_undegraded":"2024-09-16T10:31:57.640671+0000","last_fullsized":"2024-09-16T10:31:57.640671+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:12.258426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.258426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"524'10793","reported_seq":10901,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.367499+0000","last_change":"2024-09-16T10:29:35.168446+0000","last_active":"2024-09-16T10:32:03.367499+0000","last_peered":"2024-09-16T10:32:03.367499+0000","last_clean":"2024-09-16T10:32:03.367499+0000","last_became_active":"2024-09-16T10:29:35.168175+0000","last_became_peered":"2024-09-16T10:29:35.168175+0000","last_unstale":"2024-09-16T10:32:03.367499+0000","last_undegraded":"2024-09-16T10:32:03.367499+0000","last_fullsized":"2024-09-16T10:32:03.367499+0000","mapping_epoch":476,"log_start":"504'7043","ondisk_log_start":"504'7043","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3750,"log_dups_size":0,"ondisk_log_size":3750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.127276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303497216,"num_objects":4631,"num_object_clones":0,"num_object_copies":13893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10793,"num_write_kb":493568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":937,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637915+0000","last_change":"2024-09-16T10:30:11.662236+0000","last_active":"2024-09-16T10:31:57.637915+0000","last_peered":"2024-09-16T10:31:57.637915+0000","last_clean":"2024-09-16T10:31:57.637915+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:31:57.637915+0000","last_undegraded":"2024-09-16T10:31:57.637915+0000","last_fullsized":"2024-09-16T10:31:57.637915+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:11.662185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.662185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":1068,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641492+0000","last_change":"2024-09-16T10:31:14.184131+0000","last_active":"2024-09-16T10:31:57.641492+0000","last_peered":"2024-09-16T10:31:57.641492+0000","last_clean":"2024-09-16T10:31:57.641492+0000","last_became_active":"2024-09-16T10:31:14.160400+0000","last_became_peered":"2024-09-16T10:31:14.160400+0000","last_unstale":"2024-09-16T10:31:57.641492+0000","last_undegraded":"2024-09-16T10:31:57.641492+0000","last_fullsized":"2024-09-16T10:31:57.641492+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:29:27.111715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.111715+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1004,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.271103+0000","last_change":"2024-09-16T10:29:26.686298+0000","last_active":"2024-09-16T10:32:03.271103+0000","last_peered":"2024-09-16T10:32:03.271103+0000","last_clean":"2024-09-16T10:32:03.271103+0000","last_became_active":"2024-09-16T10:22:02.896504+0000","last_became_peered":"2024-09-16T10:22:02.896504+0000","last_unstale":"2024-09-16T10:32:03.271103+0000","last_undegraded":"2024-09-16T10:32:03.271103+0000","last_fullsized":"2024-09-16T10:32:03.271103+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:15.187317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":993,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637578+0000","last_change":"2024-09-16T10:30:50.670023+0000","last_active":"2024-09-16T10:31:57.637578+0000","last_peered":"2024-09-16T10:31:57.637578+0000","last_clean":"2024-09-16T10:31:57.637578+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:31:57.637578+0000","last_undegraded":"2024-09-16T10:31:57.637578+0000","last_fullsized":"2024-09-16T10:31:57.637578+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.669938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.669938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:00.286307+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":984,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.269541+0000","last_change":"2024-09-16T10:27:41.587855+0000","last_active":"2024-09-16T10:32:03.269541+0000","last_peered":"2024-09-16T10:32:03.269541+0000","last_clean":"2024-09-16T10:32:03.269541+0000","last_became_active":"2024-09-16T10:22:02.891016+0000","last_became_peered":"2024-09-16T10:22:02.891016+0000","last_unstale":"2024-09-16T10:32:03.269541+0000","last_undegraded":"2024-09-16T10:32:03.269541+0000","last_fullsized":"2024-09-16T10:32:03.269541+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"524'10928","reported_seq":11037,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.451910+0000","last_change":"2024-09-16T10:29:35.171242+0000","last_active":"2024-09-16T10:32:03.451910+0000","last_peered":"2024-09-16T10:32:03.451910+0000","last_clean":"2024-09-16T10:32:03.451910+0000","last_became_active":"2024-09-16T10:29:35.171007+0000","last_became_peered":"2024-09-16T10:29:35.171007+0000","last_unstale":"2024-09-16T10:32:03.451910+0000","last_undegraded":"2024-09-16T10:32:03.451910+0000","last_fullsized":"2024-09-16T10:32:03.451910+0000","mapping_epoch":476,"log_start":"504'7035","ondisk_log_start":"504'7035","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3893,"log_dups_size":0,"ondisk_log_size":3893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300023808,"num_objects":4578,"num_object_clones":0,"num_object_copies":13734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10928,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":885,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668550+0000","last_change":"2024-09-16T10:30:54.802360+0000","last_active":"2024-09-16T10:32:00.668550+0000","last_peered":"2024-09-16T10:32:00.668550+0000","last_clean":"2024-09-16T10:32:00.668550+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:32:00.668550+0000","last_undegraded":"2024-09-16T10:32:00.668550+0000","last_fullsized":"2024-09-16T10:32:00.668550+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:54.802277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.802277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:03.521355+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"328'42","reported_seq":1348,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.270215+0000","last_change":"2024-09-16T10:27:14.409646+0000","last_active":"2024-09-16T10:32:03.270215+0000","last_peered":"2024-09-16T10:32:03.270215+0000","last_clean":"2024-09-16T10:32:03.270215+0000","last_became_active":"2024-09-16T10:22:02.908982+0000","last_became_peered":"2024-09-16T10:22:02.908982+0000","last_unstale":"2024-09-16T10:32:03.270215+0000","last_undegraded":"2024-09-16T10:32:03.270215+0000","last_fullsized":"2024-09-16T10:32:03.270215+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:27:14.409552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:27:14.409552+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":301,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641644+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:31:57.641644+0000","last_peered":"2024-09-16T10:31:57.641644+0000","last_clean":"2024-09-16T10:31:57.641644+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:31:57.641644+0000","last_undegraded":"2024-09-16T10:31:57.641644+0000","last_fullsized":"2024-09-16T10:31:57.641644+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":1006,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669165+0000","last_change":"2024-09-16T10:30:11.895955+0000","last_active":"2024-09-16T10:32:00.669165+0000","last_peered":"2024-09-16T10:32:00.669165+0000","last_clean":"2024-09-16T10:32:00.669165+0000","last_became_active":"2024-09-16T10:13:36.081259+0000","last_became_peered":"2024-09-16T10:13:36.081259+0000","last_unstale":"2024-09-16T10:32:00.669165+0000","last_undegraded":"2024-09-16T10:32:00.669165+0000","last_fullsized":"2024-09-16T10:32:00.669165+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:30:11.895746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.895746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":969,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.642022+0000","last_change":"2024-09-16T10:30:53.380048+0000","last_active":"2024-09-16T10:31:57.642022+0000","last_peered":"2024-09-16T10:31:57.642022+0000","last_clean":"2024-09-16T10:31:57.642022+0000","last_became_active":"2024-09-16T10:13:41.117826+0000","last_became_peered":"2024-09-16T10:13:41.117826+0000","last_unstale":"2024-09-16T10:31:57.642022+0000","last_undegraded":"2024-09-16T10:31:57.642022+0000","last_fullsized":"2024-09-16T10:31:57.642022+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.379980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.379980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1010,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202812+0000","last_change":"2024-09-16T10:31:14.143712+0000","last_active":"2024-09-16T10:32:03.202812+0000","last_peered":"2024-09-16T10:32:03.202812+0000","last_clean":"2024-09-16T10:32:03.202812+0000","last_became_active":"2024-09-16T10:31:14.143349+0000","last_became_peered":"2024-09-16T10:31:14.143349+0000","last_unstale":"2024-09-16T10:32:03.202812+0000","last_undegraded":"2024-09-16T10:32:03.202812+0000","last_fullsized":"2024-09-16T10:32:03.202812+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.877913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.877913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"524'11024","reported_seq":11133,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.457767+0000","last_change":"2024-09-16T10:29:35.167788+0000","last_active":"2024-09-16T10:32:03.457767+0000","last_peered":"2024-09-16T10:32:03.457767+0000","last_clean":"2024-09-16T10:32:03.457767+0000","last_became_active":"2024-09-16T10:29:35.167515+0000","last_became_peered":"2024-09-16T10:29:35.167515+0000","last_unstale":"2024-09-16T10:32:03.457767+0000","last_undegraded":"2024-09-16T10:32:03.457767+0000","last_fullsized":"2024-09-16T10:32:03.457767+0000","mapping_epoch":476,"log_start":"504'7084","ondisk_log_start":"504'7084","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3940,"log_dups_size":0,"ondisk_log_size":3940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305922048,"num_objects":4668,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":502144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":476,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.640869+0000","last_change":"2024-09-16T10:31:17.108542+0000","last_active":"2024-09-16T10:31:57.640869+0000","last_peered":"2024-09-16T10:31:57.640869+0000","last_clean":"2024-09-16T10:31:57.640869+0000","last_became_active":"2024-09-16T10:31:17.108022+0000","last_became_peered":"2024-09-16T10:31:17.108022+0000","last_unstale":"2024-09-16T10:31:57.640869+0000","last_undegraded":"2024-09-16T10:31:57.640869+0000","last_fullsized":"2024-09-16T10:31:57.640869+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:49.434225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.434225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:57.929492+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":1003,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.274549+0000","last_change":"2024-09-16T10:30:05.864251+0000","last_active":"2024-09-16T10:32:03.274549+0000","last_peered":"2024-09-16T10:32:03.274549+0000","last_clean":"2024-09-16T10:32:03.274549+0000","last_became_active":"2024-09-16T10:13:39.094363+0000","last_became_peered":"2024-09-16T10:13:39.094363+0000","last_unstale":"2024-09-16T10:32:03.274549+0000","last_undegraded":"2024-09-16T10:32:03.274549+0000","last_fullsized":"2024-09-16T10:32:03.274549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:30:05.864159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.864159+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":989,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.197459+0000","last_change":"2024-09-16T10:29:20.691013+0000","last_active":"2024-09-16T10:32:03.197459+0000","last_peered":"2024-09-16T10:32:03.197459+0000","last_clean":"2024-09-16T10:32:03.197459+0000","last_became_active":"2024-09-16T10:22:02.896281+0000","last_became_peered":"2024-09-16T10:22:02.896281+0000","last_unstale":"2024-09-16T10:32:03.197459+0000","last_undegraded":"2024-09-16T10:32:03.197459+0000","last_fullsized":"2024-09-16T10:32:03.197459+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":515,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.635917+0000","last_change":"2024-09-16T10:26:18.549126+0000","last_active":"2024-09-16T10:31:57.635917+0000","last_peered":"2024-09-16T10:31:57.635917+0000","last_clean":"2024-09-16T10:31:57.635917+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:31:57.635917+0000","last_undegraded":"2024-09-16T10:31:57.635917+0000","last_fullsized":"2024-09-16T10:31:57.635917+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:26:18.549005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:26:18.549005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.749930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":537,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638748+0000","last_change":"2024-09-16T10:31:22.862857+0000","last_active":"2024-09-16T10:31:57.638748+0000","last_peered":"2024-09-16T10:31:57.638748+0000","last_clean":"2024-09-16T10:31:57.638748+0000","last_became_active":"2024-09-16T10:31:14.142132+0000","last_became_peered":"2024-09-16T10:31:14.142132+0000","last_unstale":"2024-09-16T10:31:57.638748+0000","last_undegraded":"2024-09-16T10:31:57.638748+0000","last_fullsized":"2024-09-16T10:31:57.638748+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:22.862677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:22.862677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:23.084617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":391,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638282+0000","last_change":"2024-09-16T10:31:14.164501+0000","last_active":"2024-09-16T10:31:57.638282+0000","last_peered":"2024-09-16T10:31:57.638282+0000","last_clean":"2024-09-16T10:31:57.638282+0000","last_became_active":"2024-09-16T10:31:14.164133+0000","last_became_peered":"2024-09-16T10:31:14.164133+0000","last_unstale":"2024-09-16T10:31:57.638282+0000","last_undegraded":"2024-09-16T10:31:57.638282+0000","last_fullsized":"2024-09-16T10:31:57.638282+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.706111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.706111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:09.595073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"524'10844","reported_seq":10953,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.444062+0000","last_change":"2024-09-16T10:29:35.171912+0000","last_active":"2024-09-16T10:32:03.444062+0000","last_peered":"2024-09-16T10:32:03.444062+0000","last_clean":"2024-09-16T10:32:03.444062+0000","last_became_active":"2024-09-16T10:29:35.171449+0000","last_became_peered":"2024-09-16T10:29:35.171449+0000","last_unstale":"2024-09-16T10:32:03.444062+0000","last_undegraded":"2024-09-16T10:32:03.444062+0000","last_fullsized":"2024-09-16T10:32:03.444062+0000","mapping_epoch":476,"log_start":"504'6972","ondisk_log_start":"504'6972","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301858816,"num_objects":4606,"num_object_clones":0,"num_object_copies":13818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":494400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":875,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.271353+0000","last_change":"2024-09-16T10:30:16.866299+0000","last_active":"2024-09-16T10:32:03.271353+0000","last_peered":"2024-09-16T10:32:03.271353+0000","last_clean":"2024-09-16T10:32:03.271353+0000","last_became_active":"2024-09-16T10:22:56.419622+0000","last_became_peered":"2024-09-16T10:22:56.419622+0000","last_unstale":"2024-09-16T10:32:03.271353+0000","last_undegraded":"2024-09-16T10:32:03.271353+0000","last_fullsized":"2024-09-16T10:32:03.271353+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:16.862456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.862456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":1083,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.642428+0000","last_change":"2024-09-16T10:30:20.268496+0000","last_active":"2024-09-16T10:31:57.642428+0000","last_peered":"2024-09-16T10:31:57.642428+0000","last_clean":"2024-09-16T10:31:57.642428+0000","last_became_active":"2024-09-16T10:22:02.897184+0000","last_became_peered":"2024-09-16T10:22:02.897184+0000","last_unstale":"2024-09-16T10:31:57.642428+0000","last_undegraded":"2024-09-16T10:31:57.642428+0000","last_fullsized":"2024-09-16T10:31:57.642428+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:30:20.268438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.268438+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1009,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641435+0000","last_change":"2024-09-16T10:30:15.229737+0000","last_active":"2024-09-16T10:31:57.641435+0000","last_peered":"2024-09-16T10:31:57.641435+0000","last_clean":"2024-09-16T10:31:57.641435+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:31:57.641435+0000","last_undegraded":"2024-09-16T10:31:57.641435+0000","last_fullsized":"2024-09-16T10:31:57.641435+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.229669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.229669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":861,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.197710+0000","last_change":"2024-09-16T10:31:14.176415+0000","last_active":"2024-09-16T10:32:03.197710+0000","last_peered":"2024-09-16T10:32:03.197710+0000","last_clean":"2024-09-16T10:32:03.197710+0000","last_became_active":"2024-09-16T10:31:14.148770+0000","last_became_peered":"2024-09-16T10:31:14.148770+0000","last_unstale":"2024-09-16T10:32:03.197710+0000","last_undegraded":"2024-09-16T10:32:03.197710+0000","last_fullsized":"2024-09-16T10:32:03.197710+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:30:39.226399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.226399+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":420,"num_read_kb":415,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":996,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669830+0000","last_change":"2024-09-16T10:30:08.854974+0000","last_active":"2024-09-16T10:32:00.669830+0000","last_peered":"2024-09-16T10:32:00.669830+0000","last_clean":"2024-09-16T10:32:00.669830+0000","last_became_active":"2024-09-16T10:13:41.112492+0000","last_became_peered":"2024-09-16T10:13:41.112492+0000","last_unstale":"2024-09-16T10:32:00.669830+0000","last_undegraded":"2024-09-16T10:32:00.669830+0000","last_fullsized":"2024-09-16T10:32:00.669830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.854924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.854924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":994,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.270048+0000","last_change":"2024-09-16T10:27:19.374485+0000","last_active":"2024-09-16T10:32:03.270048+0000","last_peered":"2024-09-16T10:32:03.270048+0000","last_clean":"2024-09-16T10:32:03.270048+0000","last_became_active":"2024-09-16T10:13:43.122676+0000","last_became_peered":"2024-09-16T10:13:43.122676+0000","last_unstale":"2024-09-16T10:32:03.270048+0000","last_undegraded":"2024-09-16T10:32:03.270048+0000","last_fullsized":"2024-09-16T10:32:03.270048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"524'10938","reported_seq":11045,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.445266+0000","last_change":"2024-09-16T10:29:35.169178+0000","last_active":"2024-09-16T10:32:03.445266+0000","last_peered":"2024-09-16T10:32:03.445266+0000","last_clean":"2024-09-16T10:32:03.445266+0000","last_became_active":"2024-09-16T10:29:35.168851+0000","last_became_peered":"2024-09-16T10:29:35.168851+0000","last_unstale":"2024-09-16T10:32:03.445266+0000","last_undegraded":"2024-09-16T10:32:03.445266+0000","last_fullsized":"2024-09-16T10:32:03.445266+0000","mapping_epoch":476,"log_start":"504'7014","ondisk_log_start":"504'7014","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300154880,"num_objects":4580,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10938,"num_write_kb":496576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"508'31","reported_seq":1272,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638418+0000","last_change":"2024-09-16T10:31:14.178826+0000","last_active":"2024-09-16T10:31:57.638418+0000","last_peered":"2024-09-16T10:31:57.638418+0000","last_clean":"2024-09-16T10:31:57.638418+0000","last_became_active":"2024-09-16T10:31:14.137441+0000","last_became_peered":"2024-09-16T10:31:14.137441+0000","last_unstale":"2024-09-16T10:31:57.638418+0000","last_undegraded":"2024-09-16T10:31:57.638418+0000","last_fullsized":"2024-09-16T10:31:57.638418+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:30:54.737043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.737043+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:11.390039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":953,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202565+0000","last_change":"2024-09-16T10:26:47.493915+0000","last_active":"2024-09-16T10:32:03.202565+0000","last_peered":"2024-09-16T10:32:03.202565+0000","last_clean":"2024-09-16T10:32:03.202565+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:32:03.202565+0000","last_undegraded":"2024-09-16T10:32:03.202565+0000","last_fullsized":"2024-09-16T10:32:03.202565+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:47.493848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:47.493848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":476,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.640877+0000","last_change":"2024-09-16T10:31:17.118674+0000","last_active":"2024-09-16T10:31:57.640877+0000","last_peered":"2024-09-16T10:31:57.640877+0000","last_clean":"2024-09-16T10:31:57.640877+0000","last_became_active":"2024-09-16T10:31:17.118308+0000","last_became_peered":"2024-09-16T10:31:17.118308+0000","last_unstale":"2024-09-16T10:31:57.640877+0000","last_undegraded":"2024-09-16T10:31:57.640877+0000","last_fullsized":"2024-09-16T10:31:57.640877+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:50.410561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.410561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:12.776555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":862,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.639200+0000","last_change":"2024-09-16T10:31:14.146544+0000","last_active":"2024-09-16T10:31:57.639200+0000","last_peered":"2024-09-16T10:31:57.639200+0000","last_clean":"2024-09-16T10:31:57.639200+0000","last_became_active":"2024-09-16T10:31:14.145333+0000","last_became_peered":"2024-09-16T10:31:14.145333+0000","last_unstale":"2024-09-16T10:31:57.639200+0000","last_undegraded":"2024-09-16T10:31:57.639200+0000","last_fullsized":"2024-09-16T10:31:57.639200+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.613780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.613780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:01.567578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":984,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669744+0000","last_change":"2024-09-16T10:30:43.705706+0000","last_active":"2024-09-16T10:32:00.669744+0000","last_peered":"2024-09-16T10:32:00.669744+0000","last_clean":"2024-09-16T10:32:00.669744+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:32:00.669744+0000","last_undegraded":"2024-09-16T10:32:00.669744+0000","last_fullsized":"2024-09-16T10:32:00.669744+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.705640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.705640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"522'10584","reported_seq":10754,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.625373+0000","last_change":"2024-09-16T10:30:40.160552+0000","last_active":"2024-09-16T10:31:58.625373+0000","last_peered":"2024-09-16T10:31:58.625373+0000","last_clean":"2024-09-16T10:31:58.625373+0000","last_became_active":"2024-09-16T10:29:35.170866+0000","last_became_peered":"2024-09-16T10:29:35.170866+0000","last_unstale":"2024-09-16T10:31:58.625373+0000","last_undegraded":"2024-09-16T10:31:58.625373+0000","last_fullsized":"2024-09-16T10:31:58.625373+0000","mapping_epoch":476,"log_start":"502'6234","ondisk_log_start":"502'6234","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'5743","last_scrub_stamp":"2024-09-16T10:30:40.160478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.160478+0000","objects_scrubbed":5687,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.130078+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327155712,"num_objects":4992,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":498432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":875,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.270231+0000","last_change":"2024-09-16T10:29:27.703960+0000","last_active":"2024-09-16T10:32:03.270231+0000","last_peered":"2024-09-16T10:32:03.270231+0000","last_clean":"2024-09-16T10:32:03.270231+0000","last_became_active":"2024-09-16T10:23:00.925713+0000","last_became_peered":"2024-09-16T10:23:00.925713+0000","last_unstale":"2024-09-16T10:32:03.270231+0000","last_undegraded":"2024-09-16T10:32:03.270231+0000","last_fullsized":"2024-09-16T10:32:03.270231+0000","mapping_epoch":311,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:27.703908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:27.703908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":914,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638280+0000","last_change":"2024-09-16T10:26:28.556079+0000","last_active":"2024-09-16T10:31:57.638280+0000","last_peered":"2024-09-16T10:31:57.638280+0000","last_clean":"2024-09-16T10:31:57.638280+0000","last_became_active":"2024-09-16T10:26:28.554604+0000","last_became_peered":"2024-09-16T10:26:28.554604+0000","last_unstale":"2024-09-16T10:31:57.638280+0000","last_undegraded":"2024-09-16T10:31:57.638280+0000","last_fullsized":"2024-09-16T10:31:57.638280+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:26:10.386128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.386128+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.928740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":806,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.635586+0000","last_change":"2024-09-16T10:31:30.867997+0000","last_active":"2024-09-16T10:31:57.635586+0000","last_peered":"2024-09-16T10:31:57.635586+0000","last_clean":"2024-09-16T10:31:57.635586+0000","last_became_active":"2024-09-16T10:31:14.134560+0000","last_became_peered":"2024-09-16T10:31:14.134560+0000","last_unstale":"2024-09-16T10:31:57.635586+0000","last_undegraded":"2024-09-16T10:31:57.635586+0000","last_fullsized":"2024-09-16T10:31:57.635586+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:30.867930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:30.867930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:48.959991+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":988,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669691+0000","last_change":"2024-09-16T10:29:30.955964+0000","last_active":"2024-09-16T10:32:00.669691+0000","last_peered":"2024-09-16T10:32:00.669691+0000","last_clean":"2024-09-16T10:32:00.669691+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:32:00.669691+0000","last_undegraded":"2024-09-16T10:32:00.669691+0000","last_fullsized":"2024-09-16T10:32:00.669691+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:30.955877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:30.955877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":925,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201716+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:32:03.201716+0000","last_peered":"2024-09-16T10:32:03.201716+0000","last_clean":"2024-09-16T10:32:03.201716+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:32:03.201716+0000","last_undegraded":"2024-09-16T10:32:03.201716+0000","last_fullsized":"2024-09-16T10:32:03.201716+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"522'10616","reported_seq":10719,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:59.338957+0000","last_change":"2024-09-16T10:29:35.171033+0000","last_active":"2024-09-16T10:31:59.338957+0000","last_peered":"2024-09-16T10:31:59.338957+0000","last_clean":"2024-09-16T10:31:59.338957+0000","last_became_active":"2024-09-16T10:29:35.170730+0000","last_became_peered":"2024-09-16T10:29:35.170730+0000","last_unstale":"2024-09-16T10:31:59.338957+0000","last_undegraded":"2024-09-16T10:31:59.338957+0000","last_fullsized":"2024-09-16T10:31:59.338957+0000","mapping_epoch":476,"log_start":"499'5527","ondisk_log_start":"499'5527","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:44.843230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318111744,"num_objects":4854,"num_object_clones":0,"num_object_copies":14562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10616,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":848,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668458+0000","last_change":"2024-09-16T10:30:05.823698+0000","last_active":"2024-09-16T10:32:00.668458+0000","last_peered":"2024-09-16T10:32:00.668458+0000","last_clean":"2024-09-16T10:32:00.668458+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:32:00.668458+0000","last_undegraded":"2024-09-16T10:32:00.668458+0000","last_fullsized":"2024-09-16T10:32:00.668458+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:05.823644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.823644+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":986,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638306+0000","last_change":"2024-09-16T10:31:18.771352+0000","last_active":"2024-09-16T10:31:57.638306+0000","last_peered":"2024-09-16T10:31:57.638306+0000","last_clean":"2024-09-16T10:31:57.638306+0000","last_became_active":"2024-09-16T10:26:28.553171+0000","last_became_peered":"2024-09-16T10:26:28.553171+0000","last_unstale":"2024-09-16T10:31:57.638306+0000","last_undegraded":"2024-09-16T10:31:57.638306+0000","last_fullsized":"2024-09-16T10:31:57.638306+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:31:18.771268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:31:18.771268+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:26.557862+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":985,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637959+0000","last_change":"2024-09-16T10:30:03.609790+0000","last_active":"2024-09-16T10:31:57.637959+0000","last_peered":"2024-09-16T10:31:57.637959+0000","last_clean":"2024-09-16T10:31:57.637959+0000","last_became_active":"2024-09-16T10:13:37.088850+0000","last_became_peered":"2024-09-16T10:13:37.088850+0000","last_unstale":"2024-09-16T10:31:57.637959+0000","last_undegraded":"2024-09-16T10:31:57.637959+0000","last_fullsized":"2024-09-16T10:31:57.637959+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.609728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.609728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":974,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.270114+0000","last_change":"2024-09-16T10:29:21.720973+0000","last_active":"2024-09-16T10:32:03.270114+0000","last_peered":"2024-09-16T10:32:03.270114+0000","last_clean":"2024-09-16T10:32:03.270114+0000","last_became_active":"2024-09-16T10:13:41.121056+0000","last_became_peered":"2024-09-16T10:13:41.121056+0000","last_unstale":"2024-09-16T10:32:03.270114+0000","last_undegraded":"2024-09-16T10:32:03.270114+0000","last_fullsized":"2024-09-16T10:32:03.270114+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.720910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.720910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.090647+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":519,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.639387+0000","last_change":"2024-09-16T10:30:40.613125+0000","last_active":"2024-09-16T10:31:57.639387+0000","last_peered":"2024-09-16T10:31:57.639387+0000","last_clean":"2024-09-16T10:31:57.639387+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:31:57.639387+0000","last_undegraded":"2024-09-16T10:31:57.639387+0000","last_fullsized":"2024-09-16T10:31:57.639387+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:40.612878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.612878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:00.522715+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"522'10850","reported_seq":25187,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.660504+0000","last_change":"2024-09-16T10:31:37.637280+0000","last_active":"2024-09-16T10:31:58.660504+0000","last_peered":"2024-09-16T10:31:58.660504+0000","last_clean":"2024-09-16T10:31:58.660504+0000","last_became_active":"2024-09-16T10:31:37.625225+0000","last_became_peered":"2024-09-16T10:31:37.625225+0000","last_unstale":"2024-09-16T10:31:58.660504+0000","last_undegraded":"2024-09-16T10:31:58.660504+0000","last_fullsized":"2024-09-16T10:31:58.660504+0000","mapping_epoch":514,"log_start":"502'6415","ondisk_log_start":"502'6415","created":476,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'7025","last_scrub_stamp":"2024-09-16T10:30:57.490342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:30:57.490342+0000","objects_scrubbed":6969,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.448746+0000","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332398592,"num_objects":5072,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10850,"num_write_kb":509504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7066,"num_bytes_recovered":462946304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":670,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202438+0000","last_change":"2024-09-16T10:31:14.142529+0000","last_active":"2024-09-16T10:32:03.202438+0000","last_peered":"2024-09-16T10:32:03.202438+0000","last_clean":"2024-09-16T10:32:03.202438+0000","last_became_active":"2024-09-16T10:31:14.140156+0000","last_became_peered":"2024-09-16T10:31:14.140156+0000","last_unstale":"2024-09-16T10:32:03.202438+0000","last_undegraded":"2024-09-16T10:32:03.202438+0000","last_fullsized":"2024-09-16T10:32:03.202438+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:07.951845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.951845+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":898,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200802+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:32:03.200802+0000","last_peered":"2024-09-16T10:32:03.200802+0000","last_clean":"2024-09-16T10:32:03.200802+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:32:03.200802+0000","last_undegraded":"2024-09-16T10:32:03.200802+0000","last_fullsized":"2024-09-16T10:32:03.200802+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":972,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.205193+0000","last_change":"2024-09-16T10:27:12.155297+0000","last_active":"2024-09-16T10:32:03.205193+0000","last_peered":"2024-09-16T10:32:03.205193+0000","last_clean":"2024-09-16T10:32:03.205193+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:32:03.205193+0000","last_undegraded":"2024-09-16T10:32:03.205193+0000","last_fullsized":"2024-09-16T10:32:03.205193+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:12.155186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:12.155186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"508'13","reported_seq":667,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638650+0000","last_change":"2024-09-16T10:31:27.828520+0000","last_active":"2024-09-16T10:31:57.638650+0000","last_peered":"2024-09-16T10:31:57.638650+0000","last_clean":"2024-09-16T10:31:57.638650+0000","last_became_active":"2024-09-16T10:31:14.138957+0000","last_became_peered":"2024-09-16T10:31:14.138957+0000","last_unstale":"2024-09-16T10:31:57.638650+0000","last_undegraded":"2024-09-16T10:31:57.638650+0000","last_fullsized":"2024-09-16T10:31:57.638650+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'13","last_scrub_stamp":"2024-09-16T10:31:27.828439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:27.828439+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:43.708396+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":789,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.273486+0000","last_change":"2024-09-16T10:30:22.007418+0000","last_active":"2024-09-16T10:32:03.273486+0000","last_peered":"2024-09-16T10:32:03.273486+0000","last_clean":"2024-09-16T10:32:03.273486+0000","last_became_active":"2024-09-16T10:26:28.550959+0000","last_became_peered":"2024-09-16T10:26:28.550959+0000","last_unstale":"2024-09-16T10:32:03.273486+0000","last_undegraded":"2024-09-16T10:32:03.273486+0000","last_fullsized":"2024-09-16T10:32:03.273486+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:22.007341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:22.007341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"522'10345","reported_seq":10466,"reported_epoch":522,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:31:58.667316+0000","last_change":"2024-09-16T10:31:45.262889+0000","last_active":"2024-09-16T10:31:58.667316+0000","last_peered":"2024-09-16T10:31:58.667316+0000","last_clean":"2024-09-16T10:31:13.061686+0000","last_became_active":"2024-09-16T10:31:15.212576+0000","last_became_peered":"2024-09-16T10:31:15.212576+0000","last_unstale":"2024-09-16T10:31:58.667316+0000","last_undegraded":"2024-09-16T10:31:58.667316+0000","last_fullsized":"2024-09-16T10:31:58.667316+0000","mapping_epoch":508,"log_start":"501'5951","ondisk_log_start":"501'5951","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320274432,"num_objects":4887,"num_object_clones":0,"num_object_copies":14661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4887,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10345,"num_write_kb":487424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[{"shards":"2,6,7","objects":4887}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":254,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201345+0000","last_change":"2024-09-16T10:31:48.588720+0000","last_active":"2024-09-16T10:32:03.201345+0000","last_peered":"2024-09-16T10:32:03.201345+0000","last_clean":"2024-09-16T10:32:03.201345+0000","last_became_active":"2024-09-16T10:31:17.119269+0000","last_became_peered":"2024-09-16T10:31:17.119269+0000","last_unstale":"2024-09-16T10:32:03.201345+0000","last_undegraded":"2024-09-16T10:32:03.201345+0000","last_fullsized":"2024-09-16T10:32:03.201345+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.900551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.900551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T10:32:17.238086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":1006,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641381+0000","last_change":"2024-09-16T10:30:19.209404+0000","last_active":"2024-09-16T10:31:57.641381+0000","last_peered":"2024-09-16T10:31:57.641381+0000","last_clean":"2024-09-16T10:31:57.641381+0000","last_became_active":"2024-09-16T10:13:39.091427+0000","last_became_peered":"2024-09-16T10:13:39.091427+0000","last_unstale":"2024-09-16T10:31:57.641381+0000","last_undegraded":"2024-09-16T10:31:57.641381+0000","last_fullsized":"2024-09-16T10:31:57.641381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:30:19.209335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:19.209335+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":996,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669277+0000","last_change":"2024-09-16T10:26:44.070285+0000","last_active":"2024-09-16T10:32:00.669277+0000","last_peered":"2024-09-16T10:32:00.669277+0000","last_clean":"2024-09-16T10:32:00.669277+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:32:00.669277+0000","last_undegraded":"2024-09-16T10:32:00.669277+0000","last_fullsized":"2024-09-16T10:32:00.669277+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:44.070218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:44.070218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:05.953782+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":511,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638930+0000","last_change":"2024-09-16T10:26:22.501028+0000","last_active":"2024-09-16T10:31:57.638930+0000","last_peered":"2024-09-16T10:31:57.638930+0000","last_clean":"2024-09-16T10:31:57.638930+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:31:57.638930+0000","last_undegraded":"2024-09-16T10:31:57.638930+0000","last_fullsized":"2024-09-16T10:31:57.638930+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:22.500963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:22.500963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:09.764402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":1032,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641439+0000","last_change":"2024-09-16T10:31:14.161646+0000","last_active":"2024-09-16T10:31:57.641439+0000","last_peered":"2024-09-16T10:31:57.641439+0000","last_clean":"2024-09-16T10:31:57.641439+0000","last_became_active":"2024-09-16T10:31:14.152023+0000","last_became_peered":"2024-09-16T10:31:14.152023+0000","last_unstale":"2024-09-16T10:31:57.641439+0000","last_undegraded":"2024-09-16T10:31:57.641439+0000","last_fullsized":"2024-09-16T10:31:57.641439+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:05.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.174161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"523'10817","reported_seq":23337,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.045771+0000","last_change":"2024-09-16T10:31:56.657757+0000","last_active":"2024-09-16T10:32:03.045771+0000","last_peered":"2024-09-16T10:32:03.045771+0000","last_clean":"2024-09-16T10:32:03.045771+0000","last_became_active":"2024-09-16T10:31:56.641359+0000","last_became_peered":"2024-09-16T10:31:56.641359+0000","last_unstale":"2024-09-16T10:32:03.045771+0000","last_undegraded":"2024-09-16T10:32:03.045771+0000","last_fullsized":"2024-09-16T10:32:03.045771+0000","mapping_epoch":520,"log_start":"502'6499","ondisk_log_start":"502'6499","created":476,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4318,"log_dups_size":0,"ondisk_log_size":4318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305201152,"num_objects":4657,"num_object_clones":0,"num_object_copies":13971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10817,"num_write_kb":495168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6198,"num_bytes_recovered":406192128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":950,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638429+0000","last_change":"2024-09-16T10:30:42.550612+0000","last_active":"2024-09-16T10:31:57.638429+0000","last_peered":"2024-09-16T10:31:57.638429+0000","last_clean":"2024-09-16T10:31:57.638429+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:31:57.638429+0000","last_undegraded":"2024-09-16T10:31:57.638429+0000","last_fullsized":"2024-09-16T10:31:57.638429+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:42.550567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.550567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":1082,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637661+0000","last_change":"2024-09-16T10:31:14.185001+0000","last_active":"2024-09-16T10:31:57.637661+0000","last_peered":"2024-09-16T10:31:57.637661+0000","last_clean":"2024-09-16T10:31:57.637661+0000","last_became_active":"2024-09-16T10:31:14.141161+0000","last_became_peered":"2024-09-16T10:31:14.141161+0000","last_unstale":"2024-09-16T10:31:57.637661+0000","last_undegraded":"2024-09-16T10:31:57.637661+0000","last_fullsized":"2024-09-16T10:31:57.637661+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:56.838892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":392,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669447+0000","last_change":"2024-09-16T10:31:14.148058+0000","last_active":"2024-09-16T10:32:00.669447+0000","last_peered":"2024-09-16T10:32:00.669447+0000","last_clean":"2024-09-16T10:32:00.669447+0000","last_became_active":"2024-09-16T10:31:14.147606+0000","last_became_peered":"2024-09-16T10:31:14.147606+0000","last_unstale":"2024-09-16T10:32:00.669447+0000","last_undegraded":"2024-09-16T10:32:00.669447+0000","last_fullsized":"2024-09-16T10:32:00.669447+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.745214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.745214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":511,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638660+0000","last_change":"2024-09-16T10:26:43.199824+0000","last_active":"2024-09-16T10:31:57.638660+0000","last_peered":"2024-09-16T10:31:57.638660+0000","last_clean":"2024-09-16T10:31:57.638660+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:31:57.638660+0000","last_undegraded":"2024-09-16T10:31:57.638660+0000","last_fullsized":"2024-09-16T10:31:57.638660+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:43.199752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:43.199752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:17.774662+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":866,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.635273+0000","last_change":"2024-09-16T10:31:14.148903+0000","last_active":"2024-09-16T10:31:57.635273+0000","last_peered":"2024-09-16T10:31:57.635273+0000","last_clean":"2024-09-16T10:31:57.635273+0000","last_became_active":"2024-09-16T10:31:14.148572+0000","last_became_peered":"2024-09-16T10:31:14.148572+0000","last_unstale":"2024-09-16T10:31:57.635273+0000","last_undegraded":"2024-09-16T10:31:57.635273+0000","last_fullsized":"2024-09-16T10:31:57.635273+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.626720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.626720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:15.346115+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"522'10792","reported_seq":10897,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.126716+0000","last_change":"2024-09-16T10:29:35.172946+0000","last_active":"2024-09-16T10:32:00.126716+0000","last_peered":"2024-09-16T10:32:00.126716+0000","last_clean":"2024-09-16T10:32:00.126716+0000","last_became_active":"2024-09-16T10:29:35.172649+0000","last_became_peered":"2024-09-16T10:29:35.172649+0000","last_unstale":"2024-09-16T10:32:00.126716+0000","last_undegraded":"2024-09-16T10:32:00.126716+0000","last_fullsized":"2024-09-16T10:32:00.126716+0000","mapping_epoch":476,"log_start":"502'6523","ondisk_log_start":"502'6523","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4269,"log_dups_size":0,"ondisk_log_size":4269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:04.288167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321912832,"num_objects":4912,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":502528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":897,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200481+0000","last_change":"2024-09-16T10:26:10.271093+0000","last_active":"2024-09-16T10:32:03.200481+0000","last_peered":"2024-09-16T10:32:03.200481+0000","last_clean":"2024-09-16T10:32:03.200481+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:32:03.200481+0000","last_undegraded":"2024-09-16T10:32:03.200481+0000","last_fullsized":"2024-09-16T10:32:03.200481+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:10.270833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:10.270833+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:04.211104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":914,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.269646+0000","last_change":"2024-09-16T10:30:20.961973+0000","last_active":"2024-09-16T10:32:03.269646+0000","last_peered":"2024-09-16T10:32:03.269646+0000","last_clean":"2024-09-16T10:32:03.269646+0000","last_became_active":"2024-09-16T10:26:28.553750+0000","last_became_peered":"2024-09-16T10:26:28.553750+0000","last_unstale":"2024-09-16T10:32:03.269646+0000","last_undegraded":"2024-09-16T10:32:03.269646+0000","last_fullsized":"2024-09-16T10:32:03.269646+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:30:20.961894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.961894+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669418+0000","last_change":"2024-09-16T10:31:14.141250+0000","last_active":"2024-09-16T10:32:00.669418+0000","last_peered":"2024-09-16T10:32:00.669418+0000","last_clean":"2024-09-16T10:32:00.669418+0000","last_became_active":"2024-09-16T10:31:14.140960+0000","last_became_peered":"2024-09-16T10:31:14.140960+0000","last_unstale":"2024-09-16T10:32:00.669418+0000","last_undegraded":"2024-09-16T10:32:00.669418+0000","last_fullsized":"2024-09-16T10:32:00.669418+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:56.301513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":519,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.635368+0000","last_change":"2024-09-16T10:30:52.624617+0000","last_active":"2024-09-16T10:31:57.635368+0000","last_peered":"2024-09-16T10:31:57.635368+0000","last_clean":"2024-09-16T10:31:57.635368+0000","last_became_active":"2024-09-16T10:22:02.893682+0000","last_became_peered":"2024-09-16T10:22:02.893682+0000","last_unstale":"2024-09-16T10:31:57.635368+0000","last_undegraded":"2024-09-16T10:31:57.635368+0000","last_fullsized":"2024-09-16T10:31:57.635368+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.624569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.624569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:31:59.763599+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1002,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200498+0000","last_change":"2024-09-16T10:31:14.149294+0000","last_active":"2024-09-16T10:32:03.200498+0000","last_peered":"2024-09-16T10:32:03.200498+0000","last_clean":"2024-09-16T10:32:03.200498+0000","last_became_active":"2024-09-16T10:31:14.148667+0000","last_became_peered":"2024-09-16T10:31:14.148667+0000","last_unstale":"2024-09-16T10:32:03.200498+0000","last_undegraded":"2024-09-16T10:32:03.200498+0000","last_fullsized":"2024-09-16T10:32:03.200498+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.230039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.230039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:14.325776+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"523'10842","reported_seq":10949,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.054835+0000","last_change":"2024-09-16T10:29:35.165138+0000","last_active":"2024-09-16T10:32:03.054835+0000","last_peered":"2024-09-16T10:32:03.054835+0000","last_clean":"2024-09-16T10:32:03.054835+0000","last_became_active":"2024-09-16T10:29:35.164843+0000","last_became_peered":"2024-09-16T10:29:35.164843+0000","last_unstale":"2024-09-16T10:32:03.054835+0000","last_undegraded":"2024-09-16T10:32:03.054835+0000","last_fullsized":"2024-09-16T10:32:03.054835+0000","mapping_epoch":476,"log_start":"502'6581","ondisk_log_start":"502'6581","created":476,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:29:34.146480+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.146480+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300154880,"num_objects":4580,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":723,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.269924+0000","last_change":"2024-09-16T10:31:17.114223+0000","last_active":"2024-09-16T10:32:03.269924+0000","last_peered":"2024-09-16T10:32:03.269924+0000","last_clean":"2024-09-16T10:32:03.269924+0000","last_became_active":"2024-09-16T10:31:17.113692+0000","last_became_peered":"2024-09-16T10:31:17.113692+0000","last_unstale":"2024-09-16T10:32:03.269924+0000","last_undegraded":"2024-09-16T10:32:03.269924+0000","last_fullsized":"2024-09-16T10:32:03.269924+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":1053,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668624+0000","last_change":"2024-09-16T10:31:14.162215+0000","last_active":"2024-09-16T10:32:00.668624+0000","last_peered":"2024-09-16T10:32:00.668624+0000","last_clean":"2024-09-16T10:32:00.668624+0000","last_became_active":"2024-09-16T10:31:14.136662+0000","last_became_peered":"2024-09-16T10:31:14.136662+0000","last_unstale":"2024-09-16T10:32:00.668624+0000","last_undegraded":"2024-09-16T10:32:00.668624+0000","last_fullsized":"2024-09-16T10:32:00.668624+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.825936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.825936+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:17.528711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":327,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.643402+0000","last_change":"2024-09-16T10:31:14.162556+0000","last_active":"2024-09-16T10:31:57.643402+0000","last_peered":"2024-09-16T10:31:57.643402+0000","last_clean":"2024-09-16T10:31:57.643402+0000","last_became_active":"2024-09-16T10:31:14.149571+0000","last_became_peered":"2024-09-16T10:31:14.149571+0000","last_unstale":"2024-09-16T10:31:57.643402+0000","last_undegraded":"2024-09-16T10:31:57.643402+0000","last_fullsized":"2024-09-16T10:31:57.643402+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:26:08.381010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:08.381010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"508'12","reported_seq":506,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637526+0000","last_change":"2024-09-16T10:31:14.166453+0000","last_active":"2024-09-16T10:31:57.637526+0000","last_peered":"2024-09-16T10:31:57.637526+0000","last_clean":"2024-09-16T10:31:57.637526+0000","last_became_active":"2024-09-16T10:31:14.131452+0000","last_became_peered":"2024-09-16T10:31:14.131452+0000","last_unstale":"2024-09-16T10:31:57.637526+0000","last_undegraded":"2024-09-16T10:31:57.637526+0000","last_fullsized":"2024-09-16T10:31:57.637526+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'11","last_scrub_stamp":"2024-09-16T10:30:53.694883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.694883+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:02.163955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":992,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.668879+0000","last_change":"2024-09-16T10:30:07.792293+0000","last_active":"2024-09-16T10:32:00.668879+0000","last_peered":"2024-09-16T10:32:00.668879+0000","last_clean":"2024-09-16T10:32:00.668879+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:32:00.668879+0000","last_undegraded":"2024-09-16T10:32:00.668879+0000","last_fullsized":"2024-09-16T10:32:00.668879+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.792060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.792060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":560,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669033+0000","last_change":"2024-09-16T10:29:28.935051+0000","last_active":"2024-09-16T10:32:00.669033+0000","last_peered":"2024-09-16T10:32:00.669033+0000","last_clean":"2024-09-16T10:32:00.669033+0000","last_became_active":"2024-09-16T10:25:00.671983+0000","last_became_peered":"2024-09-16T10:25:00.671983+0000","last_unstale":"2024-09-16T10:32:00.669033+0000","last_undegraded":"2024-09-16T10:32:00.669033+0000","last_fullsized":"2024-09-16T10:32:00.669033+0000","mapping_epoch":350,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:28.935001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:28.935001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":968,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637960+0000","last_change":"2024-09-16T10:26:20.469286+0000","last_active":"2024-09-16T10:31:57.637960+0000","last_peered":"2024-09-16T10:31:57.637960+0000","last_clean":"2024-09-16T10:31:57.637960+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:31:57.637960+0000","last_undegraded":"2024-09-16T10:31:57.637960+0000","last_fullsized":"2024-09-16T10:31:57.637960+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:20.469240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.469240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:03.848934+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":1027,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202465+0000","last_change":"2024-09-16T10:31:14.188538+0000","last_active":"2024-09-16T10:32:03.202465+0000","last_peered":"2024-09-16T10:32:03.202465+0000","last_clean":"2024-09-16T10:32:03.202465+0000","last_became_active":"2024-09-16T10:31:14.154165+0000","last_became_peered":"2024-09-16T10:31:14.154165+0000","last_unstale":"2024-09-16T10:32:03.202465+0000","last_undegraded":"2024-09-16T10:32:03.202465+0000","last_fullsized":"2024-09-16T10:32:03.202465+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:30:50.226333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.226333+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":971,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669595+0000","last_change":"2024-09-16T10:26:46.061085+0000","last_active":"2024-09-16T10:32:00.669595+0000","last_peered":"2024-09-16T10:32:00.669595+0000","last_clean":"2024-09-16T10:32:00.669595+0000","last_became_active":"2024-09-16T10:13:41.113339+0000","last_became_peered":"2024-09-16T10:13:41.113339+0000","last_unstale":"2024-09-16T10:32:00.669595+0000","last_undegraded":"2024-09-16T10:32:00.669595+0000","last_fullsized":"2024-09-16T10:32:00.669595+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":666,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637901+0000","last_change":"2024-09-16T10:31:14.137702+0000","last_active":"2024-09-16T10:31:57.637901+0000","last_peered":"2024-09-16T10:31:57.637901+0000","last_clean":"2024-09-16T10:31:57.637901+0000","last_became_active":"2024-09-16T10:31:14.136703+0000","last_became_peered":"2024-09-16T10:31:14.136703+0000","last_unstale":"2024-09-16T10:31:57.637901+0000","last_undegraded":"2024-09-16T10:31:57.637901+0000","last_fullsized":"2024-09-16T10:31:57.637901+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:39.865342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.865342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":883,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.271354+0000","last_change":"2024-09-16T10:30:48.209120+0000","last_active":"2024-09-16T10:32:03.271354+0000","last_peered":"2024-09-16T10:32:03.271354+0000","last_clean":"2024-09-16T10:32:03.271354+0000","last_became_active":"2024-09-16T10:23:31.479899+0000","last_became_peered":"2024-09-16T10:23:31.479899+0000","last_unstale":"2024-09-16T10:32:03.271354+0000","last_undegraded":"2024-09-16T10:32:03.271354+0000","last_fullsized":"2024-09-16T10:32:03.271354+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:48.209053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.209053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.123048+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":1033,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.642674+0000","last_change":"2024-09-16T10:31:14.165639+0000","last_active":"2024-09-16T10:31:57.642674+0000","last_peered":"2024-09-16T10:31:57.642674+0000","last_clean":"2024-09-16T10:31:57.642674+0000","last_became_active":"2024-09-16T10:31:14.151744+0000","last_became_peered":"2024-09-16T10:31:14.151744+0000","last_unstale":"2024-09-16T10:31:57.642674+0000","last_undegraded":"2024-09-16T10:31:57.642674+0000","last_fullsized":"2024-09-16T10:31:57.642674+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:07.157828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.157828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":329,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638209+0000","last_change":"2024-09-16T10:31:14.137225+0000","last_active":"2024-09-16T10:31:57.638209+0000","last_peered":"2024-09-16T10:31:57.638209+0000","last_clean":"2024-09-16T10:31:57.638209+0000","last_became_active":"2024-09-16T10:31:14.132504+0000","last_became_peered":"2024-09-16T10:31:14.132504+0000","last_unstale":"2024-09-16T10:31:57.638209+0000","last_undegraded":"2024-09-16T10:31:57.638209+0000","last_fullsized":"2024-09-16T10:31:57.638209+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:44.598219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:44.598219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:06.031728+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":997,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202258+0000","last_change":"2024-09-16T10:31:14.154512+0000","last_active":"2024-09-16T10:32:03.202258+0000","last_peered":"2024-09-16T10:32:03.202258+0000","last_clean":"2024-09-16T10:32:03.202258+0000","last_became_active":"2024-09-16T10:31:14.153698+0000","last_became_peered":"2024-09-16T10:31:14.153698+0000","last_unstale":"2024-09-16T10:32:03.202258+0000","last_undegraded":"2024-09-16T10:32:03.202258+0000","last_fullsized":"2024-09-16T10:32:03.202258+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:20.058297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.058297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":537,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.639661+0000","last_change":"2024-09-16T10:31:38.879647+0000","last_active":"2024-09-16T10:31:57.639661+0000","last_peered":"2024-09-16T10:31:57.639661+0000","last_clean":"2024-09-16T10:31:57.639661+0000","last_became_active":"2024-09-16T10:31:14.140004+0000","last_became_peered":"2024-09-16T10:31:14.140004+0000","last_unstale":"2024-09-16T10:31:57.639661+0000","last_undegraded":"2024-09-16T10:31:57.639661+0000","last_fullsized":"2024-09-16T10:31:57.639661+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:38.879579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:38.879579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:47.537853+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":572,"reported_epoch":523,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:32:00.668433+0000","last_change":"2024-09-16T10:31:45.130597+0000","last_active":"2024-09-16T10:32:00.668433+0000","last_peered":"2024-09-16T10:32:00.668433+0000","last_clean":"2024-09-16T10:31:04.129384+0000","last_became_active":"2024-09-16T10:31:15.129229+0000","last_became_peered":"2024-09-16T10:31:15.129229+0000","last_unstale":"2024-09-16T10:32:00.668433+0000","last_undegraded":"2024-09-16T10:32:00.668433+0000","last_fullsized":"2024-09-16T10:32:00.668433+0000","mapping_epoch":508,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,7],"avail_no_missing":["5","4","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":1020,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669596+0000","last_change":"2024-09-16T10:31:14.174635+0000","last_active":"2024-09-16T10:32:00.669596+0000","last_peered":"2024-09-16T10:32:00.669596+0000","last_clean":"2024-09-16T10:32:00.669596+0000","last_became_active":"2024-09-16T10:31:14.149019+0000","last_became_peered":"2024-09-16T10:31:14.149019+0000","last_unstale":"2024-09-16T10:32:00.669596+0000","last_undegraded":"2024-09-16T10:32:00.669596+0000","last_fullsized":"2024-09-16T10:32:00.669596+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:29:20.061184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.061184+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1002,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641118+0000","last_change":"2024-09-16T10:31:14.154526+0000","last_active":"2024-09-16T10:31:57.641118+0000","last_peered":"2024-09-16T10:31:57.641118+0000","last_clean":"2024-09-16T10:31:57.641118+0000","last_became_active":"2024-09-16T10:31:14.154145+0000","last_became_peered":"2024-09-16T10:31:14.154145+0000","last_unstale":"2024-09-16T10:31:57.641118+0000","last_undegraded":"2024-09-16T10:31:57.641118+0000","last_fullsized":"2024-09-16T10:31:57.641118+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:38.060395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:38.060395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"508'13","reported_seq":1271,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201590+0000","last_change":"2024-09-16T10:31:14.168449+0000","last_active":"2024-09-16T10:32:03.201590+0000","last_peered":"2024-09-16T10:32:03.201590+0000","last_clean":"2024-09-16T10:32:03.201590+0000","last_became_active":"2024-09-16T10:31:14.150871+0000","last_became_peered":"2024-09-16T10:31:14.150871+0000","last_unstale":"2024-09-16T10:32:03.201590+0000","last_undegraded":"2024-09-16T10:32:03.201590+0000","last_fullsized":"2024-09-16T10:32:03.201590+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:30:42.170419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.170419+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":776,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.639225+0000","last_change":"2024-09-16T10:26:45.194577+0000","last_active":"2024-09-16T10:31:57.639225+0000","last_peered":"2024-09-16T10:31:57.639225+0000","last_clean":"2024-09-16T10:31:57.639225+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:31:57.639225+0000","last_undegraded":"2024-09-16T10:31:57.639225+0000","last_fullsized":"2024-09-16T10:31:57.639225+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:45.194454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:45.194454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:10.783089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":554,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.641529+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:31:57.641529+0000","last_peered":"2024-09-16T10:31:57.641529+0000","last_clean":"2024-09-16T10:31:57.641529+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:31:57.641529+0000","last_undegraded":"2024-09-16T10:31:57.641529+0000","last_fullsized":"2024-09-16T10:31:57.641529+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:03.251453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":1051,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200702+0000","last_change":"2024-09-16T10:31:14.173911+0000","last_active":"2024-09-16T10:32:03.200702+0000","last_peered":"2024-09-16T10:32:03.200702+0000","last_clean":"2024-09-16T10:32:03.200702+0000","last_became_active":"2024-09-16T10:31:14.147045+0000","last_became_peered":"2024-09-16T10:31:14.147045+0000","last_unstale":"2024-09-16T10:32:03.200702+0000","last_undegraded":"2024-09-16T10:32:03.200702+0000","last_fullsized":"2024-09-16T10:32:03.200702+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:26:20.405507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:26:20.405507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:57.204833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":754,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638563+0000","last_change":"2024-09-16T10:31:34.935251+0000","last_active":"2024-09-16T10:31:57.638563+0000","last_peered":"2024-09-16T10:31:57.638563+0000","last_clean":"2024-09-16T10:31:57.638563+0000","last_became_active":"2024-09-16T10:31:14.132063+0000","last_became_peered":"2024-09-16T10:31:14.132063+0000","last_unstale":"2024-09-16T10:31:57.638563+0000","last_undegraded":"2024-09-16T10:31:57.638563+0000","last_fullsized":"2024-09-16T10:31:57.638563+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:34.935185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:34.935185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:55.131571+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1217,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669383+0000","last_change":"2024-09-16T10:30:56.801060+0000","last_active":"2024-09-16T10:32:00.669383+0000","last_peered":"2024-09-16T10:32:00.669383+0000","last_clean":"2024-09-16T10:32:00.669383+0000","last_became_active":"2024-09-16T10:13:41.114511+0000","last_became_peered":"2024-09-16T10:13:41.114511+0000","last_unstale":"2024-09-16T10:32:00.669383+0000","last_undegraded":"2024-09-16T10:32:00.669383+0000","last_fullsized":"2024-09-16T10:32:00.669383+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:56.800991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:56.800991+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":982,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.642150+0000","last_change":"2024-09-16T10:31:14.150959+0000","last_active":"2024-09-16T10:31:57.642150+0000","last_peered":"2024-09-16T10:31:57.642150+0000","last_clean":"2024-09-16T10:31:57.642150+0000","last_became_active":"2024-09-16T10:31:14.150453+0000","last_became_peered":"2024-09-16T10:31:14.150453+0000","last_unstale":"2024-09-16T10:31:57.642150+0000","last_undegraded":"2024-09-16T10:31:57.642150+0000","last_fullsized":"2024-09-16T10:31:57.642150+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.053102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.053102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":914,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638153+0000","last_change":"2024-09-16T10:28:02.318109+0000","last_active":"2024-09-16T10:31:57.638153+0000","last_peered":"2024-09-16T10:31:57.638153+0000","last_clean":"2024-09-16T10:31:57.638153+0000","last_became_active":"2024-09-16T10:26:28.557003+0000","last_became_peered":"2024-09-16T10:26:28.557003+0000","last_unstale":"2024-09-16T10:31:57.638153+0000","last_undegraded":"2024-09-16T10:31:57.638153+0000","last_fullsized":"2024-09-16T10:31:57.638153+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:28:02.318062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:28:02.318062+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":457,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.269952+0000","last_change":"2024-09-16T10:31:14.147632+0000","last_active":"2024-09-16T10:32:03.269952+0000","last_peered":"2024-09-16T10:32:03.269952+0000","last_clean":"2024-09-16T10:32:03.269952+0000","last_became_active":"2024-09-16T10:31:14.144517+0000","last_became_peered":"2024-09-16T10:31:14.144517+0000","last_unstale":"2024-09-16T10:32:03.269952+0000","last_undegraded":"2024-09-16T10:32:03.269952+0000","last_fullsized":"2024-09-16T10:32:03.269952+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.216239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.216239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":973,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.197836+0000","last_change":"2024-09-16T10:25:14.607173+0000","last_active":"2024-09-16T10:32:03.197836+0000","last_peered":"2024-09-16T10:32:03.197836+0000","last_clean":"2024-09-16T10:32:03.197836+0000","last_became_active":"2024-09-16T10:13:41.112618+0000","last_became_peered":"2024-09-16T10:13:41.112618+0000","last_unstale":"2024-09-16T10:32:03.197836+0000","last_undegraded":"2024-09-16T10:32:03.197836+0000","last_fullsized":"2024-09-16T10:32:03.197836+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:14.607114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:14.607114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":967,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202279+0000","last_change":"2024-09-16T10:26:14.413104+0000","last_active":"2024-09-16T10:32:03.202279+0000","last_peered":"2024-09-16T10:32:03.202279+0000","last_clean":"2024-09-16T10:32:03.202279+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:32:03.202279+0000","last_undegraded":"2024-09-16T10:32:03.202279+0000","last_fullsized":"2024-09-16T10:32:03.202279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:14.413019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:26:14.413019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:10.254762+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":1036,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.198106+0000","last_change":"2024-09-16T10:29:19.698059+0000","last_active":"2024-09-16T10:32:03.198106+0000","last_peered":"2024-09-16T10:32:03.198106+0000","last_clean":"2024-09-16T10:32:03.198106+0000","last_became_active":"2024-09-16T10:22:02.896935+0000","last_became_peered":"2024-09-16T10:22:02.896935+0000","last_unstale":"2024-09-16T10:32:03.198106+0000","last_undegraded":"2024-09-16T10:32:03.198106+0000","last_fullsized":"2024-09-16T10:32:03.198106+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":392,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201204+0000","last_change":"2024-09-16T10:31:14.144963+0000","last_active":"2024-09-16T10:32:03.201204+0000","last_peered":"2024-09-16T10:32:03.201204+0000","last_clean":"2024-09-16T10:32:03.201204+0000","last_became_active":"2024-09-16T10:31:14.144634+0000","last_became_peered":"2024-09-16T10:31:14.144634+0000","last_unstale":"2024-09-16T10:32:03.201204+0000","last_undegraded":"2024-09-16T10:32:03.201204+0000","last_fullsized":"2024-09-16T10:32:03.201204+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":936,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200596+0000","last_change":"2024-09-16T10:25:40.451657+0000","last_active":"2024-09-16T10:32:03.200596+0000","last_peered":"2024-09-16T10:32:03.200596+0000","last_clean":"2024-09-16T10:32:03.200596+0000","last_became_active":"2024-09-16T10:18:33.825360+0000","last_became_peered":"2024-09-16T10:18:33.825360+0000","last_unstale":"2024-09-16T10:32:03.200596+0000","last_undegraded":"2024-09-16T10:32:03.200596+0000","last_fullsized":"2024-09-16T10:32:03.200596+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:59.283339+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":956,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.640443+0000","last_change":"2024-09-16T10:30:52.389337+0000","last_active":"2024-09-16T10:31:57.640443+0000","last_peered":"2024-09-16T10:31:57.640443+0000","last_clean":"2024-09-16T10:31:57.640443+0000","last_became_active":"2024-09-16T10:13:43.116783+0000","last_became_peered":"2024-09-16T10:13:43.116783+0000","last_unstale":"2024-09-16T10:31:57.640443+0000","last_undegraded":"2024-09-16T10:31:57.640443+0000","last_fullsized":"2024-09-16T10:31:57.640443+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.389262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.389262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.738887+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":919,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.639047+0000","last_change":"2024-09-16T10:31:31.897756+0000","last_active":"2024-09-16T10:31:57.639047+0000","last_peered":"2024-09-16T10:31:57.639047+0000","last_clean":"2024-09-16T10:31:57.639047+0000","last_became_active":"2024-09-16T10:31:17.118844+0000","last_became_peered":"2024-09-16T10:31:17.118844+0000","last_unstale":"2024-09-16T10:31:57.639047+0000","last_undegraded":"2024-09-16T10:31:57.639047+0000","last_fullsized":"2024-09-16T10:31:57.639047+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:31:31.897681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:31:31.897681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:56.086986+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":1020,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.202563+0000","last_change":"2024-09-16T10:31:14.175984+0000","last_active":"2024-09-16T10:32:03.202563+0000","last_peered":"2024-09-16T10:32:03.202563+0000","last_clean":"2024-09-16T10:32:03.202563+0000","last_became_active":"2024-09-16T10:31:14.154997+0000","last_became_peered":"2024-09-16T10:31:14.154997+0000","last_unstale":"2024-09-16T10:32:03.202563+0000","last_undegraded":"2024-09-16T10:32:03.202563+0000","last_fullsized":"2024-09-16T10:32:03.202563+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1005,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-09-16T10:32:00.669640+0000","last_change":"2024-09-16T10:31:14.147736+0000","last_active":"2024-09-16T10:32:00.669640+0000","last_peered":"2024-09-16T10:32:00.669640+0000","last_clean":"2024-09-16T10:32:00.669640+0000","last_became_active":"2024-09-16T10:31:14.147407+0000","last_became_peered":"2024-09-16T10:31:14.147407+0000","last_unstale":"2024-09-16T10:32:00.669640+0000","last_undegraded":"2024-09-16T10:32:00.669640+0000","last_fullsized":"2024-09-16T10:32:00.669640+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:23.002518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.002518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":670,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637530+0000","last_change":"2024-09-16T10:31:14.132749+0000","last_active":"2024-09-16T10:31:57.637530+0000","last_peered":"2024-09-16T10:31:57.637530+0000","last_clean":"2024-09-16T10:31:57.637530+0000","last_became_active":"2024-09-16T10:31:14.132459+0000","last_became_peered":"2024-09-16T10:31:14.132459+0000","last_unstale":"2024-09-16T10:31:57.637530+0000","last_undegraded":"2024-09-16T10:31:57.637530+0000","last_fullsized":"2024-09-16T10:31:57.637530+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:19.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.866022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":991,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.200858+0000","last_change":"2024-09-16T10:30:45.211710+0000","last_active":"2024-09-16T10:32:03.200858+0000","last_peered":"2024-09-16T10:32:03.200858+0000","last_clean":"2024-09-16T10:32:03.200858+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:32:03.200858+0000","last_undegraded":"2024-09-16T10:32:03.200858+0000","last_fullsized":"2024-09-16T10:32:03.200858+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.211638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.211638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":1021,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201468+0000","last_change":"2024-09-16T10:25:42.513509+0000","last_active":"2024-09-16T10:32:03.201468+0000","last_peered":"2024-09-16T10:32:03.201468+0000","last_clean":"2024-09-16T10:32:03.201468+0000","last_became_active":"2024-09-16T10:18:52.841631+0000","last_became_peered":"2024-09-16T10:18:52.841631+0000","last_unstale":"2024-09-16T10:32:03.201468+0000","last_undegraded":"2024-09-16T10:32:03.201468+0000","last_fullsized":"2024-09-16T10:32:03.201468+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:25:42.513425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:42.513425+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":982,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.643648+0000","last_change":"2024-09-16T10:30:21.275187+0000","last_active":"2024-09-16T10:31:57.643648+0000","last_peered":"2024-09-16T10:31:57.643648+0000","last_clean":"2024-09-16T10:31:57.643648+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:31:57.643648+0000","last_undegraded":"2024-09-16T10:31:57.643648+0000","last_fullsized":"2024-09-16T10:31:57.643648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:30:21.275103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:21.275103+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1002,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.637571+0000","last_change":"2024-09-16T10:31:14.139140+0000","last_active":"2024-09-16T10:31:57.637571+0000","last_peered":"2024-09-16T10:31:57.637571+0000","last_clean":"2024-09-16T10:31:57.637571+0000","last_became_active":"2024-09-16T10:31:14.138772+0000","last_became_peered":"2024-09-16T10:31:14.138772+0000","last_unstale":"2024-09-16T10:31:57.637571+0000","last_undegraded":"2024-09-16T10:31:57.637571+0000","last_fullsized":"2024-09-16T10:31:57.637571+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.649610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.649610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:07.803476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":620,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638592+0000","last_change":"2024-09-16T10:31:40.861638+0000","last_active":"2024-09-16T10:31:57.638592+0000","last_peered":"2024-09-16T10:31:57.638592+0000","last_clean":"2024-09-16T10:31:57.638592+0000","last_became_active":"2024-09-16T10:31:14.152053+0000","last_became_peered":"2024-09-16T10:31:14.152053+0000","last_unstale":"2024-09-16T10:31:57.638592+0000","last_undegraded":"2024-09-16T10:31:57.638592+0000","last_fullsized":"2024-09-16T10:31:57.638592+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:40.861570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:40.861570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:55.411055+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":977,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.270117+0000","last_change":"2024-09-16T10:30:15.832984+0000","last_active":"2024-09-16T10:32:03.270117+0000","last_peered":"2024-09-16T10:32:03.270117+0000","last_clean":"2024-09-16T10:32:03.270117+0000","last_became_active":"2024-09-16T10:13:43.114439+0000","last_became_peered":"2024-09-16T10:13:43.114439+0000","last_unstale":"2024-09-16T10:32:03.270117+0000","last_undegraded":"2024-09-16T10:32:03.270117+0000","last_fullsized":"2024-09-16T10:32:03.270117+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.832904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.832904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":987,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.201608+0000","last_change":"2024-09-16T10:26:13.362416+0000","last_active":"2024-09-16T10:32:03.201608+0000","last_peered":"2024-09-16T10:32:03.201608+0000","last_clean":"2024-09-16T10:32:03.201608+0000","last_became_active":"2024-09-16T10:13:37.089537+0000","last_became_peered":"2024-09-16T10:13:37.089537+0000","last_unstale":"2024-09-16T10:32:03.201608+0000","last_undegraded":"2024-09-16T10:32:03.201608+0000","last_fullsized":"2024-09-16T10:32:03.201608+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:13.362347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:26:13.362347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:13.236069+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":1028,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.638449+0000","last_change":"2024-09-16T10:30:20.544197+0000","last_active":"2024-09-16T10:31:57.638449+0000","last_peered":"2024-09-16T10:31:57.638449+0000","last_clean":"2024-09-16T10:31:57.638449+0000","last_became_active":"2024-09-16T10:13:39.098411+0000","last_became_peered":"2024-09-16T10:13:39.098411+0000","last_unstale":"2024-09-16T10:31:57.638449+0000","last_undegraded":"2024-09-16T10:31:57.638449+0000","last_fullsized":"2024-09-16T10:31:57.638449+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:30:20.544147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.544147+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1010,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.642281+0000","last_change":"2024-09-16T10:31:14.142887+0000","last_active":"2024-09-16T10:31:57.642281+0000","last_peered":"2024-09-16T10:31:57.642281+0000","last_clean":"2024-09-16T10:31:57.642281+0000","last_became_active":"2024-09-16T10:31:14.141591+0000","last_became_peered":"2024-09-16T10:31:14.141591+0000","last_unstale":"2024-09-16T10:31:57.642281+0000","last_undegraded":"2024-09-16T10:31:57.642281+0000","last_fullsized":"2024-09-16T10:31:57.642281+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:54.399132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.399132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:32:13.946810+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4974837784,"num_objects":75911,"num_object_clones":0,"num_object_copies":227733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9903,"num_objects_unfound":0,"num_objects_dirty":75911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172637,"num_write_kb":7953473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24861,"num_bytes_recovered":1628897280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15261446144,"data_stored":15261433928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66737,"ondisk_log_size":66737,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":3400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":420,"num_read_kb":415,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":552211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":1894080577580,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528548,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3684096,"kb_used_data":2899052,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":90061056,"statfs":{"total":95995035648,"available":92222521344,"internally_reserved":0,"allocated":2968629248,"data_stored":2968241429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":803862982},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790187,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2267804,"kb_used_data":1510472,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":91477348,"statfs":{"total":95995035648,"available":93672804352,"internally_reserved":0,"allocated":1546723328,"data_stored":1546316219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986416,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1837992,"kb_used_data":1193176,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91907160,"statfs":{"total":95995035648,"available":94112931840,"internally_reserved":0,"allocated":1221812224,"data_stored":1221387680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215349,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3240700,"kb_used_data":2370852,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":90504452,"statfs":{"total":95995035648,"available":92676558848,"internally_reserved":0,"allocated":2427752448,"data_stored":2427376739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":890567112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411578,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2471052,"kb_used_data":1890100,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91274100,"statfs":{"total":95995035648,"available":93464678400,"internally_reserved":0,"allocated":1935462400,"data_stored":1935067204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575102,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3240528,"kb_used_data":2388728,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":90504624,"statfs":{"total":95995035648,"available":92676734976,"internally_reserved":0,"allocated":2446057472,"data_stored":2445660037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738627,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2707464,"kb_used_data":1833464,"kb_used_omap":1,"kb_used_meta":873854,"kb_avail":91037688,"statfs":{"total":95995035648,"available":93222592512,"internally_reserved":0,"allocated":1877467136,"data_stored":1877068409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":894826949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1861746688,"data_stored":1861746688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2439774208,"data_stored":2439774208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1923747840,"data_stored":1923743768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2420375552,"data_stored":2420375552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1209667584,"data_stored":1209663512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1526726656,"data_stored":1526726656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2934312960,"data_stored":2934308888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":945094656,"data_stored":945094656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:05.334 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-09-16T10:32:05.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:05.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:05.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:05 smithi063 ceph-mon[29993]: from='client.16761 v1:172.21.15.63:0/211300323' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:32:05.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:05 smithi063 ceph-mon[39470]: from='client.16761 v1:172.21.15.63:0/211300323' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:32:05.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:05.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:05.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:05.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:05.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:05.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:05.632 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:05.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:05.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:05.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:05.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:32:05.840 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:05.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:05.912 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:05.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:06.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:06.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:06.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:06.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:06.188 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:06.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:06.195 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:32:06.209 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:32:06.209 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:32:06.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:06.333 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:06.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:06.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:06.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:32:06.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:32:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[29993]: pgmap v993: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 9546/221892 objects misplaced (4.302%); 21 MiB/s, 343 objects/s recovering 2024-09-16T10:32:06.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4108537298' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:06.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:32:06.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:32:06.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[39470]: pgmap v993: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 9546/221892 objects misplaced (4.302%); 21 MiB/s, 343 objects/s recovering 2024-09-16T10:32:06.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4108537298' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:06.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:06.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:06.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:06 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:32:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:06 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:32:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:06 smithi165 ceph-mon[32696]: pgmap v993: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 9546/221892 objects misplaced (4.302%); 21 MiB/s, 343 objects/s recovering 2024-09-16T10:32:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4108537298' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:06.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:06.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:06.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:06.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:06.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:06.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:06.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:06.865 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:06.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:06.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:32:06.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:07.001 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:07.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:07.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:07.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:07 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:07.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:07.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:07.271 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:07.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:07.403 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:07.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:32:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4108537298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[39470]: osdmap e525: 8 total, 8 up, 7 in 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[39470]: 5.1 scrub starts 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[39470]: 5.1 scrub ok 2024-09-16T10:32:07.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:07.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4108537298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:32:07.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[29993]: osdmap e525: 8 total, 8 up, 7 in 2024-09-16T10:32:07.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:32:07.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:32:07.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[29993]: 5.1 scrub starts 2024-09-16T10:32:07.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:07 smithi063 ceph-mon[29993]: 5.1 scrub ok 2024-09-16T10:32:07.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:07.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:07.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:07 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4108537298' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:32:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:07 smithi165 ceph-mon[32696]: osdmap e525: 8 total, 8 up, 7 in 2024-09-16T10:32:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:07 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:32:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:07 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:32:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:07 smithi165 ceph-mon[32696]: 5.1 scrub starts 2024-09-16T10:32:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:07 smithi165 ceph-mon[32696]: 5.1 scrub ok 2024-09-16T10:32:07.630 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.653 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:07.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:07.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:07.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:07.820 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:32:07.821 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:32:07.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:07.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:07.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:08.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:08.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:08.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:08.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:08.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:08.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:08.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:08.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:08.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:08.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:08.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:08 smithi165 ceph-mon[32696]: 1.c scrub starts 2024-09-16T10:32:08.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:08 smithi165 ceph-mon[32696]: 1.c scrub ok 2024-09-16T10:32:08.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:08 smithi165 ceph-mon[32696]: osdmap e526: 8 total, 8 up, 7 in 2024-09-16T10:32:08.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:08 smithi165 ceph-mon[32696]: 3.16 scrub starts 2024-09-16T10:32:08.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:08 smithi165 ceph-mon[32696]: 3.16 scrub ok 2024-09-16T10:32:08.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:08 smithi165 ceph-mon[32696]: pgmap v996: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 958 op/s; 9546/221892 objects misplaced (4.302%); 5.1 MiB/s, 81 objects/s recovering 2024-09-16T10:32:08.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:08.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:32:08.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:08.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[29993]: 1.c scrub starts 2024-09-16T10:32:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[29993]: 1.c scrub ok 2024-09-16T10:32:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[29993]: osdmap e526: 8 total, 8 up, 7 in 2024-09-16T10:32:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[29993]: 3.16 scrub starts 2024-09-16T10:32:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[29993]: 3.16 scrub ok 2024-09-16T10:32:08.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[29993]: pgmap v996: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 958 op/s; 9546/221892 objects misplaced (4.302%); 5.1 MiB/s, 81 objects/s recovering 2024-09-16T10:32:08.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[39470]: 1.c scrub starts 2024-09-16T10:32:08.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[39470]: 1.c scrub ok 2024-09-16T10:32:08.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[39470]: osdmap e526: 8 total, 8 up, 7 in 2024-09-16T10:32:08.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[39470]: 3.16 scrub starts 2024-09-16T10:32:08.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[39470]: 3.16 scrub ok 2024-09-16T10:32:08.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:08 smithi063 ceph-mon[39470]: pgmap v996: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 958 op/s; 9546/221892 objects misplaced (4.302%); 5.1 MiB/s, 81 objects/s recovering 2024-09-16T10:32:08.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.819 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:08.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:08.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:08.927 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:08.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:08.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:09.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:09.100 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:09.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:09.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:09.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:09.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:09.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:09.372 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:09.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:09.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:09.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:09 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:32:09.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:09 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:32:09.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:09 smithi165 ceph-mon[32696]: osdmap e527: 8 total, 8 up, 7 in 2024-09-16T10:32:09.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:09 smithi165 ceph-mon[32696]: 4.e scrub starts 2024-09-16T10:32:09.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:09 smithi165 ceph-mon[32696]: 4.e scrub ok 2024-09-16T10:32:09.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:32:09 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:32:09.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[29993]: osdmap e527: 8 total, 8 up, 7 in 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[29993]: 4.e scrub starts 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[29993]: 4.e scrub ok 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[39470]: osdmap e527: 8 total, 8 up, 7 in 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[39470]: 4.e scrub starts 2024-09-16T10:32:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:09 smithi063 ceph-mon[39470]: 4.e scrub ok 2024-09-16T10:32:09.474 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:09.516 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:09.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:09.518 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:32:09.518 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:32:09.618 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:09.669 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:09.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:09.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:09.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:09.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:09.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:09.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:09.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:10.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:10.084 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:10.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:10.120 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:10.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:10.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:10.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:10.221 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 11 pgp_num to 16 2024-09-16T10:32:10.285 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 3.1a scrub starts 2024-09-16T10:32:10.285 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 3.1a scrub ok 2024-09-16T10:32:10.285 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:10.285 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 1.12 scrub starts 2024-09-16T10:32:10.285 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 1.12 scrub ok 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: pgmap v998: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.13k op/s; 8636/207972 objects misplaced (4.152%); 3.3 MiB/s, 53 objects/s recovering 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 4.13 scrub starts 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 4.13 scrub ok 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 3.f scrub starts 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2150449816' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: from='client.26719 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[29993]: 3.f scrub ok 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 3.1a scrub starts 2024-09-16T10:32:10.286 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 3.1a scrub ok 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 1.12 scrub starts 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 1.12 scrub ok 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: pgmap v998: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.13k op/s; 8636/207972 objects misplaced (4.152%); 3.3 MiB/s, 53 objects/s recovering 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 4.13 scrub starts 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 4.13 scrub ok 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 3.f scrub starts 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2150449816' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: from='client.26719 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:32:10.287 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:10 smithi063 ceph-mon[39470]: 3.f scrub ok 2024-09-16T10:32:10.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:10.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:10.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:10.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:10.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:10.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 3.1a scrub starts 2024-09-16T10:32:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 3.1a scrub ok 2024-09-16T10:32:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 1.12 scrub starts 2024-09-16T10:32:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 1.12 scrub ok 2024-09-16T10:32:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: pgmap v998: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.13k op/s; 8636/207972 objects misplaced (4.152%); 3.3 MiB/s, 53 objects/s recovering 2024-09-16T10:32:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 4.13 scrub starts 2024-09-16T10:32:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 4.13 scrub ok 2024-09-16T10:32:10.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 3.f scrub starts 2024-09-16T10:32:10.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2150449816' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:32:10.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: from='client.26719 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-16T10:32:10.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:10 smithi165 ceph-mon[32696]: 3.f scrub ok 2024-09-16T10:32:10.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:10.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:10.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:10.709 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:10.748 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:10.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:10.850 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:10.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:10.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:32:10.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:11.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:11.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:11.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:11.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:11.302 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:11.307 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:11.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:11.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:11.494 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:32:11.494 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: 3.b scrub starts 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: 3.b scrub ok 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: from='client.26719 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: osdmap e528: 8 total, 8 up, 7 in 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: 6.11 scrub starts 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: 6.11 scrub ok 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]} : dispatch 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]} : dispatch 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]} : dispatch 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]} : dispatch 2024-09-16T10:32:11.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: 3.b scrub starts 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: 3.b scrub ok 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: from='client.26719 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: osdmap e528: 8 total, 8 up, 7 in 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: 6.11 scrub starts 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: 6.11 scrub ok 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]} : dispatch 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]} : dispatch 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]} : dispatch 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]} : dispatch 2024-09-16T10:32:11.496 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-16T10:32:11.531 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:32:11.531 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:32:11.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: 3.b scrub starts 2024-09-16T10:32:11.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: 3.b scrub ok 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: from='client.26719 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: osdmap e528: 8 total, 8 up, 7 in 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: 6.11 scrub starts 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: 6.11 scrub ok 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]} : dispatch 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]} : dispatch 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]} : dispatch 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]} : dispatch 2024-09-16T10:32:11.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-16T10:32:11.563 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:32:11.563 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:32:11.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:11.600 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:32:11.600 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:32:11.634 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:32:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:32:11.667 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:32:11.667 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:32:11.669 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:11.704 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:11.704 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:32:11.705 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:32:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:11.739 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:32:11.740 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:32:11.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:11.838 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:11.940 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:11.942 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:32:11.942 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:32:11.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.942 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:183776 11.3 11:c12f099a:::benchmark_data_smithi063_176907_object59500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e529)", 2024-09-16T10:32:11.942 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:32:11.932213+0000", 2024-09-16T10:32:11.942 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0062083119999999997, 2024-09-16T10:32:11.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062379250000000001, 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 183776 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932213+0000", 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932213+0000", 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932215+0000", 2024-09-16T10:32:11.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.607e-06 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932220+0000", 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1710000000000001e-06 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932229+0000", 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7960000000000001e-06 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932235+0000", 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932286+0000", 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1415999999999997e-05 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932410+0000", 2024-09-16T10:32:11.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012353700000000001 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.932522+0000", 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011211 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935562+0000", 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00304002 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:183781 11.0 e529/521 11:09d387ca:::benchmark_data_smithi063_176907_object59505:head v 529'11472, mlcod=529'11469)", 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:32:11.935667+0000", 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0027541089999999998, 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028543280000000002, 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935667+0000", 2024-09-16T10:32:11.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935667+0000", 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935668+0000", 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9799999999999999e-07 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:11.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935673+0000", 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0019999999999997e-06 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935678+0000", 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5099999999999998e-06 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935683+0000", 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5059999999999997e-06 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935725+0000", 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1696999999999997e-05 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935742+0000", 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7934000000000001e-05 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:183783 11.3 11:cc8a9280:::benchmark_data_smithi063_176907_object59507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e529)", 2024-09-16T10:32:11.951 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:32:11.935670+0000", 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0027510289999999999, 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028990449999999998, 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 183783 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935670+0000", 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935670+0000", 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935672+0000", 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9039999999999999e-06 2024-09-16T10:32:11.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935677+0000", 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9810000000000003e-06 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935687+0000", 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4839999999999993e-06 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:11.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935703+0000", 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6110999999999999e-05 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935869+0000", 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016553299999999999 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936054+0000", 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000185861 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936172+0000", 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011747999999999999 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.938439+0000", 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022672370000000001 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.956 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:183779 11.5 e529/520 11:a0cbabf1:::benchmark_data_smithi063_176907_object59503:head v 529'11424, mlcod=529'11422)", 2024-09-16T10:32:11.957 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:32:11.935710+0000", 2024-09-16T10:32:11.957 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0027114959999999999, 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029221220000000001, 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935710+0000", 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935710+0000", 2024-09-16T10:32:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935711+0000", 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5199999999999997e-07 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935715+0000", 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.301e-06 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935721+0000", 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9290000000000003e-06 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935725+0000", 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2819999999999998e-06 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935851+0000", 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000126308 2024-09-16T10:32:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.935870+0000", 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8859999999999999e-05 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:183780 11.9 e529/476 11:9b5c7110:::benchmark_data_smithi063_176907_object59504:head v 529'11436, mlcod=529'11434)", 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:32:11.936745+0000", 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.001676215, 2024-09-16T10:32:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001931173, 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936745+0000", 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936745+0000", 2024-09-16T10:32:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936746+0000", 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9700000000000005e-07 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936752+0000", 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7200000000000003e-06 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936760+0000", 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.405e-06 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936767+0000", 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2949999999999999e-06 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936816+0000", 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9854000000000001e-05 2024-09-16T10:32:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.936833+0000", 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6169999999999999e-05 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:183784 11.8 e529/476 11:12d46f0b:::benchmark_data_smithi063_176907_object59508:head v 529'11625, mlcod=529'11623)", 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:32:11.937200+0000", 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0012212639999999999, 2024-09-16T10:32:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001519432, 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937200+0000", 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937200+0000", 2024-09-16T10:32:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937201+0000", 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9899999999999999e-07 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937206+0000", 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6009999999999997e-06 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937213+0000", 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1840000000000002e-06 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937218+0000", 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4480000000000002e-06 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937270+0000", 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1362000000000001e-05 2024-09-16T10:32:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937289+0000", 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9426999999999999e-05 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:183786 11.d e529/476 11:b5ff0f92:::benchmark_data_smithi063_176907_object59510:head v 529'11198, mlcod=529'11196)", 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:32:11.937735+0000", 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00068684799999999999, 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001028178, 2024-09-16T10:32:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937735+0000", 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937735+0000", 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937736+0000", 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0920000000000001e-06 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937741+0000", 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4569999999999996e-06 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937749+0000", 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1019999999999997e-06 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937755+0000", 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7200000000000003e-06 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937813+0000", 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7705999999999999e-05 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:32:11.937829+0000", 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6337e-05 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 7 2024-09-16T10:32:11.973 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:32:11.981 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:11.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:12.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:12.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:32:12.123 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:12.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:12.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:12.227 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:12.296 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:12.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:12.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:12.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:12.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:12.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: 2.1 scrub starts 2024-09-16T10:32:12.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: 2.1 scrub ok 2024-09-16T10:32:12.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: 5.12 scrub starts 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: 5.12 scrub ok 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]}]': finished 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]}]': finished 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]}]': finished 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: osdmap e529: 8 total, 8 up, 7 in 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4211346145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: pgmap v1001: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 8636/207036 objects misplaced (4.171%); 4.6 MiB/s, 73 objects/s recovering 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: 6.1e scrub starts 2024-09-16T10:32:12.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[29993]: 6.1e scrub ok 2024-09-16T10:32:12.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: 2.1 scrub starts 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: 2.1 scrub ok 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: 5.12 scrub starts 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: 5.12 scrub ok 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]}]': finished 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]}]': finished 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]}]': finished 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: osdmap e529: 8 total, 8 up, 7 in 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4211346145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: pgmap v1001: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 8636/207036 objects misplaced (4.171%); 4.6 MiB/s, 73 objects/s recovering 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: 6.1e scrub starts 2024-09-16T10:32:12.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:12 smithi063 ceph-mon[39470]: 6.1e scrub ok 2024-09-16T10:32:12.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:12.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:12.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:12.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: 2.1 scrub starts 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: 2.1 scrub ok 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: 5.12 scrub starts 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: 5.12 scrub ok 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]}]': finished 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 6]}]': finished 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]}]': finished 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: osdmap e529: 8 total, 8 up, 7 in 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4211346145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: pgmap v1001: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 8636/207036 objects misplaced (4.171%); 4.6 MiB/s, 73 objects/s recovering 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: 6.1e scrub starts 2024-09-16T10:32:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:12 smithi165 ceph-mon[32696]: 6.1e scrub ok 2024-09-16T10:32:12.643 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:12.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:12.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:12.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:12.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:12.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:12.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:12.909 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:12.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:12.951 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:12.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:13.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:13.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:13.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:13.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:32:13.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:13.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:13.232 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:13.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 5.a scrub starts 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 5.a scrub ok 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: osdmap e530: 8 total, 8 up, 7 in 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 4.10 scrub starts 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 4.10 scrub ok 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 6.3 scrub starts 2024-09-16T10:32:13.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:13 smithi165 ceph-mon[32696]: 6.3 scrub ok 2024-09-16T10:32:13.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:13.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:13.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:13.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 5.a scrub starts 2024-09-16T10:32:13.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 5.a scrub ok 2024-09-16T10:32:13.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: osdmap e530: 8 total, 8 up, 7 in 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 4.10 scrub starts 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 4.10 scrub ok 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 6.3 scrub starts 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[29993]: 6.3 scrub ok 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 5.a scrub starts 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 5.a scrub ok 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:32:13.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:32:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: osdmap e530: 8 total, 8 up, 7 in 2024-09-16T10:32:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 4.10 scrub starts 2024-09-16T10:32:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 4.10 scrub ok 2024-09-16T10:32:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 6.3 scrub starts 2024-09-16T10:32:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:13 smithi063 ceph-mon[39470]: 6.3 scrub ok 2024-09-16T10:32:13.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:13.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:13.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:13.517 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:13.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:13.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:13.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:13.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:13.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:13.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:13.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:13.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:13.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:14.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:14.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:14.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:14.193 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:14.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:14.296 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:32:14.305 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:32:14.305 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:32:14.305 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:32:14.305 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:32:14.305 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:32:14.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:14.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:14.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:14 smithi165 ceph-mon[32696]: pgmap v1003: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.53k op/s; 8255/195639 objects misplaced (4.220%); 3.8 MiB/s, 60 objects/s recovering 2024-09-16T10:32:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/931290720' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:32:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:14 smithi165 ceph-mon[32696]: 5.10 scrub starts 2024-09-16T10:32:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:14 smithi165 ceph-mon[32696]: 5.10 scrub ok 2024-09-16T10:32:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:14 smithi165 ceph-mon[32696]: 6.18 scrub starts 2024-09-16T10:32:14.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:14 smithi165 ceph-mon[32696]: 6.18 scrub ok 2024-09-16T10:32:14.554 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:14.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:32:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.595 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:14.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:14.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[29993]: pgmap v1003: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.53k op/s; 8255/195639 objects misplaced (4.220%); 3.8 MiB/s, 60 objects/s recovering 2024-09-16T10:32:14.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/931290720' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:32:14.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[29993]: 5.10 scrub starts 2024-09-16T10:32:14.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[29993]: 5.10 scrub ok 2024-09-16T10:32:14.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[29993]: 6.18 scrub starts 2024-09-16T10:32:14.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[29993]: 6.18 scrub ok 2024-09-16T10:32:14.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[39470]: pgmap v1003: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.53k op/s; 8255/195639 objects misplaced (4.220%); 3.8 MiB/s, 60 objects/s recovering 2024-09-16T10:32:14.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/931290720' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:32:14.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[39470]: 5.10 scrub starts 2024-09-16T10:32:14.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[39470]: 5.10 scrub ok 2024-09-16T10:32:14.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[39470]: 6.18 scrub starts 2024-09-16T10:32:14.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:14 smithi063 ceph-mon[39470]: 6.18 scrub ok 2024-09-16T10:32:14.696 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:14.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:14.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:14.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:14.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:14.995 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:14.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:15.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:32:15.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:15.144 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:15.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:15.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:15.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:15.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:15.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:15.424 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:15.429 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:15.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:15.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:15 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:32:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:15 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:32:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:15 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:32:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:15 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/931290720' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:32:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:15 smithi165 ceph-mon[32696]: osdmap e531: 8 total, 8 up, 7 in 2024-09-16T10:32:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:15 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:32:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:15 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:32:15.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:15.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:15.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:15.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:32:15.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:32:15.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:32:15.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/931290720' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[29993]: osdmap e531: 8 total, 8 up, 7 in 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/931290720' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[39470]: osdmap e531: 8 total, 8 up, 7 in 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:32:15.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:15 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:32:15.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:15.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:15.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:15.836 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:15.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:15.897 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:15.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:15.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:15.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:16.002 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 2] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:32:16.002 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:32:16.002 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 3, 4, 5, 6, 2] 2024-09-16T10:32:16.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-16T10:32:16.029 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88289 10.e 10:777727a3:::benchmark_data_smithi063_149043_object88288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.537589+0000", 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 348.487241955, 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0363937000000001, 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.030 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88289 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537589+0000", 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537589+0000", 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537591+0000", 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.722e-06 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537831+0000", 2024-09-16T10:32:16.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024001099999999999 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540070+0000", 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002238968 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540078+0000", 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5180000000000004e-06 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540979+0000", 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00090050899999999997 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540990+0000", 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1372000000000001e-05 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.560724+0000", 2024-09-16T10:32:16.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0197336770000001 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.562746+0000", 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002021915 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:32:16.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.562908+0000", 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162027 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568617+0000", 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057096100000000004 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568662+0000", 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4441999999999999e-05 2024-09-16T10:32:16.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573892+0000", 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052300309999999996 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573941+0000", 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8789999999999999e-05 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573983+0000", 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2107999999999997e-05 2024-09-16T10:32:16.037 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88296 10.e 10:79ee89b5:::benchmark_data_smithi063_149043_object88295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.555357+0000", 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 348.46947432600001, 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.024583236, 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.038 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88296 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555357+0000", 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555357+0000", 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555359+0000", 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9989999999999998e-06 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555394+0000", 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5635e-05 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555404+0000", 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.302e-06 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555411+0000", 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5839999999999997e-06 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555552+0000", 2024-09-16T10:32:16.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014035600000000001 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555561+0000", 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4409999999999999e-06 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.565905+0000", 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0103437479999999 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.566037+0000", 2024-09-16T10:32:16.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013203800000000001 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.566191+0000", 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015380799999999999 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568708+0000", 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025176560000000001 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579714+0000", 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011005386000000001 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579843+0000", 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012893799999999999 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579894+0000", 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1579000000000002e-05 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579940+0000", 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5766000000000002e-05 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88311 10.e 10:7eaf0035:::benchmark_data_smithi063_149043_object88310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.574876+0000", 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 348.44995526000002, 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005137081, 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.046 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88311 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574876+0000", 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574876+0000", 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574877+0000", 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9899999999999999e-07 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574916+0000", 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9264000000000002e-05 2024-09-16T10:32:16.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574922+0000", 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2940000000000002e-06 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574926+0000", 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8829999999999999e-06 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574980+0000", 2024-09-16T10:32:16.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3801999999999998e-05 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574988+0000", 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1100000000000003e-06 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568787+0000", 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99379890400000004 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568896+0000", 2024-09-16T10:32:16.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109475 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568997+0000", 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101127 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573854+0000", 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048568250000000004 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579746+0000", 2024-09-16T10:32:16.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058914609999999997 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579947+0000", 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000201199 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579983+0000", 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5907e-05 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580013+0000", 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0031000000000001e-05 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88322 10.e 10:7e8de65a:::benchmark_data_smithi063_149043_object88321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.588082+0000", 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 348.43674921000002, 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99205056999999996, 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.053 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88322 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588082+0000", 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588082+0000", 2024-09-16T10:32:16.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588083+0000", 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4020000000000001e-06 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588139+0000", 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6019e-05 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588148+0000", 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6000000000000007e-06 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588155+0000", 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9120000000000001e-06 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588218+0000", 2024-09-16T10:32:16.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3280000000000004e-05 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588226+0000", 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0479999999999999e-06 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.571192+0000", 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98296605199999998 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.571327+0000", 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013475499999999999 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.571418+0000", 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0878999999999995e-05 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579676+0000", 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082579100000000003 2024-09-16T10:32:16.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579830+0000", 2024-09-16T10:32:16.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000154139 2024-09-16T10:32:16.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580019+0000", 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018949900000000001 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580099+0000", 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9136999999999998e-05 2024-09-16T10:32:16.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580132+0000", 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3937999999999999e-05 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88330 10.e 10:7d45cc0a:::benchmark_data_smithi063_149043_object88329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.601441+0000", 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 348.42338975799998, 2024-09-16T10:32:16.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98658399699999999, 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88330 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601441+0000", 2024-09-16T10:32:16.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601441+0000", 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601442+0000", 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5099999999999998e-07 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601498+0000", 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5744e-05 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601506+0000", 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7000000000000008e-06 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601510+0000", 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.267e-06 2024-09-16T10:32:16.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601558+0000", 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8062000000000002e-05 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601565+0000", 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1149999999999996e-06 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.574017+0000", 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97245151500000004 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.574149+0000", 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013288800000000001 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.574279+0000", 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012991700000000001 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579785+0000", 2024-09-16T10:32:16.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055055340000000003 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580138+0000", 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035344700000000001 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.587670+0000", 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075319669999999997 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.587984+0000", 2024-09-16T10:32:16.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031397599999999999 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.588025+0000", 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1013999999999997e-05 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:3544 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93976561475456 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.112595+0000", 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.912236548999999, 2024-09-16T10:32:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.054160918, 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3544 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112595+0000", 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112595+0000", 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112597+0000", 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2249999999999999e-06 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112603+0000", 2024-09-16T10:32:16.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2879999999999998e-06 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112616+0000", 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2547e-05 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112624+0000", 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4540000000000007e-06 2024-09-16T10:32:16.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112682+0000", 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8171000000000003e-05 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:32:16.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112686+0000", 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2420000000000002e-06 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.126275+0000", 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013588078999999999 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.126285+0000", 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0587999999999999e-05 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.138434+0000", 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0121490390000001 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.138696+0000", 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000261506 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.148778+0000", 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010082571 2024-09-16T10:32:16.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.148871+0000", 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2923000000000005e-05 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.166755+0000", 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017884285 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:3545 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 93976561468544 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.112644+0000", 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.912186849000001, 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0541178630000001, 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3545 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112644+0000", 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112644+0000", 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112645+0000", 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000001e-07 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112652+0000", 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1779999999999997e-06 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112658+0000", 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0789999999999997e-06 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112663+0000", 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.023e-06 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112709+0000", 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5325999999999999e-05 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112714+0000", 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7509999999999997e-06 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.134894+0000", 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022180182999999999 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.134905+0000", 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0069e-05 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.135703+0000", 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000798284 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:32:16.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.137067+0000", 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001364388 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145692+0000", 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086244479999999998 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145759+0000", 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7359000000000003e-05 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.166762+0000", 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021003096999999998 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:3546 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93976566403584 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.912155421000001, 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.054477669, 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:16.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3546 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7100000000000002e-07 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112679+0000", 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3599999999999999e-06 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112684+0000", 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3340000000000001e-06 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112689+0000", 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1660000000000002e-06 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112731+0000", 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2286000000000001e-05 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112736+0000", 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2039999999999996e-06 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.141374+0000", 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028637040999999998 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.141385+0000", 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0981999999999999e-05 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.137935+0000", 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99655030600000005 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.138328+0000", 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039304800000000001 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145309+0000", 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069808719999999999 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145405+0000", 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6574000000000001e-05 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.167153+0000", 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021748025000000001 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.088 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127304 11.4 11:2e776e1e:::benchmark_data_smithi063_176907_object3028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.114285+0000", 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.910545806000002, 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1223989990000001, 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127304 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114285+0000", 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114285+0000", 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114287+0000", 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5600000000000001e-06 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114293+0000", 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.755e-06 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114314+0000", 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1268e-05 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114321+0000", 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0600000000000002e-06 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114377+0000", 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6520000000000001e-05 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114516+0000", 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000138687 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114847+0000", 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000330744 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.123432+0000", 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0085849419999999999 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.143849+0000", 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020416878999999999 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.143890+0000", 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1791e-05 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119563+0000", 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97567231899999995 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119576+0000", 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3191e-05 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218330+0000", 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0987538729999999 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218446+0000", 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011647 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236560+0000", 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018113186999999999 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236684+0000", 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012475299999999999 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127307 11.3 11:c802da4f:::benchmark_data_smithi063_176907_object3031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.114339+0000", 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.910492011999999, 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1180333509999998, 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.096 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127307 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114339+0000", 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114339+0000", 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114340+0000", 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2399999999999998e-07 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114343+0000", 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0819999999999999e-06 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114348+0000", 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114353+0000", 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9219999999999996e-06 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114437+0000", 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4216999999999997e-05 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114561+0000", 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012369400000000001 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114682+0000", 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120711 2024-09-16T10:32:16.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.124095+0000", 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094135409999999992 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.142236+0000", 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018140586 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.142250+0000", 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3715e-05 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125654+0000", 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98340460100000004 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125667+0000", 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2805e-05 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.220583+0000", 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0949163230000001 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.220891+0000", 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030743399999999998 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.232293+0000", 2024-09-16T10:32:16.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011402536 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.232372+0000", 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9097999999999994e-05 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127314 11.3 11:c1bf4bc8:::benchmark_data_smithi063_176907_object3038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.117913+0000", 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.906918203000004, 2024-09-16T10:32:16.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.118139706, 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127314 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117913+0000", 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117913+0000", 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117914+0000", 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.463e-06 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117920+0000", 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2970000000000003e-06 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117930+0000", 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0308e-05 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117937+0000", 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7819999999999996e-06 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.118037+0000", 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010067 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.118160+0000", 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122414 2024-09-16T10:32:16.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.118277+0000", 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011681 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.124100+0000", 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058237540000000004 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.144098+0000", 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019998096999999999 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.144104+0000", 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6699999999999999e-06 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125697+0000", 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98159304999999997 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125706+0000", 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6750000000000008e-06 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221153+0000", 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0954475560000001 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221311+0000", 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015769699999999999 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.235940+0000", 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014628611 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236053+0000", 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112852 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127320 11.3 11:c4c194ba:::benchmark_data_smithi063_176907_object3044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.147464+0000", 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.877366983999998, 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0989353159999999, 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127320 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.113 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147464+0000", 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147464+0000", 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147466+0000", 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.029e-06 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147472+0000", 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3630000000000002e-06 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147484+0000", 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2729e-05 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147492+0000", 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.391e-06 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147539+0000", 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7818999999999997e-05 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147546+0000", 2024-09-16T10:32:16.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6340000000000003e-06 2024-09-16T10:32:16.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125731+0000", 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97818496499999996 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125739+0000", 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9489999999999996e-06 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221514+0000", 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.095774936 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221632+0000", 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117859 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5,7", 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.222078+0000", 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000446654 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231188+0000", 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091099849999999993 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.234161+0000", 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002972293 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236228+0000", 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020670900000000002 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.246244+0000", 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010016057 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.246305+0000", 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1068999999999994e-05 2024-09-16T10:32:16.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.246399+0000", 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4493999999999993e-05 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127323 11.4 11:22126989:::benchmark_data_smithi063_176907_object3047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.155284+0000", 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.869547378999997, 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0816857770000001, 2024-09-16T10:32:16.122 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127323 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155284+0000", 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155284+0000", 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155285+0000", 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2720000000000001e-06 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155289+0000", 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8990000000000003e-06 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155298+0000", 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5279999999999992e-06 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155304+0000", 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2349999999999999e-06 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155356+0000", 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1968000000000001e-05 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155365+0000", 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8320000000000001e-06 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119602+0000", 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96423660600000005 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119610+0000", 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218745+0000", 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.099135456 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218881+0000", 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013615 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,7", 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.219298+0000", 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000416959 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.235147+0000", 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015848504999999999 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.235187+0000", 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9855999999999999e-05 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236722+0000", 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001535518 2024-09-16T10:32:16.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236762+0000", 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9521000000000003e-05 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236852+0000", 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0816999999999995e-05 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236969+0000", 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117069 2024-09-16T10:32:16.129 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127341 11.4 11:22f27910:::benchmark_data_smithi063_176907_object3065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.175921+0000", 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.848910177, 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0649205770000001, 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127341 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175921+0000", 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175921+0000", 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175923+0000", 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6109999999999999e-06 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175927+0000", 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0679999999999998e-06 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175938+0000", 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1695000000000001e-05 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175946+0000", 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5390000000000002e-06 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.176015+0000", 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9643999999999999e-05 2024-09-16T10:32:16.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.176022+0000", 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1770000000000003e-06 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119642+0000", 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94362000899999998 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119649+0000", 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5129999999999999e-06 2024-09-16T10:32:16.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231508+0000", 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1118593029999999 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231642+0000", 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013385099999999999 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,7", 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231786+0000", 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014369 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.237011+0000", 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052247439999999999 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.237153+0000", 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142595 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.238291+0000", 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001137569 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.240653+0000", 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023618630000000001 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.240715+0000", 2024-09-16T10:32:16.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1739e-05 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.240841+0000", 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012696699999999999 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149389 11.3 11:cc3dd8fd:::benchmark_data_smithi063_176907_object25113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.656263+0000", 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.368567980999998, 2024-09-16T10:32:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98310017500000002, 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149389 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656263+0000", 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656263+0000", 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656265+0000", 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.922e-06 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656271+0000", 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0349999999999998e-06 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656286+0000", 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4693e-05 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656348+0000", 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1939000000000004e-05 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656449+0000", 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101623 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656458+0000", 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7479999999999999e-06 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.634927+0000", 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.978468428 2024-09-16T10:32:16.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.635019+0000", 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2315e-05 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.635154+0000", 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013515700000000001 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639180+0000", 2024-09-16T10:32:16.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004026338 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639211+0000", 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0277000000000002e-05 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639278+0000", 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7537000000000001e-05 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639331+0000", 2024-09-16T10:32:16.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2815999999999997e-05 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639363+0000", 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2347e-05 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149392 11.3 11:c4eb902d:::benchmark_data_smithi063_176907_object25116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.659065+0000", 2024-09-16T10:32:16.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.365766086999997, 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98378280200000001, 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149392 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659065+0000", 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659065+0000", 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659067+0000", 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.573e-06 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659071+0000", 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4970000000000003e-06 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659084+0000", 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2651e-05 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659090+0000", 2024-09-16T10:32:16.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9739999999999999e-06 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659140+0000", 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0262000000000001e-05 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659148+0000", 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5939999999999996e-06 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.637503+0000", 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97835563800000003 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.637581+0000", 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7786000000000004e-05 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.637674+0000", 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2897000000000002e-05 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642747+0000", 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050729399999999997 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642756+0000", 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8429999999999999e-06 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642787+0000", 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1420000000000001e-05 2024-09-16T10:32:16.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642820+0000", 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3306000000000003e-05 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642848+0000", 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7421000000000001e-05 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149393 11.3 11:cca5cdfb:::benchmark_data_smithi063_176907_object25117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.659236+0000", 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.365595155000001, 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.987797657, 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149393 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.153 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659236+0000", 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659236+0000", 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659237+0000", 2024-09-16T10:32:16.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.111e-06 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659241+0000", 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.279e-06 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659249+0000", 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8050000000000007e-06 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659256+0000", 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8709999999999999e-06 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659305+0000", 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9387e-05 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659312+0000", 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.742e-06 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639407+0000", 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98009459399999999 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639508+0000", 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010131100000000001 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639582+0000", 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3623999999999998e-05 2024-09-16T10:32:16.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642882+0000", 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003300594 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642892+0000", 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5009999999999993e-06 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.646945+0000", 2024-09-16T10:32:16.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004053297 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.646993+0000", 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8083e-05 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647034+0000", 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0457999999999998e-05 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149430 11.3 11:cf4e2810:::benchmark_data_smithi063_176907_object25154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.692980+0000", 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.331851211, 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95419773900000004, 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149430 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692980+0000", 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692980+0000", 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692982+0000", 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5809999999999999e-06 2024-09-16T10:32:16.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692988+0000", 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8799999999999996e-06 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693003+0000", 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4433999999999999e-05 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693011+0000", 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3559999999999993e-06 2024-09-16T10:32:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693074+0000", 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2624000000000002e-05 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693084+0000", 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9559999999999992e-06 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.641063+0000", 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94797921399999996 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.641172+0000", 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108863 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.641270+0000", 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8393000000000005e-05 2024-09-16T10:32:16.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642933+0000", 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001662576 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.644750+0000", 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001817335 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647081+0000", 2024-09-16T10:32:16.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023307860000000001 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647136+0000", 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5220999999999999e-05 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647178+0000", 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1520000000000002e-05 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149459 11.3 11:c35b989e:::benchmark_data_smithi063_176907_object25183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.716639+0000", 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.308191991000001, 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93083209499999997, 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149459 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.167 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716639+0000", 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716639+0000", 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716641+0000", 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9719999999999999e-06 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716647+0000", 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7100000000000004e-06 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716659+0000", 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2567e-05 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716666+0000", 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9820000000000002e-06 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716722+0000", 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6128999999999997e-05 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716762+0000", 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9045000000000003e-05 2024-09-16T10:32:16.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642961+0000", 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92619985500000002 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.643021+0000", 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9336000000000002e-05 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:16.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.643120+0000", 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9610000000000003e-05 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647238+0000", 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041180230000000002 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647252+0000", 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3749e-05 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647381+0000", 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012919900000000001 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647433+0000", 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1696000000000001e-05 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647471+0000", 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8222000000000001e-05 2024-09-16T10:32:16.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149468 11.3 11:ce0133d3:::benchmark_data_smithi063_176907_object25192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.721717+0000", 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.303114379, 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93143064900000005, 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:16.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149468 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721717+0000", 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721717+0000", 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721719+0000", 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0109999999999999e-06 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721724+0000", 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4009999999999996e-06 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721738+0000", 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3915e-05 2024-09-16T10:32:16.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721746+0000", 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0330000000000008e-06 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721819+0000", 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2772000000000001e-05 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721828+0000", 2024-09-16T10:32:16.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0389999999999993e-06 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.644810+0000", 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92298208800000003 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.644915+0000", 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010486500000000001 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:32:16.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.645020+0000", 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104895 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647259+0000", 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022390140000000001 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647311+0000", 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1975999999999998e-05 2024-09-16T10:32:16.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.652409+0000", 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050983360000000002 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.652460+0000", 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1100999999999997e-05 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.653147+0000", 2024-09-16T10:32:16.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068720299999999995 2024-09-16T10:32:16.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:16.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:16.181 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:32:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:16.261 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:16.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:16.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:16.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:16.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:16.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:16.514 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:16.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:16 smithi165 ceph-mon[32696]: osdmap e532: 8 total, 8 up, 7 in 2024-09-16T10:32:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:16 smithi165 ceph-mon[32696]: 3.18 scrub starts 2024-09-16T10:32:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:16 smithi165 ceph-mon[32696]: 3.18 scrub ok 2024-09-16T10:32:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:16 smithi165 ceph-mon[32696]: pgmap v1006: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 7858/188766 objects misplaced (4.163%); 0 B/s, 0 objects/s recovering 2024-09-16T10:32:16.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:16.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:16.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:16.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[29993]: osdmap e532: 8 total, 8 up, 7 in 2024-09-16T10:32:16.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[29993]: 3.18 scrub starts 2024-09-16T10:32:16.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[29993]: 3.18 scrub ok 2024-09-16T10:32:16.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[29993]: pgmap v1006: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 7858/188766 objects misplaced (4.163%); 0 B/s, 0 objects/s recovering 2024-09-16T10:32:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[39470]: osdmap e532: 8 total, 8 up, 7 in 2024-09-16T10:32:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[39470]: 3.18 scrub starts 2024-09-16T10:32:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[39470]: 3.18 scrub ok 2024-09-16T10:32:16.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:16 smithi063 ceph-mon[39470]: pgmap v1006: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 7858/188766 objects misplaced (4.163%); 0 B/s, 0 objects/s recovering 2024-09-16T10:32:16.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:16.787 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:16.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:16.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:16.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:16.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:17.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:17.038 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:32:17.038 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:32:17.039 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:32:17.039 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:32:17.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:17.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:17.055 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:32:17.056 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:32:17.095 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:32:17.095 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:32:17.129 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:32:17.130 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:32:17.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:17.166 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:32:17.166 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:32:17.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:32:17.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:17.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:17.206 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:32:17.206 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:32:17.224 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:17.242 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:32:17.243 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:32:17.247 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:32:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:32:17.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:17.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:17.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:17.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:17.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:17.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:17.502 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:17.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:17.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:17 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:32:17.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:17 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:32:17.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:17 smithi165 ceph-mon[32696]: 1.a scrub starts 2024-09-16T10:32:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:17 smithi165 ceph-mon[32696]: 1.a scrub ok 2024-09-16T10:32:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:17 smithi165 ceph-mon[32696]: osdmap e533: 8 total, 8 up, 7 in 2024-09-16T10:32:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:17 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1448608583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:17.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:17.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:17.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:32:17.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:32:17.619 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[29993]: 1.a scrub starts 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[29993]: 1.a scrub ok 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[29993]: osdmap e533: 8 total, 8 up, 7 in 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1448608583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[39470]: 1.a scrub starts 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[39470]: 1.a scrub ok 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[39470]: osdmap e533: 8 total, 8 up, 7 in 2024-09-16T10:32:17.620 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:17 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1448608583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:17.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:17.719 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:17.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:17.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:32:17.821 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:17.864 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:17.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:17.965 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:18.023 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:18.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:18.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:18.171 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:18.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:18.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:18.322 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:18.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:18.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:18.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:18.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:18.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:18.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[29993]: 11.d scrub starts 2024-09-16T10:32:18.585 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:32:18.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[29993]: 1.e scrub ok 2024-09-16T10:32:18.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[29993]: pgmap v1008: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 975 B/s rd, 0 B/s wr, 1.16k op/s; 7858/188766 objects misplaced (4.163%); 0 B/s, 0 objects/s recovering 2024-09-16T10:32:18.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[39470]: 11.d scrub starts 2024-09-16T10:32:18.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:32:18.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[39470]: 1.e scrub ok 2024-09-16T10:32:18.586 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:18 smithi063 ceph-mon[39470]: pgmap v1008: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 975 B/s rd, 0 B/s wr, 1.16k op/s; 7858/188766 objects misplaced (4.163%); 0 B/s, 0 objects/s recovering 2024-09-16T10:32:18.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:18.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:18.688 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:18.705 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:18.759 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:18.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:18.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:18 smithi165 ceph-mon[32696]: 11.d scrub starts 2024-09-16T10:32:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:18 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:32:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:18 smithi165 ceph-mon[32696]: 1.e scrub ok 2024-09-16T10:32:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:18 smithi165 ceph-mon[32696]: pgmap v1008: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 975 B/s rd, 0 B/s wr, 1.16k op/s; 7858/188766 objects misplaced (4.163%); 0 B/s, 0 objects/s recovering 2024-09-16T10:32:18.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:18.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:18.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:18.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:19.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:19.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:19.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:19.156 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:32:19.013+0000 7f17f27f5640 -1 mon.a@0(leader).osd e533 definitely_dead 0 2024-09-16T10:32:19.170 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:19.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:19.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:19.350 INFO:teuthology.orchestra.run.smithi063.stderr:marked out osd.1. 2024-09-16T10:32:19.397 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.436 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:19.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:32:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:32:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[29993]: 11.d scrub ok 2024-09-16T10:32:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1010988268' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[29993]: from='client.26740 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[29993]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[39470]: 11.d scrub ok 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1010988268' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[39470]: from='client.26740 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-16T10:32:19.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:19 smithi063 ceph-mon[39470]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-16T10:32:19.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:19.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:32:19 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:32:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:19 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:32:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:19 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:32:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:19 smithi165 ceph-mon[32696]: 11.d scrub ok 2024-09-16T10:32:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1010988268' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-16T10:32:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:19 smithi165 ceph-mon[32696]: from='client.26740 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-16T10:32:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:19 smithi165 ceph-mon[32696]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-16T10:32:19.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:19.614 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-16T10:32:19.614 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:32:19.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:19.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:19.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:19.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:19.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:19.929 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:19.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:20.030 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:20.046 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:32:20.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:20.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:20.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:20.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:20.281 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:20.309 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:20.362 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:32:20.390 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:20.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:20.462 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:20.565 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:20.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:20.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[29993]: from='client.26740 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[29993]: osdmap e534: 8 total, 8 up, 6 in 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[29993]: pgmap v1010: 168 pgs: 13 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.19k op/s; 6661/173391 objects misplaced (3.842%); 6.7 MiB/s, 107 objects/s recovering 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1501057343' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[39470]: from='client.26740 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[39470]: osdmap e534: 8 total, 8 up, 6 in 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[39470]: pgmap v1010: 168 pgs: 13 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.19k op/s; 6661/173391 objects misplaced (3.842%); 6.7 MiB/s, 107 objects/s recovering 2024-09-16T10:32:20.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:20 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1501057343' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:32:20.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:20 smithi165 ceph-mon[32696]: from='client.26740 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-16T10:32:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:20 smithi165 ceph-mon[32696]: osdmap e534: 8 total, 8 up, 6 in 2024-09-16T10:32:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:20 smithi165 ceph-mon[32696]: pgmap v1010: 168 pgs: 13 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.19k op/s; 6661/173391 objects misplaced (3.842%); 6.7 MiB/s, 107 objects/s recovering 2024-09-16T10:32:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:20 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1501057343' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:32:20.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:20.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:20.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:20.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:20.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:21.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:32:21.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:21.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:21.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:21.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:21.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:21.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:21.316 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:21.344 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:21.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:21.351 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:21.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:21.481 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:21.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:21.582 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:21.612 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:21.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:21.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: 11.2 scrub starts 2024-09-16T10:32:21.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-16T10:32:21.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:21.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:21.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1501057343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:32:21.624 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: 11.e continuing backfill to osd.0 from (512'8379,533'12107] 11:71443fa5:::benchmark_data_smithi063_176907_object114347:head to 533'12107 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 533'12107 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: 11.4 starting backfill to osd.3 from (0'0,0'0] MAX to 533'11731 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[29993]: osdmap e535: 8 total, 8 up, 6 in 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: 11.2 scrub starts 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1501057343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: 11.e continuing backfill to osd.0 from (512'8379,533'12107] 11:71443fa5:::benchmark_data_smithi063_176907_object114347:head to 533'12107 2024-09-16T10:32:21.625 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 533'12107 2024-09-16T10:32:21.626 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: 11.4 starting backfill to osd.3 from (0'0,0'0] MAX to 533'11731 2024-09-16T10:32:21.626 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:21 smithi063 ceph-mon[39470]: osdmap e535: 8 total, 8 up, 6 in 2024-09-16T10:32:21.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:21.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:21.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:21.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:21.763 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:32:21.763 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:32:21.798 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:32:21.798 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:32:21.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: 11.2 scrub starts 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1501057343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: 11.e continuing backfill to osd.0 from (512'8379,533'12107] 11:71443fa5:::benchmark_data_smithi063_176907_object114347:head to 533'12107 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 533'12107 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: 11.4 starting backfill to osd.3 from (0'0,0'0] MAX to 533'11731 2024-09-16T10:32:21.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:21 smithi165 ceph-mon[32696]: osdmap e535: 8 total, 8 up, 6 in 2024-09-16T10:32:21.836 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:32:21.836 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:32:21.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:21.871 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:32:21.871 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:32:21.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:21.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:21.905 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:32:21.905 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:32:21.938 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:32:21.939 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:32:21.974 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:32:21.975 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:32:21.976 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:22.009 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:32:22.009 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:32:22.012 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:22.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:22.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:22.240 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:22.286 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:22.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:22.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:22.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: osdmap e536: 8 total, 8 up, 6 in 2024-09-16T10:32:22.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 11.3 starting backfill to osd.2 from (0'0,0'0] MAX to 533'12338 2024-09-16T10:32:22.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.12 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.b starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 533'11992 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 533'12094 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 533'12014 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.0 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 533'12106 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 533'11942 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.13 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.14 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 533'11877 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: pgmap v1013: 168 pgs: 1 unknown, 31 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 133 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 6661/172629 objects misplaced (3.859%); 6.8 MiB/s, 108 objects/s recovering 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/508802501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:32:22.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:22 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:32:22.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:22.572 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:22.669 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:32:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:32:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:32:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:32:22.670 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:32:22.673 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: osdmap e536: 8 total, 8 up, 6 in 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 11.3 starting backfill to osd.2 from (0'0,0'0] MAX to 533'12338 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.12 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.b starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 533'11992 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 533'12094 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 533'12014 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.0 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 533'12106 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 533'11942 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.13 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.14 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 533'11877 2024-09-16T10:32:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: pgmap v1013: 168 pgs: 1 unknown, 31 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 133 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 6661/172629 objects misplaced (3.859%); 6.8 MiB/s, 108 objects/s recovering 2024-09-16T10:32:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/508802501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:32:22.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: osdmap e536: 8 total, 8 up, 6 in 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 11.3 starting backfill to osd.2 from (0'0,0'0] MAX to 533'12338 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.12 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.b starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 11.7 starting backfill to osd.0 from (0'0,0'0] MAX to 533'11992 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 533'12094 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 533'12014 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.0 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 11.f starting backfill to osd.4 from (0'0,0'0] MAX to 533'12106 2024-09-16T10:32:22.704 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 533'11942 2024-09-16T10:32:22.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.13 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.14 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:32:22.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 533'11877 2024-09-16T10:32:22.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: pgmap v1013: 168 pgs: 1 unknown, 31 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 133 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 1.03k op/s; 6661/172629 objects misplaced (3.859%); 6.8 MiB/s, 108 objects/s recovering 2024-09-16T10:32:22.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/508802501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:22.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:32:22.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:22 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:32:22.716 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:22.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:22.849 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:22.950 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:22.976 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:22.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:23.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:23.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:23.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:23.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:23.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:23.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:23.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:23.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:23.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:23.462 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:32:23.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:23.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:23.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:23.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:32:23.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:23.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:23.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:23.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:23 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:32:23.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:23 smithi063 ceph-mon[29993]: osdmap e537: 8 total, 8 up, 6 in 2024-09-16T10:32:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:23 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:32:23.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:23 smithi063 ceph-mon[39470]: osdmap e537: 8 total, 8 up, 6 in 2024-09-16T10:32:23.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:23.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:23.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:23.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:23 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:32:23.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:23 smithi165 ceph-mon[32696]: osdmap e537: 8 total, 8 up, 6 in 2024-09-16T10:32:23.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:23.930 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:32:23.931 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:23.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:24.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:24.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:24.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:24.158 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:24.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:24.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:24.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:24.313 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:24.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:24.414 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:24.441 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:24.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:24.546 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:24.551 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:32:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:32:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:32:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:32:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216464 10.11 e443/442 10:8bbe5995:::benchmark_data_smithi063_149043_object84027:head v 443'11994, mlcod=389'6270)", 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.55410214299999, 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180527202, 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994108+0000", 2024-09-16T10:32:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1179999999999999e-06 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994115+0000", 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7109999999999998e-06 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994124+0000", 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8120000000000002e-06 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994131+0000", 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2929999999999997e-06 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996650+0000", 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002519777 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996675+0000", 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4800999999999999e-05 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174595+0000", 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17792013200000001 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174634+0000", 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8557999999999999e-05 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(10.1 443/442 [PushOp(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head, version: 391'6545, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head@391'6545, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.55145477100001, 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19788486499999999, 2024-09-16T10:32:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996756+0000", 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4810000000000001e-06 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996812+0000", 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6508999999999997e-05 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996831+0000", 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9009e-05 2024-09-16T10:32:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996849+0000", 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7612999999999999e-05 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193170+0000", 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19632173999999999 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193181+0000", 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0138e-05 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.194639+0000", 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001458375 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216466 10.9 e443/442 10:96e6e3aa:::benchmark_data_smithi063_149043_object84029:head v 443'12076, mlcod=386'6101)", 2024-09-16T10:32:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.55101729500001, 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18400087900000001, 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997193+0000", 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2759999999999999e-06 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997200+0000", 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8950000000000001e-06 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997209+0000", 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2010000000000005e-06 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997226+0000", 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7059e-05 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002158+0000", 2024-09-16T10:32:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049319699999999999 2024-09-16T10:32:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002173+0000", 2024-09-16T10:32:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4661000000000001e-05 2024-09-16T10:32:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181097+0000", 2024-09-16T10:32:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.178924214 2024-09-16T10:32:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181192+0000", 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5602999999999999e-05 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216473 10.9 e443/442 10:9420f438:::benchmark_data_smithi063_149043_object84036:head v 443'12077, mlcod=386'6101)", 2024-09-16T10:32:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.54738732600001, 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18039601599999999, 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000823+0000", 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.221e-06 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000830+0000", 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1319999999999996e-06 2024-09-16T10:32:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000840+0000", 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0378e-05 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000847+0000", 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4710000000000004e-06 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007645+0000", 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067982770000000001 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007677+0000", 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2172999999999998e-05 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181195+0000", 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17351757300000001 2024-09-16T10:32:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181218+0000", 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2790999999999999e-05 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216481 10.11 e443/442 10:8cd97c52:::benchmark_data_smithi063_149043_object84044:head v 443'11995, mlcod=389'6270)", 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:32:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.542229324, 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18177754400000001, 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005981+0000", 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.618e-06 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005991+0000", 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4660000000000005e-06 2024-09-16T10:32:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006004+0000", 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2964e-05 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006012+0000", 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5939999999999994e-06 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007635+0000", 2024-09-16T10:32:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001622706 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007678+0000", 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3282000000000003e-05 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187680+0000", 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180001892 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187757+0000", 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7021999999999995e-05 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216483 10.9 e443/442 10:91f62e6a:::benchmark_data_smithi063_149043_object84046:head v 443'12078, mlcod=386'6101)", 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.54108066699999, 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.185520875, 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007129+0000", 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.198e-06 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007136+0000", 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7780000000000002e-06 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007158+0000", 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2013000000000001e-05 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007166+0000", 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999999e-06 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014305+0000", 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071398319999999996 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014326+0000", 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0767999999999998e-05 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192615+0000", 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17828850800000001 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192649+0000", 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4366999999999998e-05 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216472 10.8 e443/442 10:167e16b6:::benchmark_data_smithi063_149043_object84035:head v 443'11858, mlcod=389'6215)", 2024-09-16T10:32:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.53942427999999, 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.167553278, 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008786+0000", 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.296e-06 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008820+0000", 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4329999999999998e-05 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008830+0000", 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3360000000000001e-06 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008836+0000", 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3040000000000001e-06 2024-09-16T10:32:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010938+0000", 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021016780000000001 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010959+0000", 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1454000000000002e-05 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176290+0000", 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16533061800000001 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176338+0000", 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8262e-05 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216489 10.c e443/442 10:36cb7a7c:::benchmark_data_smithi063_149043_object84052:head v 443'13145, mlcod=443'13143)", 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:32:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.53555766900001, 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16334574800000001, 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012653+0000", 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.606e-06 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012661+0000", 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0469999999999994e-06 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012673+0000", 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2490000000000001e-05 2024-09-16T10:32:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012682+0000", 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3450000000000006e-06 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012753+0000", 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1565999999999999e-05 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012773+0000", 2024-09-16T10:32:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9942999999999999e-05 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175914+0000", 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16314114399999999 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175997+0000", 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2607000000000004e-05 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216490 10.10 e443/441 10:0ed1dccc:::benchmark_data_smithi063_149043_object84053:head v 443'12109, mlcod=443'12107)", 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.53353573300001, 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16562529300000001, 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014675+0000", 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.593e-06 2024-09-16T10:32:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014687+0000", 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2071e-05 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014697+0000", 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0148e-05 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014704+0000", 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.613e-06 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174625+0000", 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15992183700000001 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6707e-05 2024-09-16T10:32:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180235+0000", 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055831880000000002 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180298+0000", 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3136e-05 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216488 10.11 e443/442 10:8a4eba15:::benchmark_data_smithi063_149043_object84051:head v 443'11996, mlcod=389'6270)", 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.532604658, 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.177066683, 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015606+0000", 2024-09-16T10:32:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.404e-06 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015617+0000", 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1769e-05 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015628+0000", 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0489000000000001e-05 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015634+0000", 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2960000000000004e-06 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174659+0000", 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15902495699999999 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174681+0000", 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1778999999999999e-05 2024-09-16T10:32:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192652+0000", 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017971311 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192671+0000", 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8678e-05 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216491 10.10 e443/441 10:0e46f635:::benchmark_data_smithi063_149043_object84054:head v 443'12110, mlcod=443'12107)", 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.53091830700001, 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.163036928, 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017292+0000", 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.443e-06 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017300+0000", 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174635+0000", 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.157335014 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174657+0000", 2024-09-16T10:32:24.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1985e-05 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176330+0000", 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016733029999999999 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176359+0000", 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9564e-05 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180301+0000", 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00394174 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180328+0000", 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6319e-05 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216495 10.c e443/442 10:33a8a0ac:::benchmark_data_smithi063_149043_object84058:head v 443'13146, mlcod=443'13143)", 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 233.52962331200001, 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.160016309, 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018587+0000", 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3820000000000001e-06 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018595+0000", 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4759999999999998e-06 2024-09-16T10:32:24.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174544+0000", 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15594872200000001 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174566+0000", 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1979000000000001e-05 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174626+0000", 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9899999999999999e-05 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.561e-05 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178550+0000", 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038981969999999999 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178602+0000", 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2043000000000003e-05 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221476 10.9 e446/442 10:900b709d:::benchmark_data_smithi063_149043_object89039:head v 446'12225, mlcod=386'6101)", 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 228.46956011500001, 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50937916299999997, 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.610 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078650+0000", 2024-09-16T10:32:24.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.091e-06 2024-09-16T10:32:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:24.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078657+0000", 2024-09-16T10:32:24.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-16T10:32:24.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078665+0000", 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6619999999999994e-06 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078671+0000", 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1700000000000002e-06 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078762+0000", 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0346999999999994e-05 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078786+0000", 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4314000000000001e-05 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587999+0000", 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.509212514 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.588028+0000", 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9326999999999999e-05 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221483 10.c e446/442 10:32ed840b:::benchmark_data_smithi063_149043_object89046:head v 446'13312, mlcod=445'13310)", 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 228.468831029, 2024-09-16T10:32:24.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50397308900000004, 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079379+0000", 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.395e-06 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079385+0000", 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5639999999999996e-06 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079392+0000", 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.481e-06 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079398+0000", 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8950000000000003e-06 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079450+0000", 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1786999999999997e-05 2024-09-16T10:32:24.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079469+0000", 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8841000000000001e-05 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583301+0000", 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50383221600000005 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583351+0000", 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9910000000000002e-05 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221482 10.1f e446/441 10:ffb425b8:::benchmark_data_smithi063_149043_object89045:head v 446'13070, mlcod=445'13068)", 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 228.468623576, 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50798407899999998, 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.621 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079587+0000", 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.187e-06 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079593+0000", 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0929999999999999e-06 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079600+0000", 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5070000000000003e-06 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079606+0000", 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7509999999999997e-06 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583349+0000", 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50374310499999997 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583378+0000", 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8649e-05 2024-09-16T10:32:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587530+0000", 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041521559999999997 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587569+0000", 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9631000000000001e-05 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.625 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:249520 10.1 10:827f4c35:::benchmark_data_smithi063_149043_object117083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 201.091244397, 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11004649, 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 249520 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456966+0000", 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.689e-06 2024-09-16T10:32:24.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456972+0000", 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3410000000000002e-06 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456983+0000", 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1039999999999999e-05 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456988+0000", 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6579999999999999e-06 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.457040+0000", 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2166000000000002e-05 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.457143+0000", 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102909 2024-09-16T10:32:24.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.458594+0000", 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001450914 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.460804+0000", 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022102369999999999 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.461370+0000", 2024-09-16T10:32:24.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056559799999999995 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.566893+0000", 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105523061 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.566950+0000", 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6637e-05 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.567011+0000", 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1240000000000003e-05 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:251245 10.1 10:810fe972:::benchmark_data_smithi063_149043_object118808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 199.657893986, 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083403509000000001, 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.632 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 251245 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890317+0000", 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.809e-06 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890322+0000", 2024-09-16T10:32:24.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5219999999999998e-06 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890333+0000", 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1069e-05 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890342+0000", 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7590000000000003e-06 2024-09-16T10:32:24.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890709+0000", 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036659099999999999 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890839+0000", 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000130147 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:32:24.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.891118+0000", 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027897099999999998 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.892553+0000", 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014351450000000001 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.892975+0000", 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042191700000000001 2024-09-16T10:32:24.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973569+0000", 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080593926999999996 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973658+0000", 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9295000000000005e-05 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973718+0000", 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0356999999999998e-05 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDelete(10.8 e460,442 10:146751e1:::benchmark_data_smithi063_149043_object32211:head,408'10091)", 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 196.43996561399999, 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111867522, 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.639 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108244+0000", 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2900000000000002e-07 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108248+0000", 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.121e-06 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219378+0000", 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11112931500000001 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219407+0000", 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9363e-05 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220044+0000", 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063718900000000003 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220053+0000", 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9199999999999993e-06 2024-09-16T10:32:24.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220111+0000", 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7685000000000001e-05 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:254612 10.22 e460/441 10:460a596d:::benchmark_data_smithi063_149043_object122175:head v 460'12456, mlcod=460'12454)", 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 196.428848657, 2024-09-16T10:32:24.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101663158, 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:32:24.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119362+0000", 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4839999999999999e-06 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119368+0000", 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.348e-06 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219363+0000", 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099994825999999995 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219389+0000", 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6383000000000001e-05 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219541+0000", 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151476 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219560+0000", 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9440999999999999e-05 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220981+0000", 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001421095 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.221023+0000", 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2104999999999998e-05 2024-09-16T10:32:24.647 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:72894 11.5 e493/476 11:ad5dc03e:::benchmark_data_smithi063_176907_object72893:head v 493'4542, mlcod=493'4539)", 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 120.816100601, 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060735732000000001, 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:32:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732109+0000", 2024-09-16T10:32:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.119e-06 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732131+0000", 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1999000000000001e-05 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732141+0000", 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4579999999999999e-06 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732147+0000", 2024-09-16T10:32:24.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2759999999999997e-06 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732195+0000", 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8288999999999998e-05 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732215+0000", 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9961000000000001e-05 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:32:24.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.792747+0000", 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060531198000000001 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.792844+0000", 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7431999999999998e-05 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:32:24.652 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:32:24.653 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:24.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:24.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[29993]: osdmap e538: 8 total, 8 up, 6 in 2024-09-16T10:32:24.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[29993]: pgmap v1016: 168 pgs: 1 unknown, 32 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 131 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 183 op/s; 12337/171186 objects misplaced (7.207%); 13 MiB/s, 203 objects/s recovering 2024-09-16T10:32:24.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:32:24.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:24.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[39470]: osdmap e538: 8 total, 8 up, 6 in 2024-09-16T10:32:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[39470]: pgmap v1016: 168 pgs: 1 unknown, 32 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 131 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 183 op/s; 12337/171186 objects misplaced (7.207%); 13 MiB/s, 203 objects/s recovering 2024-09-16T10:32:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:32:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:24.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:24.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:24.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:24.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:24.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:24 smithi165 ceph-mon[32696]: osdmap e538: 8 total, 8 up, 6 in 2024-09-16T10:32:24.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:24 smithi165 ceph-mon[32696]: pgmap v1016: 168 pgs: 1 unknown, 32 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 131 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 183 op/s; 12337/171186 objects misplaced (7.207%); 13 MiB/s, 203 objects/s recovering 2024-09-16T10:32:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:24 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:32:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:24 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:24.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:24.921 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:24.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:25.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:25.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:25.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:25.156 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:25.200 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:25.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:25.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:25.307 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 2] out_osds: [7, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T10:32:25.307 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T10:32:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:25.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:25.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:25.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:25.481 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:25.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:25.544 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T10:32:25.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:25.613 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:25 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:32:25.613 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:25 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:32:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:25 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:32:25.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:25 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:32:25.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:25.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:25.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:25.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:25.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:25.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:25 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:32:25.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:25 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:32:25.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:25.894 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:25.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:25.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:25.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:26.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:26.126 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:26.151 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:26.253 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:26.291 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:26.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:26.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:26.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:26.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:26.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:26.657 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:26.698 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:26.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:26 smithi063 ceph-mon[39470]: pgmap v1017: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 3.1 KiB/s wr, 522 op/s; 31490/164565 objects misplaced (19.135%); 30 MiB/s, 499 objects/s recovering 2024-09-16T10:32:26.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:26 smithi063 ceph-mon[39470]: Health check update: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-16T10:32:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:26 smithi063 ceph-mon[29993]: pgmap v1017: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 3.1 KiB/s wr, 522 op/s; 31490/164565 objects misplaced (19.135%); 30 MiB/s, 499 objects/s recovering 2024-09-16T10:32:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:26 smithi063 ceph-mon[29993]: Health check update: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-16T10:32:26.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:26.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:26 smithi165 ceph-mon[32696]: pgmap v1017: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 3.1 KiB/s wr, 522 op/s; 31490/164565 objects misplaced (19.135%); 30 MiB/s, 499 objects/s recovering 2024-09-16T10:32:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:26 smithi165 ceph-mon[32696]: Health check update: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-16T10:32:26.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:26.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:26.970 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:26.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:27.074 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.107 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:27.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:27.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:27.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:27.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:27.367 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:27.439 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:32:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:32:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:32:27.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:27.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:27.598 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.629 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:27.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:27.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:27 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:27.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:27 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 10 pgs peering) 2024-09-16T10:32:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:27 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:27.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:27 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 10 pgs peering) 2024-09-16T10:32:27.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:27.773 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:27.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:27 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:27 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 10 pgs peering) 2024-09-16T10:32:27.875 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:27.922 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:27.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:28.024 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:28.065 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:28.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:28.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:28.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:28.323 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:28.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:28.452 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:28.554 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:28.585 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:28.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:28.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:28 smithi063 ceph-mon[29993]: pgmap v1018: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 2.6 KiB/s wr, 437 op/s; 31490/164565 objects misplaced (19.135%); 25 MiB/s, 418 objects/s recovering 2024-09-16T10:32:28.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:28 smithi063 ceph-mon[39470]: pgmap v1018: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 2.6 KiB/s wr, 437 op/s; 31490/164565 objects misplaced (19.135%); 25 MiB/s, 418 objects/s recovering 2024-09-16T10:32:28.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:28.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:28 smithi165 ceph-mon[32696]: pgmap v1018: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 2.6 KiB/s wr, 437 op/s; 31490/164565 objects misplaced (19.135%); 25 MiB/s, 418 objects/s recovering 2024-09-16T10:32:28.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:28.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:28.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:28.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:28.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:29.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:29.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:29.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:29.201 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:29.224 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:29.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:29.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:29.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:29.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:29.501 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:29.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:29.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:32:29 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:32:29.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:29.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:29.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:29.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:29.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:29.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:29.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:29.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:29.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:29.983 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:30.027 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:30.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:30.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:30.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.305 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:30.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:30.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:30.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:30.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:30.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:30.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.730 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:30.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:30.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:30.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:30.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:30 smithi063 ceph-mon[39470]: pgmap v1019: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 2.0 KiB/s wr, 839 op/s; 26137/152466 objects misplaced (17.143%); 44 MiB/s, 725 objects/s recovering 2024-09-16T10:32:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:30.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:30 smithi063 ceph-mon[29993]: pgmap v1019: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 2.0 KiB/s wr, 839 op/s; 26137/152466 objects misplaced (17.143%); 44 MiB/s, 725 objects/s recovering 2024-09-16T10:32:30.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:30.996 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:30.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:30 smithi165 ceph-mon[32696]: pgmap v1019: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 2.0 KiB/s wr, 839 op/s; 26137/152466 objects misplaced (17.143%); 44 MiB/s, 725 objects/s recovering 2024-09-16T10:32:31.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:31.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:31.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:31.231 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:31.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:31.372 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:31.443 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:31.469 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:31.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:31.469 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:31.570 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:31.634 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:31.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:31.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:31.791 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:31.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:31.793 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:31.845 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:31 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:32:31.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:31 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:32:31.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:31.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:31.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:32.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:32.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:31 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:32:32.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:32.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:32.154 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:32.202 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:32.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:32.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:32.341 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:32.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:32.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:32.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:32.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:32.571 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:32.616 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:32.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:32.718 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:32.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:32.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[29993]: pgmap v1020: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.8 KiB/s wr, 766 op/s; 25877/151686 objects misplaced (17.060%); 39 MiB/s, 637 objects/s recovering 2024-09-16T10:32:32.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:32:32.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:32:32.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:32:32.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:32:32.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:32.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[39470]: pgmap v1020: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.8 KiB/s wr, 766 op/s; 25877/151686 objects misplaced (17.060%); 39 MiB/s, 637 objects/s recovering 2024-09-16T10:32:32.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:32:32.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:32:32.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:32:32.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:32:32.854 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:32.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:32.913 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:32.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:32.937 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:32:32.937 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:32:32.937 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 0 2024-09-16T10:32:32.938 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:32:33.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:33.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:33.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:32 smithi165 ceph-mon[32696]: pgmap v1020: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.8 KiB/s wr, 766 op/s; 25877/151686 objects misplaced (17.060%); 39 MiB/s, 637 objects/s recovering 2024-09-16T10:32:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:32 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:32:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:32 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:32:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:32:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:32:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:33.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:33.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:33.285 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:33.327 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:33.365 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:32:33.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:33.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:33.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:33.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:33.610 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T10:32:33.671 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:33 smithi063 ceph-mon[29993]: osdmap e539: 8 total, 8 up, 6 in 2024-09-16T10:32:33.671 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4162198901' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:32:33.671 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:33 smithi063 ceph-mon[29993]: from='client.26755 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:32:33.672 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:33 smithi063 ceph-mon[39470]: osdmap e539: 8 total, 8 up, 6 in 2024-09-16T10:32:33.672 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4162198901' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:32:33.672 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:33 smithi063 ceph-mon[39470]: from='client.26755 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:32:33.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:33.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:33.754 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T10:32:33.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:33.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:33.944 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:33.996 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:33.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:33.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:33 smithi165 ceph-mon[32696]: osdmap e539: 8 total, 8 up, 6 in 2024-09-16T10:32:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4162198901' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:32:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:33 smithi165 ceph-mon[32696]: from='client.26755 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:32:34.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:34.147 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:34.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:34.249 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:34.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:34.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:34.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:34.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:34.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:34.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T10:32:34.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:34.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:34.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:34.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:34 smithi063 ceph-mon[29993]: pgmap v1022: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 307 B/s rd, 1.6 KiB/s wr, 730 op/s; 24701/149280 objects misplaced (16.547%); 34 MiB/s, 554 objects/s recovering 2024-09-16T10:32:34.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:34 smithi063 ceph-mon[29993]: from='client.26755 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:32:34.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:34 smithi063 ceph-mon[29993]: osdmap e540: 8 total, 8 up, 6 in 2024-09-16T10:32:34.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:34 smithi063 ceph-mon[39470]: pgmap v1022: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 307 B/s rd, 1.6 KiB/s wr, 730 op/s; 24701/149280 objects misplaced (16.547%); 34 MiB/s, 554 objects/s recovering 2024-09-16T10:32:34.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:34 smithi063 ceph-mon[39470]: from='client.26755 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:32:34.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:34 smithi063 ceph-mon[39470]: osdmap e540: 8 total, 8 up, 6 in 2024-09-16T10:32:34.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:34.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:34.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:34.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:34.838 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:34.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:34.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:34.875 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:34.960 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:35.029 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:35.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:35.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:34 smithi165 ceph-mon[32696]: pgmap v1022: 168 pgs: 3 active+clean+scrubbing, 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 155 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 307 B/s rd, 1.6 KiB/s wr, 730 op/s; 24701/149280 objects misplaced (16.547%); 34 MiB/s, 554 objects/s recovering 2024-09-16T10:32:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:34 smithi165 ceph-mon[32696]: from='client.26755 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:32:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:34 smithi165 ceph-mon[32696]: osdmap e540: 8 total, 8 up, 6 in 2024-09-16T10:32:35.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:35.203 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:35.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:35.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:35.350 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:35.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:35.452 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:35.494 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:35.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:35.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:35.640 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:35.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:35.641 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:32:35.704 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:32:35.704 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:32:35.704 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:32:35.705 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:32:35.705 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:32:35.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:35.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:35.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:35.781 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2398404822' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:32:35.783 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2398404822' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:32:35.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:35.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:35.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:36.001 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:36.049 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:36.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:36.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2398404822' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:32:36.152 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:36.196 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:36.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:36.297 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:36.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:36.445 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:36.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:36.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:36.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:36.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:36.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:36.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:36.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:36.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:36.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:36.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:36.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[29993]: pgmap v1024: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 20732/137961 objects misplaced (15.027%); 48 MiB/s, 775 objects/s recovering 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2398404822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[29993]: osdmap e541: 8 total, 8 up, 6 in 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3912943597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[39470]: pgmap v1024: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 20732/137961 objects misplaced (15.027%); 48 MiB/s, 775 objects/s recovering 2024-09-16T10:32:36.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:36.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2398404822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:32:36.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[39470]: osdmap e541: 8 total, 8 up, 6 in 2024-09-16T10:32:36.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3912943597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:36.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:36 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:36.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:37.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:37.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:36 smithi165 ceph-mon[32696]: pgmap v1024: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 20732/137961 objects misplaced (15.027%); 48 MiB/s, 775 objects/s recovering 2024-09-16T10:32:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:36 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2398404822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:32:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:36 smithi165 ceph-mon[32696]: osdmap e541: 8 total, 8 up, 6 in 2024-09-16T10:32:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:36 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3912943597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:36 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:37.116 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:37.155 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:37.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:37.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:37.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.419 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:37.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:37.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:32:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:32:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:32:37.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.558 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:37.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:37.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:37.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:37.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:37.818 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:37.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:37.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:37.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:38.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:38.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:38.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:38.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:38.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:38.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:38.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:38.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:38.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:38.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:38.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:38.587 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:38.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:38.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:38.754 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:38.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:38.856 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:38.897 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:38.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:38.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:38 smithi063 ceph-mon[29993]: pgmap v1026: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 763 op/s; 20732/137961 objects misplaced (15.027%); 31 MiB/s, 495 objects/s recovering 2024-09-16T10:32:38.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:38 smithi063 ceph-mon[39470]: pgmap v1026: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 763 op/s; 20732/137961 objects misplaced (15.027%); 31 MiB/s, 495 objects/s recovering 2024-09-16T10:32:38.999 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:39.042 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:39.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:39.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:38 smithi165 ceph-mon[32696]: pgmap v1026: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.8 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 763 op/s; 20732/137961 objects misplaced (15.027%); 31 MiB/s, 495 objects/s recovering 2024-09-16T10:32:39.085 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:32:39.086 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:32:39.086 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:32:39.086 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:32:39.086 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:32:39.086 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:32:39.086 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:32:39.086 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:32:39.087 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:32:39.144 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:39.186 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:39.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:39.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:39.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:39.432 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:39.471 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:39.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:39.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:32:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:32:39.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:39.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:39.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:39.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:39.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:39.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:39.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:39.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:39.970 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:40.009 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:40.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:40.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:40.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:40.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:40.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:40.414 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:40.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:40.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[29993]: pgmap v1027: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 740 B/s rd, 0 B/s wr, 1.31k op/s; 15916/124458 objects misplaced (12.788%); 55 MiB/s, 886 objects/s recovering 2024-09-16T10:32:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:40.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:40.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:40.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:40 smithi063 ceph-mon[39470]: pgmap v1027: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 740 B/s rd, 0 B/s wr, 1.31k op/s; 15916/124458 objects misplaced (12.788%); 55 MiB/s, 886 objects/s recovering 2024-09-16T10:32:40.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:40.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:40.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:40.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:40.757 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T10:32:40.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:40.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:40.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:40 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:32:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:40 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:40.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:40 smithi165 ceph-mon[32696]: pgmap v1027: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 740 B/s rd, 0 B/s wr, 1.31k op/s; 15916/124458 objects misplaced (12.788%); 55 MiB/s, 886 objects/s recovering 2024-09-16T10:32:40.826 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:40.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:40.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:41.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:41.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:41.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:41.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:41.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.355 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:41.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:41.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:41.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:41.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:41.607 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:41.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:41 smithi063 ceph-mon[39470]: osdmap e542: 8 total, 8 up, 6 in 2024-09-16T10:32:41.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:41 smithi063 ceph-mon[39470]: osdmap e543: 8 total, 8 up, 6 in 2024-09-16T10:32:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:41 smithi063 ceph-mon[29993]: osdmap e542: 8 total, 8 up, 6 in 2024-09-16T10:32:41.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:41 smithi063 ceph-mon[29993]: osdmap e543: 8 total, 8 up, 6 in 2024-09-16T10:32:41.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:41.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:41.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:41 smithi165 ceph-mon[32696]: osdmap e542: 8 total, 8 up, 6 in 2024-09-16T10:32:41.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:41 smithi165 ceph-mon[32696]: osdmap e543: 8 total, 8 up, 6 in 2024-09-16T10:32:41.833 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.856 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:41.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:41.957 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:41.982 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:41.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:42.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:42.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:42.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:42.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:42.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:42.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:42.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:42.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:32:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:42.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:42.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:42.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:42.481 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:42.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:42.574 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:42.582 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:42.584 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:42.603 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:42 smithi063 ceph-mon[39470]: pgmap v1030: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 806 op/s; 15582/123456 objects misplaced (12.622%); 33 MiB/s, 525 objects/s recovering 2024-09-16T10:32:42.603 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:42 smithi063 ceph-mon[39470]: osdmap e544: 8 total, 8 up, 6 in 2024-09-16T10:32:42.604 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:42 smithi063 ceph-mon[29993]: pgmap v1030: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 806 op/s; 15582/123456 objects misplaced (12.622%); 33 MiB/s, 525 objects/s recovering 2024-09-16T10:32:42.604 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:42 smithi063 ceph-mon[29993]: osdmap e544: 8 total, 8 up, 6 in 2024-09-16T10:32:42.639 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:42.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:42.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:42 smithi165 ceph-mon[32696]: pgmap v1030: 168 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 806 op/s; 15582/123456 objects misplaced (12.622%); 33 MiB/s, 525 objects/s recovering 2024-09-16T10:32:42.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:42 smithi165 ceph-mon[32696]: osdmap e544: 8 total, 8 up, 6 in 2024-09-16T10:32:42.727 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:42.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:42.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:42.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:42.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:42.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:42.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:42.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:43.012 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:43.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:43.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:43.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:43.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:43.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:43.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:43.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:43.547 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:43.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:43.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:43.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:43.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:43.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:43.950 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:43.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:44.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:44.090 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:44.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:44.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:44.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:44.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:44.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:44.338 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:44.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:44.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:44 smithi063 ceph-mon[29993]: osdmap e545: 8 total, 8 up, 6 in 2024-09-16T10:32:44.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:44 smithi063 ceph-mon[29993]: pgmap v1033: 168 pgs: 1 peering, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 215 op/s; 14431/121890 objects misplaced (11.839%); 12 MiB/s, 196 objects/s recovering 2024-09-16T10:32:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:44 smithi063 ceph-mon[39470]: osdmap e545: 8 total, 8 up, 6 in 2024-09-16T10:32:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:44 smithi063 ceph-mon[39470]: pgmap v1033: 168 pgs: 1 peering, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 215 op/s; 14431/121890 objects misplaced (11.839%); 12 MiB/s, 196 objects/s recovering 2024-09-16T10:32:44.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:44.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:44 smithi165 ceph-mon[32696]: osdmap e545: 8 total, 8 up, 6 in 2024-09-16T10:32:44.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:44 smithi165 ceph-mon[32696]: pgmap v1033: 168 pgs: 1 peering, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 158 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 215 op/s; 14431/121890 objects misplaced (11.839%); 12 MiB/s, 196 objects/s recovering 2024-09-16T10:32:44.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:44.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:32:44.617 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:44.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:44.719 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:44.769 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:44.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:44.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:44.919 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:44.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:45.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:45.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:45.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:45.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:45.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:45.270 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:45.307 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:45.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:45.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:45.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:45.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:45.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:45.683 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:45.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:45.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:45.910 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:45.933 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:45.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:46.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:46.057 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:46.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:46.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:46.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:46.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:46.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:46.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:46.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:46 smithi063 ceph-mon[29993]: osdmap e546: 8 total, 8 up, 6 in 2024-09-16T10:32:46.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:46 smithi063 ceph-mon[29993]: pgmap v1035: 168 pgs: 1 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 161 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 645 op/s; 11388/114951 objects misplaced (9.907%); 45 MiB/s, 727 objects/s recovering 2024-09-16T10:32:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:46 smithi063 ceph-mon[39470]: osdmap e546: 8 total, 8 up, 6 in 2024-09-16T10:32:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:46 smithi063 ceph-mon[39470]: pgmap v1035: 168 pgs: 1 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 161 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 645 op/s; 11388/114951 objects misplaced (9.907%); 45 MiB/s, 727 objects/s recovering 2024-09-16T10:32:46.535 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:46.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:46 smithi165 ceph-mon[32696]: osdmap e546: 8 total, 8 up, 6 in 2024-09-16T10:32:46.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:46 smithi165 ceph-mon[32696]: pgmap v1035: 168 pgs: 1 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 161 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 645 op/s; 11388/114951 objects misplaced (9.907%); 45 MiB/s, 727 objects/s recovering 2024-09-16T10:32:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:46.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:46.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:46.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:46.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:46.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:46.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:46.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:46.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:47.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:47.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:47.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:47.157 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T10:32:47.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:47.244 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:47.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:47.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:47.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:47.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:47.374 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:32:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:32:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:32:47.374 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:47 smithi063 ceph-mon[29993]: osdmap e547: 8 total, 8 up, 6 in 2024-09-16T10:32:47.375 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1673121045' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:32:47.375 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:47 smithi063 ceph-mon[39470]: osdmap e547: 8 total, 8 up, 6 in 2024-09-16T10:32:47.375 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1673121045' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:32:47.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:47.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:47.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:47.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:47 smithi165 ceph-mon[32696]: osdmap e547: 8 total, 8 up, 6 in 2024-09-16T10:32:47.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1673121045' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:32:47.564 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:32:47.564 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:88281 10.c 10:356f8e40:::benchmark_data_smithi063_149043_object88280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 380.021243857, 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.020604785, 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:32:47.565 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 88281 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537287+0000", 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537289+0000", 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3520000000000001e-06 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537316+0000", 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7038000000000001e-05 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.567 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537335+0000", 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8247e-05 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537343+0000", 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7239999999999998e-06 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543029+0000", 2024-09-16T10:32:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056852450000000002 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543036+0000", 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2749999999999998e-06 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.559029+0000", 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015993265 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.559039+0000", 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0185e-05 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.556377+0000", 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99733790600000005 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.557892+0000", 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0015145479999999999 2024-09-16T10:32:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:88285 10.c 10:348a9cd0:::benchmark_data_smithi063_149043_object88284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 380.021174895, 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0211189380000001, 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 88285 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537356+0000", 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537357+0000", 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9500000000000002e-07 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537424+0000", 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7220999999999999e-05 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537459+0000", 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5497999999999998e-05 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.537467+0000", 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6450000000000005e-06 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543085+0000", 2024-09-16T10:32:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056183730000000003 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting_for_map not empty", 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.543092+0000", 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2899999999999999e-06 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.562892+0000", 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.019800512999999999 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:27.562904+0000", 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1368e-05 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.558397+0000", 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.99549375600000001 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:28.558475+0000", 2024-09-16T10:32:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.7578999999999997e-05 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:98186 10.c 10:30a2e491:::benchmark_data_smithi063_149043_object98185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 370.59047094699997, 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0526488009999999, 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:32:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 98186 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968060+0000", 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968062+0000", 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.928e-06 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968086+0000", 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3671000000000001e-05 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968096+0000", 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0839000000000001e-05 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.968104+0000", 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8890000000000002e-06 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.974858+0000", 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067535399999999997 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.975006+0000", 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014837499999999999 2024-09-16T10:32:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.975142+0000", 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013551899999999999 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.986786+0000", 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011644408 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.992120+0000", 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0053334860000000001 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.012439+0000", 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.020319265999999999 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.012449+0000", 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0223e-05 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020605+0000", 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0081562230000001 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020709+0000", 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103434 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.16152.0:98194 10.c 10:3f81be78:::benchmark_data_smithi063_149043_object98193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 370.580470521, 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.050357231, 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.16152", 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 98194 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978060+0000", 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978063+0000", 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4260000000000002e-06 2024-09-16T10:32:47.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978090+0000", 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7475e-05 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978103+0000", 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2544000000000001e-05 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.978110+0000", 2024-09-16T10:32:47.587 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0060000000000003e-06 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980486+0000", 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023762700000000002 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980605+0000", 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011847 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,7", 2024-09-16T10:32:47.588 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.980747+0000", 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014267800000000001 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:36.992126+0000", 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011379142 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.015722+0000", 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.023596004 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:37.015734+0000", 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2058999999999999e-05 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.020985+0000", 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.005250813 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.021617+0000", 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00063140899999999999 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.028349+0000", 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.006732839 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:26:38.028418+0000", 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8096000000000007e-05 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.591 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204629 11.2 e541/476 11:4f1a5dea:::benchmark_data_smithi063_176907_object80353:head v 541'12694, mlcod=541'12692)", 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.874950+0000", 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.683580789000001, 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015825622000000001, 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.874950+0000", 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.592 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.874950+0000", 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.874950+0000", 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8999999999999998e-07 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.593 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.874955+0000", 2024-09-16T10:32:47.597 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4290000000000003e-06 2024-09-16T10:32:47.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.597 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.597 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.874960+0000", 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8910000000000003e-06 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.874964+0000", 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3220000000000003e-06 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.875063+0000", 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.9199999999999999e-05 2024-09-16T10:32:47.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.875084+0000", 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0902000000000001e-05 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.890632+0000", 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015548378 2024-09-16T10:32:47.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.890776+0000", 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014321000000000001 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:204635 11.6 11:673c6067:::benchmark_data_smithi063_176907_object80359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.876830+0000", 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.681701202999999, 2024-09-16T10:32:47.601 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.014573188000000001, 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 204635 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.876830+0000", 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.876830+0000", 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.876831+0000", 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.623e-06 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.876836+0000", 2024-09-16T10:32:47.603 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8579999999999999e-06 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.876846+0000", 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4199999999999996e-06 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.876852+0000", 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7379999999999997e-06 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.604 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.883141+0000", 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0062888570000000001 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.883272+0000", 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013130800000000001 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,4", 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.883398+0000", 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012591100000000001 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.889070+0000", 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056720219999999997 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.889108+0000", 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7245e-05 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.889141+0000", 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3633000000000001e-05 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.891220+0000", 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0020788170000000002 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.891283+0000", 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3015999999999994e-05 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.891403+0000", 2024-09-16T10:32:47.607 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011974 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204630 11.2 e541/476 11:4b59edbc:::benchmark_data_smithi063_176907_object80354:head v 541'12695, mlcod=541'12692)", 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.878985+0000", 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.679545851, 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011827466, 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.878985+0000", 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.878985+0000", 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.878986+0000", 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0499999999999999e-06 2024-09-16T10:32:47.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.878991+0000", 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3970000000000002e-06 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.879000+0000", 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5299999999999996e-06 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.879006+0000", 2024-09-16T10:32:47.610 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6219999999999997e-06 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.887308+0000", 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0083020890000000003 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.887328+0000", 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0386000000000001e-05 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.611 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.890779+0000", 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003451108 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.890812+0000", 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3284000000000002e-05 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204631 11.e e541/534 11:798007b2:::benchmark_data_smithi063_176907_object80355:head v 541'12896, mlcod=541'12894)", 2024-09-16T10:32:47.612 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.884799+0000", 2024-09-16T10:32:47.613 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.673732353, 2024-09-16T10:32:47.613 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010489477000000001, 2024-09-16T10:32:47.613 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.613 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:47.614 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.614 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.614 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.614 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.884799+0000", 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.884799+0000", 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.884800+0000", 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.257e-06 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.615 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.884821+0000", 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1648999999999999e-05 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.884832+0000", 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0210999999999999e-05 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.884838+0000", 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5490000000000003e-06 2024-09-16T10:32:47.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.885704+0000", 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00086600399999999997 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.885724+0000", 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0205999999999999e-05 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.617 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.895245+0000", 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0095205789999999995 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.895288+0000", 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3022000000000002e-05 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:204649 11.6 11:67651659:::benchmark_data_smithi063_176907_object80373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-09-16T10:32:47.618 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.892159+0000", 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.666371902, 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.018665774, 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 204649 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.619 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.892159+0000", 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.892159+0000", 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.892161+0000", 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.629e-06 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.892165+0000", 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4900000000000002e-06 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.892174+0000", 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1290000000000002e-06 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.892181+0000", 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7859999999999999e-06 2024-09-16T10:32:47.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.895206+0000", 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030253419999999999 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.895403+0000", 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019624700000000001 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,4", 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.895520+0000", 2024-09-16T10:32:47.622 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011769000000000001 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897868+0000", 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002348213 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901959+0000", 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004090247 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901998+0000", 2024-09-16T10:32:47.623 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9406999999999997e-05 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.909242+0000", 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.007244103 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.910699+0000", 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001457093 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.624 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.910825+0000", 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012539800000000001 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204651 11.e e541/534 11:76927418:::benchmark_data_smithi063_176907_object80375:head v 541'12898, mlcod=541'12894)", 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.897798+0000", 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.660732479, 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016496156000000001, 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.625 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897798+0000", 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897798+0000", 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.626 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897800+0000", 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5650000000000001e-06 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897814+0000", 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3597e-05 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897825+0000", 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1007e-05 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.627 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897832+0000", 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0040000000000002e-06 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897897+0000", 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5834999999999996e-05 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.897917+0000", 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0001e-05 2024-09-16T10:32:47.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.914178+0000", 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016260299999999998 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.914295+0000", 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116847 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204648 11.f e541/535 11:f2079f42:::benchmark_data_smithi063_176907_object80372:head v 541'12950, mlcod=541'12948)", 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.898307+0000", 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.660223679, 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015481749, 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898307+0000", 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898307+0000", 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898308+0000", 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.057e-06 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898314+0000", 2024-09-16T10:32:47.631 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9190000000000004e-06 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898323+0000", 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9809999999999992e-06 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898329+0000", 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2890000000000003e-06 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.632 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898385+0000", 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5850000000000002e-05 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898409+0000", 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4131999999999999e-05 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.913744+0000", 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015334342000000001 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.913789+0000", 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5179000000000002e-05 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:204653 11.6 11:66aaba7c:::benchmark_data_smithi063_176907_object80377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.898511+0000", 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.660020226, 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.020209015, 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.634 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 204653 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898511+0000", 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.635 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898511+0000", 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898513+0000", 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.883e-06 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898517+0000", 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9080000000000003e-06 2024-09-16T10:32:47.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898527+0000", 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.1859999999999998e-06 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.898533+0000", 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.054e-06 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.902032+0000", 2024-09-16T10:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034989610000000001 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.902151+0000", 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119583 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,4", 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.902273+0000", 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000121637 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.638 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.904964+0000", 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0026907070000000001 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.912010+0000", 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0070465340000000001 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.914305+0000", 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022949210000000001 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.639 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.918535+0000", 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042301029999999998 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.918610+0000", 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5167999999999996e-05 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.918720+0000", 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010937 2024-09-16T10:32:47.640 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204641 11.9 e541/476 11:9eb29bed:::benchmark_data_smithi063_176907_object80365:head v 541'12718, mlcod=541'12716)", 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.900418+0000", 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.658113245999999, 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.011307381, 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.641 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900418+0000", 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900418+0000", 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900419+0000", 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.238e-06 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900425+0000", 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0129999999999999e-06 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900435+0000", 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6949999999999995e-06 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900441+0000", 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0549999999999996e-06 2024-09-16T10:32:47.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900725+0000", 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00028472700000000003 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900745+0000", 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9389e-05 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.644 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.911687+0000", 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010942557 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.911725+0000", 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7707000000000003e-05 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "MOSDPGPush(11.7 541/535 [PushOp(11:e866d818:::benchmark_data_smithi063_176907_object89481:head, version: 499'5609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:e866d818:::benchmark_data_smithi063_176907_object89481:head@499'5609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T10:32:47.645 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.900468+0000", 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.658062534999999, 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.013272124999999999, 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900468+0000", 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.646 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900468+0000", 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900469+0000", 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8800000000000002e-07 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900503+0000", 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3748999999999998e-05 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.647 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900518+0000", 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5726000000000002e-05 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.900533+0000", 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4752999999999999e-05 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.913528+0000", 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012995132 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.913542+0000", 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3261000000000001e-05 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.913740+0000", 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00019881599999999999 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.649 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204650 11.0 e541/535 11:0f4c00e7:::benchmark_data_smithi063_176907_object80374:head v 541'12829, mlcod=541'12826)", 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.901756+0000", 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.656775241, 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.010061949000000001, 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901756+0000", 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.650 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901756+0000", 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901757+0000", 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0139999999999999e-06 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901763+0000", 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0839999999999997e-06 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901772+0000", 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8910000000000001e-06 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901778+0000", 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1750000000000002e-06 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.909019+0000", 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0072413399999999998 2024-09-16T10:32:47.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.909039+0000", 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0268e-05 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.911768+0000", 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027290040000000002 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.911818+0000", 2024-09-16T10:32:47.653 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9172999999999998e-05 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204655 11.7 e541/535 11:eca41c24:::benchmark_data_smithi063_176907_object80379:head v 541'12788, mlcod=541'12786)", 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.901821+0000", 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.656710142, 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.012004502, 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901821+0000", 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901821+0000", 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901822+0000", 2024-09-16T10:32:47.655 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7199999999999997e-07 2024-09-16T10:32:47.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901827+0000", 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2970000000000003e-06 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901835+0000", 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.1110000000000008e-06 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901841+0000", 2024-09-16T10:32:47.657 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6110000000000001e-06 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.907261+0000", 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0054199749999999996 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.907282+0000", 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.1016999999999999e-05 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.658 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.913792+0000", 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0065099779999999996 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.913825+0000", 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3640999999999999e-05 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:204656 11.b e541/535 11:d23699dd:::benchmark_data_smithi063_176907_object80380:head v 541'12737, mlcod=541'12735)", 2024-09-16T10:32:47.659 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.901965+0000", 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.656566073, 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0097302550000000002, 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901965+0000", 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.660 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901965+0000", 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901966+0000", 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2800000000000005e-07 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901971+0000", 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.412e-06 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901979+0000", 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.3420000000000008e-06 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.901985+0000", 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9619999999999998e-06 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.903662+0000", 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001676786 2024-09-16T10:32:47.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.903686+0000", 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3306e-05 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.911643+0000", 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0079579349999999993 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.911695+0000", 2024-09-16T10:32:47.663 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1583999999999999e-05 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:204660 11.6 11:6d39add7:::benchmark_data_smithi063_176907_object80384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.906589+0000", 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.651941746, 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.014384245, 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.664 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 204660 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.906589+0000", 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.906589+0000", 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.906592+0000", 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.695e-06 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.906598+0000", 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4819999999999999e-06 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.906617+0000", 2024-09-16T10:32:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8280000000000001e-05 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.906625+0000", 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.9060000000000008e-06 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.906878+0000", 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00025266900000000002 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.667 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.907014+0000", 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000136172 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,4", 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.907146+0000", 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013140800000000001 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.912214+0000", 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0050681249999999997 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.914112+0000", 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001897777 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.914313+0000", 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020118499999999999 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.919882+0000", 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0055696590000000002 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.920573+0000", 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00069076399999999996 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.920973+0000", 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000400123 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:204689 11.6 11:63364206:::benchmark_data_smithi063_176907_object80413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.934954+0000", 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.623576821, 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.017189494, 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 204689 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.934954+0000", 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.934954+0000", 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.934956+0000", 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6339999999999999e-06 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.934960+0000", 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9110000000000003e-06 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.934967+0000", 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6420000000000007e-06 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.934972+0000", 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2070000000000003e-06 2024-09-16T10:32:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.936360+0000", 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001387313 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.936494+0000", 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013438999999999999 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,4", 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.936626+0000", 2024-09-16T10:32:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013160299999999999 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.939568+0000", 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0029417879999999999 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.949512+0000", 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0099442450000000009 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.675 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.951915+0000", 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024031999999999999 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.951967+0000", 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1892999999999999e-05 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.952028+0000", 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0875999999999998e-05 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.676 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.952144+0000", 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011579200000000001 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:204694 11.6 11:640237f9:::benchmark_data_smithi063_176907_object80418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:35.939742+0000", 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 11.618788622, 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.014943454, 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:47.677 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 204694 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.939742+0000", 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.678 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.939742+0000", 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.939744+0000", 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.962e-06 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.939749+0000", 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.8239999999999999e-06 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.939758+0000", 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0440000000000001e-06 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.939764+0000", 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3110000000000002e-06 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.941964+0000", 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0021992299999999999 2024-09-16T10:32:47.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.942081+0000", 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011781999999999999 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 1,2,4", 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.942204+0000", 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012296400000000001 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.951929+0000", 2024-09-16T10:32:47.681 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0097243529999999998 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.952190+0000", 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00026119499999999998 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.952234+0000", 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.3560999999999998e-05 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:47.682 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.952274+0000", 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0574000000000002e-05 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.952331+0000", 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6433e-05 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:35.954686+0000", 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0023551829999999998 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:47.684 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:47.684 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:32:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:47.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:47.804 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:47.845 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:47.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:47.947 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:47.988 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:47.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:48.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:48.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:48.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:48.221 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:48.265 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:48.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:48.366 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:48.401 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:32:48.401 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:32:48.401 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:32:48.402 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:32:48.403 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:32:48.403 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:32:48.405 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:48.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:48.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1673121045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:32:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[39470]: osdmap e548: 8 total, 8 up, 6 in 2024-09-16T10:32:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1597373857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[39470]: pgmap v1038: 168 pgs: 1 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 161 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 572 op/s; 11388/114951 objects misplaced (9.907%); 34 MiB/s, 550 objects/s recovering 2024-09-16T10:32:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1673121045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:32:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[29993]: osdmap e548: 8 total, 8 up, 6 in 2024-09-16T10:32:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1597373857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:48 smithi063 ceph-mon[29993]: pgmap v1038: 168 pgs: 1 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 161 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 572 op/s; 11388/114951 objects misplaced (9.907%); 34 MiB/s, 550 objects/s recovering 2024-09-16T10:32:48.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:48.545 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:48.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:48.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1673121045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:32:48.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:48 smithi165 ceph-mon[32696]: osdmap e548: 8 total, 8 up, 6 in 2024-09-16T10:32:48.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1597373857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:48 smithi165 ceph-mon[32696]: pgmap v1038: 168 pgs: 1 peering, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 161 active+clean; 2.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 572 op/s; 11388/114951 objects misplaced (9.907%); 34 MiB/s, 550 objects/s recovering 2024-09-16T10:32:48.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:48.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:48.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:48.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:48.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:48.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:48.900 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:48.940 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:48.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:49.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:49.083 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:49.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:49.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:49.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:49.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:49.311 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:49.370 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T10:32:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:49.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:49.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:49.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:32:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:32:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:49.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:49.662 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:49.720 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:49.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:49.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:49.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T10:32:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:49.881 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:49.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:49.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:49 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:49.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:49 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:49.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:50.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:50.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:49 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:50.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:50.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:50.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:50.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:50.358 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:50.370 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.431 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:50.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:50.433 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:50.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:50.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:50.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:50.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:50.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:50.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:50.917 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:50.925 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:50.925 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:50 smithi063 ceph-mon[29993]: pgmap v1039: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 10270/102321 objects misplaced (10.037%); 26 MiB/s, 413 objects/s recovering 2024-09-16T10:32:50.925 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:50.925 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:50 smithi063 ceph-mon[39470]: pgmap v1039: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 10270/102321 objects misplaced (10.037%); 26 MiB/s, 413 objects/s recovering 2024-09-16T10:32:50.964 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:50.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:51.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:32:51.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:50 smithi165 ceph-mon[32696]: pgmap v1039: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 10270/102321 objects misplaced (10.037%); 26 MiB/s, 413 objects/s recovering 2024-09-16T10:32:51.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:51.106 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:51.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:51.208 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:51.250 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:51.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:51.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:51.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:51.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:51.429 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:51.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:51.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:51.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:51.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:51.639 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:32:51.645 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:51.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:51.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:51.793 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:51.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:51.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3720072629' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:51.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:51 smithi063 ceph-mon[39470]: from='client.26785 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:51.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3720072629' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:51.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:51 smithi063 ceph-mon[29993]: from='client.26785 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:51.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:51.924 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:51.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:52.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:52.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3720072629' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:52.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:51 smithi165 ceph-mon[32696]: from='client.26785 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:32:52.071 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:52.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:52.173 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:52.197 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:32:52.197 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:32:52.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.197 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:218533 11.e 11:745893c8:::benchmark_data_smithi063_176907_object94257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-09-16T10:32:52.197 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:52.170317+0000", 2024-09-16T10:32:52.197 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.023599041000000001, 2024-09-16T10:32:52.198 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.023631493999999999, 2024-09-16T10:32:52.198 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:52.198 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:52.198 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:32:52.198 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:52.198 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:52.198 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 218533 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170317+0000", 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170317+0000", 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170319+0000", 2024-09-16T10:32:52.200 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3769999999999999e-06 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170326+0000", 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3589999999999996e-06 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170344+0000", 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7910999999999998e-05 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.201 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170353+0000", 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3309999999999993e-06 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170557+0000", 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00020398500000000001 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.170925+0000", 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000367456 2024-09-16T10:32:52.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3,4,7", 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.171340+0000", 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00041527699999999997 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185928+0000", 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.014588564 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185989+0000", 2024-09-16T10:32:52.203 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0454000000000001e-05 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.186034+0000", 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4702e-05 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189441+0000", 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0034079290000000001 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.204 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189655+0000", 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00021357400000000001 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:218545 11.e 11:72bc35c9:::benchmark_data_smithi063_176907_object94269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:52.185257+0000", 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0086587899999999995, 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0087843629999999999, 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:52.205 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 218545 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185257+0000", 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:52.206 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185257+0000", 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185259+0000", 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.849e-06 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185264+0000", 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.853e-06 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.207 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185272+0000", 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9680000000000006e-06 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.185278+0000", 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6840000000000001e-06 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.186062+0000", 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00078439500000000001 2024-09-16T10:32:52.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.186205+0000", 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00014281499999999999 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,3,4,7", 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.186342+0000", 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013711100000000001 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.209 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189578+0000", 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00323596 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.26491.0:218545 11.e e549/534 ondisk, result = 0)", 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:52.189333+0000", 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0045833540000000004, 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047726660000000001, 2024-09-16T10:32:52.210 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189333+0000", 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189333+0000", 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189335+0000", 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.86e-06 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189344+0000", 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.7340000000000003e-06 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:52.212 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189365+0000", 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.086e-05 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189375+0000", 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4520000000000003e-06 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189526+0000", 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015117200000000001 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.213 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.189554+0000", 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.7976000000000001e-05 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.26491.0:218545 11.e e549/534)", 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:52.190573+0000", 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.003342982, 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0035777890000000001, 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:32:52.214 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.190573+0000", 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.190573+0000", 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:32:52.215 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.190575+0000", 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.559e-06 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.190584+0000", 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.6730000000000004e-06 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.190602+0000", 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.7544999999999999e-05 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:32:52.216 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:52.190611+0000", 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.3530000000000001e-06 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 4 2024-09-16T10:32:52.217 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:32:52.226 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:52.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:52.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:52.373 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:52.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:52.427 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:32:52.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:52.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:52.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:52.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:52.658 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:52.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:52.686 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T10:32:52.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:52.822 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:52.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:52.924 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[29993]: pgmap v1040: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 740 op/s; 9943/101340 objects misplaced (9.812%); 633 KiB/s, 9 objects/s recovering 2024-09-16T10:32:52.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[29993]: from='client.26785 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:32:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[29993]: osdmap e549: 8 total, 8 up, 6 in 2024-09-16T10:32:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/851626302' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:32:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[29993]: from='client.46667 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:32:52.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[29993]: 11.4 scrub starts 2024-09-16T10:32:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[39470]: pgmap v1040: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 740 op/s; 9943/101340 objects misplaced (9.812%); 633 KiB/s, 9 objects/s recovering 2024-09-16T10:32:52.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:52.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[39470]: from='client.26785 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:32:52.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[39470]: osdmap e549: 8 total, 8 up, 6 in 2024-09-16T10:32:52.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/851626302' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:32:52.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[39470]: from='client.46667 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:32:52.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:52 smithi063 ceph-mon[39470]: 11.4 scrub starts 2024-09-16T10:32:52.966 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:52.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:53.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:52 smithi165 ceph-mon[32696]: pgmap v1040: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 740 op/s; 9943/101340 objects misplaced (9.812%); 633 KiB/s, 9 objects/s recovering 2024-09-16T10:32:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:52 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:32:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:52 smithi165 ceph-mon[32696]: from='client.26785 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:32:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:52 smithi165 ceph-mon[32696]: osdmap e549: 8 total, 8 up, 6 in 2024-09-16T10:32:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:52 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/851626302' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:32:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:52 smithi165 ceph-mon[32696]: from='client.46667 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:32:53.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:52 smithi165 ceph-mon[32696]: 11.4 scrub starts 2024-09-16T10:32:53.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:53.122 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:53.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:53.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:53.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:53.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:53.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:53.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:53.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:53.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:53.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:53.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:53.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:53.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:53.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:53.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:53.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:53.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T10:32:53.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:53.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:53.885 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:53.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:53.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:53 smithi063 ceph-mon[29993]: 11.4 scrub ok 2024-09-16T10:32:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:53 smithi063 ceph-mon[29993]: from='client.46667 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:32:53.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:53 smithi063 ceph-mon[29993]: osdmap e550: 8 total, 8 up, 6 in 2024-09-16T10:32:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:53 smithi063 ceph-mon[39470]: 11.4 scrub ok 2024-09-16T10:32:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:53 smithi063 ceph-mon[39470]: from='client.46667 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:32:53.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:53 smithi063 ceph-mon[39470]: osdmap e550: 8 total, 8 up, 6 in 2024-09-16T10:32:54.042 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:54.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:53 smithi165 ceph-mon[32696]: 11.4 scrub ok 2024-09-16T10:32:54.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:53 smithi165 ceph-mon[32696]: from='client.46667 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:32:54.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:53 smithi165 ceph-mon[32696]: osdmap e550: 8 total, 8 up, 6 in 2024-09-16T10:32:54.098 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:54.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:54.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:54.244 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:54.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:54.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:54.383 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:54.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:54.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T10:32:54.485 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:54.488 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:54.570 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:54.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:54.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:54.719 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:54.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:54.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:54.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:54.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:54.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[29993]: pgmap v1043: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.0 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 803 B/s rd, 0 B/s wr, 871 op/s; 9339/98310 objects misplaced (9.500%); 633 KiB/s, 9 objects/s recovering 2024-09-16T10:32:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:32:54.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[29993]: 1.b scrub ok 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[39470]: pgmap v1043: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.0 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 803 B/s rd, 0 B/s wr, 871 op/s; 9339/98310 objects misplaced (9.500%); 633 KiB/s, 9 objects/s recovering 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[39470]: 1.b scrub ok 2024-09-16T10:32:54.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:54 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:54.997 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:54.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:55.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:32:55.051 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:54 smithi165 ceph-mon[32696]: pgmap v1043: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 2.0 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 803 B/s rd, 0 B/s wr, 871 op/s; 9339/98310 objects misplaced (9.500%); 633 KiB/s, 9 objects/s recovering 2024-09-16T10:32:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:54 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:32:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:54 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:32:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:54 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:32:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:54 smithi165 ceph-mon[32696]: 1.b scrub ok 2024-09-16T10:32:55.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:54 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:32:55.094 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:55.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:55.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:55.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:55.196 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:32:55.196 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:32:55.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:55.234 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:32:55.234 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:32:55.273 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:32:55.273 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:32:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:55.286 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:55.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:55.312 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:32:55.313 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:32:55.352 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:32:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:32:55.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:55.391 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:32:55.391 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:32:55.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:55.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:55.428 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:32:55.429 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:32:55.466 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:32:55.466 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:32:55.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:32:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:55.566 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:55.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:32:55.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:55.707 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:55.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:55.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:55 smithi165 ceph-mon[32696]: 3.a scrub starts 2024-09-16T10:32:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:55 smithi165 ceph-mon[32696]: 3.a scrub ok 2024-09-16T10:32:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:55 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2063253340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:55 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:32:55.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:55 smithi165 ceph-mon[32696]: 1.13 scrub ok 2024-09-16T10:32:55.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:55.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:55.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:55.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:55.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[29993]: 3.a scrub starts 2024-09-16T10:32:55.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[29993]: 3.a scrub ok 2024-09-16T10:32:55.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2063253340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:55.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:32:55.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[29993]: 1.13 scrub ok 2024-09-16T10:32:55.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[39470]: 3.a scrub starts 2024-09-16T10:32:55.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[39470]: 3.a scrub ok 2024-09-16T10:32:55.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2063253340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:32:55.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:32:55.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:55 smithi063 ceph-mon[39470]: 1.13 scrub ok 2024-09-16T10:32:55.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:55.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:56.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:56.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:56.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:56.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:56.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:56.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:56.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:56.362 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:56.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:56.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:56.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:56.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:56.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:56.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:56.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:56.627 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:32:56.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:56.767 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:56.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:56.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:56.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:56.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[29993]: pgmap v1044: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 7275/85056 objects misplaced (8.553%); 9.2 MiB/s, 146 objects/s recovering 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1021361006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[39470]: pgmap v1044: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 7275/85056 objects misplaced (8.553%); 9.2 MiB/s, 146 objects/s recovering 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:32:56.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:32:56.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1021361006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:32:56.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:57.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:56 smithi165 ceph-mon[32696]: pgmap v1044: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 7275/85056 objects misplaced (8.553%); 9.2 MiB/s, 146 objects/s recovering 2024-09-16T10:32:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:56 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:32:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:56 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:32:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:56 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:32:57.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1021361006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:32:57.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:57.129 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T10:32:57.179 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:57.281 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:57.314 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:57.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:57.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:57.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:32:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:32:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:32:57.549 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:57.581 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:57.683 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:57.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.718 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:57.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:57.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.855 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:57.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:57.956 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:57.996 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:57.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:58.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:58.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:58.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:58.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:58.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:58.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:58.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:58.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:58.394 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:58.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:58.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:32:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:32:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1021361006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:32:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[29993]: osdmap e551: 8 total, 8 up, 6 in 2024-09-16T10:32:58.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[29993]: pgmap v1046: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 905 op/s; 7275/85056 objects misplaced (8.553%); 12 MiB/s, 185 objects/s recovering 2024-09-16T10:32:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:32:58.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:32:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1021361006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:32:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[39470]: osdmap e551: 8 total, 8 up, 6 in 2024-09-16T10:32:58.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:58 smithi063 ceph-mon[39470]: pgmap v1046: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 905 op/s; 7275/85056 objects misplaced (8.553%); 12 MiB/s, 185 objects/s recovering 2024-09-16T10:32:58.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:58.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:58.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:32:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:58 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:32:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:58 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:32:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:58 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1021361006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:32:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:58 smithi165 ceph-mon[32696]: osdmap e551: 8 total, 8 up, 6 in 2024-09-16T10:32:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:58 smithi165 ceph-mon[32696]: pgmap v1046: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 1.7 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 905 op/s; 7275/85056 objects misplaced (8.553%); 12 MiB/s, 185 objects/s recovering 2024-09-16T10:32:58.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:32:58.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:58.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:32:58.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:58.790 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:58.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:58.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:58.920 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:58.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:59.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:59.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:59.152 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.189 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:59.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:59.289 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:32:59.314 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:32:59.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:32:59.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:32:59.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:32:59 smithi063 ceph-mon[29993]: 11.7 scrub starts 2024-09-16T10:32:59.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:32:59 smithi063 ceph-mon[39470]: 11.7 scrub starts 2024-09-16T10:32:59.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:32:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:32:59 smithi165 ceph-mon[32696]: 11.7 scrub starts 2024-09-16T10:32:59.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:32:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:32:59.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:32:59.676 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:32:59.720 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:32:59.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:32:59.860 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:32:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:32:59.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:32:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:32:59.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:32:59.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:00.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:00.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:00.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:00.220 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:00.267 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:00.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:00.369 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:00.411 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:00.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:00.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:00 smithi063 ceph-mon[29993]: 11.7 scrub ok 2024-09-16T10:33:00.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:00.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:00 smithi063 ceph-mon[29993]: pgmap v1047: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 389 B/s rd, 0 B/s wr, 1.27k op/s; 4613/71283 objects misplaced (6.471%); 21 MiB/s, 334 objects/s recovering 2024-09-16T10:33:00.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:00 smithi063 ceph-mon[39470]: 11.7 scrub ok 2024-09-16T10:33:00.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:00.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:00 smithi063 ceph-mon[39470]: pgmap v1047: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 389 B/s rd, 0 B/s wr, 1.27k op/s; 4613/71283 objects misplaced (6.471%); 21 MiB/s, 334 objects/s recovering 2024-09-16T10:33:00.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:00.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:00 smithi165 ceph-mon[32696]: 11.7 scrub ok 2024-09-16T10:33:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:00.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:00 smithi165 ceph-mon[32696]: pgmap v1047: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 389 B/s rd, 0 B/s wr, 1.27k op/s; 4613/71283 objects misplaced (6.471%); 21 MiB/s, 334 objects/s recovering 2024-09-16T10:33:00.559 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:00.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:00.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:00.703 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:00.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:00.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:00.848 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:00.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:00.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:00.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:00.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:01.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:01.114 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:01.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:01.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:01.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:01.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:01.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:01.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:01.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:01.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:01 smithi063 ceph-mon[29993]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:33:01.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:01 smithi063 ceph-mon[39470]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:33:01.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:01.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:01.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:01.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:01 smithi165 ceph-mon[32696]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-16T10:33:01.600 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:01.641 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:01.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:01.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:01.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:01.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:01.874 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:01.917 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:01.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:01.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:02.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:02.055 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:02.156 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:02.190 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:02.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:02.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:02.422 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:02 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:33:02.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:02 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:33:02.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:02 smithi063 ceph-mon[29993]: pgmap v1048: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 925 B/s rd, 0 B/s wr, 1.17k op/s; 3835/70326 objects misplaced (5.453%); 23 MiB/s, 365 objects/s recovering 2024-09-16T10:33:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:02 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:33:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:02 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:33:02.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:02 smithi063 ceph-mon[39470]: pgmap v1048: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 925 B/s rd, 0 B/s wr, 1.17k op/s; 3835/70326 objects misplaced (5.453%); 23 MiB/s, 365 objects/s recovering 2024-09-16T10:33:02.461 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:02.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:02 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:33:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:02 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:33:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:02 smithi165 ceph-mon[32696]: pgmap v1048: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 925 B/s rd, 0 B/s wr, 1.17k op/s; 3835/70326 objects misplaced (5.453%); 23 MiB/s, 365 objects/s recovering 2024-09-16T10:33:02.563 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:02.589 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:02.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:02.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:02.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:02.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:02.952 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:02.988 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:02.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:03.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:03.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:03.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:03.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:03.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:03.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:03.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:03.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:03.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:03.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[29993]: 3.5 scrub starts 2024-09-16T10:33:03.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[29993]: 3.5 scrub ok 2024-09-16T10:33:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:33:03.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:33:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[39470]: 3.5 scrub starts 2024-09-16T10:33:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[39470]: 3.5 scrub ok 2024-09-16T10:33:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:33:03.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:03 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:33:03.475 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:03.508 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:03.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:03.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:03 smithi165 ceph-mon[32696]: 3.5 scrub starts 2024-09-16T10:33:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:03 smithi165 ceph-mon[32696]: 3.5 scrub ok 2024-09-16T10:33:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:03 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:33:03.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:03 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:33:03.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:03.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:03.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:03.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:03.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:03.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:03.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:03.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:03.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:03.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:04.005 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:04.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:04.128 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:04.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:04.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:04.379 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:04.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[29993]: osdmap e552: 8 total, 8 up, 6 in 2024-09-16T10:33:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[29993]: pgmap v1050: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 725 op/s; 3007/67647 objects misplaced (4.445%); 18 MiB/s, 294 objects/s recovering 2024-09-16T10:33:04.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[29993]: 11.2 scrub starts 2024-09-16T10:33:04.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:04.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:04.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[39470]: osdmap e552: 8 total, 8 up, 6 in 2024-09-16T10:33:04.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[39470]: pgmap v1050: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 725 op/s; 3007/67647 objects misplaced (4.445%); 18 MiB/s, 294 objects/s recovering 2024-09-16T10:33:04.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:04 smithi063 ceph-mon[39470]: 11.2 scrub starts 2024-09-16T10:33:04.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:04.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:04.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:04 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:04 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:04 smithi165 ceph-mon[32696]: osdmap e552: 8 total, 8 up, 6 in 2024-09-16T10:33:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:04 smithi165 ceph-mon[32696]: pgmap v1050: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 1.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 725 op/s; 3007/67647 objects misplaced (4.445%); 18 MiB/s, 294 objects/s recovering 2024-09-16T10:33:04.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:04 smithi165 ceph-mon[32696]: 11.2 scrub starts 2024-09-16T10:33:04.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:04.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:04.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:04.754 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:04.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:04.886 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:04.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:04.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:05.014 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:05.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:05.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:05.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:05.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:05.373 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:05.395 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:05.497 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:05.519 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:05.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:05 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:05 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:05 smithi165 ceph-mon[32696]: 11.2 scrub ok 2024-09-16T10:33:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:05 smithi165 ceph-mon[32696]: osdmap e553: 8 total, 8 up, 6 in 2024-09-16T10:33:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:05 smithi165 ceph-mon[32696]: 4.0 scrub starts 2024-09-16T10:33:05.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:05 smithi165 ceph-mon[32696]: 4.0 scrub ok 2024-09-16T10:33:05.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:05.643 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:05.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:05.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[29993]: 11.2 scrub ok 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[29993]: osdmap e553: 8 total, 8 up, 6 in 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[29993]: 4.0 scrub starts 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[29993]: 4.0 scrub ok 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[39470]: 11.2 scrub ok 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[39470]: osdmap e553: 8 total, 8 up, 6 in 2024-09-16T10:33:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[39470]: 4.0 scrub starts 2024-09-16T10:33:05.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:05 smithi063 ceph-mon[39470]: 4.0 scrub ok 2024-09-16T10:33:05.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:05.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:05.847 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:33:05.848 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:33:05.868 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:05.891 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:05.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:05.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:06.016 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:06.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:06.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:06.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:06.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:06.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:06.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:06.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:06.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:06.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:06 smithi165 ceph-mon[32696]: 6.16 scrub starts 2024-09-16T10:33:06.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:06 smithi165 ceph-mon[32696]: 6.16 scrub ok 2024-09-16T10:33:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:06 smithi165 ceph-mon[32696]: 4.14 scrub starts 2024-09-16T10:33:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:06 smithi165 ceph-mon[32696]: 4.14 scrub ok 2024-09-16T10:33:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:06 smithi165 ceph-mon[32696]: osdmap e554: 8 total, 8 up, 6 in 2024-09-16T10:33:06.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:06 smithi165 ceph-mon[32696]: pgmap v1053: 168 pgs: 1 peering, 1 active+remapped+backfilling, 166 active+clean; 1.1 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 795 op/s; 15/56991 objects misplaced (0.026%); 31 MiB/s, 503 objects/s recovering 2024-09-16T10:33:06.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:06.672 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[29993]: 6.16 scrub starts 2024-09-16T10:33:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[29993]: 6.16 scrub ok 2024-09-16T10:33:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[29993]: 4.14 scrub starts 2024-09-16T10:33:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[29993]: 4.14 scrub ok 2024-09-16T10:33:06.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[29993]: osdmap e554: 8 total, 8 up, 6 in 2024-09-16T10:33:06.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[29993]: pgmap v1053: 168 pgs: 1 peering, 1 active+remapped+backfilling, 166 active+clean; 1.1 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 795 op/s; 15/56991 objects misplaced (0.026%); 31 MiB/s, 503 objects/s recovering 2024-09-16T10:33:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[39470]: 6.16 scrub starts 2024-09-16T10:33:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[39470]: 6.16 scrub ok 2024-09-16T10:33:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[39470]: 4.14 scrub starts 2024-09-16T10:33:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[39470]: 4.14 scrub ok 2024-09-16T10:33:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[39470]: osdmap e554: 8 total, 8 up, 6 in 2024-09-16T10:33:06.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:06 smithi063 ceph-mon[39470]: pgmap v1053: 168 pgs: 1 peering, 1 active+remapped+backfilling, 166 active+clean; 1.1 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 795 op/s; 15/56991 objects misplaced (0.026%); 31 MiB/s, 503 objects/s recovering 2024-09-16T10:33:06.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:06.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:06.906 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:06.944 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:06.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:07.047 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:07.081 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:07.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:07.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:07.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:33:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:33:07.235 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:07.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:07.337 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:07.378 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:07.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:07.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[29993]: 5.c scrub starts 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[29993]: 5.c scrub ok 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[29993]: 4.f scrub starts 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[29993]: 4.f scrub ok 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[29993]: osdmap e555: 8 total, 8 up, 6 in 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[29993]: 4.15 scrub starts 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[29993]: 4.15 scrub ok 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[39470]: 5.c scrub starts 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[39470]: 5.c scrub ok 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[39470]: 4.f scrub starts 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[39470]: 4.f scrub ok 2024-09-16T10:33:07.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[39470]: osdmap e555: 8 total, 8 up, 6 in 2024-09-16T10:33:07.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[39470]: 4.15 scrub starts 2024-09-16T10:33:07.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:07 smithi063 ceph-mon[39470]: 4.15 scrub ok 2024-09-16T10:33:07.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:07.513 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:07.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:07.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:07 smithi165 ceph-mon[32696]: 5.c scrub starts 2024-09-16T10:33:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:07 smithi165 ceph-mon[32696]: 5.c scrub ok 2024-09-16T10:33:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:07 smithi165 ceph-mon[32696]: 4.f scrub starts 2024-09-16T10:33:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:07 smithi165 ceph-mon[32696]: 4.f scrub ok 2024-09-16T10:33:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:07 smithi165 ceph-mon[32696]: osdmap e555: 8 total, 8 up, 6 in 2024-09-16T10:33:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:07 smithi165 ceph-mon[32696]: 4.15 scrub starts 2024-09-16T10:33:07.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:07 smithi165 ceph-mon[32696]: 4.15 scrub ok 2024-09-16T10:33:07.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:07.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:07.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:07.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:07.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:07.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:07.792 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T10:33:07.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:07.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:07.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:08.007 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:08.057 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:08.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:08.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:08.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:08.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:08.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:08.329 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:08.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:08.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:08.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[29993]: 11.f scrub starts 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[29993]: 11.f scrub ok 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[29993]: pgmap v1055: 168 pgs: 1 peering, 1 active+remapped+backfilling, 166 active+clean; 1.1 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s; 15/56991 objects misplaced (0.026%); 35 MiB/s, 558 objects/s recovering 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[29993]: 3.10 scrub starts 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[29993]: 3.10 scrub ok 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[39470]: 11.f scrub starts 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[39470]: 11.f scrub ok 2024-09-16T10:33:08.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:33:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:33:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[39470]: pgmap v1055: 168 pgs: 1 peering, 1 active+remapped+backfilling, 166 active+clean; 1.1 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s; 15/56991 objects misplaced (0.026%); 35 MiB/s, 558 objects/s recovering 2024-09-16T10:33:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[39470]: 3.10 scrub starts 2024-09-16T10:33:08.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:08 smithi063 ceph-mon[39470]: 3.10 scrub ok 2024-09-16T10:33:08.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:08.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:08.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:08 smithi165 ceph-mon[32696]: 11.f scrub starts 2024-09-16T10:33:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:08 smithi165 ceph-mon[32696]: 11.f scrub ok 2024-09-16T10:33:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:08 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:33:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:08 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:33:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:08 smithi165 ceph-mon[32696]: pgmap v1055: 168 pgs: 1 peering, 1 active+remapped+backfilling, 166 active+clean; 1.1 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s; 15/56991 objects misplaced (0.026%); 35 MiB/s, 558 objects/s recovering 2024-09-16T10:33:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:08 smithi165 ceph-mon[32696]: 3.10 scrub starts 2024-09-16T10:33:08.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:08 smithi165 ceph-mon[32696]: 3.10 scrub ok 2024-09-16T10:33:08.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:08.599 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:08.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:08.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:08.727 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:08.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:08.829 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:08.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T10:33:08.889 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:08.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:08.911 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:08.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:33:08.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:09.049 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:09.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:09.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:09.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:09.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:09.231 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:33:09.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:09.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:09.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:09.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:09.446 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:09.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[29993]: 6.1f scrub starts 2024-09-16T10:33:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[29993]: 6.1f scrub ok 2024-09-16T10:33:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4254887427' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:09.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[29993]: from='client.26806 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[39470]: 6.1f scrub starts 2024-09-16T10:33:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[39470]: 6.1f scrub ok 2024-09-16T10:33:09.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4254887427' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:09.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:09 smithi063 ceph-mon[39470]: from='client.26806 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:09.548 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:09.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:33:09 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:33:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:09 smithi165 ceph-mon[32696]: 6.1f scrub starts 2024-09-16T10:33:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:09 smithi165 ceph-mon[32696]: 6.1f scrub ok 2024-09-16T10:33:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4254887427' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:09 smithi165 ceph-mon[32696]: from='client.26806 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:09.593 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:09.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:09.694 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:09.735 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:33:09.736 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:33:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:09.745 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:09.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:09.846 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:09.888 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:09.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:09.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.026 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:10.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:10.127 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:10.271 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:10.311 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:10.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:10.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:10.543 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:10.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:33:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:33:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: from='client.26806 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: osdmap e556: 8 total, 8 up, 6 in 2024-09-16T10:33:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:10.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:10 smithi165 ceph-mon[32696]: pgmap v1057: 168 pgs: 1 peering, 167 active+clean; 952 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 959 B/s rd, 0 B/s wr, 910 op/s; 18 MiB/s, 291 objects/s recovering 2024-09-16T10:33:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.582 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:10.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:10.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:33:10.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:33:10.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: from='client.26806 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:10.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: osdmap e556: 8 total, 8 up, 6 in 2024-09-16T10:33:10.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:10.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:10.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:10.645 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[29993]: pgmap v1057: 168 pgs: 1 peering, 167 active+clean; 952 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 959 B/s rd, 0 B/s wr, 910 op/s; 18 MiB/s, 291 objects/s recovering 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: from='client.26806 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: osdmap e556: 8 total, 8 up, 6 in 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:10.646 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:10 smithi063 ceph-mon[39470]: pgmap v1057: 168 pgs: 1 peering, 167 active+clean; 952 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 959 B/s rd, 0 B/s wr, 910 op/s; 18 MiB/s, 291 objects/s recovering 2024-09-16T10:33:10.660 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:10.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:10.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:10.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:10.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:10.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:10.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:10.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:11.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:11.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:11.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:11.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T10:33:11.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:11.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:11.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:11.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:11.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:11.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:11.399 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:11.471 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:11.510 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:11.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:11.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:11.660 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:11.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:11.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:11.764 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:33:11.764 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:33:11.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:11.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:11.796 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:33:11.797 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:33:11.833 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:33:11.834 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:33:11.866 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:33:11.867 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:33:11.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:11.906 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:33:11.906 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:33:11.936 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:33:11.936 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:33:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:11.941 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:11.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:11.976 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:33:11.976 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:33:12.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:12.049 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:33:12.049 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:33:12.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:12.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:12.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:12.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:12 smithi063 ceph-mon[29993]: pgmap v1058: 168 pgs: 168 active+clean; 906 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 717 op/s; 10 MiB/s, 166 objects/s recovering 2024-09-16T10:33:12.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:12 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/557285035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:12 smithi063 ceph-mon[39470]: pgmap v1058: 168 pgs: 168 active+clean; 906 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 717 op/s; 10 MiB/s, 166 objects/s recovering 2024-09-16T10:33:12.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:12 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/557285035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:12.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:12.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:12.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:12.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:12.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:12.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:12.505 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:12.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:12 smithi165 ceph-mon[32696]: pgmap v1058: 168 pgs: 168 active+clean; 906 MiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 717 op/s; 10 MiB/s, 166 objects/s recovering 2024-09-16T10:33:12.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:12 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/557285035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:12.606 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:12.649 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:12.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:12.715 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T10:33:12.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:12.800 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:12.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:12.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:12.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:12.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:12.942 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:13.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:13.072 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:13.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:13.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:13.151 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T10:33:13.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:13.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:13.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:13.307 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:13.345 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:13.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:13.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[29993]: 4.3 scrub starts 2024-09-16T10:33:13.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[29993]: 4.3 scrub ok 2024-09-16T10:33:13.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3295282491' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:33:13.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[29993]: from='client.26815 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:33:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[39470]: 4.3 scrub starts 2024-09-16T10:33:13.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[39470]: 4.3 scrub ok 2024-09-16T10:33:13.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3295282491' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:33:13.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:13 smithi063 ceph-mon[39470]: from='client.26815 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:33:13.481 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:13.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:13 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:13 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:13 smithi165 ceph-mon[32696]: 4.3 scrub starts 2024-09-16T10:33:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:13 smithi165 ceph-mon[32696]: 4.3 scrub ok 2024-09-16T10:33:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:13 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3295282491' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:33:13.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:13 smithi165 ceph-mon[32696]: from='client.26815 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:33:13.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:13.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:13.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:13.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:13.763 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:13.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:13.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:13.903 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:13.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:14.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:14.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:14.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:14.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:14.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:14.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:14.277 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:14.320 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:14.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:14.422 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:14.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:14 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:33:14.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:14 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:33:14.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:14 smithi165 ceph-mon[32696]: from='client.26815 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:33:14.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:14 smithi165 ceph-mon[32696]: osdmap e557: 8 total, 8 up, 6 in 2024-09-16T10:33:14.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:14 smithi165 ceph-mon[32696]: 11.5 scrub starts 2024-09-16T10:33:14.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:14 smithi165 ceph-mon[32696]: pgmap v1060: 168 pgs: 168 active+clean; 850 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 699 B/s rd, 0 B/s wr, 713 op/s; 122 KiB/s, 1 objects/s recovering 2024-09-16T10:33:14.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:33:14.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[29993]: from='client.26815 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[29993]: osdmap e557: 8 total, 8 up, 6 in 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[29993]: 11.5 scrub starts 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[29993]: pgmap v1060: 168 pgs: 168 active+clean; 850 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 699 B/s rd, 0 B/s wr, 713 op/s; 122 KiB/s, 1 objects/s recovering 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[39470]: from='client.26815 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[39470]: osdmap e557: 8 total, 8 up, 6 in 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[39470]: 11.5 scrub starts 2024-09-16T10:33:14.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:14 smithi063 ceph-mon[39470]: pgmap v1060: 168 pgs: 168 active+clean; 850 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 699 B/s rd, 0 B/s wr, 713 op/s; 122 KiB/s, 1 objects/s recovering 2024-09-16T10:33:14.458 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:14.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:14.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:14.615 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:14.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:14.718 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:14.747 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:14.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:14.849 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:14.885 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:14.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:14.944 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T10:33:14.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:15.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:15.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:15.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:15.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:15.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:15.366 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:15.409 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:15.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[29993]: 5.8 scrub starts 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[29993]: 5.8 scrub ok 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[29993]: 11.5 scrub ok 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[29993]: 4.1e scrub starts 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[29993]: 4.1e scrub ok 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[39470]: 5.8 scrub starts 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[39470]: 5.8 scrub ok 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[39470]: 11.5 scrub ok 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[39470]: 4.1e scrub starts 2024-09-16T10:33:15.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:15 smithi063 ceph-mon[39470]: 4.1e scrub ok 2024-09-16T10:33:15.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:15.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:15.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:15 smithi165 ceph-mon[32696]: 5.8 scrub starts 2024-09-16T10:33:15.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:15 smithi165 ceph-mon[32696]: 5.8 scrub ok 2024-09-16T10:33:15.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:15 smithi165 ceph-mon[32696]: 11.5 scrub ok 2024-09-16T10:33:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:15 smithi165 ceph-mon[32696]: 4.1e scrub starts 2024-09-16T10:33:15.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:15 smithi165 ceph-mon[32696]: 4.1e scrub ok 2024-09-16T10:33:15.643 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:15.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:15.790 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:33:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:33:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:242271 11.c 11:31082b56:::benchmark_data_smithi063_176907_object117995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-09-16T10:33:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:14.698477+0000", 2024-09-16T10:33:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0097076690000000004, 2024-09-16T10:33:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0097372340000000009, 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 242271 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698477+0000", 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698477+0000", 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698479+0000", 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8109999999999999e-06 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698484+0000", 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7899999999999999e-06 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698496+0000", 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2415000000000001e-05 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698503+0000", 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9330000000000004e-06 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.699904+0000", 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0014004880000000001 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.700008+0000", 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000104045 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,5", 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.700111+0000", 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010355499999999999 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.706715+0000", 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0066042160000000004 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.706769+0000", 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3372000000000003e-05 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.708073+0000", 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001304576 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:242272 11.c 11:3a137dcf:::benchmark_data_smithi063_176907_object117996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-09-16T10:33:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:14.698515+0000", 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0096697929999999994, 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.009795013, 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 242272 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698515+0000", 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698515+0000", 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698516+0000", 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1500000000000004e-07 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698519+0000", 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.8399999999999999e-06 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698528+0000", 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.4690000000000003e-06 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.698533+0000", 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0440000000000003e-06 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.702244+0000", 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037104870000000002 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.702417+0000", 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00017328600000000001 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,5", 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.702527+0000", 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000110591 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:242279 11.d 11:b8c66313:::benchmark_data_smithi063_176907_object118003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:14.701187+0000", 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0069976450000000003, 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0071911479999999996, 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "waiting for sub ops", 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 242279 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.701187+0000", 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.701187+0000", 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.701189+0000", 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.945e-06 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.701194+0000", 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.152e-06 2024-09-16T10:33:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.701210+0000", 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6050999999999999e-05 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.701219+0000", 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5580000000000001e-06 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.701284+0000", 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4597999999999994e-05 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.702217+0000", 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00093386000000000001 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 4,6", 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.702330+0000", 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112613 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705865+0000", 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0035351279999999998 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:242278 11.4 e557/542 11:24c4e58b:::benchmark_data_smithi063_176907_object118002:head v 557'14861, mlcod=557'14858)", 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:14.704401+0000", 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0037833739999999999, 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0040485920000000002, 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704401+0000", 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704401+0000", 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704402+0000", 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.78e-07 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704411+0000", 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.5909999999999996e-06 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704418+0000", 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3509999999999996e-06 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704423+0000", 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0579999999999997e-06 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.706792+0000", 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002368602 2024-09-16T10:33:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.706816+0000", 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.3649999999999999e-05 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:242283 11.4 e557/542)", 2024-09-16T10:33:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:14.704433+0000", 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.003752263, 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.004065333, 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704433+0000", 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704433+0000", 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704433+0000", 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9499999999999998e-07 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704436+0000", 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.3400000000000002e-06 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704441+0000", 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4510000000000002e-06 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.704445+0000", 2024-09-16T10:33:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.8580000000000002e-06 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.26491.0:242272 11.c e557/544)", 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:14.705717+0000", 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0024674739999999999, 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028345169999999999, 2024-09-16T10:33:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "reached pg", 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705717+0000", 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705717+0000", 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705719+0000", 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2079999999999999e-06 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705723+0000", 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5619999999999997e-06 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705736+0000", 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2974e-05 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705743+0000", 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.781e-06 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.708553+0000", 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028101990000000002 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop_reply(client.26491.0:242272 11.c e557/544)", 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:14.705812+0000", 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0023727750000000001, 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0027887570000000002, 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "queued for pg", 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705812+0000", 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705812+0000", 2024-09-16T10:33:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705813+0000", 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.2699999999999998e-07 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705817+0000", 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1810000000000003e-06 2024-09-16T10:33:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705831+0000", 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.3899e-05 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:14.705837+0000", 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7209999999999999e-06 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-09-16T10:33:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "num_ops": 7 2024-09-16T10:33:15.823 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:33:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:15.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:15.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:15.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:15.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:16.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:16.094 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:16.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:16.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:16.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:16.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:16.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:16.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:16.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:16.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:16.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:16.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:16 smithi063 ceph-mon[29993]: 5.0 scrub starts 2024-09-16T10:33:16.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:16 smithi063 ceph-mon[29993]: 5.0 scrub ok 2024-09-16T10:33:16.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:16 smithi063 ceph-mon[29993]: pgmap v1061: 168 pgs: 168 active+clean; 455 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 112 KiB/s, 1 objects/s recovering 2024-09-16T10:33:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:16 smithi063 ceph-mon[39470]: 5.0 scrub starts 2024-09-16T10:33:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:16 smithi063 ceph-mon[39470]: 5.0 scrub ok 2024-09-16T10:33:16.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:16 smithi063 ceph-mon[39470]: pgmap v1061: 168 pgs: 168 active+clean; 455 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 112 KiB/s, 1 objects/s recovering 2024-09-16T10:33:16.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:16.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:16.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:16.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:16 smithi165 ceph-mon[32696]: 5.0 scrub starts 2024-09-16T10:33:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:16 smithi165 ceph-mon[32696]: 5.0 scrub ok 2024-09-16T10:33:16.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:16 smithi165 ceph-mon[32696]: pgmap v1061: 168 pgs: 168 active+clean; 455 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 112 KiB/s, 1 objects/s recovering 2024-09-16T10:33:16.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:16.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:16.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:16.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:16.757 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:16.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:16.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:16.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:16.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:16.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:17.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:17.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:17.166 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:17.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:17.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.447 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:17.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:17.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:33:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:33:17.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[39470]: osdmap e558: 8 total, 8 up, 6 in 2024-09-16T10:33:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[39470]: 11.e scrub starts 2024-09-16T10:33:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:33:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:33:17.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[39470]: 11.e scrub ok 2024-09-16T10:33:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[29993]: osdmap e558: 8 total, 8 up, 6 in 2024-09-16T10:33:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[29993]: 11.e scrub starts 2024-09-16T10:33:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:33:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:33:17.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:17 smithi063 ceph-mon[29993]: 11.e scrub ok 2024-09-16T10:33:17.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:17.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:17 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:17.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:17 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:17.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:17 smithi165 ceph-mon[32696]: osdmap e558: 8 total, 8 up, 6 in 2024-09-16T10:33:17.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:17 smithi165 ceph-mon[32696]: 11.e scrub starts 2024-09-16T10:33:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:17 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:33:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:17 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:33:17.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:17 smithi165 ceph-mon[32696]: 11.e scrub ok 2024-09-16T10:33:17.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:17.671 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:17.694 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:17.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:17.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:17.926 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:17.970 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:17.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:18.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:18.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:18.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:18.198 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:18.243 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:18.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:18.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:18.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:18.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[29993]: 6.f scrub starts 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[29993]: 6.f scrub ok 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[29993]: 3.1f scrub starts 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[29993]: 3.1f scrub ok 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[29993]: 1.8 scrub starts 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[29993]: 1.8 scrub ok 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[29993]: pgmap v1063: 168 pgs: 168 active+clean; 455 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[39470]: 6.f scrub starts 2024-09-16T10:33:18.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[39470]: 6.f scrub ok 2024-09-16T10:33:18.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[39470]: 3.1f scrub starts 2024-09-16T10:33:18.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[39470]: 3.1f scrub ok 2024-09-16T10:33:18.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[39470]: 1.8 scrub starts 2024-09-16T10:33:18.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[39470]: 1.8 scrub ok 2024-09-16T10:33:18.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:18 smithi063 ceph-mon[39470]: pgmap v1063: 168 pgs: 168 active+clean; 455 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-16T10:33:18.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:18.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:18.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:18.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:18 smithi165 ceph-mon[32696]: 6.f scrub starts 2024-09-16T10:33:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:18 smithi165 ceph-mon[32696]: 6.f scrub ok 2024-09-16T10:33:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:18 smithi165 ceph-mon[32696]: 3.1f scrub starts 2024-09-16T10:33:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:18 smithi165 ceph-mon[32696]: 3.1f scrub ok 2024-09-16T10:33:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:18 smithi165 ceph-mon[32696]: 1.8 scrub starts 2024-09-16T10:33:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:18 smithi165 ceph-mon[32696]: 1.8 scrub ok 2024-09-16T10:33:18.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:18 smithi165 ceph-mon[32696]: pgmap v1063: 168 pgs: 168 active+clean; 455 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-16T10:33:18.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:18.654 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:18.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:18.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:18.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:18.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:18.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:18.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:18.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:18.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:19.006 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:19.047 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:19.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:19.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:19.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:19.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:19.410 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.450 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:19.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:19 smithi063 ceph-mon[29993]: 3.12 scrub starts 2024-09-16T10:33:19.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:19 smithi063 ceph-mon[29993]: 3.12 scrub ok 2024-09-16T10:33:19.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:19 smithi063 ceph-mon[39470]: 3.12 scrub starts 2024-09-16T10:33:19.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:19 smithi063 ceph-mon[39470]: 3.12 scrub ok 2024-09-16T10:33:19.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:19 smithi165 ceph-mon[32696]: 3.12 scrub starts 2024-09-16T10:33:19.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:19 smithi165 ceph-mon[32696]: 3.12 scrub ok 2024-09-16T10:33:19.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:33:19 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:33:19.575 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Removed 124273 objects 2024-09-16T10:33:19.575 INFO:tasks.radosbench.radosbench.0.smithi063.stdout:Clean up completed and total clean up time :129.24 2024-09-16T10:33:19.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:19.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:19.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:19.804 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.828 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:19.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:19.917 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-16T10:33:19.917 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-16T10:33:19.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:19.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:19.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:20.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T10:33:20.053 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:20.103 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:20.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:20.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:20.228 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:20.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:20.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:20.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:20.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:20.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:20.494 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:20 smithi063 ceph-mon[39470]: pgmap v1064: 168 pgs: 168 active+clean; 95 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.62k op/s 2024-09-16T10:33:20.495 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:20.495 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:20 smithi063 ceph-mon[29993]: pgmap v1064: 168 pgs: 168 active+clean; 95 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.62k op/s 2024-09-16T10:33:20.496 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:20.520 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:20.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:20.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:20 smithi165 ceph-mon[32696]: pgmap v1064: 168 pgs: 168 active+clean; 95 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.62k op/s 2024-09-16T10:33:20.588 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:20.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:20.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T10:33:20.643 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:20.650 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:20.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:20.752 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:20.777 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:20.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:20.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:20.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:20.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:21.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:21.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:21.089 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:21.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:21.158 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:21.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:21.283 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:21.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:21.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:21.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[29993]: 11.6 scrub starts 2024-09-16T10:33:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[29993]: 11.6 scrub ok 2024-09-16T10:33:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:33:21.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:33:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[39470]: 11.6 scrub starts 2024-09-16T10:33:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[39470]: 11.6 scrub ok 2024-09-16T10:33:21.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:33:21.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:21 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:33:21.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:21.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:21.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:21 smithi165 ceph-mon[32696]: 11.6 scrub starts 2024-09-16T10:33:21.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:21 smithi165 ceph-mon[32696]: 11.6 scrub ok 2024-09-16T10:33:21.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:21 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:33:21.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:21 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:33:21.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:21.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:21.772 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:21.781 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:33:21.781 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:33:21.788 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:21.798 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:21.802 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:33:21.802 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:33:21.820 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:33:21.820 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:33:21.836 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:33:21.836 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:33:21.854 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:33:21.854 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:33:21.870 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:33:21.871 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:33:21.888 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:33:21.888 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:33:21.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:21.905 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:33:21.905 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:33:21.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:21.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:22.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:22.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:22.156 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:22.180 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:22.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:22.205 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is set 2024-09-16T10:33:22.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:22.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:22.371 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/862864713' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:33:22.371 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[29993]: from='client.26821 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:33:22.371 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[29993]: pgmap v1065: 168 pgs: 168 active+clean; 52 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-16T10:33:22.371 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1086038225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:22.371 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/862864713' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:33:22.371 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[39470]: from='client.26821 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:33:22.371 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[39470]: pgmap v1065: 168 pgs: 168 active+clean; 52 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-16T10:33:22.372 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:22 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1086038225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:22.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:22.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:22.452 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:33:22.452 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:33:22.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:22.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:22 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/862864713' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:33:22.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:22 smithi165 ceph-mon[32696]: from='client.26821 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-16T10:33:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:22 smithi165 ceph-mon[32696]: pgmap v1065: 168 pgs: 168 active+clean; 52 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-16T10:33:22.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:22 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1086038225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:22.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:22.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:22.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:22.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:22.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:22.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:22.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:22.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:23.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.060 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:23.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:23.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:23.184 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:23.199 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[29993]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:23.199 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[29993]: from='client.26821 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:33:23.199 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[29993]: osdmap e559: 8 total, 8 up, 6 in 2024-09-16T10:33:23.199 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:33:23.203 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T10:33:23.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:23.310 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:23.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:23.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:23.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1168812397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:33:23.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[39470]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:23.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[39470]: from='client.26821 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:33:23.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[39470]: osdmap e559: 8 total, 8 up, 6 in 2024-09-16T10:33:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:33:23.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:23 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1168812397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:33:23.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:23 smithi165 ceph-mon[32696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:23.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:23 smithi165 ceph-mon[32696]: from='client.26821 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T10:33:23.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:23 smithi165 ceph-mon[32696]: osdmap e559: 8 total, 8 up, 6 in 2024-09-16T10:33:23.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:23 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:33:23.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:23 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1168812397' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:33:23.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:23.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:23.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:23.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:23.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:23.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:23.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:23.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:23.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:23.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:24.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:24.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:24.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:24.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:24.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:24.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:24.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:24.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:24.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1168812397' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:33:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:24 smithi063 ceph-mon[29993]: osdmap e560: 8 total, 8 up, 6 in 2024-09-16T10:33:24.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:24 smithi063 ceph-mon[29993]: pgmap v1068: 168 pgs: 168 active+clean; 37 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 696 B/s rd, 0 B/s wr, 909 op/s 2024-09-16T10:33:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1168812397' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:33:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:24 smithi063 ceph-mon[39470]: osdmap e560: 8 total, 8 up, 6 in 2024-09-16T10:33:24.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:24 smithi063 ceph-mon[39470]: pgmap v1068: 168 pgs: 168 active+clean; 37 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 696 B/s rd, 0 B/s wr, 909 op/s 2024-09-16T10:33:24.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1168812397' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:33:24.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:24 smithi165 ceph-mon[32696]: osdmap e560: 8 total, 8 up, 6 in 2024-09-16T10:33:24.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:24 smithi165 ceph-mon[32696]: pgmap v1068: 168 pgs: 168 active+clean; 37 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 696 B/s rd, 0 B/s wr, 909 op/s 2024-09-16T10:33:24.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T10:33:24.527 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:24.580 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:24.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:24.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:24.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:24.829 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:24.854 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:24.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:24.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:24.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:25.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:25.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:25.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:25.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:25.252 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:25.283 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T10:33:25.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:33:25.301 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.349 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:25.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:25.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:25.472 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:25 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:25.557 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:25 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:25.557 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:25 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:25.557 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:25 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:25.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:25.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:25.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T10:33:25.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:25.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:25.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:25 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:25.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:25 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:25.825 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:25.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:25.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:25.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:25.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:25.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:25.999 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T10:33:26.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.098 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:26.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:26.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4 2024-09-16T10:33:26.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.4' 2024-09-16T10:33:26.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:26.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:26.399 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'unique_pool_4' removed 2024-09-16T10:33:26.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:26.467 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:26.500 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:33:26.500 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:33:26.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:26.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:26.607 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:26 smithi063 ceph-mon[39470]: pgmap v1069: 168 pgs: 168 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 909 op/s 2024-09-16T10:33:26.607 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:26 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4068491657' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:33:26.607 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:26 smithi063 ceph-mon[39470]: from='client.46715 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:33:26.607 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:26 smithi063 ceph-mon[29993]: pgmap v1069: 168 pgs: 168 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 909 op/s 2024-09-16T10:33:26.607 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:26 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4068491657' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:33:26.607 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:26 smithi063 ceph-mon[29993]: from='client.46715 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:33:26.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:26.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:26.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:26 smithi165 ceph-mon[32696]: pgmap v1069: 168 pgs: 168 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 909 op/s 2024-09-16T10:33:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:26 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4068491657' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:33:26.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:26 smithi165 ceph-mon[32696]: from='client.46715 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T10:33:26.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:26.839 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:26.911 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:33:26.913 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:33:26.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:26.963 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:26.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:27.065 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:27.088 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:27.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:27.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:33:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:33:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:33:27.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:27.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:27.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:27.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:27.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:27.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:27.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:27.572 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:27 smithi063 ceph-mon[39470]: from='client.46715 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:33:27.573 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:27 smithi063 ceph-mon[39470]: osdmap e561: 8 total, 8 up, 6 in 2024-09-16T10:33:27.573 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:27 smithi063 ceph-mon[39470]: from='client.16887 v1:172.21.15.63:0/2280958433' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:27.573 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:27 smithi063 ceph-mon[29993]: from='client.46715 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:33:27.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:27 smithi063 ceph-mon[29993]: osdmap e561: 8 total, 8 up, 6 in 2024-09-16T10:33:27.574 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:27 smithi063 ceph-mon[29993]: from='client.16887 v1:172.21.15.63:0/2280958433' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:27.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:27.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T10:33:27.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1 2024-09-16T10:33:27.706 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.1' 2024-09-16T10:33:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:27 smithi165 ceph-mon[32696]: from='client.46715 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T10:33:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:27 smithi165 ceph-mon[32696]: osdmap e561: 8 total, 8 up, 6 in 2024-09-16T10:33:27.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:27 smithi165 ceph-mon[32696]: from='client.16887 v1:172.21.15.63:0/2280958433' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:27.808 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:27.834 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:27.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7 2024-09-16T10:33:27.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:27.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.7' 2024-09-16T10:33:28.026 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":1070,"stamp":"2024-09-16T10:33:26.388010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1188,"num_read_kb":940,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":533032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1024,"ondisk_log_size":1024,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4767252,"kb_used_data":7056,"kb_used_omap":9,"kb_used_meta":4760054,"kb_avail":557703660,"statfs":{"total":575970213888,"available":571088547840,"internally_reserved":0,"allocated":7225344,"data_stored":4632662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":4874295970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-98828312,"num_objects":-1509,"num_object_clones":0,"num_object_copies":-4527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1509,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-247039,"num_write_kb":-7953473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-46483,"num_bytes_recovered":-3045785600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.855505"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1084,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227591+0000","last_change":"2024-09-16T10:33:17.292872+0000","last_active":"2024-09-16T10:33:23.227591+0000","last_peered":"2024-09-16T10:33:23.227591+0000","last_clean":"2024-09-16T10:33:23.227591+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:33:23.227591+0000","last_undegraded":"2024-09-16T10:33:23.227591+0000","last_fullsized":"2024-09-16T10:33:23.227591+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:17.292796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:17.292796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:40.930967+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":1046,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225057+0000","last_change":"2024-09-16T10:32:56.707159+0000","last_active":"2024-09-16T10:33:23.225057+0000","last_peered":"2024-09-16T10:33:23.225057+0000","last_clean":"2024-09-16T10:33:23.225057+0000","last_became_active":"2024-09-16T10:32:20.414426+0000","last_became_peered":"2024-09-16T10:32:20.414426+0000","last_unstale":"2024-09-16T10:33:23.225057+0000","last_undegraded":"2024-09-16T10:33:23.225057+0000","last_fullsized":"2024-09-16T10:33:23.225057+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:32:56.707086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:56.707086+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:01.126649+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1026,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724497+0000","last_change":"2024-09-16T10:32:20.401571+0000","last_active":"2024-09-16T10:33:22.724497+0000","last_peered":"2024-09-16T10:33:22.724497+0000","last_clean":"2024-09-16T10:33:22.724497+0000","last_became_active":"2024-09-16T10:32:20.401119+0000","last_became_peered":"2024-09-16T10:32:20.401119+0000","last_unstale":"2024-09-16T10:33:22.724497+0000","last_undegraded":"2024-09-16T10:33:22.724497+0000","last_fullsized":"2024-09-16T10:33:22.724497+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.872906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.872906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":390,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724375+0000","last_change":"2024-09-16T10:32:20.381952+0000","last_active":"2024-09-16T10:33:22.724375+0000","last_peered":"2024-09-16T10:33:22.724375+0000","last_clean":"2024-09-16T10:33:22.724375+0000","last_became_active":"2024-09-16T10:32:20.380629+0000","last_became_peered":"2024-09-16T10:32:20.380629+0000","last_unstale":"2024-09-16T10:33:22.724375+0000","last_undegraded":"2024-09-16T10:33:22.724375+0000","last_fullsized":"2024-09-16T10:33:22.724375+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:17.879610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:17.879610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1064,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225239+0000","last_change":"2024-09-16T10:32:20.435331+0000","last_active":"2024-09-16T10:33:23.225239+0000","last_peered":"2024-09-16T10:33:23.225239+0000","last_clean":"2024-09-16T10:33:23.225239+0000","last_became_active":"2024-09-16T10:32:20.433546+0000","last_became_peered":"2024-09-16T10:32:20.433546+0000","last_unstale":"2024-09-16T10:33:23.225239+0000","last_undegraded":"2024-09-16T10:33:23.225239+0000","last_fullsized":"2024-09-16T10:33:23.225239+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1083,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226249+0000","last_change":"2024-09-16T10:30:48.140925+0000","last_active":"2024-09-16T10:33:23.226249+0000","last_peered":"2024-09-16T10:33:23.226249+0000","last_clean":"2024-09-16T10:33:23.226249+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:33:23.226249+0000","last_undegraded":"2024-09-16T10:33:23.226249+0000","last_fullsized":"2024-09-16T10:33:23.226249+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":1153,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225936+0000","last_change":"2024-09-16T10:30:15.998707+0000","last_active":"2024-09-16T10:33:23.225936+0000","last_peered":"2024-09-16T10:33:23.225936+0000","last_clean":"2024-09-16T10:33:23.225936+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:33:23.225936+0000","last_undegraded":"2024-09-16T10:33:23.225936+0000","last_fullsized":"2024-09-16T10:33:23.225936+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:15.998614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.998614+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":400,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224787+0000","last_change":"2024-09-16T10:33:02.705141+0000","last_active":"2024-09-16T10:33:23.224787+0000","last_peered":"2024-09-16T10:33:23.224787+0000","last_clean":"2024-09-16T10:33:23.224787+0000","last_became_active":"2024-09-16T10:32:20.392399+0000","last_became_peered":"2024-09-16T10:32:20.392399+0000","last_unstale":"2024-09-16T10:33:23.224787+0000","last_undegraded":"2024-09-16T10:33:23.224787+0000","last_fullsized":"2024-09-16T10:33:23.224787+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:02.705078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:02.705078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:18.211849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1091,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725293+0000","last_change":"2024-09-16T10:33:16.405006+0000","last_active":"2024-09-16T10:33:22.725293+0000","last_peered":"2024-09-16T10:33:22.725293+0000","last_clean":"2024-09-16T10:33:22.725293+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:33:22.725293+0000","last_undegraded":"2024-09-16T10:33:22.725293+0000","last_fullsized":"2024-09-16T10:33:22.725293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:16.404954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:16.404954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:30.180621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":979,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227041+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:33:23.227041+0000","last_peered":"2024-09-16T10:33:23.227041+0000","last_clean":"2024-09-16T10:33:23.227041+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:33:23.227041+0000","last_undegraded":"2024-09-16T10:33:23.227041+0000","last_fullsized":"2024-09-16T10:33:23.227041+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1077,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724775+0000","last_change":"2024-09-16T10:31:14.144852+0000","last_active":"2024-09-16T10:33:22.724775+0000","last_peered":"2024-09-16T10:33:22.724775+0000","last_clean":"2024-09-16T10:33:22.724775+0000","last_became_active":"2024-09-16T10:31:14.144519+0000","last_became_peered":"2024-09-16T10:31:14.144519+0000","last_unstale":"2024-09-16T10:33:22.724775+0000","last_undegraded":"2024-09-16T10:33:22.724775+0000","last_fullsized":"2024-09-16T10:33:22.724775+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:41.762023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:41.762023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1082,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224723+0000","last_change":"2024-09-16T10:32:20.421334+0000","last_active":"2024-09-16T10:33:23.224723+0000","last_peered":"2024-09-16T10:33:23.224723+0000","last_clean":"2024-09-16T10:33:23.224723+0000","last_became_active":"2024-09-16T10:32:20.413663+0000","last_became_peered":"2024-09-16T10:32:20.413663+0000","last_unstale":"2024-09-16T10:33:23.224723+0000","last_undegraded":"2024-09-16T10:33:23.224723+0000","last_fullsized":"2024-09-16T10:33:23.224723+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:13.749174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:13.749174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:32.838025+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1042,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724976+0000","last_change":"2024-09-16T10:32:20.377023+0000","last_active":"2024-09-16T10:33:22.724976+0000","last_peered":"2024-09-16T10:33:22.724976+0000","last_clean":"2024-09-16T10:33:22.724976+0000","last_became_active":"2024-09-16T10:32:20.376673+0000","last_became_peered":"2024-09-16T10:32:20.376673+0000","last_unstale":"2024-09-16T10:33:22.724976+0000","last_undegraded":"2024-09-16T10:33:22.724976+0000","last_fullsized":"2024-09-16T10:33:22.724976+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:13.941032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:13.941032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":980,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225610+0000","last_change":"2024-09-16T10:32:20.470950+0000","last_active":"2024-09-16T10:33:23.225610+0000","last_peered":"2024-09-16T10:33:23.225610+0000","last_clean":"2024-09-16T10:33:23.225610+0000","last_became_active":"2024-09-16T10:32:20.434528+0000","last_became_peered":"2024-09-16T10:32:20.434528+0000","last_unstale":"2024-09-16T10:33:23.225610+0000","last_undegraded":"2024-09-16T10:33:23.225610+0000","last_fullsized":"2024-09-16T10:33:23.225610+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:30:39.082450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.082450+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":711,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225234+0000","last_change":"2024-09-16T10:32:20.408057+0000","last_active":"2024-09-16T10:33:23.225234+0000","last_peered":"2024-09-16T10:33:23.225234+0000","last_clean":"2024-09-16T10:33:23.225234+0000","last_became_active":"2024-09-16T10:32:20.403637+0000","last_became_peered":"2024-09-16T10:32:20.403637+0000","last_unstale":"2024-09-16T10:33:23.225234+0000","last_undegraded":"2024-09-16T10:33:23.225234+0000","last_fullsized":"2024-09-16T10:33:23.225234+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.995349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.995349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1078,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724542+0000","last_change":"2024-09-16T10:32:20.401456+0000","last_active":"2024-09-16T10:33:22.724542+0000","last_peered":"2024-09-16T10:33:22.724542+0000","last_clean":"2024-09-16T10:33:22.724542+0000","last_became_active":"2024-09-16T10:32:20.401126+0000","last_became_peered":"2024-09-16T10:32:20.401126+0000","last_unstale":"2024-09-16T10:33:22.724542+0000","last_undegraded":"2024-09-16T10:33:22.724542+0000","last_fullsized":"2024-09-16T10:33:22.724542+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.741362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.741362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":386,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226032+0000","last_change":"2024-09-16T10:32:11.660805+0000","last_active":"2024-09-16T10:33:23.226032+0000","last_peered":"2024-09-16T10:33:23.226032+0000","last_clean":"2024-09-16T10:33:23.226032+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:33:23.226032+0000","last_undegraded":"2024-09-16T10:33:23.226032+0000","last_fullsized":"2024-09-16T10:33:23.226032+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:11.660728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.660728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1077,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226188+0000","last_change":"2024-09-16T10:32:20.417609+0000","last_active":"2024-09-16T10:33:23.226188+0000","last_peered":"2024-09-16T10:33:23.226188+0000","last_clean":"2024-09-16T10:33:23.226188+0000","last_became_active":"2024-09-16T10:32:20.416791+0000","last_became_peered":"2024-09-16T10:32:20.416791+0000","last_unstale":"2024-09-16T10:33:23.226188+0000","last_undegraded":"2024-09-16T10:33:23.226188+0000","last_fullsized":"2024-09-16T10:33:23.226188+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.446934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.446934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":652,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206301+0000","last_change":"2024-09-16T10:32:20.499898+0000","last_active":"2024-09-16T10:33:23.206301+0000","last_peered":"2024-09-16T10:33:23.206301+0000","last_clean":"2024-09-16T10:33:23.206301+0000","last_became_active":"2024-09-16T10:32:20.490019+0000","last_became_peered":"2024-09-16T10:32:20.490019+0000","last_unstale":"2024-09-16T10:33:23.206301+0000","last_undegraded":"2024-09-16T10:33:23.206301+0000","last_fullsized":"2024-09-16T10:33:23.206301+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.599903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.599903+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1053,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225427+0000","last_change":"2024-09-16T10:32:20.415199+0000","last_active":"2024-09-16T10:33:23.225427+0000","last_peered":"2024-09-16T10:33:23.225427+0000","last_clean":"2024-09-16T10:33:23.225427+0000","last_became_active":"2024-09-16T10:32:20.414861+0000","last_became_peered":"2024-09-16T10:32:20.414861+0000","last_unstale":"2024-09-16T10:33:23.225427+0000","last_undegraded":"2024-09-16T10:33:23.225427+0000","last_fullsized":"2024-09-16T10:33:23.225427+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:09.918371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:09.918371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1014,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225655+0000","last_change":"2024-09-16T10:33:08.153114+0000","last_active":"2024-09-16T10:33:23.225655+0000","last_peered":"2024-09-16T10:33:23.225655+0000","last_clean":"2024-09-16T10:33:23.225655+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:33:23.225655+0000","last_undegraded":"2024-09-16T10:33:23.225655+0000","last_fullsized":"2024-09-16T10:33:23.225655+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:08.153044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:08.153044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:22.272748+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":537,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205385+0000","last_change":"2024-09-16T10:32:20.424078+0000","last_active":"2024-09-16T10:33:23.205385+0000","last_peered":"2024-09-16T10:33:23.205385+0000","last_clean":"2024-09-16T10:33:23.205385+0000","last_became_active":"2024-09-16T10:32:20.423662+0000","last_became_peered":"2024-09-16T10:32:20.423662+0000","last_unstale":"2024-09-16T10:33:23.205385+0000","last_undegraded":"2024-09-16T10:33:23.205385+0000","last_fullsized":"2024-09-16T10:33:23.205385+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:09.147080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.147080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:25.390018+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":1141,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226779+0000","last_change":"2024-09-16T10:32:20.461961+0000","last_active":"2024-09-16T10:33:23.226779+0000","last_peered":"2024-09-16T10:33:23.226779+0000","last_clean":"2024-09-16T10:33:23.226779+0000","last_became_active":"2024-09-16T10:32:20.429076+0000","last_became_peered":"2024-09-16T10:32:20.429076+0000","last_unstale":"2024-09-16T10:33:23.226779+0000","last_undegraded":"2024-09-16T10:33:23.226779+0000","last_fullsized":"2024-09-16T10:33:23.226779+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:30:38.072460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:38.072460+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":849,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227374+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:33:23.227374+0000","last_peered":"2024-09-16T10:33:23.227374+0000","last_clean":"2024-09-16T10:33:23.227374+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:33:23.227374+0000","last_undegraded":"2024-09-16T10:33:23.227374+0000","last_fullsized":"2024-09-16T10:33:23.227374+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:48.236715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":481,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724763+0000","last_change":"2024-09-16T10:31:14.169546+0000","last_active":"2024-09-16T10:33:22.724763+0000","last_peered":"2024-09-16T10:33:22.724763+0000","last_clean":"2024-09-16T10:33:22.724763+0000","last_became_active":"2024-09-16T10:31:14.151297+0000","last_became_peered":"2024-09-16T10:31:14.151297+0000","last_unstale":"2024-09-16T10:33:22.724763+0000","last_undegraded":"2024-09-16T10:33:22.724763+0000","last_fullsized":"2024-09-16T10:33:22.724763+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:40.788922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.788922+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":79,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226069+0000","last_change":"2024-09-16T10:32:20.469065+0000","last_active":"2024-09-16T10:33:23.226069+0000","last_peered":"2024-09-16T10:33:23.226069+0000","last_clean":"2024-09-16T10:33:23.226069+0000","last_became_active":"2024-09-16T10:32:20.432046+0000","last_became_peered":"2024-09-16T10:32:20.432046+0000","last_unstale":"2024-09-16T10:33:23.226069+0000","last_undegraded":"2024-09-16T10:33:23.226069+0000","last_fullsized":"2024-09-16T10:33:23.226069+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:30:12.928274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.928274+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":1145,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227329+0000","last_change":"2024-09-16T10:33:14.278928+0000","last_active":"2024-09-16T10:33:23.227329+0000","last_peered":"2024-09-16T10:33:23.227329+0000","last_clean":"2024-09-16T10:33:23.227329+0000","last_became_active":"2024-09-16T10:32:20.407208+0000","last_became_peered":"2024-09-16T10:32:20.407208+0000","last_unstale":"2024-09-16T10:33:23.227329+0000","last_undegraded":"2024-09-16T10:33:23.227329+0000","last_fullsized":"2024-09-16T10:33:23.227329+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:14.278843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:14.278843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:33.769863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"508'11","reported_seq":1327,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226561+0000","last_change":"2024-09-16T10:31:14.167033+0000","last_active":"2024-09-16T10:33:23.226561+0000","last_peered":"2024-09-16T10:33:23.226561+0000","last_clean":"2024-09-16T10:33:23.226561+0000","last_became_active":"2024-09-16T10:31:14.137776+0000","last_became_peered":"2024-09-16T10:31:14.137776+0000","last_unstale":"2024-09-16T10:33:23.226561+0000","last_undegraded":"2024-09-16T10:33:23.226561+0000","last_fullsized":"2024-09-16T10:33:23.226561+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.694030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":914,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724401+0000","last_change":"2024-09-16T10:32:16.263470+0000","last_active":"2024-09-16T10:33:22.724401+0000","last_peered":"2024-09-16T10:33:22.724401+0000","last_clean":"2024-09-16T10:33:22.724401+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:33:22.724401+0000","last_undegraded":"2024-09-16T10:33:22.724401+0000","last_fullsized":"2024-09-16T10:33:22.724401+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:16.263399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:16.263399+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:27.317800+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":694,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225883+0000","last_change":"2024-09-16T10:33:06.135162+0000","last_active":"2024-09-16T10:33:23.225883+0000","last_peered":"2024-09-16T10:33:23.225883+0000","last_clean":"2024-09-16T10:33:23.225883+0000","last_became_active":"2024-09-16T10:32:20.414383+0000","last_became_peered":"2024-09-16T10:32:20.414383+0000","last_unstale":"2024-09-16T10:33:23.225883+0000","last_undegraded":"2024-09-16T10:33:23.225883+0000","last_fullsized":"2024-09-16T10:33:23.225883+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:33:06.135084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.135084+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:20.735523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1094,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225321+0000","last_change":"2024-09-16T10:32:20.421467+0000","last_active":"2024-09-16T10:33:23.225321+0000","last_peered":"2024-09-16T10:33:23.225321+0000","last_clean":"2024-09-16T10:33:23.225321+0000","last_became_active":"2024-09-16T10:32:20.417850+0000","last_became_peered":"2024-09-16T10:32:20.417850+0000","last_unstale":"2024-09-16T10:33:23.225321+0000","last_undegraded":"2024-09-16T10:33:23.225321+0000","last_fullsized":"2024-09-16T10:33:23.225321+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.213821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.213821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":493,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725225+0000","last_change":"2024-09-16T10:30:16.857737+0000","last_active":"2024-09-16T10:33:22.725225+0000","last_peered":"2024-09-16T10:33:22.725225+0000","last_clean":"2024-09-16T10:33:22.725225+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:33:22.725225+0000","last_undegraded":"2024-09-16T10:33:22.725225+0000","last_fullsized":"2024-09-16T10:33:22.725225+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:30:16.857676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.857676+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1063,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724910+0000","last_change":"2024-09-16T10:32:20.407177+0000","last_active":"2024-09-16T10:33:22.724910+0000","last_peered":"2024-09-16T10:33:22.724910+0000","last_clean":"2024-09-16T10:33:22.724910+0000","last_became_active":"2024-09-16T10:32:20.406800+0000","last_became_peered":"2024-09-16T10:32:20.406800+0000","last_unstale":"2024-09-16T10:33:22.724910+0000","last_undegraded":"2024-09-16T10:33:22.724910+0000","last_fullsized":"2024-09-16T10:33:22.724910+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:14.925284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:14.925284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":490,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227102+0000","last_change":"2024-09-16T10:30:18.655233+0000","last_active":"2024-09-16T10:33:23.227102+0000","last_peered":"2024-09-16T10:33:23.227102+0000","last_clean":"2024-09-16T10:33:23.227102+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:33:23.227102+0000","last_undegraded":"2024-09-16T10:33:23.227102+0000","last_fullsized":"2024-09-16T10:33:23.227102+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:18.655172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:18.655172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":451,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225338+0000","last_change":"2024-09-16T10:33:04.704385+0000","last_active":"2024-09-16T10:33:23.225338+0000","last_peered":"2024-09-16T10:33:23.225338+0000","last_clean":"2024-09-16T10:33:23.225338+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:33:23.225338+0000","last_undegraded":"2024-09-16T10:33:23.225338+0000","last_fullsized":"2024-09-16T10:33:23.225338+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:33:04.704308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:04.704308+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:14.562749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1051,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226190+0000","last_change":"2024-09-16T10:31:14.146422+0000","last_active":"2024-09-16T10:33:23.226190+0000","last_peered":"2024-09-16T10:33:23.226190+0000","last_clean":"2024-09-16T10:33:23.226190+0000","last_became_active":"2024-09-16T10:31:14.146107+0000","last_became_peered":"2024-09-16T10:31:14.146107+0000","last_unstale":"2024-09-16T10:33:23.226190+0000","last_undegraded":"2024-09-16T10:33:23.226190+0000","last_fullsized":"2024-09-16T10:33:23.226190+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:55.254993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.254993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":818,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225532+0000","last_change":"2024-09-16T10:32:06.770331+0000","last_active":"2024-09-16T10:33:23.225532+0000","last_peered":"2024-09-16T10:33:23.225532+0000","last_clean":"2024-09-16T10:33:23.225532+0000","last_became_active":"2024-09-16T10:31:14.164506+0000","last_became_peered":"2024-09-16T10:31:14.164506+0000","last_unstale":"2024-09-16T10:33:23.225532+0000","last_undegraded":"2024-09-16T10:33:23.225532+0000","last_fullsized":"2024-09-16T10:33:23.225532+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:06.770275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.770275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:34.696037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":865,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226943+0000","last_change":"2024-09-16T10:33:23.226943+0000","last_active":"2024-09-16T10:33:23.226943+0000","last_peered":"2024-09-16T10:33:23.226943+0000","last_clean":"2024-09-16T10:33:23.226943+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:33:23.226943+0000","last_undegraded":"2024-09-16T10:33:23.226943+0000","last_fullsized":"2024-09-16T10:33:23.226943+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:52.274606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":987,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227325+0000","last_change":"2024-09-16T10:31:17.119206+0000","last_active":"2024-09-16T10:33:23.227325+0000","last_peered":"2024-09-16T10:33:23.227325+0000","last_clean":"2024-09-16T10:33:23.227325+0000","last_became_active":"2024-09-16T10:31:17.118842+0000","last_became_peered":"2024-09-16T10:31:17.118842+0000","last_unstale":"2024-09-16T10:33:23.227325+0000","last_undegraded":"2024-09-16T10:33:23.227325+0000","last_fullsized":"2024-09-16T10:33:23.227325+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.198923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.198923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":1005,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724610+0000","last_change":"2024-09-16T10:30:12.893400+0000","last_active":"2024-09-16T10:33:22.724610+0000","last_peered":"2024-09-16T10:33:22.724610+0000","last_clean":"2024-09-16T10:33:22.724610+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:33:22.724610+0000","last_undegraded":"2024-09-16T10:33:22.724610+0000","last_fullsized":"2024-09-16T10:33:22.724610+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:30:12.893340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.893340+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":1066,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227515+0000","last_change":"2024-09-16T10:32:20.425958+0000","last_active":"2024-09-16T10:33:23.227515+0000","last_peered":"2024-09-16T10:33:23.227515+0000","last_clean":"2024-09-16T10:33:23.227515+0000","last_became_active":"2024-09-16T10:32:20.393386+0000","last_became_peered":"2024-09-16T10:32:20.393386+0000","last_unstale":"2024-09-16T10:33:23.227515+0000","last_undegraded":"2024-09-16T10:33:23.227515+0000","last_fullsized":"2024-09-16T10:33:23.227515+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:46.894131+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"536'9","reported_seq":1321,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225016+0000","last_change":"2024-09-16T10:33:15.204469+0000","last_active":"2024-09-16T10:33:23.225016+0000","last_peered":"2024-09-16T10:33:23.225016+0000","last_clean":"2024-09-16T10:33:23.225016+0000","last_became_active":"2024-09-16T10:32:20.418792+0000","last_became_peered":"2024-09-16T10:32:20.418792+0000","last_unstale":"2024-09-16T10:33:23.225016+0000","last_undegraded":"2024-09-16T10:33:23.225016+0000","last_fullsized":"2024-09-16T10:33:23.225016+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'9","last_scrub_stamp":"2024-09-16T10:33:15.201123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:15.201123+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:28.015856+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":935,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226996+0000","last_change":"2024-09-16T10:32:20.413255+0000","last_active":"2024-09-16T10:33:23.226996+0000","last_peered":"2024-09-16T10:33:23.226996+0000","last_clean":"2024-09-16T10:33:23.226996+0000","last_became_active":"2024-09-16T10:32:20.412898+0000","last_became_peered":"2024-09-16T10:32:20.412898+0000","last_unstale":"2024-09-16T10:33:23.226996+0000","last_undegraded":"2024-09-16T10:33:23.226996+0000","last_fullsized":"2024-09-16T10:33:23.226996+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:12.688413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.688413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:40.735663+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":1045,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226455+0000","last_change":"2024-09-16T10:32:16.334773+0000","last_active":"2024-09-16T10:33:23.226455+0000","last_peered":"2024-09-16T10:33:23.226455+0000","last_clean":"2024-09-16T10:33:23.226455+0000","last_became_active":"2024-09-16T10:32:16.334470+0000","last_became_peered":"2024-09-16T10:32:16.334470+0000","last_unstale":"2024-09-16T10:33:23.226455+0000","last_undegraded":"2024-09-16T10:33:23.226455+0000","last_fullsized":"2024-09-16T10:33:23.226455+0000","mapping_epoch":532,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.663450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.663450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":691,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724628+0000","last_change":"2024-09-16T10:32:20.446784+0000","last_active":"2024-09-16T10:33:22.724628+0000","last_peered":"2024-09-16T10:33:22.724628+0000","last_clean":"2024-09-16T10:33:22.724628+0000","last_became_active":"2024-09-16T10:32:20.395496+0000","last_became_peered":"2024-09-16T10:32:20.395496+0000","last_unstale":"2024-09-16T10:33:22.724628+0000","last_undegraded":"2024-09-16T10:33:22.724628+0000","last_fullsized":"2024-09-16T10:33:22.724628+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:50.360324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1083,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725388+0000","last_change":"2024-09-16T10:33:02.347961+0000","last_active":"2024-09-16T10:33:22.725388+0000","last_peered":"2024-09-16T10:33:22.725388+0000","last_clean":"2024-09-16T10:33:22.725388+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:33:22.725388+0000","last_undegraded":"2024-09-16T10:33:22.725388+0000","last_fullsized":"2024-09-16T10:33:22.725388+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:02.347917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:02.347917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:09.667103+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1292,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226748+0000","last_change":"2024-09-16T10:30:08.183366+0000","last_active":"2024-09-16T10:33:23.226748+0000","last_peered":"2024-09-16T10:33:23.226748+0000","last_clean":"2024-09-16T10:33:23.226748+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:33:23.226748+0000","last_undegraded":"2024-09-16T10:33:23.226748+0000","last_fullsized":"2024-09-16T10:33:23.226748+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:08.182949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.182949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1072,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224845+0000","last_change":"2024-09-16T10:30:12.258492+0000","last_active":"2024-09-16T10:33:23.224845+0000","last_peered":"2024-09-16T10:33:23.224845+0000","last_clean":"2024-09-16T10:33:23.224845+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:33:23.224845+0000","last_undegraded":"2024-09-16T10:33:23.224845+0000","last_fullsized":"2024-09-16T10:33:23.224845+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:12.258426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.258426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":1022,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225941+0000","last_change":"2024-09-16T10:32:55.757529+0000","last_active":"2024-09-16T10:33:23.225941+0000","last_peered":"2024-09-16T10:33:23.225941+0000","last_clean":"2024-09-16T10:33:23.225941+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:33:23.225941+0000","last_undegraded":"2024-09-16T10:33:23.225941+0000","last_fullsized":"2024-09-16T10:33:23.225941+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:55.757468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:55.757468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:58.838895+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":1178,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226567+0000","last_change":"2024-09-16T10:33:12.305837+0000","last_active":"2024-09-16T10:33:23.226567+0000","last_peered":"2024-09-16T10:33:23.226567+0000","last_clean":"2024-09-16T10:33:23.226567+0000","last_became_active":"2024-09-16T10:32:20.413038+0000","last_became_peered":"2024-09-16T10:32:20.413038+0000","last_unstale":"2024-09-16T10:33:23.226567+0000","last_undegraded":"2024-09-16T10:33:23.226567+0000","last_fullsized":"2024-09-16T10:33:23.226567+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:33:12.305749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:12.305749+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:35.711805+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":553,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206174+0000","last_change":"2024-09-16T10:32:20.426707+0000","last_active":"2024-09-16T10:33:23.206174+0000","last_peered":"2024-09-16T10:33:23.206174+0000","last_clean":"2024-09-16T10:33:23.206174+0000","last_became_active":"2024-09-16T10:32:20.426427+0000","last_became_peered":"2024-09-16T10:32:20.426427+0000","last_unstale":"2024-09-16T10:33:23.206174+0000","last_undegraded":"2024-09-16T10:33:23.206174+0000","last_fullsized":"2024-09-16T10:33:23.206174+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.303693+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1070,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226183+0000","last_change":"2024-09-16T10:30:50.670023+0000","last_active":"2024-09-16T10:33:23.226183+0000","last_peered":"2024-09-16T10:33:23.226183+0000","last_clean":"2024-09-16T10:33:23.226183+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:33:23.226183+0000","last_undegraded":"2024-09-16T10:33:23.226183+0000","last_fullsized":"2024-09-16T10:33:23.226183+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.669938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.669938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":540,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225719+0000","last_change":"2024-09-16T10:32:20.405226+0000","last_active":"2024-09-16T10:33:23.225719+0000","last_peered":"2024-09-16T10:33:23.225719+0000","last_clean":"2024-09-16T10:33:23.225719+0000","last_became_active":"2024-09-16T10:32:20.404614+0000","last_became_peered":"2024-09-16T10:32:20.404614+0000","last_unstale":"2024-09-16T10:33:23.225719+0000","last_undegraded":"2024-09-16T10:33:23.225719+0000","last_fullsized":"2024-09-16T10:33:23.225719+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:42.743403+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":958,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724343+0000","last_change":"2024-09-16T10:30:54.802360+0000","last_active":"2024-09-16T10:33:22.724343+0000","last_peered":"2024-09-16T10:33:22.724343+0000","last_clean":"2024-09-16T10:33:22.724343+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:33:22.724343+0000","last_undegraded":"2024-09-16T10:33:22.724343+0000","last_fullsized":"2024-09-16T10:33:22.724343+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:54.802277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.802277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"537'43","reported_seq":932,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206275+0000","last_change":"2024-09-16T10:32:20.529917+0000","last_active":"2024-09-16T10:33:23.206275+0000","last_peered":"2024-09-16T10:33:23.206275+0000","last_clean":"2024-09-16T10:33:23.206275+0000","last_became_active":"2024-09-16T10:32:20.435824+0000","last_became_peered":"2024-09-16T10:32:20.435824+0000","last_unstale":"2024-09-16T10:33:23.206275+0000","last_undegraded":"2024-09-16T10:33:23.206275+0000","last_fullsized":"2024-09-16T10:33:23.206275+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:32:12.197914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.197914+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":378,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226323+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:33:23.226323+0000","last_peered":"2024-09-16T10:33:23.226323+0000","last_clean":"2024-09-16T10:33:23.226323+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:33:23.226323+0000","last_undegraded":"2024-09-16T10:33:23.226323+0000","last_fullsized":"2024-09-16T10:33:23.226323+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:49.909210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":1090,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724862+0000","last_change":"2024-09-16T10:32:20.428905+0000","last_active":"2024-09-16T10:33:22.724862+0000","last_peered":"2024-09-16T10:33:22.724862+0000","last_clean":"2024-09-16T10:33:22.724862+0000","last_became_active":"2024-09-16T10:32:20.396067+0000","last_became_peered":"2024-09-16T10:32:20.396067+0000","last_unstale":"2024-09-16T10:33:22.724862+0000","last_undegraded":"2024-09-16T10:33:22.724862+0000","last_fullsized":"2024-09-16T10:33:22.724862+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:30:11.895746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.895746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1053,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226021+0000","last_change":"2024-09-16T10:32:20.395692+0000","last_active":"2024-09-16T10:33:23.226021+0000","last_peered":"2024-09-16T10:33:23.226021+0000","last_clean":"2024-09-16T10:33:23.226021+0000","last_became_active":"2024-09-16T10:32:20.395374+0000","last_became_peered":"2024-09-16T10:32:20.395374+0000","last_unstale":"2024-09-16T10:33:23.226021+0000","last_undegraded":"2024-09-16T10:33:23.226021+0000","last_fullsized":"2024-09-16T10:33:23.226021+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.379980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.379980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1083,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225644+0000","last_change":"2024-09-16T10:31:14.143712+0000","last_active":"2024-09-16T10:33:23.225644+0000","last_peered":"2024-09-16T10:33:23.225644+0000","last_clean":"2024-09-16T10:33:23.225644+0000","last_became_active":"2024-09-16T10:31:14.143349+0000","last_became_peered":"2024-09-16T10:31:14.143349+0000","last_unstale":"2024-09-16T10:33:23.225644+0000","last_undegraded":"2024-09-16T10:33:23.225644+0000","last_fullsized":"2024-09-16T10:33:23.225644+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.877913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.877913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":553,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225488+0000","last_change":"2024-09-16T10:31:17.108542+0000","last_active":"2024-09-16T10:33:23.225488+0000","last_peered":"2024-09-16T10:33:23.225488+0000","last_clean":"2024-09-16T10:33:23.225488+0000","last_became_active":"2024-09-16T10:31:17.108022+0000","last_became_peered":"2024-09-16T10:31:17.108022+0000","last_unstale":"2024-09-16T10:33:23.225488+0000","last_undegraded":"2024-09-16T10:33:23.225488+0000","last_fullsized":"2024-09-16T10:33:23.225488+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:49.434225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.434225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":660,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725294+0000","last_change":"2024-09-16T10:32:20.453683+0000","last_active":"2024-09-16T10:33:22.725294+0000","last_peered":"2024-09-16T10:33:22.725294+0000","last_clean":"2024-09-16T10:33:22.725294+0000","last_became_active":"2024-09-16T10:32:20.435036+0000","last_became_peered":"2024-09-16T10:32:20.435036+0000","last_unstale":"2024-09-16T10:33:22.725294+0000","last_undegraded":"2024-09-16T10:33:22.725294+0000","last_fullsized":"2024-09-16T10:33:22.725294+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:30:05.864159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.864159+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":545,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205142+0000","last_change":"2024-09-16T10:32:20.419274+0000","last_active":"2024-09-16T10:33:23.205142+0000","last_peered":"2024-09-16T10:33:23.205142+0000","last_clean":"2024-09-16T10:33:23.205142+0000","last_became_active":"2024-09-16T10:32:20.418545+0000","last_became_peered":"2024-09-16T10:32:20.418545+0000","last_unstale":"2024-09-16T10:33:23.205142+0000","last_undegraded":"2024-09-16T10:33:23.205142+0000","last_fullsized":"2024-09-16T10:33:23.205142+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:43.359607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":600,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205572+0000","last_change":"2024-09-16T10:32:11.066183+0000","last_active":"2024-09-16T10:33:23.205572+0000","last_peered":"2024-09-16T10:33:23.205572+0000","last_clean":"2024-09-16T10:33:23.205572+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:33:23.205572+0000","last_undegraded":"2024-09-16T10:33:23.205572+0000","last_fullsized":"2024-09-16T10:33:23.205572+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:32:11.066076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.066076+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:27.244502+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":614,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206737+0000","last_change":"2024-09-16T10:31:22.862857+0000","last_active":"2024-09-16T10:33:23.206737+0000","last_peered":"2024-09-16T10:33:23.206737+0000","last_clean":"2024-09-16T10:33:23.206737+0000","last_became_active":"2024-09-16T10:31:14.142132+0000","last_became_peered":"2024-09-16T10:31:14.142132+0000","last_unstale":"2024-09-16T10:33:23.206737+0000","last_undegraded":"2024-09-16T10:33:23.206737+0000","last_fullsized":"2024-09-16T10:33:23.206737+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:22.862677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:22.862677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":468,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226951+0000","last_change":"2024-09-16T10:31:14.164501+0000","last_active":"2024-09-16T10:33:23.226951+0000","last_peered":"2024-09-16T10:33:23.226951+0000","last_clean":"2024-09-16T10:33:23.226951+0000","last_became_active":"2024-09-16T10:31:14.164133+0000","last_became_peered":"2024-09-16T10:31:14.164133+0000","last_unstale":"2024-09-16T10:33:23.226951+0000","last_undegraded":"2024-09-16T10:33:23.226951+0000","last_fullsized":"2024-09-16T10:33:23.226951+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.706111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.706111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":162,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226027+0000","last_change":"2024-09-16T10:32:53.774953+0000","last_active":"2024-09-16T10:33:23.226027+0000","last_peered":"2024-09-16T10:33:23.226027+0000","last_clean":"2024-09-16T10:33:23.226027+0000","last_became_active":"2024-09-16T10:32:23.420463+0000","last_became_peered":"2024-09-16T10:32:23.420463+0000","last_unstale":"2024-09-16T10:33:23.226027+0000","last_undegraded":"2024-09-16T10:33:23.226027+0000","last_fullsized":"2024-09-16T10:33:23.226027+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:53.774903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:53.774903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.401981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":1181,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227352+0000","last_change":"2024-09-16T10:32:20.455220+0000","last_active":"2024-09-16T10:33:23.227352+0000","last_peered":"2024-09-16T10:33:23.227352+0000","last_clean":"2024-09-16T10:33:23.227352+0000","last_became_active":"2024-09-16T10:32:20.413651+0000","last_became_peered":"2024-09-16T10:32:20.413651+0000","last_unstale":"2024-09-16T10:33:23.227352+0000","last_undegraded":"2024-09-16T10:33:23.227352+0000","last_fullsized":"2024-09-16T10:33:23.227352+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:30:20.268438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.268438+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1086,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226070+0000","last_change":"2024-09-16T10:30:15.229737+0000","last_active":"2024-09-16T10:33:23.226070+0000","last_peered":"2024-09-16T10:33:23.226070+0000","last_clean":"2024-09-16T10:33:23.226070+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:33:23.226070+0000","last_undegraded":"2024-09-16T10:33:23.226070+0000","last_fullsized":"2024-09-16T10:33:23.226070+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.229669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.229669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":175,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.765760+0000","last_change":"2024-09-16T10:32:20.443363+0000","last_active":"2024-09-16T10:33:23.765760+0000","last_peered":"2024-09-16T10:33:23.765760+0000","last_clean":"2024-09-16T10:33:23.765760+0000","last_became_active":"2024-09-16T10:32:20.396791+0000","last_became_peered":"2024-09-16T10:32:20.396791+0000","last_unstale":"2024-09-16T10:33:23.765760+0000","last_undegraded":"2024-09-16T10:33:23.765760+0000","last_fullsized":"2024-09-16T10:33:23.765760+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:30:39.226399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.226399+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1084,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725271+0000","last_change":"2024-09-16T10:33:20.360912+0000","last_active":"2024-09-16T10:33:22.725271+0000","last_peered":"2024-09-16T10:33:22.725271+0000","last_clean":"2024-09-16T10:33:22.725271+0000","last_became_active":"2024-09-16T10:32:20.395732+0000","last_became_peered":"2024-09-16T10:32:20.395732+0000","last_unstale":"2024-09-16T10:33:22.725271+0000","last_undegraded":"2024-09-16T10:33:22.725271+0000","last_fullsized":"2024-09-16T10:33:22.725271+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:20.360862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:20.360862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:30.710303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":62,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725024+0000","last_change":"2024-09-16T10:32:20.435415+0000","last_active":"2024-09-16T10:33:22.725024+0000","last_peered":"2024-09-16T10:33:22.725024+0000","last_clean":"2024-09-16T10:33:22.725024+0000","last_became_active":"2024-09-16T10:32:20.434853+0000","last_became_peered":"2024-09-16T10:32:20.434853+0000","last_unstale":"2024-09-16T10:33:22.725024+0000","last_undegraded":"2024-09-16T10:33:22.725024+0000","last_fullsized":"2024-09-16T10:33:22.725024+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:47.361715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"508'31","reported_seq":1366,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227009+0000","last_change":"2024-09-16T10:31:14.178826+0000","last_active":"2024-09-16T10:33:23.227009+0000","last_peered":"2024-09-16T10:33:23.227009+0000","last_clean":"2024-09-16T10:33:23.227009+0000","last_became_active":"2024-09-16T10:31:14.137441+0000","last_became_peered":"2024-09-16T10:31:14.137441+0000","last_unstale":"2024-09-16T10:33:23.227009+0000","last_undegraded":"2024-09-16T10:33:23.227009+0000","last_fullsized":"2024-09-16T10:33:23.227009+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:30:54.737043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.737043+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1034,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227135+0000","last_change":"2024-09-16T10:32:07.379538+0000","last_active":"2024-09-16T10:33:23.227135+0000","last_peered":"2024-09-16T10:33:23.227135+0000","last_clean":"2024-09-16T10:33:23.227135+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:33:23.227135+0000","last_undegraded":"2024-09-16T10:33:23.227135+0000","last_fullsized":"2024-09-16T10:33:23.227135+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:07.379336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:07.379336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":570,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224956+0000","last_change":"2024-09-16T10:32:23.435277+0000","last_active":"2024-09-16T10:33:23.224956+0000","last_peered":"2024-09-16T10:33:23.224956+0000","last_clean":"2024-09-16T10:33:23.224956+0000","last_became_active":"2024-09-16T10:32:23.434945+0000","last_became_peered":"2024-09-16T10:32:23.434945+0000","last_unstale":"2024-09-16T10:33:23.224956+0000","last_undegraded":"2024-09-16T10:33:23.224956+0000","last_fullsized":"2024-09-16T10:33:23.224956+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:50.410561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.410561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":939,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206024+0000","last_change":"2024-09-16T10:31:14.146544+0000","last_active":"2024-09-16T10:33:23.206024+0000","last_peered":"2024-09-16T10:33:23.206024+0000","last_clean":"2024-09-16T10:33:23.206024+0000","last_became_active":"2024-09-16T10:31:14.145333+0000","last_became_peered":"2024-09-16T10:31:14.145333+0000","last_unstale":"2024-09-16T10:33:23.206024+0000","last_undegraded":"2024-09-16T10:33:23.206024+0000","last_fullsized":"2024-09-16T10:33:23.206024+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.613780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.613780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:23.684845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1057,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725399+0000","last_change":"2024-09-16T10:30:43.705706+0000","last_active":"2024-09-16T10:33:22.725399+0000","last_peered":"2024-09-16T10:33:22.725399+0000","last_clean":"2024-09-16T10:33:22.725399+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:33:22.725399+0000","last_undegraded":"2024-09-16T10:33:22.725399+0000","last_fullsized":"2024-09-16T10:33:22.725399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.705640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.705640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":498,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226970+0000","last_change":"2024-09-16T10:32:53.999521+0000","last_active":"2024-09-16T10:33:23.226970+0000","last_peered":"2024-09-16T10:33:23.226970+0000","last_clean":"2024-09-16T10:33:23.226970+0000","last_became_active":"2024-09-16T10:32:23.432634+0000","last_became_peered":"2024-09-16T10:32:23.432634+0000","last_unstale":"2024-09-16T10:33:23.226970+0000","last_undegraded":"2024-09-16T10:33:23.226970+0000","last_fullsized":"2024-09-16T10:33:23.226970+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:53.999450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:53.999450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.072558+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":1018,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226132+0000","last_change":"2024-09-16T10:32:20.480209+0000","last_active":"2024-09-16T10:33:23.226132+0000","last_peered":"2024-09-16T10:33:23.226132+0000","last_clean":"2024-09-16T10:33:23.226132+0000","last_became_active":"2024-09-16T10:32:20.395805+0000","last_became_peered":"2024-09-16T10:32:20.395805+0000","last_unstale":"2024-09-16T10:33:23.226132+0000","last_undegraded":"2024-09-16T10:33:23.226132+0000","last_fullsized":"2024-09-16T10:33:23.226132+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:32:08.692093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:08.692093+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:37.858412+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":890,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205663+0000","last_change":"2024-09-16T10:32:20.409986+0000","last_active":"2024-09-16T10:33:23.205663+0000","last_peered":"2024-09-16T10:33:23.205663+0000","last_clean":"2024-09-16T10:33:23.205663+0000","last_became_active":"2024-09-16T10:32:20.406762+0000","last_became_peered":"2024-09-16T10:32:20.406762+0000","last_unstale":"2024-09-16T10:33:23.205663+0000","last_undegraded":"2024-09-16T10:33:23.205663+0000","last_fullsized":"2024-09-16T10:33:23.205663+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:30.867930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:30.867930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1068,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725474+0000","last_change":"2024-09-16T10:33:07.435064+0000","last_active":"2024-09-16T10:33:22.725474+0000","last_peered":"2024-09-16T10:33:22.725474+0000","last_clean":"2024-09-16T10:33:22.725474+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:33:22.725474+0000","last_undegraded":"2024-09-16T10:33:22.725474+0000","last_fullsized":"2024-09-16T10:33:22.725474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:07.434963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:07.434963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:11.222708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":997,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227237+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:33:23.227237+0000","last_peered":"2024-09-16T10:33:23.227237+0000","last_clean":"2024-09-16T10:33:23.227237+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:33:23.227237+0000","last_undegraded":"2024-09-16T10:33:23.227237+0000","last_fullsized":"2024-09-16T10:33:23.227237+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":929,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724328+0000","last_change":"2024-09-16T10:33:17.369390+0000","last_active":"2024-09-16T10:33:22.724328+0000","last_peered":"2024-09-16T10:33:22.724328+0000","last_clean":"2024-09-16T10:33:22.724328+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:33:22.724328+0000","last_undegraded":"2024-09-16T10:33:22.724328+0000","last_fullsized":"2024-09-16T10:33:22.724328+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:17.369317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:17.369317+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:29.192042+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":1088,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205985+0000","last_change":"2024-09-16T10:32:20.440999+0000","last_active":"2024-09-16T10:33:23.205985+0000","last_peered":"2024-09-16T10:33:23.205985+0000","last_clean":"2024-09-16T10:33:23.205985+0000","last_became_active":"2024-09-16T10:32:20.402914+0000","last_became_peered":"2024-09-16T10:32:20.402914+0000","last_unstale":"2024-09-16T10:33:23.205985+0000","last_undegraded":"2024-09-16T10:33:23.205985+0000","last_fullsized":"2024-09-16T10:33:23.205985+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:31:18.771268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:31:18.771268+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1077,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226786+0000","last_change":"2024-09-16T10:32:54.752441+0000","last_active":"2024-09-16T10:33:23.226786+0000","last_peered":"2024-09-16T10:33:23.226786+0000","last_clean":"2024-09-16T10:33:23.226786+0000","last_became_active":"2024-09-16T10:32:20.406154+0000","last_became_peered":"2024-09-16T10:32:20.406154+0000","last_unstale":"2024-09-16T10:33:23.226786+0000","last_undegraded":"2024-09-16T10:33:23.226786+0000","last_fullsized":"2024-09-16T10:33:23.226786+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:54.752376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:54.752376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.545772+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":71,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225464+0000","last_change":"2024-09-16T10:33:06.130249+0000","last_active":"2024-09-16T10:33:23.225464+0000","last_peered":"2024-09-16T10:33:23.225464+0000","last_clean":"2024-09-16T10:33:23.225464+0000","last_became_active":"2024-09-16T10:32:20.395860+0000","last_became_peered":"2024-09-16T10:32:20.395860+0000","last_unstale":"2024-09-16T10:33:23.225464+0000","last_undegraded":"2024-09-16T10:33:23.225464+0000","last_fullsized":"2024-09-16T10:33:23.225464+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:06.130189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.130189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:17.390775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":604,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206386+0000","last_change":"2024-09-16T10:33:16.677107+0000","last_active":"2024-09-16T10:33:23.206386+0000","last_peered":"2024-09-16T10:33:23.206386+0000","last_clean":"2024-09-16T10:33:23.206386+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:33:23.206386+0000","last_undegraded":"2024-09-16T10:33:23.206386+0000","last_fullsized":"2024-09-16T10:33:23.206386+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:16.677041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:16.677041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:39.426050+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":742,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226594+0000","last_change":"2024-09-16T10:31:14.142529+0000","last_active":"2024-09-16T10:33:23.226594+0000","last_peered":"2024-09-16T10:33:23.226594+0000","last_clean":"2024-09-16T10:33:23.226594+0000","last_became_active":"2024-09-16T10:31:14.140156+0000","last_became_peered":"2024-09-16T10:31:14.140156+0000","last_unstale":"2024-09-16T10:33:23.226594+0000","last_undegraded":"2024-09-16T10:33:23.226594+0000","last_fullsized":"2024-09-16T10:33:23.226594+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:07.951845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.951845+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":971,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226881+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:33:23.226881+0000","last_peered":"2024-09-16T10:33:23.226881+0000","last_clean":"2024-09-16T10:33:23.226881+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:33:23.226881+0000","last_undegraded":"2024-09-16T10:33:23.226881+0000","last_fullsized":"2024-09-16T10:33:23.226881+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1052,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226351+0000","last_change":"2024-09-16T10:32:09.398843+0000","last_active":"2024-09-16T10:33:23.226351+0000","last_peered":"2024-09-16T10:33:23.226351+0000","last_clean":"2024-09-16T10:33:23.226351+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:33:23.226351+0000","last_undegraded":"2024-09-16T10:33:23.226351+0000","last_fullsized":"2024-09-16T10:33:23.226351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:09.398706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.398706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"508'13","reported_seq":761,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206141+0000","last_change":"2024-09-16T10:31:27.828520+0000","last_active":"2024-09-16T10:33:23.206141+0000","last_peered":"2024-09-16T10:33:23.206141+0000","last_clean":"2024-09-16T10:33:23.206141+0000","last_became_active":"2024-09-16T10:31:14.138957+0000","last_became_peered":"2024-09-16T10:31:14.138957+0000","last_unstale":"2024-09-16T10:33:23.206141+0000","last_undegraded":"2024-09-16T10:33:23.206141+0000","last_fullsized":"2024-09-16T10:33:23.206141+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'13","last_scrub_stamp":"2024-09-16T10:31:27.828439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:27.828439+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":549,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206595+0000","last_change":"2024-09-16T10:33:12.679629+0000","last_active":"2024-09-16T10:33:23.206595+0000","last_peered":"2024-09-16T10:33:23.206595+0000","last_clean":"2024-09-16T10:33:23.206595+0000","last_became_active":"2024-09-16T10:32:20.392061+0000","last_became_peered":"2024-09-16T10:32:20.392061+0000","last_unstale":"2024-09-16T10:33:23.206595+0000","last_undegraded":"2024-09-16T10:33:23.206595+0000","last_fullsized":"2024-09-16T10:33:23.206595+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:12.679545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:12.679545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:31.979630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":334,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225397+0000","last_change":"2024-09-16T10:32:17.363539+0000","last_active":"2024-09-16T10:33:23.225397+0000","last_peered":"2024-09-16T10:33:23.225397+0000","last_clean":"2024-09-16T10:33:23.225397+0000","last_became_active":"2024-09-16T10:31:17.119269+0000","last_became_peered":"2024-09-16T10:31:17.119269+0000","last_unstale":"2024-09-16T10:33:23.225397+0000","last_undegraded":"2024-09-16T10:33:23.225397+0000","last_fullsized":"2024-09-16T10:33:23.225397+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:17.363464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:17.363464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.071616+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":1099,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226660+0000","last_change":"2024-09-16T10:32:20.468825+0000","last_active":"2024-09-16T10:33:23.226660+0000","last_peered":"2024-09-16T10:33:23.226660+0000","last_clean":"2024-09-16T10:33:23.226660+0000","last_became_active":"2024-09-16T10:32:20.417250+0000","last_became_peered":"2024-09-16T10:32:20.417250+0000","last_unstale":"2024-09-16T10:33:23.226660+0000","last_undegraded":"2024-09-16T10:33:23.226660+0000","last_fullsized":"2024-09-16T10:33:23.226660+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:30:19.209335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:19.209335+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1076,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725071+0000","last_change":"2024-09-16T10:32:06.250968+0000","last_active":"2024-09-16T10:33:22.725071+0000","last_peered":"2024-09-16T10:33:22.725071+0000","last_clean":"2024-09-16T10:33:22.725071+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:33:22.725071+0000","last_undegraded":"2024-09-16T10:33:22.725071+0000","last_fullsized":"2024-09-16T10:33:22.725071+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:06.250899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.250899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:26.201157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":596,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205791+0000","last_change":"2024-09-16T10:32:12.023979+0000","last_active":"2024-09-16T10:33:23.205791+0000","last_peered":"2024-09-16T10:33:23.205791+0000","last_clean":"2024-09-16T10:33:23.205791+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:33:23.205791+0000","last_undegraded":"2024-09-16T10:33:23.205791+0000","last_fullsized":"2024-09-16T10:33:23.205791+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:12.023740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.023740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:31.931461+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":1109,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224876+0000","last_change":"2024-09-16T10:31:14.161646+0000","last_active":"2024-09-16T10:33:23.224876+0000","last_peered":"2024-09-16T10:33:23.224876+0000","last_clean":"2024-09-16T10:33:23.224876+0000","last_became_active":"2024-09-16T10:31:14.152023+0000","last_became_peered":"2024-09-16T10:31:14.152023+0000","last_unstale":"2024-09-16T10:33:23.224876+0000","last_undegraded":"2024-09-16T10:33:23.224876+0000","last_fullsized":"2024-09-16T10:33:23.224876+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:05.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.174161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":1026,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226649+0000","last_change":"2024-09-16T10:30:42.550612+0000","last_active":"2024-09-16T10:33:23.226649+0000","last_peered":"2024-09-16T10:33:23.226649+0000","last_clean":"2024-09-16T10:33:23.226649+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:33:23.226649+0000","last_undegraded":"2024-09-16T10:33:23.226649+0000","last_fullsized":"2024-09-16T10:33:23.226649+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:42.550567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.550567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":1186,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226545+0000","last_change":"2024-09-16T10:32:20.470230+0000","last_active":"2024-09-16T10:33:23.226545+0000","last_peered":"2024-09-16T10:33:23.226545+0000","last_clean":"2024-09-16T10:33:23.226545+0000","last_became_active":"2024-09-16T10:32:20.412893+0000","last_became_peered":"2024-09-16T10:32:20.412893+0000","last_unstale":"2024-09-16T10:33:23.226545+0000","last_undegraded":"2024-09-16T10:33:23.226545+0000","last_fullsized":"2024-09-16T10:33:23.226545+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:41.740678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":464,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725122+0000","last_change":"2024-09-16T10:31:14.148058+0000","last_active":"2024-09-16T10:33:22.725122+0000","last_peered":"2024-09-16T10:33:22.725122+0000","last_clean":"2024-09-16T10:33:22.725122+0000","last_became_active":"2024-09-16T10:31:14.147606+0000","last_became_peered":"2024-09-16T10:31:14.147606+0000","last_unstale":"2024-09-16T10:33:22.725122+0000","last_undegraded":"2024-09-16T10:33:22.725122+0000","last_fullsized":"2024-09-16T10:33:22.725122+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.745214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.745214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":596,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206137+0000","last_change":"2024-09-16T10:32:18.059677+0000","last_active":"2024-09-16T10:33:23.206137+0000","last_peered":"2024-09-16T10:33:23.206137+0000","last_clean":"2024-09-16T10:33:23.206137+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:33:23.206137+0000","last_undegraded":"2024-09-16T10:33:23.206137+0000","last_fullsized":"2024-09-16T10:33:23.206137+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:18.059602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:18.059602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:35.489290+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":942,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205334+0000","last_change":"2024-09-16T10:31:14.148903+0000","last_active":"2024-09-16T10:33:23.205334+0000","last_peered":"2024-09-16T10:33:23.205334+0000","last_clean":"2024-09-16T10:33:23.205334+0000","last_became_active":"2024-09-16T10:31:14.148572+0000","last_became_peered":"2024-09-16T10:31:14.148572+0000","last_unstale":"2024-09-16T10:33:23.205334+0000","last_undegraded":"2024-09-16T10:33:23.205334+0000","last_fullsized":"2024-09-16T10:33:23.205334+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.626720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.626720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":977,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225747+0000","last_change":"2024-09-16T10:32:06.364289+0000","last_active":"2024-09-16T10:33:23.225747+0000","last_peered":"2024-09-16T10:33:23.225747+0000","last_clean":"2024-09-16T10:33:23.225747+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:33:23.225747+0000","last_undegraded":"2024-09-16T10:33:23.225747+0000","last_fullsized":"2024-09-16T10:33:23.225747+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:06.364207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.364207+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":676,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226163+0000","last_change":"2024-09-16T10:32:20.466527+0000","last_active":"2024-09-16T10:33:23.226163+0000","last_peered":"2024-09-16T10:33:23.226163+0000","last_clean":"2024-09-16T10:33:23.226163+0000","last_became_active":"2024-09-16T10:32:20.431532+0000","last_became_peered":"2024-09-16T10:32:20.431532+0000","last_unstale":"2024-09-16T10:33:23.226163+0000","last_undegraded":"2024-09-16T10:33:23.226163+0000","last_fullsized":"2024-09-16T10:33:23.226163+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:30:20.961894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.961894+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":66,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227268+0000","last_change":"2024-09-16T10:32:20.390993+0000","last_active":"2024-09-16T10:33:23.227268+0000","last_peered":"2024-09-16T10:33:23.227268+0000","last_clean":"2024-09-16T10:33:23.227268+0000","last_became_active":"2024-09-16T10:32:20.390515+0000","last_became_peered":"2024-09-16T10:32:20.390515+0000","last_unstale":"2024-09-16T10:33:23.227268+0000","last_undegraded":"2024-09-16T10:33:23.227268+0000","last_fullsized":"2024-09-16T10:33:23.227268+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.911596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":611,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205337+0000","last_change":"2024-09-16T10:33:13.716266+0000","last_active":"2024-09-16T10:33:23.205337+0000","last_peered":"2024-09-16T10:33:23.205337+0000","last_clean":"2024-09-16T10:33:23.205337+0000","last_became_active":"2024-09-16T10:32:20.418945+0000","last_became_peered":"2024-09-16T10:32:20.418945+0000","last_unstale":"2024-09-16T10:33:23.205337+0000","last_undegraded":"2024-09-16T10:33:23.205337+0000","last_fullsized":"2024-09-16T10:33:23.205337+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:13.715922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:13.715922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:16.525289+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1081,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225433+0000","last_change":"2024-09-16T10:32:20.433825+0000","last_active":"2024-09-16T10:33:23.225433+0000","last_peered":"2024-09-16T10:33:23.225433+0000","last_clean":"2024-09-16T10:33:23.225433+0000","last_became_active":"2024-09-16T10:32:20.423737+0000","last_became_peered":"2024-09-16T10:32:20.423737+0000","last_unstale":"2024-09-16T10:33:23.225433+0000","last_undegraded":"2024-09-16T10:33:23.225433+0000","last_fullsized":"2024-09-16T10:33:23.225433+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.230039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.230039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":754,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205960+0000","last_change":"2024-09-16T10:32:23.425365+0000","last_active":"2024-09-16T10:33:23.205960+0000","last_peered":"2024-09-16T10:33:23.205960+0000","last_clean":"2024-09-16T10:33:23.205960+0000","last_became_active":"2024-09-16T10:32:23.424848+0000","last_became_peered":"2024-09-16T10:32:23.424848+0000","last_unstale":"2024-09-16T10:33:23.205960+0000","last_undegraded":"2024-09-16T10:33:23.205960+0000","last_fullsized":"2024-09-16T10:33:23.205960+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:46.291175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":1126,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724475+0000","last_change":"2024-09-16T10:31:14.162215+0000","last_active":"2024-09-16T10:33:22.724475+0000","last_peered":"2024-09-16T10:33:22.724475+0000","last_clean":"2024-09-16T10:33:22.724475+0000","last_became_active":"2024-09-16T10:31:14.136662+0000","last_became_peered":"2024-09-16T10:31:14.136662+0000","last_unstale":"2024-09-16T10:33:22.724475+0000","last_undegraded":"2024-09-16T10:33:22.724475+0000","last_fullsized":"2024-09-16T10:33:22.724475+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.825936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.825936+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":425,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227536+0000","last_change":"2024-09-16T10:32:20.422676+0000","last_active":"2024-09-16T10:33:23.227536+0000","last_peered":"2024-09-16T10:33:23.227536+0000","last_clean":"2024-09-16T10:33:23.227536+0000","last_became_active":"2024-09-16T10:32:20.391995+0000","last_became_peered":"2024-09-16T10:32:20.391995+0000","last_unstale":"2024-09-16T10:33:23.227536+0000","last_undegraded":"2024-09-16T10:33:23.227536+0000","last_fullsized":"2024-09-16T10:33:23.227536+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:32:09.730242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.730242+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"536'13","reported_seq":610,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225625+0000","last_change":"2024-09-16T10:32:20.449596+0000","last_active":"2024-09-16T10:33:23.225625+0000","last_peered":"2024-09-16T10:33:23.225625+0000","last_clean":"2024-09-16T10:33:23.225625+0000","last_became_active":"2024-09-16T10:32:20.398138+0000","last_became_peered":"2024-09-16T10:32:20.398138+0000","last_unstale":"2024-09-16T10:33:23.225625+0000","last_undegraded":"2024-09-16T10:33:23.225625+0000","last_fullsized":"2024-09-16T10:33:23.225625+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'11","last_scrub_stamp":"2024-09-16T10:30:53.694883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.694883+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1065,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724525+0000","last_change":"2024-09-16T10:30:07.792293+0000","last_active":"2024-09-16T10:33:22.724525+0000","last_peered":"2024-09-16T10:33:22.724525+0000","last_clean":"2024-09-16T10:33:22.724525+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:33:22.724525+0000","last_undegraded":"2024-09-16T10:33:22.724525+0000","last_fullsized":"2024-09-16T10:33:22.724525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.792060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.792060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":657,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724606+0000","last_change":"2024-09-16T10:32:23.427047+0000","last_active":"2024-09-16T10:33:22.724606+0000","last_peered":"2024-09-16T10:33:22.724606+0000","last_clean":"2024-09-16T10:33:22.724606+0000","last_became_active":"2024-09-16T10:32:23.426646+0000","last_became_peered":"2024-09-16T10:32:23.426646+0000","last_unstale":"2024-09-16T10:33:22.724606+0000","last_undegraded":"2024-09-16T10:33:22.724606+0000","last_fullsized":"2024-09-16T10:33:22.724606+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:09.269989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.269989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:23.872255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1053,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224747+0000","last_change":"2024-09-16T10:33:07.769547+0000","last_active":"2024-09-16T10:33:23.224747+0000","last_peered":"2024-09-16T10:33:23.224747+0000","last_clean":"2024-09-16T10:33:23.224747+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:33:23.224747+0000","last_undegraded":"2024-09-16T10:33:23.224747+0000","last_fullsized":"2024-09-16T10:33:23.224747+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:07.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:07.769371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:15.987044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":1115,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226039+0000","last_change":"2024-09-16T10:32:20.455088+0000","last_active":"2024-09-16T10:33:23.226039+0000","last_peered":"2024-09-16T10:33:23.226039+0000","last_clean":"2024-09-16T10:33:23.226039+0000","last_became_active":"2024-09-16T10:32:20.425287+0000","last_became_peered":"2024-09-16T10:32:20.425287+0000","last_unstale":"2024-09-16T10:33:23.226039+0000","last_undegraded":"2024-09-16T10:33:23.226039+0000","last_fullsized":"2024-09-16T10:33:23.226039+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:30:50.226333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.226333+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1051,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725343+0000","last_change":"2024-09-16T10:32:20.474305+0000","last_active":"2024-09-16T10:33:22.725343+0000","last_peered":"2024-09-16T10:33:22.725343+0000","last_clean":"2024-09-16T10:33:22.725343+0000","last_became_active":"2024-09-16T10:32:20.473984+0000","last_became_peered":"2024-09-16T10:32:20.473984+0000","last_unstale":"2024-09-16T10:33:22.725343+0000","last_undegraded":"2024-09-16T10:33:22.725343+0000","last_fullsized":"2024-09-16T10:33:22.725343+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":742,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226921+0000","last_change":"2024-09-16T10:31:14.137702+0000","last_active":"2024-09-16T10:33:23.226921+0000","last_peered":"2024-09-16T10:33:23.226921+0000","last_clean":"2024-09-16T10:33:23.226921+0000","last_became_active":"2024-09-16T10:31:14.136703+0000","last_became_peered":"2024-09-16T10:31:14.136703+0000","last_unstale":"2024-09-16T10:33:23.226921+0000","last_undegraded":"2024-09-16T10:33:23.226921+0000","last_fullsized":"2024-09-16T10:33:23.226921+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:39.865342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.865342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":523,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225540+0000","last_change":"2024-09-16T10:32:55.165175+0000","last_active":"2024-09-16T10:33:23.225540+0000","last_peered":"2024-09-16T10:33:23.225540+0000","last_clean":"2024-09-16T10:33:23.225540+0000","last_became_active":"2024-09-16T10:32:23.433435+0000","last_became_peered":"2024-09-16T10:32:23.433435+0000","last_unstale":"2024-09-16T10:33:23.225540+0000","last_undegraded":"2024-09-16T10:33:23.225540+0000","last_fullsized":"2024-09-16T10:33:23.225540+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:55.165009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:55.165009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:12.152556+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":1110,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227438+0000","last_change":"2024-09-16T10:31:14.165639+0000","last_active":"2024-09-16T10:33:23.227438+0000","last_peered":"2024-09-16T10:33:23.227438+0000","last_clean":"2024-09-16T10:33:23.227438+0000","last_became_active":"2024-09-16T10:31:14.151744+0000","last_became_peered":"2024-09-16T10:31:14.151744+0000","last_unstale":"2024-09-16T10:33:23.227438+0000","last_undegraded":"2024-09-16T10:33:23.227438+0000","last_fullsized":"2024-09-16T10:33:23.227438+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:07.157828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.157828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":405,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226060+0000","last_change":"2024-09-16T10:31:14.137225+0000","last_active":"2024-09-16T10:33:23.226060+0000","last_peered":"2024-09-16T10:33:23.226060+0000","last_clean":"2024-09-16T10:33:23.226060+0000","last_became_active":"2024-09-16T10:31:14.132504+0000","last_became_peered":"2024-09-16T10:31:14.132504+0000","last_unstale":"2024-09-16T10:33:23.226060+0000","last_undegraded":"2024-09-16T10:33:23.226060+0000","last_fullsized":"2024-09-16T10:33:23.226060+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:44.598219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:44.598219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1084,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225776+0000","last_change":"2024-09-16T10:32:20.425604+0000","last_active":"2024-09-16T10:33:23.225776+0000","last_peered":"2024-09-16T10:33:23.225776+0000","last_clean":"2024-09-16T10:33:23.225776+0000","last_became_active":"2024-09-16T10:32:20.424781+0000","last_became_peered":"2024-09-16T10:32:20.424781+0000","last_unstale":"2024-09-16T10:33:23.225776+0000","last_undegraded":"2024-09-16T10:33:23.225776+0000","last_fullsized":"2024-09-16T10:33:23.225776+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:20.058297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.058297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":613,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206371+0000","last_change":"2024-09-16T10:31:38.879647+0000","last_active":"2024-09-16T10:33:23.206371+0000","last_peered":"2024-09-16T10:33:23.206371+0000","last_clean":"2024-09-16T10:33:23.206371+0000","last_became_active":"2024-09-16T10:31:14.140004+0000","last_became_peered":"2024-09-16T10:31:14.140004+0000","last_unstale":"2024-09-16T10:33:23.206371+0000","last_undegraded":"2024-09-16T10:33:23.206371+0000","last_fullsized":"2024-09-16T10:33:23.206371+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:38.879579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:38.879579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":663,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724288+0000","last_change":"2024-09-16T10:32:46.160022+0000","last_active":"2024-09-16T10:33:22.724288+0000","last_peered":"2024-09-16T10:33:22.724288+0000","last_clean":"2024-09-16T10:33:22.724288+0000","last_became_active":"2024-09-16T10:32:46.159664+0000","last_became_peered":"2024-09-16T10:32:46.159664+0000","last_unstale":"2024-09-16T10:33:22.724288+0000","last_undegraded":"2024-09-16T10:33:22.724288+0000","last_fullsized":"2024-09-16T10:33:22.724288+0000","mapping_epoch":546,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:51.390508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":1101,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725272+0000","last_change":"2024-09-16T10:33:06.436155+0000","last_active":"2024-09-16T10:33:22.725272+0000","last_peered":"2024-09-16T10:33:22.725272+0000","last_clean":"2024-09-16T10:33:22.725272+0000","last_became_active":"2024-09-16T10:31:14.149019+0000","last_became_peered":"2024-09-16T10:31:14.149019+0000","last_unstale":"2024-09-16T10:33:22.725272+0000","last_undegraded":"2024-09-16T10:33:22.725272+0000","last_fullsized":"2024-09-16T10:33:22.725272+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:33:06.436076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.436076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:19.257155+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1094,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225751+0000","last_change":"2024-09-16T10:33:18.297728+0000","last_active":"2024-09-16T10:33:23.225751+0000","last_peered":"2024-09-16T10:33:23.225751+0000","last_clean":"2024-09-16T10:33:23.225751+0000","last_became_active":"2024-09-16T10:32:20.416989+0000","last_became_peered":"2024-09-16T10:32:20.416989+0000","last_unstale":"2024-09-16T10:33:23.225751+0000","last_undegraded":"2024-09-16T10:33:23.225751+0000","last_fullsized":"2024-09-16T10:33:23.225751+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:18.297660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:18.297660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:35.219614+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"536'14","reported_seq":1371,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227140+0000","last_change":"2024-09-16T10:32:20.493322+0000","last_active":"2024-09-16T10:33:23.227140+0000","last_peered":"2024-09-16T10:33:23.227140+0000","last_clean":"2024-09-16T10:33:23.227140+0000","last_became_active":"2024-09-16T10:32:20.489447+0000","last_became_peered":"2024-09-16T10:32:20.489447+0000","last_unstale":"2024-09-16T10:33:23.227140+0000","last_undegraded":"2024-09-16T10:33:23.227140+0000","last_fullsized":"2024-09-16T10:33:23.227140+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:30:42.170419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.170419+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":860,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206212+0000","last_change":"2024-09-16T10:32:16.061021+0000","last_active":"2024-09-16T10:33:23.206212+0000","last_peered":"2024-09-16T10:33:23.206212+0000","last_clean":"2024-09-16T10:33:23.206212+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:33:23.206212+0000","last_undegraded":"2024-09-16T10:33:23.206212+0000","last_fullsized":"2024-09-16T10:33:23.206212+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:16.060965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:16.060965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":631,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224757+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:33:23.224757+0000","last_peered":"2024-09-16T10:33:23.224757+0000","last_clean":"2024-09-16T10:33:23.224757+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:33:23.224757+0000","last_undegraded":"2024-09-16T10:33:23.224757+0000","last_fullsized":"2024-09-16T10:33:23.224757+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:44.949384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":1151,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226762+0000","last_change":"2024-09-16T10:33:05.157157+0000","last_active":"2024-09-16T10:33:23.226762+0000","last_peered":"2024-09-16T10:33:23.226762+0000","last_clean":"2024-09-16T10:33:23.226762+0000","last_became_active":"2024-09-16T10:32:20.488822+0000","last_became_peered":"2024-09-16T10:32:20.488822+0000","last_unstale":"2024-09-16T10:33:23.226762+0000","last_undegraded":"2024-09-16T10:33:23.226762+0000","last_fullsized":"2024-09-16T10:33:23.226762+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:05.157106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.157106+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:06.420060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":830,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.206054+0000","last_change":"2024-09-16T10:31:34.935251+0000","last_active":"2024-09-16T10:33:23.206054+0000","last_peered":"2024-09-16T10:33:23.206054+0000","last_clean":"2024-09-16T10:33:23.206054+0000","last_became_active":"2024-09-16T10:31:14.132063+0000","last_became_peered":"2024-09-16T10:31:14.132063+0000","last_unstale":"2024-09-16T10:33:23.206054+0000","last_undegraded":"2024-09-16T10:33:23.206054+0000","last_fullsized":"2024-09-16T10:33:23.206054+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:34.935185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:34.935185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"536'9","reported_seq":1317,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724993+0000","last_change":"2024-09-16T10:32:20.449280+0000","last_active":"2024-09-16T10:33:22.724993+0000","last_peered":"2024-09-16T10:33:22.724993+0000","last_clean":"2024-09-16T10:33:22.724993+0000","last_became_active":"2024-09-16T10:32:20.436067+0000","last_became_peered":"2024-09-16T10:32:20.436067+0000","last_unstale":"2024-09-16T10:33:22.724993+0000","last_undegraded":"2024-09-16T10:33:22.724993+0000","last_fullsized":"2024-09-16T10:33:22.724993+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:56.800991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:56.800991+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1067,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227076+0000","last_change":"2024-09-16T10:33:05.136554+0000","last_active":"2024-09-16T10:33:23.227076+0000","last_peered":"2024-09-16T10:33:23.227076+0000","last_clean":"2024-09-16T10:33:23.227076+0000","last_became_active":"2024-09-16T10:31:14.150453+0000","last_became_peered":"2024-09-16T10:31:14.150453+0000","last_unstale":"2024-09-16T10:33:23.227076+0000","last_undegraded":"2024-09-16T10:33:23.227076+0000","last_fullsized":"2024-09-16T10:33:23.227076+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:05.136512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.136512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:09.897407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":1017,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227101+0000","last_change":"2024-09-16T10:32:20.399950+0000","last_active":"2024-09-16T10:33:23.227101+0000","last_peered":"2024-09-16T10:33:23.227101+0000","last_clean":"2024-09-16T10:33:23.227101+0000","last_became_active":"2024-09-16T10:32:20.390257+0000","last_became_peered":"2024-09-16T10:32:20.390257+0000","last_unstale":"2024-09-16T10:33:23.227101+0000","last_undegraded":"2024-09-16T10:33:23.227101+0000","last_fullsized":"2024-09-16T10:33:23.227101+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:32:09.683973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.683973+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":478,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226702+0000","last_change":"2024-09-16T10:32:20.406554+0000","last_active":"2024-09-16T10:33:23.226702+0000","last_peered":"2024-09-16T10:33:23.226702+0000","last_clean":"2024-09-16T10:33:23.226702+0000","last_became_active":"2024-09-16T10:32:20.405826+0000","last_became_peered":"2024-09-16T10:32:20.405826+0000","last_unstale":"2024-09-16T10:33:23.226702+0000","last_undegraded":"2024-09-16T10:33:23.226702+0000","last_fullsized":"2024-09-16T10:33:23.226702+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.216239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.216239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":61,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.724533+0000","last_change":"2024-09-16T10:32:20.434897+0000","last_active":"2024-09-16T10:33:22.724533+0000","last_peered":"2024-09-16T10:33:22.724533+0000","last_clean":"2024-09-16T10:33:22.724533+0000","last_became_active":"2024-09-16T10:32:20.434526+0000","last_became_peered":"2024-09-16T10:32:20.434526+0000","last_unstale":"2024-09-16T10:33:22.724533+0000","last_undegraded":"2024-09-16T10:33:22.724533+0000","last_fullsized":"2024-09-16T10:33:22.724533+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:11.168381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.168381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:23.152496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1048,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226705+0000","last_change":"2024-09-16T10:32:10.448088+0000","last_active":"2024-09-16T10:33:23.226705+0000","last_peered":"2024-09-16T10:33:23.226705+0000","last_clean":"2024-09-16T10:33:23.226705+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:33:23.226705+0000","last_undegraded":"2024-09-16T10:33:23.226705+0000","last_fullsized":"2024-09-16T10:33:23.226705+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:10.447999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:10.447999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":702,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226699+0000","last_change":"2024-09-16T10:32:20.464025+0000","last_active":"2024-09-16T10:33:23.226699+0000","last_peered":"2024-09-16T10:33:23.226699+0000","last_clean":"2024-09-16T10:33:23.226699+0000","last_became_active":"2024-09-16T10:32:20.413318+0000","last_became_peered":"2024-09-16T10:32:20.413318+0000","last_unstale":"2024-09-16T10:33:23.226699+0000","last_undegraded":"2024-09-16T10:33:23.226699+0000","last_fullsized":"2024-09-16T10:33:23.226699+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:43.705811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":465,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226949+0000","last_change":"2024-09-16T10:31:14.144963+0000","last_active":"2024-09-16T10:33:23.226949+0000","last_peered":"2024-09-16T10:33:23.226949+0000","last_clean":"2024-09-16T10:33:23.226949+0000","last_became_active":"2024-09-16T10:31:14.144634+0000","last_became_peered":"2024-09-16T10:31:14.144634+0000","last_unstale":"2024-09-16T10:33:23.226949+0000","last_undegraded":"2024-09-16T10:33:23.226949+0000","last_fullsized":"2024-09-16T10:33:23.226949+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1016,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225515+0000","last_change":"2024-09-16T10:32:20.415681+0000","last_active":"2024-09-16T10:33:23.225515+0000","last_peered":"2024-09-16T10:33:23.225515+0000","last_clean":"2024-09-16T10:33:23.225515+0000","last_became_active":"2024-09-16T10:32:20.415323+0000","last_became_peered":"2024-09-16T10:32:20.415323+0000","last_unstale":"2024-09-16T10:33:23.225515+0000","last_undegraded":"2024-09-16T10:33:23.225515+0000","last_fullsized":"2024-09-16T10:33:23.225515+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.144732+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1040,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.224618+0000","last_change":"2024-09-16T10:32:20.402165+0000","last_active":"2024-09-16T10:33:23.224618+0000","last_peered":"2024-09-16T10:33:23.224618+0000","last_clean":"2024-09-16T10:33:23.224618+0000","last_became_active":"2024-09-16T10:32:20.396229+0000","last_became_peered":"2024-09-16T10:32:20.396229+0000","last_unstale":"2024-09-16T10:33:23.224618+0000","last_undegraded":"2024-09-16T10:33:23.224618+0000","last_fullsized":"2024-09-16T10:33:23.224618+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.389262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.389262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":1013,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205877+0000","last_change":"2024-09-16T10:32:23.440126+0000","last_active":"2024-09-16T10:33:23.205877+0000","last_peered":"2024-09-16T10:33:23.205877+0000","last_clean":"2024-09-16T10:33:23.205877+0000","last_became_active":"2024-09-16T10:32:23.439740+0000","last_became_peered":"2024-09-16T10:32:23.439740+0000","last_unstale":"2024-09-16T10:33:23.205877+0000","last_undegraded":"2024-09-16T10:33:23.205877+0000","last_fullsized":"2024-09-16T10:33:23.205877+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:31:31.897681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:31:31.897681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":1093,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226490+0000","last_change":"2024-09-16T10:31:14.175984+0000","last_active":"2024-09-16T10:33:23.226490+0000","last_peered":"2024-09-16T10:33:23.226490+0000","last_clean":"2024-09-16T10:33:23.226490+0000","last_became_active":"2024-09-16T10:31:14.154997+0000","last_became_peered":"2024-09-16T10:31:14.154997+0000","last_unstale":"2024-09-16T10:33:23.226490+0000","last_undegraded":"2024-09-16T10:33:23.226490+0000","last_fullsized":"2024-09-16T10:33:23.226490+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1093,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725406+0000","last_change":"2024-09-16T10:32:20.474361+0000","last_active":"2024-09-16T10:33:22.725406+0000","last_peered":"2024-09-16T10:33:22.725406+0000","last_clean":"2024-09-16T10:33:22.725406+0000","last_became_active":"2024-09-16T10:32:20.474067+0000","last_became_peered":"2024-09-16T10:32:20.474067+0000","last_unstale":"2024-09-16T10:33:22.725406+0000","last_undegraded":"2024-09-16T10:33:22.725406+0000","last_fullsized":"2024-09-16T10:33:22.725406+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:07.250964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:07.250964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":754,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225189+0000","last_change":"2024-09-16T10:32:13.704695+0000","last_active":"2024-09-16T10:33:23.225189+0000","last_peered":"2024-09-16T10:33:23.225189+0000","last_clean":"2024-09-16T10:33:23.225189+0000","last_became_active":"2024-09-16T10:31:14.132459+0000","last_became_peered":"2024-09-16T10:31:14.132459+0000","last_unstale":"2024-09-16T10:33:23.225189+0000","last_undegraded":"2024-09-16T10:33:23.225189+0000","last_fullsized":"2024-09-16T10:33:23.225189+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:13.704652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:13.704652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:36.717155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1064,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225644+0000","last_change":"2024-09-16T10:30:45.211710+0000","last_active":"2024-09-16T10:33:23.225644+0000","last_peered":"2024-09-16T10:33:23.225644+0000","last_clean":"2024-09-16T10:33:23.225644+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:33:23.225644+0000","last_undegraded":"2024-09-16T10:33:23.225644+0000","last_fullsized":"2024-09-16T10:33:23.225644+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.211638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.211638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":1128,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225095+0000","last_change":"2024-09-16T10:32:20.445935+0000","last_active":"2024-09-16T10:33:23.225095+0000","last_peered":"2024-09-16T10:33:23.225095+0000","last_clean":"2024-09-16T10:33:23.225095+0000","last_became_active":"2024-09-16T10:32:20.396707+0000","last_became_peered":"2024-09-16T10:32:20.396707+0000","last_unstale":"2024-09-16T10:33:23.225095+0000","last_undegraded":"2024-09-16T10:33:23.225095+0000","last_fullsized":"2024-09-16T10:33:23.225095+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:32:12.365003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.365003+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:36.671931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":1059,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227617+0000","last_change":"2024-09-16T10:30:21.275187+0000","last_active":"2024-09-16T10:33:23.227617+0000","last_peered":"2024-09-16T10:33:23.227617+0000","last_clean":"2024-09-16T10:33:23.227617+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:33:23.227617+0000","last_undegraded":"2024-09-16T10:33:23.227617+0000","last_fullsized":"2024-09-16T10:33:23.227617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:30:21.275103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:21.275103+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1079,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225820+0000","last_change":"2024-09-16T10:31:14.139140+0000","last_active":"2024-09-16T10:33:23.225820+0000","last_peered":"2024-09-16T10:33:23.225820+0000","last_clean":"2024-09-16T10:33:23.225820+0000","last_became_active":"2024-09-16T10:31:14.138772+0000","last_became_peered":"2024-09-16T10:31:14.138772+0000","last_unstale":"2024-09-16T10:33:23.225820+0000","last_undegraded":"2024-09-16T10:33:23.225820+0000","last_fullsized":"2024-09-16T10:33:23.225820+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.649610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.649610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":697,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.205598+0000","last_change":"2024-09-16T10:31:40.861638+0000","last_active":"2024-09-16T10:33:23.205598+0000","last_peered":"2024-09-16T10:33:23.205598+0000","last_clean":"2024-09-16T10:33:23.205598+0000","last_became_active":"2024-09-16T10:31:14.152053+0000","last_became_peered":"2024-09-16T10:31:14.152053+0000","last_unstale":"2024-09-16T10:33:23.205598+0000","last_undegraded":"2024-09-16T10:33:23.205598+0000","last_fullsized":"2024-09-16T10:33:23.205598+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:40.861570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:40.861570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":62,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.725160+0000","last_change":"2024-09-16T10:32:20.401586+0000","last_active":"2024-09-16T10:33:22.725160+0000","last_peered":"2024-09-16T10:33:22.725160+0000","last_clean":"2024-09-16T10:33:22.725160+0000","last_became_active":"2024-09-16T10:32:20.401249+0000","last_became_peered":"2024-09-16T10:32:20.401249+0000","last_unstale":"2024-09-16T10:33:22.725160+0000","last_undegraded":"2024-09-16T10:33:22.725160+0000","last_fullsized":"2024-09-16T10:33:22.725160+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.832904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.832904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1075,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.225178+0000","last_change":"2024-09-16T10:32:20.428692+0000","last_active":"2024-09-16T10:33:23.225178+0000","last_peered":"2024-09-16T10:33:23.225178+0000","last_clean":"2024-09-16T10:33:23.225178+0000","last_became_active":"2024-09-16T10:32:20.428421+0000","last_became_peered":"2024-09-16T10:32:20.428421+0000","last_unstale":"2024-09-16T10:33:23.225178+0000","last_undegraded":"2024-09-16T10:33:23.225178+0000","last_fullsized":"2024-09-16T10:33:23.225178+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:15.364448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:15.364448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:24.890266+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":1132,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.226738+0000","last_change":"2024-09-16T10:33:00.718946+0000","last_active":"2024-09-16T10:33:23.226738+0000","last_peered":"2024-09-16T10:33:23.226738+0000","last_clean":"2024-09-16T10:33:23.226738+0000","last_became_active":"2024-09-16T10:32:20.395308+0000","last_became_peered":"2024-09-16T10:32:20.395308+0000","last_unstale":"2024-09-16T10:33:23.226738+0000","last_undegraded":"2024-09-16T10:33:23.226738+0000","last_fullsized":"2024-09-16T10:33:23.226738+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:33:00.718864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:00.718864+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:18.336822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1093,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.227178+0000","last_change":"2024-09-16T10:32:20.414016+0000","last_active":"2024-09-16T10:33:23.227178+0000","last_peered":"2024-09-16T10:33:23.227178+0000","last_clean":"2024-09-16T10:33:23.227178+0000","last_became_active":"2024-09-16T10:32:20.413184+0000","last_became_peered":"2024-09-16T10:32:20.413184+0000","last_unstale":"2024-09-16T10:33:23.227178+0000","last_undegraded":"2024-09-16T10:33:23.227178+0000","last_fullsized":"2024-09-16T10:33:23.227178+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:54.399132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.399132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1110,"num_read_kb":867,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":6068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":5551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":1894080577597,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528565,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834828,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92910324,"statfs":{"total":95995035648,"available":95140171776,"internally_reserved":0,"allocated":1236992,"data_stored":800800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":853604806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790203,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776432,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92968720,"statfs":{"total":95995035648,"available":95199969280,"internally_reserved":0,"allocated":1146880,"data_stored":729398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986432,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681808,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063344,"statfs":{"total":95995035648,"available":95296864256,"internally_reserved":0,"allocated":909312,"data_stored":479788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215365,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917040,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828112,"statfs":{"total":95995035648,"available":95055986688,"internally_reserved":0,"allocated":1531904,"data_stored":1072350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":937490888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411595,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622340,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93122812,"statfs":{"total":95995035648,"available":95357759488,"internally_reserved":0,"allocated":1159168,"data_stored":742036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575118,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738644,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934804,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92810348,"statfs":{"total":95995035648,"available":95037796352,"internally_reserved":0,"allocated":1241088,"data_stored":808290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:28.033 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:33:28.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:28.097 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:28.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:28.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0 2024-09-16T10:33:28.222 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:28.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.0' 2024-09-16T10:33:28.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5 2024-09-16T10:33:28.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:28.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.5' 2024-09-16T10:33:28.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3 2024-09-16T10:33:28.473 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:28.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.3' 2024-09-16T10:33:28.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2 2024-09-16T10:33:28.598 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: docker: command not found 2024-09-16T10:33:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:28.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi063 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.2' 2024-09-16T10:33:28.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:28 smithi063 ceph-mon[29993]: pgmap v1071: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:33:28.695 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:28 smithi063 ceph-mon[39470]: pgmap v1071: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:33:28.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6 2024-09-16T10:33:28.723 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T10:33:28.732 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T10:33:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:33:28.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-16T10:33:28.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd.6' 2024-09-16T10:33:28.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:28 smithi165 ceph-mon[32696]: pgmap v1071: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:33:28.836 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:28.846 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:28.846 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:28.849 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:29.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:33:29 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:33:30.175 INFO:teuthology.orchestra.run.smithi063.stdout:ERROR: (22) Invalid argument 2024-09-16T10:33:30.175 INFO:teuthology.orchestra.run.smithi063.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T10:33:30.599 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is set 2024-09-16T10:33:30.743 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:30.743 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:30 smithi063 ceph-mon[29993]: pgmap v1072: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 558 B/s rd, 0 op/s 2024-09-16T10:33:30.744 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:30 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2123791827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:30.744 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:30.744 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:30 smithi063 ceph-mon[39470]: pgmap v1072: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 558 B/s rd, 0 op/s 2024-09-16T10:33:30.744 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:30 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2123791827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:31.004 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T10:33:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:30 smithi165 ceph-mon[32696]: pgmap v1072: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 558 B/s rd, 0 op/s 2024-09-16T10:33:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:30 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2123791827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:31.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[29993]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:31.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2123791827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:31.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[29993]: osdmap e562: 8 total, 8 up, 6 in 2024-09-16T10:33:31.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[29993]: osdmap e563: 8 total, 8 up, 6 in 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2051549608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[39470]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2123791827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[39470]: osdmap e562: 8 total, 8 up, 6 in 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[39470]: osdmap e563: 8 total, 8 up, 6 in 2024-09-16T10:33:31.629 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2051549608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:31.834 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0: {} 2024-09-16T10:33:31.834 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1: {} 2024-09-16T10:33:31.834 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2: {} 2024-09-16T10:33:31.834 INFO:teuthology.orchestra.run.smithi063.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3: {} 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4: {} 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5: {} 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6: {} 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7: {} 2024-09-16T10:33:31.835 INFO:teuthology.orchestra.run.smithi063.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T10:33:31.836 INFO:teuthology.orchestra.run.smithi063.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T10:33:31.836 INFO:teuthology.orchestra.run.smithi063.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T10:33:31.836 INFO:teuthology.orchestra.run.smithi063.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T10:33:31.836 INFO:teuthology.orchestra.run.smithi063.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T10:33:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:31 smithi165 ceph-mon[32696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2123791827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:31 smithi165 ceph-mon[32696]: osdmap e562: 8 total, 8 up, 6 in 2024-09-16T10:33:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:31 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:31 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:31 smithi165 ceph-mon[32696]: osdmap e563: 8 total, 8 up, 6 in 2024-09-16T10:33:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2051549608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-16T10:33:32.768 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:32.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:32 smithi063 ceph-mon[29993]: pgmap v1075: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:33:32.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:32 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:33:32.776 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:32 smithi063 ceph-mon[39470]: pgmap v1075: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:33:32.776 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:32 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:33:32.930 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:33:32.932 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:33:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:32 smithi165 ceph-mon[32696]: pgmap v1075: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T10:33:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:32 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:33:33.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T10:33:33.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[29993]: from='client.16905 v1:172.21.15.63:0/2442933523' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:33.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:33:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:33:33.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[39470]: from='client.16905 v1:172.21.15.63:0/2442933523' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:33:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:33:33.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:33 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:33.783 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":1075,"stamp":"2024-09-16T10:33:31.536370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1193,"num_read_kb":945,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":533032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1024,"ondisk_log_size":1024,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":473,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4767360,"kb_used_data":7172,"kb_used_omap":9,"kb_used_meta":4760054,"kb_avail":557703552,"statfs":{"total":575970213888,"available":571088437248,"internally_reserved":0,"allocated":7344128,"data_stored":4688790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":4874295970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-248548,"num_write_kb":-7953473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-46483,"num_bytes_recovered":-3045785600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001991"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1086,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.831322+0000","last_change":"2024-09-16T10:33:17.292872+0000","last_active":"2024-09-16T10:33:27.831322+0000","last_peered":"2024-09-16T10:33:27.831322+0000","last_clean":"2024-09-16T10:33:27.831322+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:33:27.831322+0000","last_undegraded":"2024-09-16T10:33:27.831322+0000","last_fullsized":"2024-09-16T10:33:27.831322+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:17.292796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:17.292796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:40.930967+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":1048,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657334+0000","last_change":"2024-09-16T10:32:56.707159+0000","last_active":"2024-09-16T10:33:26.657334+0000","last_peered":"2024-09-16T10:33:26.657334+0000","last_clean":"2024-09-16T10:33:26.657334+0000","last_became_active":"2024-09-16T10:32:20.414426+0000","last_became_peered":"2024-09-16T10:32:20.414426+0000","last_unstale":"2024-09-16T10:33:26.657334+0000","last_undegraded":"2024-09-16T10:33:26.657334+0000","last_fullsized":"2024-09-16T10:33:26.657334+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:32:56.707086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:56.707086+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:01.126649+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1030,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406126+0000","last_change":"2024-09-16T10:32:20.401571+0000","last_active":"2024-09-16T10:33:27.406126+0000","last_peered":"2024-09-16T10:33:27.406126+0000","last_clean":"2024-09-16T10:33:27.406126+0000","last_became_active":"2024-09-16T10:32:20.401119+0000","last_became_peered":"2024-09-16T10:32:20.401119+0000","last_unstale":"2024-09-16T10:33:27.406126+0000","last_undegraded":"2024-09-16T10:33:27.406126+0000","last_fullsized":"2024-09-16T10:33:27.406126+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.872906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.872906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":394,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406017+0000","last_change":"2024-09-16T10:32:20.381952+0000","last_active":"2024-09-16T10:33:27.406017+0000","last_peered":"2024-09-16T10:33:27.406017+0000","last_clean":"2024-09-16T10:33:27.406017+0000","last_became_active":"2024-09-16T10:32:20.380629+0000","last_became_peered":"2024-09-16T10:32:20.380629+0000","last_unstale":"2024-09-16T10:33:27.406017+0000","last_undegraded":"2024-09-16T10:33:27.406017+0000","last_fullsized":"2024-09-16T10:33:27.406017+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:17.879610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:17.879610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1066,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.392601+0000","last_change":"2024-09-16T10:32:20.435331+0000","last_active":"2024-09-16T10:33:26.392601+0000","last_peered":"2024-09-16T10:33:26.392601+0000","last_clean":"2024-09-16T10:33:26.392601+0000","last_became_active":"2024-09-16T10:32:20.433546+0000","last_became_peered":"2024-09-16T10:32:20.433546+0000","last_unstale":"2024-09-16T10:33:26.392601+0000","last_undegraded":"2024-09-16T10:33:26.392601+0000","last_fullsized":"2024-09-16T10:33:26.392601+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1085,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.397317+0000","last_change":"2024-09-16T10:30:48.140925+0000","last_active":"2024-09-16T10:33:26.397317+0000","last_peered":"2024-09-16T10:33:26.397317+0000","last_clean":"2024-09-16T10:33:26.397317+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:33:26.397317+0000","last_undegraded":"2024-09-16T10:33:26.397317+0000","last_fullsized":"2024-09-16T10:33:26.397317+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":1155,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.391845+0000","last_change":"2024-09-16T10:30:15.998707+0000","last_active":"2024-09-16T10:33:26.391845+0000","last_peered":"2024-09-16T10:33:26.391845+0000","last_clean":"2024-09-16T10:33:26.391845+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:33:26.391845+0000","last_undegraded":"2024-09-16T10:33:26.391845+0000","last_fullsized":"2024-09-16T10:33:26.391845+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:15.998614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.998614+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":402,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657200+0000","last_change":"2024-09-16T10:33:02.705141+0000","last_active":"2024-09-16T10:33:26.657200+0000","last_peered":"2024-09-16T10:33:26.657200+0000","last_clean":"2024-09-16T10:33:26.657200+0000","last_became_active":"2024-09-16T10:32:20.392399+0000","last_became_peered":"2024-09-16T10:32:20.392399+0000","last_unstale":"2024-09-16T10:33:26.657200+0000","last_undegraded":"2024-09-16T10:33:26.657200+0000","last_fullsized":"2024-09-16T10:33:26.657200+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:02.705078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:02.705078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:18.211849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1095,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407665+0000","last_change":"2024-09-16T10:33:16.405006+0000","last_active":"2024-09-16T10:33:27.407665+0000","last_peered":"2024-09-16T10:33:27.407665+0000","last_clean":"2024-09-16T10:33:27.407665+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:33:27.407665+0000","last_undegraded":"2024-09-16T10:33:27.407665+0000","last_fullsized":"2024-09-16T10:33:27.407665+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:16.404954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:16.404954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:30.180621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":981,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393758+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:33:26.393758+0000","last_peered":"2024-09-16T10:33:26.393758+0000","last_clean":"2024-09-16T10:33:26.393758+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:33:26.393758+0000","last_undegraded":"2024-09-16T10:33:26.393758+0000","last_fullsized":"2024-09-16T10:33:26.393758+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1081,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406815+0000","last_change":"2024-09-16T10:31:14.144852+0000","last_active":"2024-09-16T10:33:27.406815+0000","last_peered":"2024-09-16T10:33:27.406815+0000","last_clean":"2024-09-16T10:33:27.406815+0000","last_became_active":"2024-09-16T10:31:14.144519+0000","last_became_peered":"2024-09-16T10:31:14.144519+0000","last_unstale":"2024-09-16T10:33:27.406815+0000","last_undegraded":"2024-09-16T10:33:27.406815+0000","last_fullsized":"2024-09-16T10:33:27.406815+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:41.762023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:41.762023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1084,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825821+0000","last_change":"2024-09-16T10:32:20.421334+0000","last_active":"2024-09-16T10:33:27.825821+0000","last_peered":"2024-09-16T10:33:27.825821+0000","last_clean":"2024-09-16T10:33:27.825821+0000","last_became_active":"2024-09-16T10:32:20.413663+0000","last_became_peered":"2024-09-16T10:32:20.413663+0000","last_unstale":"2024-09-16T10:33:27.825821+0000","last_undegraded":"2024-09-16T10:33:27.825821+0000","last_fullsized":"2024-09-16T10:33:27.825821+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:13.749174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:13.749174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:32.838025+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1046,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406764+0000","last_change":"2024-09-16T10:32:20.377023+0000","last_active":"2024-09-16T10:33:27.406764+0000","last_peered":"2024-09-16T10:33:27.406764+0000","last_clean":"2024-09-16T10:33:27.406764+0000","last_became_active":"2024-09-16T10:32:20.376673+0000","last_became_peered":"2024-09-16T10:32:20.376673+0000","last_unstale":"2024-09-16T10:33:27.406764+0000","last_undegraded":"2024-09-16T10:33:27.406764+0000","last_fullsized":"2024-09-16T10:33:27.406764+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:13.941032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:13.941032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":982,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.395811+0000","last_change":"2024-09-16T10:32:20.470950+0000","last_active":"2024-09-16T10:33:26.395811+0000","last_peered":"2024-09-16T10:33:26.395811+0000","last_clean":"2024-09-16T10:33:26.395811+0000","last_became_active":"2024-09-16T10:32:20.434528+0000","last_became_peered":"2024-09-16T10:32:20.434528+0000","last_unstale":"2024-09-16T10:33:26.395811+0000","last_undegraded":"2024-09-16T10:33:26.395811+0000","last_fullsized":"2024-09-16T10:33:26.395811+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:30:39.082450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.082450+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":713,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825773+0000","last_change":"2024-09-16T10:32:20.408057+0000","last_active":"2024-09-16T10:33:27.825773+0000","last_peered":"2024-09-16T10:33:27.825773+0000","last_clean":"2024-09-16T10:33:27.825773+0000","last_became_active":"2024-09-16T10:32:20.403637+0000","last_became_peered":"2024-09-16T10:32:20.403637+0000","last_unstale":"2024-09-16T10:33:27.825773+0000","last_undegraded":"2024-09-16T10:33:27.825773+0000","last_fullsized":"2024-09-16T10:33:27.825773+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.995349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.995349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1082,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406139+0000","last_change":"2024-09-16T10:32:20.401456+0000","last_active":"2024-09-16T10:33:27.406139+0000","last_peered":"2024-09-16T10:33:27.406139+0000","last_clean":"2024-09-16T10:33:27.406139+0000","last_became_active":"2024-09-16T10:32:20.401126+0000","last_became_peered":"2024-09-16T10:32:20.401126+0000","last_unstale":"2024-09-16T10:33:27.406139+0000","last_undegraded":"2024-09-16T10:33:27.406139+0000","last_fullsized":"2024-09-16T10:33:27.406139+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.741362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.741362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":388,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658176+0000","last_change":"2024-09-16T10:32:11.660805+0000","last_active":"2024-09-16T10:33:26.658176+0000","last_peered":"2024-09-16T10:33:26.658176+0000","last_clean":"2024-09-16T10:33:26.658176+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:33:26.658176+0000","last_undegraded":"2024-09-16T10:33:26.658176+0000","last_fullsized":"2024-09-16T10:33:26.658176+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:11.660728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.660728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1079,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826102+0000","last_change":"2024-09-16T10:32:20.417609+0000","last_active":"2024-09-16T10:33:27.826102+0000","last_peered":"2024-09-16T10:33:27.826102+0000","last_clean":"2024-09-16T10:33:27.826102+0000","last_became_active":"2024-09-16T10:32:20.416791+0000","last_became_peered":"2024-09-16T10:32:20.416791+0000","last_unstale":"2024-09-16T10:33:27.826102+0000","last_undegraded":"2024-09-16T10:33:27.826102+0000","last_fullsized":"2024-09-16T10:33:27.826102+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.446934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.446934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":654,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.727382+0000","last_change":"2024-09-16T10:32:20.499898+0000","last_active":"2024-09-16T10:33:26.727382+0000","last_peered":"2024-09-16T10:33:26.727382+0000","last_clean":"2024-09-16T10:33:26.727382+0000","last_became_active":"2024-09-16T10:32:20.490019+0000","last_became_peered":"2024-09-16T10:32:20.490019+0000","last_unstale":"2024-09-16T10:33:26.727382+0000","last_undegraded":"2024-09-16T10:33:26.727382+0000","last_fullsized":"2024-09-16T10:33:26.727382+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.599903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.599903+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:24.641849+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1055,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.392564+0000","last_change":"2024-09-16T10:32:20.415199+0000","last_active":"2024-09-16T10:33:26.392564+0000","last_peered":"2024-09-16T10:33:26.392564+0000","last_clean":"2024-09-16T10:33:26.392564+0000","last_became_active":"2024-09-16T10:32:20.414861+0000","last_became_peered":"2024-09-16T10:32:20.414861+0000","last_unstale":"2024-09-16T10:33:26.392564+0000","last_undegraded":"2024-09-16T10:33:26.392564+0000","last_fullsized":"2024-09-16T10:33:26.392564+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:09.918371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:09.918371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1016,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393046+0000","last_change":"2024-09-16T10:33:08.153114+0000","last_active":"2024-09-16T10:33:26.393046+0000","last_peered":"2024-09-16T10:33:26.393046+0000","last_clean":"2024-09-16T10:33:26.393046+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:33:26.393046+0000","last_undegraded":"2024-09-16T10:33:26.393046+0000","last_fullsized":"2024-09-16T10:33:26.393046+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:08.153044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:08.153044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:22.272748+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":539,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725109+0000","last_change":"2024-09-16T10:32:20.424078+0000","last_active":"2024-09-16T10:33:26.725109+0000","last_peered":"2024-09-16T10:33:26.725109+0000","last_clean":"2024-09-16T10:33:26.725109+0000","last_became_active":"2024-09-16T10:32:20.423662+0000","last_became_peered":"2024-09-16T10:32:20.423662+0000","last_unstale":"2024-09-16T10:33:26.725109+0000","last_undegraded":"2024-09-16T10:33:26.725109+0000","last_fullsized":"2024-09-16T10:33:26.725109+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:09.147080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.147080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":1143,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.394701+0000","last_change":"2024-09-16T10:32:20.461961+0000","last_active":"2024-09-16T10:33:26.394701+0000","last_peered":"2024-09-16T10:33:26.394701+0000","last_clean":"2024-09-16T10:33:26.394701+0000","last_became_active":"2024-09-16T10:32:20.429076+0000","last_became_peered":"2024-09-16T10:32:20.429076+0000","last_unstale":"2024-09-16T10:33:26.394701+0000","last_undegraded":"2024-09-16T10:33:26.394701+0000","last_fullsized":"2024-09-16T10:33:26.394701+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:30:38.072460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:38.072460+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":851,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.399487+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:33:26.399487+0000","last_peered":"2024-09-16T10:33:26.399487+0000","last_clean":"2024-09-16T10:33:26.399487+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:33:26.399487+0000","last_undegraded":"2024-09-16T10:33:26.399487+0000","last_fullsized":"2024-09-16T10:33:26.399487+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:48.236715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":485,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406361+0000","last_change":"2024-09-16T10:31:14.169546+0000","last_active":"2024-09-16T10:33:27.406361+0000","last_peered":"2024-09-16T10:33:27.406361+0000","last_clean":"2024-09-16T10:33:27.406361+0000","last_became_active":"2024-09-16T10:31:14.151297+0000","last_became_peered":"2024-09-16T10:31:14.151297+0000","last_unstale":"2024-09-16T10:33:27.406361+0000","last_undegraded":"2024-09-16T10:33:27.406361+0000","last_fullsized":"2024-09-16T10:33:27.406361+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:40.788922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.788922+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":81,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393915+0000","last_change":"2024-09-16T10:32:20.469065+0000","last_active":"2024-09-16T10:33:26.393915+0000","last_peered":"2024-09-16T10:33:26.393915+0000","last_clean":"2024-09-16T10:33:26.393915+0000","last_became_active":"2024-09-16T10:32:20.432046+0000","last_became_peered":"2024-09-16T10:32:20.432046+0000","last_unstale":"2024-09-16T10:33:26.393915+0000","last_undegraded":"2024-09-16T10:33:26.393915+0000","last_fullsized":"2024-09-16T10:33:26.393915+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:30:12.928274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.928274+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":1147,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.827514+0000","last_change":"2024-09-16T10:33:14.278928+0000","last_active":"2024-09-16T10:33:27.827514+0000","last_peered":"2024-09-16T10:33:27.827514+0000","last_clean":"2024-09-16T10:33:27.827514+0000","last_became_active":"2024-09-16T10:32:20.407208+0000","last_became_peered":"2024-09-16T10:32:20.407208+0000","last_unstale":"2024-09-16T10:33:27.827514+0000","last_undegraded":"2024-09-16T10:33:27.827514+0000","last_fullsized":"2024-09-16T10:33:27.827514+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:14.278843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:14.278843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:33.769863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"508'11","reported_seq":1330,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658460+0000","last_change":"2024-09-16T10:31:14.167033+0000","last_active":"2024-09-16T10:33:26.658460+0000","last_peered":"2024-09-16T10:33:26.658460+0000","last_clean":"2024-09-16T10:33:26.658460+0000","last_became_active":"2024-09-16T10:31:14.137776+0000","last_became_peered":"2024-09-16T10:31:14.137776+0000","last_unstale":"2024-09-16T10:33:26.658460+0000","last_undegraded":"2024-09-16T10:33:26.658460+0000","last_fullsized":"2024-09-16T10:33:26.658460+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.694030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":918,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406011+0000","last_change":"2024-09-16T10:32:16.263470+0000","last_active":"2024-09-16T10:33:27.406011+0000","last_peered":"2024-09-16T10:33:27.406011+0000","last_clean":"2024-09-16T10:33:27.406011+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:33:27.406011+0000","last_undegraded":"2024-09-16T10:33:27.406011+0000","last_fullsized":"2024-09-16T10:33:27.406011+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:16.263399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:16.263399+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":696,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393212+0000","last_change":"2024-09-16T10:33:06.135162+0000","last_active":"2024-09-16T10:33:26.393212+0000","last_peered":"2024-09-16T10:33:26.393212+0000","last_clean":"2024-09-16T10:33:26.393212+0000","last_became_active":"2024-09-16T10:32:20.414383+0000","last_became_peered":"2024-09-16T10:32:20.414383+0000","last_unstale":"2024-09-16T10:33:26.393212+0000","last_undegraded":"2024-09-16T10:33:26.393212+0000","last_fullsized":"2024-09-16T10:33:26.393212+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:33:06.135084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.135084+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:20.735523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1096,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.401316+0000","last_change":"2024-09-16T10:32:20.421467+0000","last_active":"2024-09-16T10:33:26.401316+0000","last_peered":"2024-09-16T10:33:26.401316+0000","last_clean":"2024-09-16T10:33:26.401316+0000","last_became_active":"2024-09-16T10:32:20.417850+0000","last_became_peered":"2024-09-16T10:32:20.417850+0000","last_unstale":"2024-09-16T10:33:26.401316+0000","last_undegraded":"2024-09-16T10:33:26.401316+0000","last_fullsized":"2024-09-16T10:33:26.401316+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.213821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.213821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":498,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406868+0000","last_change":"2024-09-16T10:30:16.857737+0000","last_active":"2024-09-16T10:33:27.406868+0000","last_peered":"2024-09-16T10:33:27.406868+0000","last_clean":"2024-09-16T10:33:27.406868+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:33:27.406868+0000","last_undegraded":"2024-09-16T10:33:27.406868+0000","last_fullsized":"2024-09-16T10:33:27.406868+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:30:16.857676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.857676+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1067,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406416+0000","last_change":"2024-09-16T10:32:20.407177+0000","last_active":"2024-09-16T10:33:27.406416+0000","last_peered":"2024-09-16T10:33:27.406416+0000","last_clean":"2024-09-16T10:33:27.406416+0000","last_became_active":"2024-09-16T10:32:20.406800+0000","last_became_peered":"2024-09-16T10:32:20.406800+0000","last_unstale":"2024-09-16T10:33:27.406416+0000","last_undegraded":"2024-09-16T10:33:27.406416+0000","last_fullsized":"2024-09-16T10:33:27.406416+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:14.925284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:14.925284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":492,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657048+0000","last_change":"2024-09-16T10:30:18.655233+0000","last_active":"2024-09-16T10:33:26.657048+0000","last_peered":"2024-09-16T10:33:26.657048+0000","last_clean":"2024-09-16T10:33:26.657048+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:33:26.657048+0000","last_undegraded":"2024-09-16T10:33:26.657048+0000","last_fullsized":"2024-09-16T10:33:26.657048+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:18.655172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:18.655172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:24.722180+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":453,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657721+0000","last_change":"2024-09-16T10:33:04.704385+0000","last_active":"2024-09-16T10:33:26.657721+0000","last_peered":"2024-09-16T10:33:26.657721+0000","last_clean":"2024-09-16T10:33:26.657721+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:33:26.657721+0000","last_undegraded":"2024-09-16T10:33:26.657721+0000","last_fullsized":"2024-09-16T10:33:26.657721+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:33:04.704308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:04.704308+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:14.562749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1053,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393472+0000","last_change":"2024-09-16T10:31:14.146422+0000","last_active":"2024-09-16T10:33:26.393472+0000","last_peered":"2024-09-16T10:33:26.393472+0000","last_clean":"2024-09-16T10:33:26.393472+0000","last_became_active":"2024-09-16T10:31:14.146107+0000","last_became_peered":"2024-09-16T10:31:14.146107+0000","last_unstale":"2024-09-16T10:33:26.393472+0000","last_undegraded":"2024-09-16T10:33:26.393472+0000","last_fullsized":"2024-09-16T10:33:26.393472+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:55.254993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.254993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":820,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.660082+0000","last_change":"2024-09-16T10:32:06.770331+0000","last_active":"2024-09-16T10:33:26.660082+0000","last_peered":"2024-09-16T10:33:26.660082+0000","last_clean":"2024-09-16T10:33:26.660082+0000","last_became_active":"2024-09-16T10:31:14.164506+0000","last_became_peered":"2024-09-16T10:31:14.164506+0000","last_unstale":"2024-09-16T10:33:26.660082+0000","last_undegraded":"2024-09-16T10:33:26.660082+0000","last_fullsized":"2024-09-16T10:33:26.660082+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:06.770275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.770275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:34.696037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":867,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393667+0000","last_change":"2024-09-16T10:33:23.226943+0000","last_active":"2024-09-16T10:33:26.393667+0000","last_peered":"2024-09-16T10:33:26.393667+0000","last_clean":"2024-09-16T10:33:26.393667+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:33:26.393667+0000","last_undegraded":"2024-09-16T10:33:26.393667+0000","last_fullsized":"2024-09-16T10:33:26.393667+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:52.274606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":989,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.399763+0000","last_change":"2024-09-16T10:31:17.119206+0000","last_active":"2024-09-16T10:33:26.399763+0000","last_peered":"2024-09-16T10:33:26.399763+0000","last_clean":"2024-09-16T10:33:26.399763+0000","last_became_active":"2024-09-16T10:31:17.118842+0000","last_became_peered":"2024-09-16T10:31:17.118842+0000","last_unstale":"2024-09-16T10:33:26.399763+0000","last_undegraded":"2024-09-16T10:33:26.399763+0000","last_fullsized":"2024-09-16T10:33:26.399763+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.198923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.198923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":1009,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406215+0000","last_change":"2024-09-16T10:30:12.893400+0000","last_active":"2024-09-16T10:33:27.406215+0000","last_peered":"2024-09-16T10:33:27.406215+0000","last_clean":"2024-09-16T10:33:27.406215+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:33:27.406215+0000","last_undegraded":"2024-09-16T10:33:27.406215+0000","last_fullsized":"2024-09-16T10:33:27.406215+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:30:12.893340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.893340+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":1068,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.829327+0000","last_change":"2024-09-16T10:32:20.425958+0000","last_active":"2024-09-16T10:33:27.829327+0000","last_peered":"2024-09-16T10:33:27.829327+0000","last_clean":"2024-09-16T10:33:27.829327+0000","last_became_active":"2024-09-16T10:32:20.393386+0000","last_became_peered":"2024-09-16T10:32:20.393386+0000","last_unstale":"2024-09-16T10:33:27.829327+0000","last_undegraded":"2024-09-16T10:33:27.829327+0000","last_fullsized":"2024-09-16T10:33:27.829327+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:46.894131+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"536'9","reported_seq":1324,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.397449+0000","last_change":"2024-09-16T10:33:15.204469+0000","last_active":"2024-09-16T10:33:26.397449+0000","last_peered":"2024-09-16T10:33:26.397449+0000","last_clean":"2024-09-16T10:33:26.397449+0000","last_became_active":"2024-09-16T10:32:20.418792+0000","last_became_peered":"2024-09-16T10:32:20.418792+0000","last_unstale":"2024-09-16T10:33:26.397449+0000","last_undegraded":"2024-09-16T10:33:26.397449+0000","last_fullsized":"2024-09-16T10:33:26.397449+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'9","last_scrub_stamp":"2024-09-16T10:33:15.201123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:15.201123+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:28.015856+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":937,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658128+0000","last_change":"2024-09-16T10:32:20.413255+0000","last_active":"2024-09-16T10:33:26.658128+0000","last_peered":"2024-09-16T10:33:26.658128+0000","last_clean":"2024-09-16T10:33:26.658128+0000","last_became_active":"2024-09-16T10:32:20.412898+0000","last_became_peered":"2024-09-16T10:32:20.412898+0000","last_unstale":"2024-09-16T10:33:26.658128+0000","last_undegraded":"2024-09-16T10:33:26.658128+0000","last_fullsized":"2024-09-16T10:33:26.658128+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:12.688413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.688413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:40.735663+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":1047,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658167+0000","last_change":"2024-09-16T10:32:16.334773+0000","last_active":"2024-09-16T10:33:26.658167+0000","last_peered":"2024-09-16T10:33:26.658167+0000","last_clean":"2024-09-16T10:33:26.658167+0000","last_became_active":"2024-09-16T10:32:16.334470+0000","last_became_peered":"2024-09-16T10:32:16.334470+0000","last_unstale":"2024-09-16T10:33:26.658167+0000","last_undegraded":"2024-09-16T10:33:26.658167+0000","last_fullsized":"2024-09-16T10:33:26.658167+0000","mapping_epoch":532,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.663450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.663450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":695,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406403+0000","last_change":"2024-09-16T10:32:20.446784+0000","last_active":"2024-09-16T10:33:27.406403+0000","last_peered":"2024-09-16T10:33:27.406403+0000","last_clean":"2024-09-16T10:33:27.406403+0000","last_became_active":"2024-09-16T10:32:20.395496+0000","last_became_peered":"2024-09-16T10:32:20.395496+0000","last_unstale":"2024-09-16T10:33:27.406403+0000","last_undegraded":"2024-09-16T10:33:27.406403+0000","last_fullsized":"2024-09-16T10:33:27.406403+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:50.360324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1087,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407788+0000","last_change":"2024-09-16T10:33:02.347961+0000","last_active":"2024-09-16T10:33:27.407788+0000","last_peered":"2024-09-16T10:33:27.407788+0000","last_clean":"2024-09-16T10:33:27.407788+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:33:27.407788+0000","last_undegraded":"2024-09-16T10:33:27.407788+0000","last_fullsized":"2024-09-16T10:33:27.407788+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:02.347917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:02.347917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:09.667103+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1295,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.827037+0000","last_change":"2024-09-16T10:30:08.183366+0000","last_active":"2024-09-16T10:33:27.827037+0000","last_peered":"2024-09-16T10:33:27.827037+0000","last_clean":"2024-09-16T10:33:27.827037+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:33:27.827037+0000","last_undegraded":"2024-09-16T10:33:27.827037+0000","last_fullsized":"2024-09-16T10:33:27.827037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:08.182949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.182949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1074,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825925+0000","last_change":"2024-09-16T10:30:12.258492+0000","last_active":"2024-09-16T10:33:27.825925+0000","last_peered":"2024-09-16T10:33:27.825925+0000","last_clean":"2024-09-16T10:33:27.825925+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:33:27.825925+0000","last_undegraded":"2024-09-16T10:33:27.825925+0000","last_fullsized":"2024-09-16T10:33:27.825925+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:12.258426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.258426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":1024,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658166+0000","last_change":"2024-09-16T10:32:55.757529+0000","last_active":"2024-09-16T10:33:26.658166+0000","last_peered":"2024-09-16T10:33:26.658166+0000","last_clean":"2024-09-16T10:33:26.658166+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:33:26.658166+0000","last_undegraded":"2024-09-16T10:33:26.658166+0000","last_fullsized":"2024-09-16T10:33:26.658166+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:55.757468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:55.757468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:58.838895+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":1180,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826932+0000","last_change":"2024-09-16T10:33:12.305837+0000","last_active":"2024-09-16T10:33:27.826932+0000","last_peered":"2024-09-16T10:33:27.826932+0000","last_clean":"2024-09-16T10:33:27.826932+0000","last_became_active":"2024-09-16T10:32:20.413038+0000","last_became_peered":"2024-09-16T10:32:20.413038+0000","last_unstale":"2024-09-16T10:33:27.826932+0000","last_undegraded":"2024-09-16T10:33:27.826932+0000","last_fullsized":"2024-09-16T10:33:27.826932+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:33:12.305749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:12.305749+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:35.711805+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":555,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.727122+0000","last_change":"2024-09-16T10:32:20.426707+0000","last_active":"2024-09-16T10:33:26.727122+0000","last_peered":"2024-09-16T10:33:26.727122+0000","last_clean":"2024-09-16T10:33:26.727122+0000","last_became_active":"2024-09-16T10:32:20.426427+0000","last_became_peered":"2024-09-16T10:32:20.426427+0000","last_unstale":"2024-09-16T10:33:26.727122+0000","last_undegraded":"2024-09-16T10:33:26.727122+0000","last_fullsized":"2024-09-16T10:33:26.727122+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.303693+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1072,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657479+0000","last_change":"2024-09-16T10:30:50.670023+0000","last_active":"2024-09-16T10:33:26.657479+0000","last_peered":"2024-09-16T10:33:26.657479+0000","last_clean":"2024-09-16T10:33:26.657479+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:33:26.657479+0000","last_undegraded":"2024-09-16T10:33:26.657479+0000","last_fullsized":"2024-09-16T10:33:26.657479+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.669938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.669938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":542,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.660264+0000","last_change":"2024-09-16T10:32:20.405226+0000","last_active":"2024-09-16T10:33:26.660264+0000","last_peered":"2024-09-16T10:33:26.660264+0000","last_clean":"2024-09-16T10:33:26.660264+0000","last_became_active":"2024-09-16T10:32:20.404614+0000","last_became_peered":"2024-09-16T10:32:20.404614+0000","last_unstale":"2024-09-16T10:33:26.660264+0000","last_undegraded":"2024-09-16T10:33:26.660264+0000","last_fullsized":"2024-09-16T10:33:26.660264+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:42.743403+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":962,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406008+0000","last_change":"2024-09-16T10:30:54.802360+0000","last_active":"2024-09-16T10:33:27.406008+0000","last_peered":"2024-09-16T10:33:27.406008+0000","last_clean":"2024-09-16T10:33:27.406008+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:33:27.406008+0000","last_undegraded":"2024-09-16T10:33:27.406008+0000","last_fullsized":"2024-09-16T10:33:27.406008+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:54.802277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.802277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"537'43","reported_seq":935,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.727335+0000","last_change":"2024-09-16T10:32:20.529917+0000","last_active":"2024-09-16T10:33:26.727335+0000","last_peered":"2024-09-16T10:33:26.727335+0000","last_clean":"2024-09-16T10:33:26.727335+0000","last_became_active":"2024-09-16T10:32:20.435824+0000","last_became_peered":"2024-09-16T10:32:20.435824+0000","last_unstale":"2024-09-16T10:33:26.727335+0000","last_undegraded":"2024-09-16T10:33:26.727335+0000","last_fullsized":"2024-09-16T10:33:26.727335+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:32:12.197914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.197914+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":380,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826294+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:33:27.826294+0000","last_peered":"2024-09-16T10:33:27.826294+0000","last_clean":"2024-09-16T10:33:27.826294+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:33:27.826294+0000","last_undegraded":"2024-09-16T10:33:27.826294+0000","last_fullsized":"2024-09-16T10:33:27.826294+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:49.909210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":1094,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407322+0000","last_change":"2024-09-16T10:32:20.428905+0000","last_active":"2024-09-16T10:33:27.407322+0000","last_peered":"2024-09-16T10:33:27.407322+0000","last_clean":"2024-09-16T10:33:27.407322+0000","last_became_active":"2024-09-16T10:32:20.396067+0000","last_became_peered":"2024-09-16T10:32:20.396067+0000","last_unstale":"2024-09-16T10:33:27.407322+0000","last_undegraded":"2024-09-16T10:33:27.407322+0000","last_fullsized":"2024-09-16T10:33:27.407322+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:30:11.895746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.895746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1055,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826704+0000","last_change":"2024-09-16T10:32:20.395692+0000","last_active":"2024-09-16T10:33:27.826704+0000","last_peered":"2024-09-16T10:33:27.826704+0000","last_clean":"2024-09-16T10:33:27.826704+0000","last_became_active":"2024-09-16T10:32:20.395374+0000","last_became_peered":"2024-09-16T10:32:20.395374+0000","last_unstale":"2024-09-16T10:33:27.826704+0000","last_undegraded":"2024-09-16T10:33:27.826704+0000","last_fullsized":"2024-09-16T10:33:27.826704+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.379980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.379980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1085,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.396213+0000","last_change":"2024-09-16T10:31:14.143712+0000","last_active":"2024-09-16T10:33:26.396213+0000","last_peered":"2024-09-16T10:33:26.396213+0000","last_clean":"2024-09-16T10:33:26.396213+0000","last_became_active":"2024-09-16T10:31:14.143349+0000","last_became_peered":"2024-09-16T10:31:14.143349+0000","last_unstale":"2024-09-16T10:33:26.396213+0000","last_undegraded":"2024-09-16T10:33:26.396213+0000","last_fullsized":"2024-09-16T10:33:26.396213+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.877913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.877913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":555,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826018+0000","last_change":"2024-09-16T10:31:17.108542+0000","last_active":"2024-09-16T10:33:27.826018+0000","last_peered":"2024-09-16T10:33:27.826018+0000","last_clean":"2024-09-16T10:33:27.826018+0000","last_became_active":"2024-09-16T10:31:17.108022+0000","last_became_peered":"2024-09-16T10:31:17.108022+0000","last_unstale":"2024-09-16T10:33:27.826018+0000","last_undegraded":"2024-09-16T10:33:27.826018+0000","last_fullsized":"2024-09-16T10:33:27.826018+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:49.434225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.434225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":664,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406837+0000","last_change":"2024-09-16T10:32:20.453683+0000","last_active":"2024-09-16T10:33:27.406837+0000","last_peered":"2024-09-16T10:33:27.406837+0000","last_clean":"2024-09-16T10:33:27.406837+0000","last_became_active":"2024-09-16T10:32:20.435036+0000","last_became_peered":"2024-09-16T10:32:20.435036+0000","last_unstale":"2024-09-16T10:33:27.406837+0000","last_undegraded":"2024-09-16T10:33:27.406837+0000","last_fullsized":"2024-09-16T10:33:27.406837+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:30:05.864159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.864159+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":547,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725058+0000","last_change":"2024-09-16T10:32:20.419274+0000","last_active":"2024-09-16T10:33:26.725058+0000","last_peered":"2024-09-16T10:33:26.725058+0000","last_clean":"2024-09-16T10:33:26.725058+0000","last_became_active":"2024-09-16T10:32:20.418545+0000","last_became_peered":"2024-09-16T10:32:20.418545+0000","last_unstale":"2024-09-16T10:33:26.725058+0000","last_undegraded":"2024-09-16T10:33:26.725058+0000","last_fullsized":"2024-09-16T10:33:26.725058+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:43.359607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":602,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725292+0000","last_change":"2024-09-16T10:32:11.066183+0000","last_active":"2024-09-16T10:33:26.725292+0000","last_peered":"2024-09-16T10:33:26.725292+0000","last_clean":"2024-09-16T10:33:26.725292+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:33:26.725292+0000","last_undegraded":"2024-09-16T10:33:26.725292+0000","last_fullsized":"2024-09-16T10:33:26.725292+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:32:11.066076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.066076+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:27.244502+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":616,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.726520+0000","last_change":"2024-09-16T10:31:22.862857+0000","last_active":"2024-09-16T10:33:26.726520+0000","last_peered":"2024-09-16T10:33:26.726520+0000","last_clean":"2024-09-16T10:33:26.726520+0000","last_became_active":"2024-09-16T10:31:14.142132+0000","last_became_peered":"2024-09-16T10:31:14.142132+0000","last_unstale":"2024-09-16T10:33:26.726520+0000","last_undegraded":"2024-09-16T10:33:26.726520+0000","last_fullsized":"2024-09-16T10:33:26.726520+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:22.862677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:22.862677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":470,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.659079+0000","last_change":"2024-09-16T10:31:14.164501+0000","last_active":"2024-09-16T10:33:26.659079+0000","last_peered":"2024-09-16T10:33:26.659079+0000","last_clean":"2024-09-16T10:33:26.659079+0000","last_became_active":"2024-09-16T10:31:14.164133+0000","last_became_peered":"2024-09-16T10:31:14.164133+0000","last_unstale":"2024-09-16T10:33:26.659079+0000","last_undegraded":"2024-09-16T10:33:26.659079+0000","last_fullsized":"2024-09-16T10:33:26.659079+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.706111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.706111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":164,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657430+0000","last_change":"2024-09-16T10:32:53.774953+0000","last_active":"2024-09-16T10:33:26.657430+0000","last_peered":"2024-09-16T10:33:26.657430+0000","last_clean":"2024-09-16T10:33:26.657430+0000","last_became_active":"2024-09-16T10:32:23.420463+0000","last_became_peered":"2024-09-16T10:32:23.420463+0000","last_unstale":"2024-09-16T10:33:26.657430+0000","last_undegraded":"2024-09-16T10:33:26.657430+0000","last_fullsized":"2024-09-16T10:33:26.657430+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:53.774903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:53.774903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.401981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":1183,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.827603+0000","last_change":"2024-09-16T10:32:20.455220+0000","last_active":"2024-09-16T10:33:27.827603+0000","last_peered":"2024-09-16T10:33:27.827603+0000","last_clean":"2024-09-16T10:33:27.827603+0000","last_became_active":"2024-09-16T10:32:20.413651+0000","last_became_peered":"2024-09-16T10:32:20.413651+0000","last_unstale":"2024-09-16T10:33:27.827603+0000","last_undegraded":"2024-09-16T10:33:27.827603+0000","last_fullsized":"2024-09-16T10:33:27.827603+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:30:20.268438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.268438+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1088,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826292+0000","last_change":"2024-09-16T10:30:15.229737+0000","last_active":"2024-09-16T10:33:27.826292+0000","last_peered":"2024-09-16T10:33:27.826292+0000","last_clean":"2024-09-16T10:33:27.826292+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:33:27.826292+0000","last_undegraded":"2024-09-16T10:33:27.826292+0000","last_fullsized":"2024-09-16T10:33:27.826292+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.229669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.229669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":182,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:28.774649+0000","last_change":"2024-09-16T10:32:20.443363+0000","last_active":"2024-09-16T10:33:28.774649+0000","last_peered":"2024-09-16T10:33:28.774649+0000","last_clean":"2024-09-16T10:33:28.774649+0000","last_became_active":"2024-09-16T10:32:20.396791+0000","last_became_peered":"2024-09-16T10:32:20.396791+0000","last_unstale":"2024-09-16T10:33:28.774649+0000","last_undegraded":"2024-09-16T10:33:28.774649+0000","last_fullsized":"2024-09-16T10:33:28.774649+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:30:39.226399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.226399+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1088,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406773+0000","last_change":"2024-09-16T10:33:20.360912+0000","last_active":"2024-09-16T10:33:27.406773+0000","last_peered":"2024-09-16T10:33:27.406773+0000","last_clean":"2024-09-16T10:33:27.406773+0000","last_became_active":"2024-09-16T10:32:20.395732+0000","last_became_peered":"2024-09-16T10:32:20.395732+0000","last_unstale":"2024-09-16T10:33:27.406773+0000","last_undegraded":"2024-09-16T10:33:27.406773+0000","last_fullsized":"2024-09-16T10:33:27.406773+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:20.360862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:20.360862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:30.710303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":66,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407223+0000","last_change":"2024-09-16T10:32:20.435415+0000","last_active":"2024-09-16T10:33:27.407223+0000","last_peered":"2024-09-16T10:33:27.407223+0000","last_clean":"2024-09-16T10:33:27.407223+0000","last_became_active":"2024-09-16T10:32:20.434853+0000","last_became_peered":"2024-09-16T10:32:20.434853+0000","last_unstale":"2024-09-16T10:33:27.407223+0000","last_undegraded":"2024-09-16T10:33:27.407223+0000","last_fullsized":"2024-09-16T10:33:27.407223+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:47.361715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"508'31","reported_seq":1369,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.659254+0000","last_change":"2024-09-16T10:31:14.178826+0000","last_active":"2024-09-16T10:33:26.659254+0000","last_peered":"2024-09-16T10:33:26.659254+0000","last_clean":"2024-09-16T10:33:26.659254+0000","last_became_active":"2024-09-16T10:31:14.137441+0000","last_became_peered":"2024-09-16T10:31:14.137441+0000","last_unstale":"2024-09-16T10:33:26.659254+0000","last_undegraded":"2024-09-16T10:33:26.659254+0000","last_fullsized":"2024-09-16T10:33:26.659254+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:30:54.737043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.737043+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1036,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.397548+0000","last_change":"2024-09-16T10:32:07.379538+0000","last_active":"2024-09-16T10:33:26.397548+0000","last_peered":"2024-09-16T10:33:26.397548+0000","last_clean":"2024-09-16T10:33:26.397548+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:33:26.397548+0000","last_undegraded":"2024-09-16T10:33:26.397548+0000","last_fullsized":"2024-09-16T10:33:26.397548+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:07.379336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:07.379336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":572,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825189+0000","last_change":"2024-09-16T10:32:23.435277+0000","last_active":"2024-09-16T10:33:27.825189+0000","last_peered":"2024-09-16T10:33:27.825189+0000","last_clean":"2024-09-16T10:33:27.825189+0000","last_became_active":"2024-09-16T10:32:23.434945+0000","last_became_peered":"2024-09-16T10:32:23.434945+0000","last_unstale":"2024-09-16T10:33:27.825189+0000","last_undegraded":"2024-09-16T10:33:27.825189+0000","last_fullsized":"2024-09-16T10:33:27.825189+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:50.410561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.410561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":941,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.726124+0000","last_change":"2024-09-16T10:31:14.146544+0000","last_active":"2024-09-16T10:33:26.726124+0000","last_peered":"2024-09-16T10:33:26.726124+0000","last_clean":"2024-09-16T10:33:26.726124+0000","last_became_active":"2024-09-16T10:31:14.145333+0000","last_became_peered":"2024-09-16T10:31:14.145333+0000","last_unstale":"2024-09-16T10:33:26.726124+0000","last_undegraded":"2024-09-16T10:33:26.726124+0000","last_fullsized":"2024-09-16T10:33:26.726124+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.613780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.613780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:23.684845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1061,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.405772+0000","last_change":"2024-09-16T10:30:43.705706+0000","last_active":"2024-09-16T10:33:27.405772+0000","last_peered":"2024-09-16T10:33:27.405772+0000","last_clean":"2024-09-16T10:33:27.405772+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:33:27.405772+0000","last_undegraded":"2024-09-16T10:33:27.405772+0000","last_fullsized":"2024-09-16T10:33:27.405772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.705640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.705640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":500,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825249+0000","last_change":"2024-09-16T10:32:53.999521+0000","last_active":"2024-09-16T10:33:27.825249+0000","last_peered":"2024-09-16T10:33:27.825249+0000","last_clean":"2024-09-16T10:33:27.825249+0000","last_became_active":"2024-09-16T10:32:23.432634+0000","last_became_peered":"2024-09-16T10:32:23.432634+0000","last_unstale":"2024-09-16T10:33:27.825249+0000","last_undegraded":"2024-09-16T10:33:27.825249+0000","last_fullsized":"2024-09-16T10:33:27.825249+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:53.999450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:53.999450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.072558+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":1020,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658280+0000","last_change":"2024-09-16T10:32:20.480209+0000","last_active":"2024-09-16T10:33:26.658280+0000","last_peered":"2024-09-16T10:33:26.658280+0000","last_clean":"2024-09-16T10:33:26.658280+0000","last_became_active":"2024-09-16T10:32:20.395805+0000","last_became_peered":"2024-09-16T10:32:20.395805+0000","last_unstale":"2024-09-16T10:33:26.658280+0000","last_undegraded":"2024-09-16T10:33:26.658280+0000","last_fullsized":"2024-09-16T10:33:26.658280+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:32:08.692093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:08.692093+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:37.858412+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":892,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725093+0000","last_change":"2024-09-16T10:32:20.409986+0000","last_active":"2024-09-16T10:33:26.725093+0000","last_peered":"2024-09-16T10:33:26.725093+0000","last_clean":"2024-09-16T10:33:26.725093+0000","last_became_active":"2024-09-16T10:32:20.406762+0000","last_became_peered":"2024-09-16T10:32:20.406762+0000","last_unstale":"2024-09-16T10:33:26.725093+0000","last_undegraded":"2024-09-16T10:33:26.725093+0000","last_fullsized":"2024-09-16T10:33:26.725093+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:30.867930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:30.867930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1072,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407047+0000","last_change":"2024-09-16T10:33:07.435064+0000","last_active":"2024-09-16T10:33:27.407047+0000","last_peered":"2024-09-16T10:33:27.407047+0000","last_clean":"2024-09-16T10:33:27.407047+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:33:27.407047+0000","last_undegraded":"2024-09-16T10:33:27.407047+0000","last_fullsized":"2024-09-16T10:33:27.407047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:07.434963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:07.434963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:11.222708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":999,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.399671+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:33:26.399671+0000","last_peered":"2024-09-16T10:33:26.399671+0000","last_clean":"2024-09-16T10:33:26.399671+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:33:26.399671+0000","last_undegraded":"2024-09-16T10:33:26.399671+0000","last_fullsized":"2024-09-16T10:33:26.399671+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":933,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406048+0000","last_change":"2024-09-16T10:33:17.369390+0000","last_active":"2024-09-16T10:33:27.406048+0000","last_peered":"2024-09-16T10:33:27.406048+0000","last_clean":"2024-09-16T10:33:27.406048+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:33:27.406048+0000","last_undegraded":"2024-09-16T10:33:27.406048+0000","last_fullsized":"2024-09-16T10:33:27.406048+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:17.369317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:17.369317+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:29.192042+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":1090,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725530+0000","last_change":"2024-09-16T10:32:20.440999+0000","last_active":"2024-09-16T10:33:26.725530+0000","last_peered":"2024-09-16T10:33:26.725530+0000","last_clean":"2024-09-16T10:33:26.725530+0000","last_became_active":"2024-09-16T10:32:20.402914+0000","last_became_peered":"2024-09-16T10:32:20.402914+0000","last_unstale":"2024-09-16T10:33:26.725530+0000","last_undegraded":"2024-09-16T10:33:26.725530+0000","last_fullsized":"2024-09-16T10:33:26.725530+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:31:18.771268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:31:18.771268+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1079,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658106+0000","last_change":"2024-09-16T10:32:54.752441+0000","last_active":"2024-09-16T10:33:26.658106+0000","last_peered":"2024-09-16T10:33:26.658106+0000","last_clean":"2024-09-16T10:33:26.658106+0000","last_became_active":"2024-09-16T10:32:20.406154+0000","last_became_peered":"2024-09-16T10:32:20.406154+0000","last_unstale":"2024-09-16T10:33:26.658106+0000","last_undegraded":"2024-09-16T10:33:26.658106+0000","last_fullsized":"2024-09-16T10:33:26.658106+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:54.752376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:54.752376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.545772+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":73,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826529+0000","last_change":"2024-09-16T10:33:06.130249+0000","last_active":"2024-09-16T10:33:27.826529+0000","last_peered":"2024-09-16T10:33:27.826529+0000","last_clean":"2024-09-16T10:33:27.826529+0000","last_became_active":"2024-09-16T10:32:20.395860+0000","last_became_peered":"2024-09-16T10:32:20.395860+0000","last_unstale":"2024-09-16T10:33:27.826529+0000","last_undegraded":"2024-09-16T10:33:27.826529+0000","last_fullsized":"2024-09-16T10:33:27.826529+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:06.130189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.130189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:17.390775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":606,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.727021+0000","last_change":"2024-09-16T10:33:16.677107+0000","last_active":"2024-09-16T10:33:26.727021+0000","last_peered":"2024-09-16T10:33:26.727021+0000","last_clean":"2024-09-16T10:33:26.727021+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:33:26.727021+0000","last_undegraded":"2024-09-16T10:33:26.727021+0000","last_fullsized":"2024-09-16T10:33:26.727021+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:16.677041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:16.677041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:39.426050+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":744,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.394825+0000","last_change":"2024-09-16T10:31:14.142529+0000","last_active":"2024-09-16T10:33:26.394825+0000","last_peered":"2024-09-16T10:33:26.394825+0000","last_clean":"2024-09-16T10:33:26.394825+0000","last_became_active":"2024-09-16T10:31:14.140156+0000","last_became_peered":"2024-09-16T10:31:14.140156+0000","last_unstale":"2024-09-16T10:33:26.394825+0000","last_undegraded":"2024-09-16T10:33:26.394825+0000","last_fullsized":"2024-09-16T10:33:26.394825+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:07.951845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.951845+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":973,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.401898+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:33:26.401898+0000","last_peered":"2024-09-16T10:33:26.401898+0000","last_clean":"2024-09-16T10:33:26.401898+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:33:26.401898+0000","last_undegraded":"2024-09-16T10:33:26.401898+0000","last_fullsized":"2024-09-16T10:33:26.401898+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1054,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.395992+0000","last_change":"2024-09-16T10:32:09.398843+0000","last_active":"2024-09-16T10:33:26.395992+0000","last_peered":"2024-09-16T10:33:26.395992+0000","last_clean":"2024-09-16T10:33:26.395992+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:33:26.395992+0000","last_undegraded":"2024-09-16T10:33:26.395992+0000","last_fullsized":"2024-09-16T10:33:26.395992+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:09.398706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.398706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"508'13","reported_seq":764,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725773+0000","last_change":"2024-09-16T10:31:27.828520+0000","last_active":"2024-09-16T10:33:26.725773+0000","last_peered":"2024-09-16T10:33:26.725773+0000","last_clean":"2024-09-16T10:33:26.725773+0000","last_became_active":"2024-09-16T10:31:14.138957+0000","last_became_peered":"2024-09-16T10:31:14.138957+0000","last_unstale":"2024-09-16T10:33:26.725773+0000","last_undegraded":"2024-09-16T10:33:26.725773+0000","last_fullsized":"2024-09-16T10:33:26.725773+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'13","last_scrub_stamp":"2024-09-16T10:31:27.828439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:27.828439+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":551,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.726239+0000","last_change":"2024-09-16T10:33:12.679629+0000","last_active":"2024-09-16T10:33:26.726239+0000","last_peered":"2024-09-16T10:33:26.726239+0000","last_clean":"2024-09-16T10:33:26.726239+0000","last_became_active":"2024-09-16T10:32:20.392061+0000","last_became_peered":"2024-09-16T10:32:20.392061+0000","last_unstale":"2024-09-16T10:33:26.726239+0000","last_undegraded":"2024-09-16T10:33:26.726239+0000","last_fullsized":"2024-09-16T10:33:26.726239+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:12.679545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:12.679545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:31.979630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":337,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.398917+0000","last_change":"2024-09-16T10:32:17.363539+0000","last_active":"2024-09-16T10:33:26.398917+0000","last_peered":"2024-09-16T10:33:26.398917+0000","last_clean":"2024-09-16T10:33:26.398917+0000","last_became_active":"2024-09-16T10:31:17.119269+0000","last_became_peered":"2024-09-16T10:31:17.119269+0000","last_unstale":"2024-09-16T10:33:26.398917+0000","last_undegraded":"2024-09-16T10:33:26.398917+0000","last_fullsized":"2024-09-16T10:33:26.398917+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:17.363464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:17.363464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.071616+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":1101,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826954+0000","last_change":"2024-09-16T10:32:20.468825+0000","last_active":"2024-09-16T10:33:27.826954+0000","last_peered":"2024-09-16T10:33:27.826954+0000","last_clean":"2024-09-16T10:33:27.826954+0000","last_became_active":"2024-09-16T10:32:20.417250+0000","last_became_peered":"2024-09-16T10:32:20.417250+0000","last_unstale":"2024-09-16T10:33:27.826954+0000","last_undegraded":"2024-09-16T10:33:27.826954+0000","last_fullsized":"2024-09-16T10:33:27.826954+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:30:19.209335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:19.209335+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1080,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407315+0000","last_change":"2024-09-16T10:32:06.250968+0000","last_active":"2024-09-16T10:33:27.407315+0000","last_peered":"2024-09-16T10:33:27.407315+0000","last_clean":"2024-09-16T10:33:27.407315+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:33:27.407315+0000","last_undegraded":"2024-09-16T10:33:27.407315+0000","last_fullsized":"2024-09-16T10:33:27.407315+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:06.250899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.250899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":598,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.726416+0000","last_change":"2024-09-16T10:32:12.023979+0000","last_active":"2024-09-16T10:33:26.726416+0000","last_peered":"2024-09-16T10:33:26.726416+0000","last_clean":"2024-09-16T10:33:26.726416+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:33:26.726416+0000","last_undegraded":"2024-09-16T10:33:26.726416+0000","last_fullsized":"2024-09-16T10:33:26.726416+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:12.023740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.023740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:31.931461+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":1111,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825778+0000","last_change":"2024-09-16T10:31:14.161646+0000","last_active":"2024-09-16T10:33:27.825778+0000","last_peered":"2024-09-16T10:33:27.825778+0000","last_clean":"2024-09-16T10:33:27.825778+0000","last_became_active":"2024-09-16T10:31:14.152023+0000","last_became_peered":"2024-09-16T10:31:14.152023+0000","last_unstale":"2024-09-16T10:33:27.825778+0000","last_undegraded":"2024-09-16T10:33:27.825778+0000","last_fullsized":"2024-09-16T10:33:27.825778+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:05.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.174161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":1028,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658388+0000","last_change":"2024-09-16T10:30:42.550612+0000","last_active":"2024-09-16T10:33:26.658388+0000","last_peered":"2024-09-16T10:33:26.658388+0000","last_clean":"2024-09-16T10:33:26.658388+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:33:26.658388+0000","last_undegraded":"2024-09-16T10:33:26.658388+0000","last_fullsized":"2024-09-16T10:33:26.658388+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:42.550567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.550567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":1188,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657572+0000","last_change":"2024-09-16T10:32:20.470230+0000","last_active":"2024-09-16T10:33:26.657572+0000","last_peered":"2024-09-16T10:33:26.657572+0000","last_clean":"2024-09-16T10:33:26.657572+0000","last_became_active":"2024-09-16T10:32:20.412893+0000","last_became_peered":"2024-09-16T10:32:20.412893+0000","last_unstale":"2024-09-16T10:33:26.657572+0000","last_undegraded":"2024-09-16T10:33:26.657572+0000","last_fullsized":"2024-09-16T10:33:26.657572+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:41.740678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":468,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406980+0000","last_change":"2024-09-16T10:31:14.148058+0000","last_active":"2024-09-16T10:33:27.406980+0000","last_peered":"2024-09-16T10:33:27.406980+0000","last_clean":"2024-09-16T10:33:27.406980+0000","last_became_active":"2024-09-16T10:31:14.147606+0000","last_became_peered":"2024-09-16T10:31:14.147606+0000","last_unstale":"2024-09-16T10:33:27.406980+0000","last_undegraded":"2024-09-16T10:33:27.406980+0000","last_fullsized":"2024-09-16T10:33:27.406980+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.745214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.745214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":598,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.726285+0000","last_change":"2024-09-16T10:32:18.059677+0000","last_active":"2024-09-16T10:33:26.726285+0000","last_peered":"2024-09-16T10:33:26.726285+0000","last_clean":"2024-09-16T10:33:26.726285+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:33:26.726285+0000","last_undegraded":"2024-09-16T10:33:26.726285+0000","last_fullsized":"2024-09-16T10:33:26.726285+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:18.059602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:18.059602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:35.489290+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":944,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725245+0000","last_change":"2024-09-16T10:31:14.148903+0000","last_active":"2024-09-16T10:33:26.725245+0000","last_peered":"2024-09-16T10:33:26.725245+0000","last_clean":"2024-09-16T10:33:26.725245+0000","last_became_active":"2024-09-16T10:31:14.148572+0000","last_became_peered":"2024-09-16T10:31:14.148572+0000","last_unstale":"2024-09-16T10:33:26.725245+0000","last_undegraded":"2024-09-16T10:33:26.725245+0000","last_fullsized":"2024-09-16T10:33:26.725245+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.626720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.626720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":979,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393943+0000","last_change":"2024-09-16T10:32:06.364289+0000","last_active":"2024-09-16T10:33:26.393943+0000","last_peered":"2024-09-16T10:33:26.393943+0000","last_clean":"2024-09-16T10:33:26.393943+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:33:26.393943+0000","last_undegraded":"2024-09-16T10:33:26.393943+0000","last_fullsized":"2024-09-16T10:33:26.393943+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:06.364207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.364207+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":678,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.394020+0000","last_change":"2024-09-16T10:32:20.466527+0000","last_active":"2024-09-16T10:33:26.394020+0000","last_peered":"2024-09-16T10:33:26.394020+0000","last_clean":"2024-09-16T10:33:26.394020+0000","last_became_active":"2024-09-16T10:32:20.431532+0000","last_became_peered":"2024-09-16T10:32:20.431532+0000","last_unstale":"2024-09-16T10:33:26.394020+0000","last_undegraded":"2024-09-16T10:33:26.394020+0000","last_fullsized":"2024-09-16T10:33:26.394020+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:30:20.961894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.961894+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":68,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.827423+0000","last_change":"2024-09-16T10:32:20.390993+0000","last_active":"2024-09-16T10:33:27.827423+0000","last_peered":"2024-09-16T10:33:27.827423+0000","last_clean":"2024-09-16T10:33:27.827423+0000","last_became_active":"2024-09-16T10:32:20.390515+0000","last_became_peered":"2024-09-16T10:32:20.390515+0000","last_unstale":"2024-09-16T10:33:27.827423+0000","last_undegraded":"2024-09-16T10:33:27.827423+0000","last_fullsized":"2024-09-16T10:33:27.827423+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.911596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":613,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725268+0000","last_change":"2024-09-16T10:33:13.716266+0000","last_active":"2024-09-16T10:33:26.725268+0000","last_peered":"2024-09-16T10:33:26.725268+0000","last_clean":"2024-09-16T10:33:26.725268+0000","last_became_active":"2024-09-16T10:32:20.418945+0000","last_became_peered":"2024-09-16T10:32:20.418945+0000","last_unstale":"2024-09-16T10:33:26.725268+0000","last_undegraded":"2024-09-16T10:33:26.725268+0000","last_fullsized":"2024-09-16T10:33:26.725268+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:13.715922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:13.715922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:16.525289+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1083,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.395359+0000","last_change":"2024-09-16T10:32:20.433825+0000","last_active":"2024-09-16T10:33:26.395359+0000","last_peered":"2024-09-16T10:33:26.395359+0000","last_clean":"2024-09-16T10:33:26.395359+0000","last_became_active":"2024-09-16T10:32:20.423737+0000","last_became_peered":"2024-09-16T10:32:20.423737+0000","last_unstale":"2024-09-16T10:33:26.395359+0000","last_undegraded":"2024-09-16T10:33:26.395359+0000","last_fullsized":"2024-09-16T10:33:26.395359+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.230039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.230039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":756,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725481+0000","last_change":"2024-09-16T10:32:23.425365+0000","last_active":"2024-09-16T10:33:26.725481+0000","last_peered":"2024-09-16T10:33:26.725481+0000","last_clean":"2024-09-16T10:33:26.725481+0000","last_became_active":"2024-09-16T10:32:23.424848+0000","last_became_peered":"2024-09-16T10:32:23.424848+0000","last_unstale":"2024-09-16T10:33:26.725481+0000","last_undegraded":"2024-09-16T10:33:26.725481+0000","last_fullsized":"2024-09-16T10:33:26.725481+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:46.291175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":1130,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406360+0000","last_change":"2024-09-16T10:31:14.162215+0000","last_active":"2024-09-16T10:33:27.406360+0000","last_peered":"2024-09-16T10:33:27.406360+0000","last_clean":"2024-09-16T10:33:27.406360+0000","last_became_active":"2024-09-16T10:31:14.136662+0000","last_became_peered":"2024-09-16T10:31:14.136662+0000","last_unstale":"2024-09-16T10:33:27.406360+0000","last_undegraded":"2024-09-16T10:33:27.406360+0000","last_fullsized":"2024-09-16T10:33:27.406360+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.825936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.825936+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":427,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.831189+0000","last_change":"2024-09-16T10:32:20.422676+0000","last_active":"2024-09-16T10:33:27.831189+0000","last_peered":"2024-09-16T10:33:27.831189+0000","last_clean":"2024-09-16T10:33:27.831189+0000","last_became_active":"2024-09-16T10:32:20.391995+0000","last_became_peered":"2024-09-16T10:32:20.391995+0000","last_unstale":"2024-09-16T10:33:27.831189+0000","last_undegraded":"2024-09-16T10:33:27.831189+0000","last_fullsized":"2024-09-16T10:33:27.831189+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:32:09.730242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.730242+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"536'13","reported_seq":613,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.660218+0000","last_change":"2024-09-16T10:32:20.449596+0000","last_active":"2024-09-16T10:33:26.660218+0000","last_peered":"2024-09-16T10:33:26.660218+0000","last_clean":"2024-09-16T10:33:26.660218+0000","last_became_active":"2024-09-16T10:32:20.398138+0000","last_became_peered":"2024-09-16T10:32:20.398138+0000","last_unstale":"2024-09-16T10:33:26.660218+0000","last_undegraded":"2024-09-16T10:33:26.660218+0000","last_fullsized":"2024-09-16T10:33:26.660218+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'11","last_scrub_stamp":"2024-09-16T10:30:53.694883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.694883+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1069,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.405746+0000","last_change":"2024-09-16T10:30:07.792293+0000","last_active":"2024-09-16T10:33:27.405746+0000","last_peered":"2024-09-16T10:33:27.405746+0000","last_clean":"2024-09-16T10:33:27.405746+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:33:27.405746+0000","last_undegraded":"2024-09-16T10:33:27.405746+0000","last_fullsized":"2024-09-16T10:33:27.405746+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.792060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.792060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":661,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406291+0000","last_change":"2024-09-16T10:32:23.427047+0000","last_active":"2024-09-16T10:33:27.406291+0000","last_peered":"2024-09-16T10:33:27.406291+0000","last_clean":"2024-09-16T10:33:27.406291+0000","last_became_active":"2024-09-16T10:32:23.426646+0000","last_became_peered":"2024-09-16T10:32:23.426646+0000","last_unstale":"2024-09-16T10:33:27.406291+0000","last_undegraded":"2024-09-16T10:33:27.406291+0000","last_fullsized":"2024-09-16T10:33:27.406291+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:09.269989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.269989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1055,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657173+0000","last_change":"2024-09-16T10:33:07.769547+0000","last_active":"2024-09-16T10:33:26.657173+0000","last_peered":"2024-09-16T10:33:26.657173+0000","last_clean":"2024-09-16T10:33:26.657173+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:33:26.657173+0000","last_undegraded":"2024-09-16T10:33:26.657173+0000","last_fullsized":"2024-09-16T10:33:26.657173+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:07.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:07.769371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:15.987044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":1117,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393345+0000","last_change":"2024-09-16T10:32:20.455088+0000","last_active":"2024-09-16T10:33:26.393345+0000","last_peered":"2024-09-16T10:33:26.393345+0000","last_clean":"2024-09-16T10:33:26.393345+0000","last_became_active":"2024-09-16T10:32:20.425287+0000","last_became_peered":"2024-09-16T10:32:20.425287+0000","last_unstale":"2024-09-16T10:33:26.393345+0000","last_undegraded":"2024-09-16T10:33:26.393345+0000","last_fullsized":"2024-09-16T10:33:26.393345+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:30:50.226333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.226333+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1055,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406975+0000","last_change":"2024-09-16T10:32:20.474305+0000","last_active":"2024-09-16T10:33:27.406975+0000","last_peered":"2024-09-16T10:33:27.406975+0000","last_clean":"2024-09-16T10:33:27.406975+0000","last_became_active":"2024-09-16T10:32:20.473984+0000","last_became_peered":"2024-09-16T10:32:20.473984+0000","last_unstale":"2024-09-16T10:33:27.406975+0000","last_undegraded":"2024-09-16T10:33:27.406975+0000","last_fullsized":"2024-09-16T10:33:27.406975+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:25.381252+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":744,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.659002+0000","last_change":"2024-09-16T10:31:14.137702+0000","last_active":"2024-09-16T10:33:26.659002+0000","last_peered":"2024-09-16T10:33:26.659002+0000","last_clean":"2024-09-16T10:33:26.659002+0000","last_became_active":"2024-09-16T10:31:14.136703+0000","last_became_peered":"2024-09-16T10:31:14.136703+0000","last_unstale":"2024-09-16T10:33:26.659002+0000","last_undegraded":"2024-09-16T10:33:26.659002+0000","last_fullsized":"2024-09-16T10:33:26.659002+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:39.865342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.865342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":525,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.395706+0000","last_change":"2024-09-16T10:32:55.165175+0000","last_active":"2024-09-16T10:33:26.395706+0000","last_peered":"2024-09-16T10:33:26.395706+0000","last_clean":"2024-09-16T10:33:26.395706+0000","last_became_active":"2024-09-16T10:32:23.433435+0000","last_became_peered":"2024-09-16T10:32:23.433435+0000","last_unstale":"2024-09-16T10:33:26.395706+0000","last_undegraded":"2024-09-16T10:33:26.395706+0000","last_fullsized":"2024-09-16T10:33:26.395706+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:55.165009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:55.165009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:12.152556+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":1112,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.829213+0000","last_change":"2024-09-16T10:31:14.165639+0000","last_active":"2024-09-16T10:33:27.829213+0000","last_peered":"2024-09-16T10:33:27.829213+0000","last_clean":"2024-09-16T10:33:27.829213+0000","last_became_active":"2024-09-16T10:31:14.151744+0000","last_became_peered":"2024-09-16T10:31:14.151744+0000","last_unstale":"2024-09-16T10:33:27.829213+0000","last_undegraded":"2024-09-16T10:33:27.829213+0000","last_fullsized":"2024-09-16T10:33:27.829213+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:07.157828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.157828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:25.317922+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":408,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.660538+0000","last_change":"2024-09-16T10:31:14.137225+0000","last_active":"2024-09-16T10:33:26.660538+0000","last_peered":"2024-09-16T10:33:26.660538+0000","last_clean":"2024-09-16T10:33:26.660538+0000","last_became_active":"2024-09-16T10:31:14.132504+0000","last_became_peered":"2024-09-16T10:31:14.132504+0000","last_unstale":"2024-09-16T10:33:26.660538+0000","last_undegraded":"2024-09-16T10:33:26.660538+0000","last_fullsized":"2024-09-16T10:33:26.660538+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:44.598219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:44.598219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1086,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.393132+0000","last_change":"2024-09-16T10:32:20.425604+0000","last_active":"2024-09-16T10:33:26.393132+0000","last_peered":"2024-09-16T10:33:26.393132+0000","last_clean":"2024-09-16T10:33:26.393132+0000","last_became_active":"2024-09-16T10:32:20.424781+0000","last_became_peered":"2024-09-16T10:32:20.424781+0000","last_unstale":"2024-09-16T10:33:26.393132+0000","last_undegraded":"2024-09-16T10:33:26.393132+0000","last_fullsized":"2024-09-16T10:33:26.393132+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:20.058297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.058297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":615,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.727464+0000","last_change":"2024-09-16T10:31:38.879647+0000","last_active":"2024-09-16T10:33:26.727464+0000","last_peered":"2024-09-16T10:33:26.727464+0000","last_clean":"2024-09-16T10:33:26.727464+0000","last_became_active":"2024-09-16T10:31:14.140004+0000","last_became_peered":"2024-09-16T10:31:14.140004+0000","last_unstale":"2024-09-16T10:33:26.727464+0000","last_undegraded":"2024-09-16T10:33:26.727464+0000","last_fullsized":"2024-09-16T10:33:26.727464+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:38.879579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:38.879579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":667,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.405948+0000","last_change":"2024-09-16T10:32:46.160022+0000","last_active":"2024-09-16T10:33:27.405948+0000","last_peered":"2024-09-16T10:33:27.405948+0000","last_clean":"2024-09-16T10:33:27.405948+0000","last_became_active":"2024-09-16T10:32:46.159664+0000","last_became_peered":"2024-09-16T10:32:46.159664+0000","last_unstale":"2024-09-16T10:33:27.405948+0000","last_undegraded":"2024-09-16T10:33:27.405948+0000","last_fullsized":"2024-09-16T10:33:27.405948+0000","mapping_epoch":546,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:51.390508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":1105,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407576+0000","last_change":"2024-09-16T10:33:06.436155+0000","last_active":"2024-09-16T10:33:27.407576+0000","last_peered":"2024-09-16T10:33:27.407576+0000","last_clean":"2024-09-16T10:33:27.407576+0000","last_became_active":"2024-09-16T10:31:14.149019+0000","last_became_peered":"2024-09-16T10:31:14.149019+0000","last_unstale":"2024-09-16T10:33:27.407576+0000","last_undegraded":"2024-09-16T10:33:27.407576+0000","last_fullsized":"2024-09-16T10:33:27.407576+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:33:06.436076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.436076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:19.257155+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1096,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.826356+0000","last_change":"2024-09-16T10:33:18.297728+0000","last_active":"2024-09-16T10:33:27.826356+0000","last_peered":"2024-09-16T10:33:27.826356+0000","last_clean":"2024-09-16T10:33:27.826356+0000","last_became_active":"2024-09-16T10:32:20.416989+0000","last_became_peered":"2024-09-16T10:32:20.416989+0000","last_unstale":"2024-09-16T10:33:27.826356+0000","last_undegraded":"2024-09-16T10:33:27.826356+0000","last_fullsized":"2024-09-16T10:33:27.826356+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:18.297660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:18.297660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:35.219614+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"536'14","reported_seq":1374,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.399622+0000","last_change":"2024-09-16T10:32:20.493322+0000","last_active":"2024-09-16T10:33:26.399622+0000","last_peered":"2024-09-16T10:33:26.399622+0000","last_clean":"2024-09-16T10:33:26.399622+0000","last_became_active":"2024-09-16T10:32:20.489447+0000","last_became_peered":"2024-09-16T10:32:20.489447+0000","last_unstale":"2024-09-16T10:33:26.399622+0000","last_undegraded":"2024-09-16T10:33:26.399622+0000","last_fullsized":"2024-09-16T10:33:26.399622+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:30:42.170419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.170419+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":862,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.726886+0000","last_change":"2024-09-16T10:32:16.061021+0000","last_active":"2024-09-16T10:33:26.726886+0000","last_peered":"2024-09-16T10:33:26.726886+0000","last_clean":"2024-09-16T10:33:26.726886+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:33:26.726886+0000","last_undegraded":"2024-09-16T10:33:26.726886+0000","last_fullsized":"2024-09-16T10:33:26.726886+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:16.060965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:16.060965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":633,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825551+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:33:27.825551+0000","last_peered":"2024-09-16T10:33:27.825551+0000","last_clean":"2024-09-16T10:33:27.825551+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:33:27.825551+0000","last_undegraded":"2024-09-16T10:33:27.825551+0000","last_fullsized":"2024-09-16T10:33:27.825551+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:44.949384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":1153,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.394172+0000","last_change":"2024-09-16T10:33:05.157157+0000","last_active":"2024-09-16T10:33:26.394172+0000","last_peered":"2024-09-16T10:33:26.394172+0000","last_clean":"2024-09-16T10:33:26.394172+0000","last_became_active":"2024-09-16T10:32:20.488822+0000","last_became_peered":"2024-09-16T10:32:20.488822+0000","last_unstale":"2024-09-16T10:33:26.394172+0000","last_undegraded":"2024-09-16T10:33:26.394172+0000","last_fullsized":"2024-09-16T10:33:26.394172+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:05.157106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.157106+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:06.420060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":832,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.726199+0000","last_change":"2024-09-16T10:31:34.935251+0000","last_active":"2024-09-16T10:33:26.726199+0000","last_peered":"2024-09-16T10:33:26.726199+0000","last_clean":"2024-09-16T10:33:26.726199+0000","last_became_active":"2024-09-16T10:31:14.132063+0000","last_became_peered":"2024-09-16T10:31:14.132063+0000","last_unstale":"2024-09-16T10:33:26.726199+0000","last_undegraded":"2024-09-16T10:33:26.726199+0000","last_fullsized":"2024-09-16T10:33:26.726199+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:34.935185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:34.935185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"536'9","reported_seq":1322,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406610+0000","last_change":"2024-09-16T10:32:20.449280+0000","last_active":"2024-09-16T10:33:27.406610+0000","last_peered":"2024-09-16T10:33:27.406610+0000","last_clean":"2024-09-16T10:33:27.406610+0000","last_became_active":"2024-09-16T10:32:20.436067+0000","last_became_peered":"2024-09-16T10:32:20.436067+0000","last_unstale":"2024-09-16T10:33:27.406610+0000","last_undegraded":"2024-09-16T10:33:27.406610+0000","last_fullsized":"2024-09-16T10:33:27.406610+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:56.800991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:56.800991+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1069,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.827024+0000","last_change":"2024-09-16T10:33:05.136554+0000","last_active":"2024-09-16T10:33:27.827024+0000","last_peered":"2024-09-16T10:33:27.827024+0000","last_clean":"2024-09-16T10:33:27.827024+0000","last_became_active":"2024-09-16T10:31:14.150453+0000","last_became_peered":"2024-09-16T10:31:14.150453+0000","last_unstale":"2024-09-16T10:33:27.827024+0000","last_undegraded":"2024-09-16T10:33:27.827024+0000","last_fullsized":"2024-09-16T10:33:27.827024+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:05.136512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.136512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:09.897407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":1020,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658828+0000","last_change":"2024-09-16T10:32:20.399950+0000","last_active":"2024-09-16T10:33:26.658828+0000","last_peered":"2024-09-16T10:33:26.658828+0000","last_clean":"2024-09-16T10:33:26.658828+0000","last_became_active":"2024-09-16T10:32:20.390257+0000","last_became_peered":"2024-09-16T10:32:20.390257+0000","last_unstale":"2024-09-16T10:33:26.658828+0000","last_undegraded":"2024-09-16T10:33:26.658828+0000","last_fullsized":"2024-09-16T10:33:26.658828+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:32:09.683973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.683973+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":480,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658647+0000","last_change":"2024-09-16T10:32:20.406554+0000","last_active":"2024-09-16T10:33:26.658647+0000","last_peered":"2024-09-16T10:33:26.658647+0000","last_clean":"2024-09-16T10:33:26.658647+0000","last_became_active":"2024-09-16T10:32:20.405826+0000","last_became_peered":"2024-09-16T10:32:20.405826+0000","last_unstale":"2024-09-16T10:33:26.658647+0000","last_undegraded":"2024-09-16T10:33:26.658647+0000","last_fullsized":"2024-09-16T10:33:26.658647+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.216239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.216239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":65,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406228+0000","last_change":"2024-09-16T10:32:20.434897+0000","last_active":"2024-09-16T10:33:27.406228+0000","last_peered":"2024-09-16T10:33:27.406228+0000","last_clean":"2024-09-16T10:33:27.406228+0000","last_became_active":"2024-09-16T10:32:20.434526+0000","last_became_peered":"2024-09-16T10:32:20.434526+0000","last_unstale":"2024-09-16T10:33:27.406228+0000","last_undegraded":"2024-09-16T10:33:27.406228+0000","last_fullsized":"2024-09-16T10:33:27.406228+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:11.168381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.168381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1050,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.394978+0000","last_change":"2024-09-16T10:32:10.448088+0000","last_active":"2024-09-16T10:33:26.394978+0000","last_peered":"2024-09-16T10:33:26.394978+0000","last_clean":"2024-09-16T10:33:26.394978+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:33:26.394978+0000","last_undegraded":"2024-09-16T10:33:26.394978+0000","last_fullsized":"2024-09-16T10:33:26.394978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:10.447999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:10.447999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":704,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657984+0000","last_change":"2024-09-16T10:32:20.464025+0000","last_active":"2024-09-16T10:33:26.657984+0000","last_peered":"2024-09-16T10:33:26.657984+0000","last_clean":"2024-09-16T10:33:26.657984+0000","last_became_active":"2024-09-16T10:32:20.413318+0000","last_became_peered":"2024-09-16T10:32:20.413318+0000","last_unstale":"2024-09-16T10:33:26.657984+0000","last_undegraded":"2024-09-16T10:33:26.657984+0000","last_fullsized":"2024-09-16T10:33:26.657984+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:43.705811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":467,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.394870+0000","last_change":"2024-09-16T10:31:14.144963+0000","last_active":"2024-09-16T10:33:26.394870+0000","last_peered":"2024-09-16T10:33:26.394870+0000","last_clean":"2024-09-16T10:33:26.394870+0000","last_became_active":"2024-09-16T10:31:14.144634+0000","last_became_peered":"2024-09-16T10:31:14.144634+0000","last_unstale":"2024-09-16T10:33:26.394870+0000","last_undegraded":"2024-09-16T10:33:26.394870+0000","last_fullsized":"2024-09-16T10:33:26.394870+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1018,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.395510+0000","last_change":"2024-09-16T10:32:20.415681+0000","last_active":"2024-09-16T10:33:26.395510+0000","last_peered":"2024-09-16T10:33:26.395510+0000","last_clean":"2024-09-16T10:33:26.395510+0000","last_became_active":"2024-09-16T10:32:20.415323+0000","last_became_peered":"2024-09-16T10:32:20.415323+0000","last_unstale":"2024-09-16T10:33:26.395510+0000","last_undegraded":"2024-09-16T10:33:26.395510+0000","last_fullsized":"2024-09-16T10:33:26.395510+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.144732+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1042,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.825646+0000","last_change":"2024-09-16T10:32:20.402165+0000","last_active":"2024-09-16T10:33:27.825646+0000","last_peered":"2024-09-16T10:33:27.825646+0000","last_clean":"2024-09-16T10:33:27.825646+0000","last_became_active":"2024-09-16T10:32:20.396229+0000","last_became_peered":"2024-09-16T10:32:20.396229+0000","last_unstale":"2024-09-16T10:33:27.825646+0000","last_undegraded":"2024-09-16T10:33:27.825646+0000","last_fullsized":"2024-09-16T10:33:27.825646+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.389262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.389262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":1015,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725966+0000","last_change":"2024-09-16T10:32:23.440126+0000","last_active":"2024-09-16T10:33:26.725966+0000","last_peered":"2024-09-16T10:33:26.725966+0000","last_clean":"2024-09-16T10:33:26.725966+0000","last_became_active":"2024-09-16T10:32:23.439740+0000","last_became_peered":"2024-09-16T10:32:23.439740+0000","last_unstale":"2024-09-16T10:33:26.725966+0000","last_undegraded":"2024-09-16T10:33:26.725966+0000","last_fullsized":"2024-09-16T10:33:26.725966+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:31:31.897681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:31:31.897681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":1095,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.394706+0000","last_change":"2024-09-16T10:31:14.175984+0000","last_active":"2024-09-16T10:33:26.394706+0000","last_peered":"2024-09-16T10:33:26.394706+0000","last_clean":"2024-09-16T10:33:26.394706+0000","last_became_active":"2024-09-16T10:31:14.154997+0000","last_became_peered":"2024-09-16T10:31:14.154997+0000","last_unstale":"2024-09-16T10:33:26.394706+0000","last_undegraded":"2024-09-16T10:33:26.394706+0000","last_fullsized":"2024-09-16T10:33:26.394706+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:25.298353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1097,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.407120+0000","last_change":"2024-09-16T10:32:20.474361+0000","last_active":"2024-09-16T10:33:27.407120+0000","last_peered":"2024-09-16T10:33:27.407120+0000","last_clean":"2024-09-16T10:33:27.407120+0000","last_became_active":"2024-09-16T10:32:20.474067+0000","last_became_peered":"2024-09-16T10:32:20.474067+0000","last_unstale":"2024-09-16T10:33:27.407120+0000","last_undegraded":"2024-09-16T10:33:27.407120+0000","last_fullsized":"2024-09-16T10:33:27.407120+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:07.250964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:07.250964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":756,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.657564+0000","last_change":"2024-09-16T10:32:13.704695+0000","last_active":"2024-09-16T10:33:26.657564+0000","last_peered":"2024-09-16T10:33:26.657564+0000","last_clean":"2024-09-16T10:33:26.657564+0000","last_became_active":"2024-09-16T10:31:14.132459+0000","last_became_peered":"2024-09-16T10:31:14.132459+0000","last_unstale":"2024-09-16T10:33:26.657564+0000","last_undegraded":"2024-09-16T10:33:26.657564+0000","last_fullsized":"2024-09-16T10:33:26.657564+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:13.704652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:13.704652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:36.717155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1066,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.395905+0000","last_change":"2024-09-16T10:30:45.211710+0000","last_active":"2024-09-16T10:33:26.395905+0000","last_peered":"2024-09-16T10:33:26.395905+0000","last_clean":"2024-09-16T10:33:26.395905+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:33:26.395905+0000","last_undegraded":"2024-09-16T10:33:26.395905+0000","last_fullsized":"2024-09-16T10:33:26.395905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.211638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.211638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":1130,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.397498+0000","last_change":"2024-09-16T10:32:20.445935+0000","last_active":"2024-09-16T10:33:26.397498+0000","last_peered":"2024-09-16T10:33:26.397498+0000","last_clean":"2024-09-16T10:33:26.397498+0000","last_became_active":"2024-09-16T10:32:20.396707+0000","last_became_peered":"2024-09-16T10:32:20.396707+0000","last_unstale":"2024-09-16T10:33:26.397498+0000","last_undegraded":"2024-09-16T10:33:26.397498+0000","last_fullsized":"2024-09-16T10:33:26.397498+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:32:12.365003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.365003+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:36.671931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":1061,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.831399+0000","last_change":"2024-09-16T10:30:21.275187+0000","last_active":"2024-09-16T10:33:27.831399+0000","last_peered":"2024-09-16T10:33:27.831399+0000","last_clean":"2024-09-16T10:33:27.831399+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:33:27.831399+0000","last_undegraded":"2024-09-16T10:33:27.831399+0000","last_fullsized":"2024-09-16T10:33:27.831399+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:30:21.275103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:21.275103+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1081,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.660345+0000","last_change":"2024-09-16T10:31:14.139140+0000","last_active":"2024-09-16T10:33:26.660345+0000","last_peered":"2024-09-16T10:33:26.660345+0000","last_clean":"2024-09-16T10:33:26.660345+0000","last_became_active":"2024-09-16T10:31:14.138772+0000","last_became_peered":"2024-09-16T10:31:14.138772+0000","last_unstale":"2024-09-16T10:33:26.660345+0000","last_undegraded":"2024-09-16T10:33:26.660345+0000","last_fullsized":"2024-09-16T10:33:26.660345+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.649610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.649610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":699,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.725385+0000","last_change":"2024-09-16T10:31:40.861638+0000","last_active":"2024-09-16T10:33:26.725385+0000","last_peered":"2024-09-16T10:33:26.725385+0000","last_clean":"2024-09-16T10:33:26.725385+0000","last_became_active":"2024-09-16T10:31:14.152053+0000","last_became_peered":"2024-09-16T10:31:14.152053+0000","last_unstale":"2024-09-16T10:33:26.725385+0000","last_undegraded":"2024-09-16T10:33:26.725385+0000","last_fullsized":"2024-09-16T10:33:26.725385+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:40.861570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:40.861570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":66,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.406902+0000","last_change":"2024-09-16T10:32:20.401586+0000","last_active":"2024-09-16T10:33:27.406902+0000","last_peered":"2024-09-16T10:33:27.406902+0000","last_clean":"2024-09-16T10:33:27.406902+0000","last_became_active":"2024-09-16T10:32:20.401249+0000","last_became_peered":"2024-09-16T10:32:20.401249+0000","last_unstale":"2024-09-16T10:33:27.406902+0000","last_undegraded":"2024-09-16T10:33:27.406902+0000","last_fullsized":"2024-09-16T10:33:27.406902+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.832904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.832904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:27.404759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1077,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.399010+0000","last_change":"2024-09-16T10:32:20.428692+0000","last_active":"2024-09-16T10:33:26.399010+0000","last_peered":"2024-09-16T10:33:26.399010+0000","last_clean":"2024-09-16T10:33:26.399010+0000","last_became_active":"2024-09-16T10:32:20.428421+0000","last_became_peered":"2024-09-16T10:32:20.428421+0000","last_unstale":"2024-09-16T10:33:26.399010+0000","last_undegraded":"2024-09-16T10:33:26.399010+0000","last_fullsized":"2024-09-16T10:33:26.399010+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:15.364448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:15.364448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":1134,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:26.658534+0000","last_change":"2024-09-16T10:33:00.718946+0000","last_active":"2024-09-16T10:33:26.658534+0000","last_peered":"2024-09-16T10:33:26.658534+0000","last_clean":"2024-09-16T10:33:26.658534+0000","last_became_active":"2024-09-16T10:32:20.395308+0000","last_became_peered":"2024-09-16T10:32:20.395308+0000","last_unstale":"2024-09-16T10:33:26.658534+0000","last_undegraded":"2024-09-16T10:33:26.658534+0000","last_fullsized":"2024-09-16T10:33:26.658534+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:33:00.718864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:00.718864+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:18.336822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1095,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-16T10:33:27.827191+0000","last_change":"2024-09-16T10:32:20.414016+0000","last_active":"2024-09-16T10:33:27.827191+0000","last_peered":"2024-09-16T10:33:27.827191+0000","last_clean":"2024-09-16T10:33:27.827191+0000","last_became_active":"2024-09-16T10:32:20.413184+0000","last_became_peered":"2024-09-16T10:32:20.413184+0000","last_unstale":"2024-09-16T10:33:27.827191+0000","last_undegraded":"2024-09-16T10:33:27.827191+0000","last_fullsized":"2024-09-16T10:33:27.827191+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:54.399132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.399132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1110,"num_read_kb":867,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":6068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":5551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":1894080577598,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528566,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834856,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92910296,"statfs":{"total":95995035648,"available":95140143104,"internally_reserved":0,"allocated":1265664,"data_stored":814834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":853604806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790204,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776464,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92968688,"statfs":{"total":95995035648,"available":95199936512,"internally_reserved":0,"allocated":1171456,"data_stored":743424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986433,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681820,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063332,"statfs":{"total":95995035648,"available":95296851968,"internally_reserved":0,"allocated":925696,"data_stored":486805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215366,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917052,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828100,"statfs":{"total":95995035648,"available":95055974400,"internally_reserved":0,"allocated":1548288,"data_stored":1079367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":937490888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411596,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622352,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93122800,"statfs":{"total":95995035648,"available":95357747200,"internally_reserved":0,"allocated":1175552,"data_stored":749053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575119,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934816,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92810336,"statfs":{"total":95995035648,"available":95037784064,"internally_reserved":0,"allocated":1257472,"data_stored":815307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:33.790 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T10:33:33.790 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T10:33:33.790 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-16T10:33:33.790 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-16T10:33:33.791 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:33:34.025 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:33:34.026 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [], 2024-09-16T10:33:34.026 INFO:teuthology.orchestra.run.smithi063.stdout: "complaint_time": 30, 2024-09-16T10:33:34.026 INFO:teuthology.orchestra.run.smithi063.stdout: "num_blocked_ops": 0 2024-09-16T10:33:34.026 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:33:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:33 smithi165 ceph-mon[32696]: from='client.16905 v1:172.21.15.63:0/2442933523' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:33 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:33:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:33 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:33:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:33 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:34.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T10:33:34.786 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:34.787 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:34.969 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:34 smithi063 ceph-mon[29993]: pgmap v1076: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2024-09-16T10:33:34.969 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:34 smithi063 ceph-mon[39470]: pgmap v1076: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2024-09-16T10:33:35.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:34 smithi165 ceph-mon[32696]: pgmap v1076: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2024-09-16T10:33:35.693 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:33:35.695 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:33:35.942 INFO:teuthology.orchestra.run.smithi063.stderr:noscrub is unset 2024-09-16T10:33:36.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:35 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2106607388' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:33:36.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:35 smithi063 ceph-mon[29993]: from='client.46751 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:33:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:35 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2106607388' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:33:36.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:35 smithi063 ceph-mon[39470]: from='client.46751 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:33:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:35 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2106607388' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:33:36.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:35 smithi165 ceph-mon[32696]: from='client.46751 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-16T10:33:36.709 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T10:33:37.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:33:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:33:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[29993]: pgmap v1077: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:33:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[29993]: from='client.46745 v1:172.21.15.63:0/1241333132' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[29993]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[29993]: from='client.46751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:33:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[29993]: osdmap e564: 8 total, 8 up, 6 in 2024-09-16T10:33:37.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[29993]: osdmap e565: 8 total, 8 up, 6 in 2024-09-16T10:33:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[39470]: pgmap v1077: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:33:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[39470]: from='client.46745 v1:172.21.15.63:0/1241333132' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[39470]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[39470]: from='client.46751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:33:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[39470]: osdmap e564: 8 total, 8 up, 6 in 2024-09-16T10:33:37.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:36 smithi063 ceph-mon[39470]: osdmap e565: 8 total, 8 up, 6 in 2024-09-16T10:33:37.207 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":1077,"stamp":"2024-09-16T10:33:35.537194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1198,"num_read_kb":950,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":533032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1024,"ondisk_log_size":1024,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4767536,"kb_used_data":7332,"kb_used_omap":9,"kb_used_meta":4760054,"kb_avail":557703376,"statfs":{"total":575970213888,"available":571088257024,"internally_reserved":0,"allocated":7507968,"data_stored":4775835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":4874295970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001844"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1090,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135022+0000","last_change":"2024-09-16T10:33:17.292872+0000","last_active":"2024-09-16T10:33:31.135022+0000","last_peered":"2024-09-16T10:33:31.135022+0000","last_clean":"2024-09-16T10:33:31.135022+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:33:31.135022+0000","last_undegraded":"2024-09-16T10:33:31.135022+0000","last_fullsized":"2024-09-16T10:33:31.135022+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:17.292796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:17.292796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:40.930967+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"327'15","reported_seq":1052,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260453+0000","last_change":"2024-09-16T10:32:56.707159+0000","last_active":"2024-09-16T10:33:31.260453+0000","last_peered":"2024-09-16T10:33:31.260453+0000","last_clean":"2024-09-16T10:33:31.260453+0000","last_became_active":"2024-09-16T10:32:20.414426+0000","last_became_peered":"2024-09-16T10:32:20.414426+0000","last_unstale":"2024-09-16T10:33:31.260453+0000","last_undegraded":"2024-09-16T10:33:31.260453+0000","last_fullsized":"2024-09-16T10:33:31.260453+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:32:56.707086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:56.707086+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:01.126649+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1034,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807314+0000","last_change":"2024-09-16T10:32:20.401571+0000","last_active":"2024-09-16T10:33:31.807314+0000","last_peered":"2024-09-16T10:33:31.807314+0000","last_clean":"2024-09-16T10:33:31.807314+0000","last_became_active":"2024-09-16T10:32:20.401119+0000","last_became_peered":"2024-09-16T10:32:20.401119+0000","last_unstale":"2024-09-16T10:33:31.807314+0000","last_undegraded":"2024-09-16T10:33:31.807314+0000","last_fullsized":"2024-09-16T10:33:31.807314+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:08.872906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.872906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:29.429076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":398,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807126+0000","last_change":"2024-09-16T10:32:20.381952+0000","last_active":"2024-09-16T10:33:31.807126+0000","last_peered":"2024-09-16T10:33:31.807126+0000","last_clean":"2024-09-16T10:33:31.807126+0000","last_became_active":"2024-09-16T10:32:20.380629+0000","last_became_peered":"2024-09-16T10:32:20.380629+0000","last_unstale":"2024-09-16T10:33:31.807126+0000","last_undegraded":"2024-09-16T10:33:31.807126+0000","last_fullsized":"2024-09-16T10:33:31.807126+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:17.879610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:17.879610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1070,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807996+0000","last_change":"2024-09-16T10:32:20.435331+0000","last_active":"2024-09-16T10:33:31.807996+0000","last_peered":"2024-09-16T10:33:31.807996+0000","last_clean":"2024-09-16T10:33:31.807996+0000","last_became_active":"2024-09-16T10:32:20.433546+0000","last_became_peered":"2024-09-16T10:32:20.433546+0000","last_unstale":"2024-09-16T10:33:31.807996+0000","last_undegraded":"2024-09-16T10:33:31.807996+0000","last_fullsized":"2024-09-16T10:33:31.807996+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:21.916666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:21.916666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1089,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.809213+0000","last_change":"2024-09-16T10:30:48.140925+0000","last_active":"2024-09-16T10:33:31.809213+0000","last_peered":"2024-09-16T10:33:31.809213+0000","last_clean":"2024-09-16T10:33:31.809213+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:33:31.809213+0000","last_undegraded":"2024-09-16T10:33:31.809213+0000","last_fullsized":"2024-09-16T10:33:31.809213+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"328'27","reported_seq":1159,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807612+0000","last_change":"2024-09-16T10:30:15.998707+0000","last_active":"2024-09-16T10:33:31.807612+0000","last_peered":"2024-09-16T10:33:31.807612+0000","last_clean":"2024-09-16T10:33:31.807612+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:33:31.807612+0000","last_undegraded":"2024-09-16T10:33:31.807612+0000","last_fullsized":"2024-09-16T10:33:31.807612+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:15.998614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.998614+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":406,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260324+0000","last_change":"2024-09-16T10:33:02.705141+0000","last_active":"2024-09-16T10:33:31.260324+0000","last_peered":"2024-09-16T10:33:31.260324+0000","last_clean":"2024-09-16T10:33:31.260324+0000","last_became_active":"2024-09-16T10:32:20.392399+0000","last_became_peered":"2024-09-16T10:32:20.392399+0000","last_unstale":"2024-09-16T10:33:31.260324+0000","last_undegraded":"2024-09-16T10:33:31.260324+0000","last_fullsized":"2024-09-16T10:33:31.260324+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:02.705078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:02.705078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:18.211849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1099,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808088+0000","last_change":"2024-09-16T10:33:16.405006+0000","last_active":"2024-09-16T10:33:31.808088+0000","last_peered":"2024-09-16T10:33:31.808088+0000","last_clean":"2024-09-16T10:33:31.808088+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:33:31.808088+0000","last_undegraded":"2024-09-16T10:33:31.808088+0000","last_fullsized":"2024-09-16T10:33:31.808088+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:16.404954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:16.404954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:30.180621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"327'15","reported_seq":985,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808563+0000","last_change":"2024-09-16T10:29:35.742990+0000","last_active":"2024-09-16T10:33:31.808563+0000","last_peered":"2024-09-16T10:33:31.808563+0000","last_clean":"2024-09-16T10:33:31.808563+0000","last_became_active":"2024-09-16T10:26:28.550640+0000","last_became_peered":"2024-09-16T10:26:28.550640+0000","last_unstale":"2024-09-16T10:33:31.808563+0000","last_undegraded":"2024-09-16T10:33:31.808563+0000","last_fullsized":"2024-09-16T10:33:31.808563+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:35.742935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:35.742935+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1085,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807672+0000","last_change":"2024-09-16T10:31:14.144852+0000","last_active":"2024-09-16T10:33:31.807672+0000","last_peered":"2024-09-16T10:33:31.807672+0000","last_clean":"2024-09-16T10:33:31.807672+0000","last_became_active":"2024-09-16T10:31:14.144519+0000","last_became_peered":"2024-09-16T10:31:14.144519+0000","last_unstale":"2024-09-16T10:33:31.807672+0000","last_undegraded":"2024-09-16T10:33:31.807672+0000","last_fullsized":"2024-09-16T10:33:31.807672+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:41.762023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:41.762023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1088,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134267+0000","last_change":"2024-09-16T10:32:20.421334+0000","last_active":"2024-09-16T10:33:31.134267+0000","last_peered":"2024-09-16T10:33:31.134267+0000","last_clean":"2024-09-16T10:33:31.134267+0000","last_became_active":"2024-09-16T10:32:20.413663+0000","last_became_peered":"2024-09-16T10:32:20.413663+0000","last_unstale":"2024-09-16T10:33:31.134267+0000","last_undegraded":"2024-09-16T10:33:31.134267+0000","last_fullsized":"2024-09-16T10:33:31.134267+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:13.749174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:13.749174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:32.838025+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1050,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807935+0000","last_change":"2024-09-16T10:32:20.377023+0000","last_active":"2024-09-16T10:33:31.807935+0000","last_peered":"2024-09-16T10:33:31.807935+0000","last_clean":"2024-09-16T10:33:31.807935+0000","last_became_active":"2024-09-16T10:32:20.376673+0000","last_became_peered":"2024-09-16T10:32:20.376673+0000","last_unstale":"2024-09-16T10:33:31.807935+0000","last_undegraded":"2024-09-16T10:33:31.807935+0000","last_fullsized":"2024-09-16T10:33:31.807935+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:13.941032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:13.941032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"327'7","reported_seq":986,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808203+0000","last_change":"2024-09-16T10:32:20.470950+0000","last_active":"2024-09-16T10:33:31.808203+0000","last_peered":"2024-09-16T10:33:31.808203+0000","last_clean":"2024-09-16T10:33:31.808203+0000","last_became_active":"2024-09-16T10:32:20.434528+0000","last_became_peered":"2024-09-16T10:32:20.434528+0000","last_unstale":"2024-09-16T10:33:31.808203+0000","last_undegraded":"2024-09-16T10:33:31.808203+0000","last_fullsized":"2024-09-16T10:33:31.808203+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'7","last_scrub_stamp":"2024-09-16T10:30:39.082450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.082450+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":717,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135777+0000","last_change":"2024-09-16T10:32:20.408057+0000","last_active":"2024-09-16T10:33:31.135777+0000","last_peered":"2024-09-16T10:33:31.135777+0000","last_clean":"2024-09-16T10:33:31.135777+0000","last_became_active":"2024-09-16T10:32:20.403637+0000","last_became_peered":"2024-09-16T10:32:20.403637+0000","last_unstale":"2024-09-16T10:33:31.135777+0000","last_undegraded":"2024-09-16T10:33:31.135777+0000","last_fullsized":"2024-09-16T10:33:31.135777+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.995349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.995349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1086,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807381+0000","last_change":"2024-09-16T10:32:20.401456+0000","last_active":"2024-09-16T10:33:31.807381+0000","last_peered":"2024-09-16T10:33:31.807381+0000","last_clean":"2024-09-16T10:33:31.807381+0000","last_became_active":"2024-09-16T10:32:20.401126+0000","last_became_peered":"2024-09-16T10:32:20.401126+0000","last_unstale":"2024-09-16T10:33:31.807381+0000","last_undegraded":"2024-09-16T10:33:31.807381+0000","last_fullsized":"2024-09-16T10:33:31.807381+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.741362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.741362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":392,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261345+0000","last_change":"2024-09-16T10:32:11.660805+0000","last_active":"2024-09-16T10:33:31.261345+0000","last_peered":"2024-09-16T10:33:31.261345+0000","last_clean":"2024-09-16T10:33:31.261345+0000","last_became_active":"2024-09-16T10:26:28.555702+0000","last_became_peered":"2024-09-16T10:26:28.555702+0000","last_unstale":"2024-09-16T10:33:31.261345+0000","last_undegraded":"2024-09-16T10:33:31.261345+0000","last_fullsized":"2024-09-16T10:33:31.261345+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:11.660728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.660728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1083,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134191+0000","last_change":"2024-09-16T10:32:20.417609+0000","last_active":"2024-09-16T10:33:31.134191+0000","last_peered":"2024-09-16T10:33:31.134191+0000","last_clean":"2024-09-16T10:33:31.134191+0000","last_became_active":"2024-09-16T10:32:20.416791+0000","last_became_peered":"2024-09-16T10:32:20.416791+0000","last_unstale":"2024-09-16T10:33:31.134191+0000","last_undegraded":"2024-09-16T10:33:31.134191+0000","last_fullsized":"2024-09-16T10:33:31.134191+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.446934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.446934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"328'27","reported_seq":658,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807567+0000","last_change":"2024-09-16T10:32:20.499898+0000","last_active":"2024-09-16T10:33:31.807567+0000","last_peered":"2024-09-16T10:33:31.807567+0000","last_clean":"2024-09-16T10:33:31.807567+0000","last_became_active":"2024-09-16T10:32:20.490019+0000","last_became_peered":"2024-09-16T10:32:20.490019+0000","last_unstale":"2024-09-16T10:33:31.807567+0000","last_undegraded":"2024-09-16T10:33:31.807567+0000","last_fullsized":"2024-09-16T10:33:31.807567+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.599903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.599903+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:24.641849+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1059,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808592+0000","last_change":"2024-09-16T10:32:20.415199+0000","last_active":"2024-09-16T10:33:31.808592+0000","last_peered":"2024-09-16T10:33:31.808592+0000","last_clean":"2024-09-16T10:33:31.808592+0000","last_became_active":"2024-09-16T10:32:20.414861+0000","last_became_peered":"2024-09-16T10:32:20.414861+0000","last_unstale":"2024-09-16T10:33:31.808592+0000","last_undegraded":"2024-09-16T10:33:31.808592+0000","last_fullsized":"2024-09-16T10:33:31.808592+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:09.918371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:09.918371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1020,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808548+0000","last_change":"2024-09-16T10:33:08.153114+0000","last_active":"2024-09-16T10:33:31.808548+0000","last_peered":"2024-09-16T10:33:31.808548+0000","last_clean":"2024-09-16T10:33:31.808548+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:33:31.808548+0000","last_undegraded":"2024-09-16T10:33:31.808548+0000","last_fullsized":"2024-09-16T10:33:31.808548+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:08.153044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:08.153044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:22.272748+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":543,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806649+0000","last_change":"2024-09-16T10:32:20.424078+0000","last_active":"2024-09-16T10:33:31.806649+0000","last_peered":"2024-09-16T10:33:31.806649+0000","last_clean":"2024-09-16T10:33:31.806649+0000","last_became_active":"2024-09-16T10:32:20.423662+0000","last_became_peered":"2024-09-16T10:32:20.423662+0000","last_unstale":"2024-09-16T10:33:31.806649+0000","last_undegraded":"2024-09-16T10:33:31.806649+0000","last_fullsized":"2024-09-16T10:33:31.806649+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:09.147080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.147080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"328'18","reported_seq":1147,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808918+0000","last_change":"2024-09-16T10:32:20.461961+0000","last_active":"2024-09-16T10:33:31.808918+0000","last_peered":"2024-09-16T10:33:31.808918+0000","last_clean":"2024-09-16T10:33:31.808918+0000","last_became_active":"2024-09-16T10:32:20.429076+0000","last_became_peered":"2024-09-16T10:32:20.429076+0000","last_unstale":"2024-09-16T10:33:31.808918+0000","last_undegraded":"2024-09-16T10:33:31.808918+0000","last_fullsized":"2024-09-16T10:33:31.808918+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'18","last_scrub_stamp":"2024-09-16T10:30:38.072460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:38.072460+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":855,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808429+0000","last_change":"2024-09-16T10:26:28.553036+0000","last_active":"2024-09-16T10:33:31.808429+0000","last_peered":"2024-09-16T10:33:31.808429+0000","last_clean":"2024-09-16T10:33:31.808429+0000","last_became_active":"2024-09-16T10:26:28.551913+0000","last_became_peered":"2024-09-16T10:26:28.551913+0000","last_unstale":"2024-09-16T10:33:31.808429+0000","last_undegraded":"2024-09-16T10:33:31.808429+0000","last_fullsized":"2024-09-16T10:33:31.808429+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:09.384740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:09.384740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:48.236715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":489,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807367+0000","last_change":"2024-09-16T10:31:14.169546+0000","last_active":"2024-09-16T10:33:31.807367+0000","last_peered":"2024-09-16T10:33:31.807367+0000","last_clean":"2024-09-16T10:33:31.807367+0000","last_became_active":"2024-09-16T10:31:14.151297+0000","last_became_peered":"2024-09-16T10:31:14.151297+0000","last_unstale":"2024-09-16T10:33:31.807367+0000","last_undegraded":"2024-09-16T10:33:31.807367+0000","last_fullsized":"2024-09-16T10:33:31.807367+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:40.788922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:40.788922+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":85,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807718+0000","last_change":"2024-09-16T10:32:20.469065+0000","last_active":"2024-09-16T10:33:31.807718+0000","last_peered":"2024-09-16T10:33:31.807718+0000","last_clean":"2024-09-16T10:33:31.807718+0000","last_became_active":"2024-09-16T10:32:20.432046+0000","last_became_peered":"2024-09-16T10:32:20.432046+0000","last_unstale":"2024-09-16T10:33:31.807718+0000","last_undegraded":"2024-09-16T10:33:31.807718+0000","last_fullsized":"2024-09-16T10:33:31.807718+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:30:12.928274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.928274+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"328'20","reported_seq":1151,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134596+0000","last_change":"2024-09-16T10:33:14.278928+0000","last_active":"2024-09-16T10:33:31.134596+0000","last_peered":"2024-09-16T10:33:31.134596+0000","last_clean":"2024-09-16T10:33:31.134596+0000","last_became_active":"2024-09-16T10:32:20.407208+0000","last_became_peered":"2024-09-16T10:32:20.407208+0000","last_unstale":"2024-09-16T10:33:31.134596+0000","last_undegraded":"2024-09-16T10:33:31.134596+0000","last_fullsized":"2024-09-16T10:33:31.134596+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:14.278843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:14.278843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:33.769863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"508'11","reported_seq":1335,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261700+0000","last_change":"2024-09-16T10:31:14.167033+0000","last_active":"2024-09-16T10:33:31.261700+0000","last_peered":"2024-09-16T10:33:31.261700+0000","last_clean":"2024-09-16T10:33:31.261700+0000","last_became_active":"2024-09-16T10:31:14.137776+0000","last_became_peered":"2024-09-16T10:31:14.137776+0000","last_unstale":"2024-09-16T10:33:31.261700+0000","last_undegraded":"2024-09-16T10:33:31.261700+0000","last_fullsized":"2024-09-16T10:33:31.261700+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'10","last_scrub_stamp":"2024-09-16T10:29:37.281095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:29:37.281095+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.694030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":922,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807086+0000","last_change":"2024-09-16T10:32:16.263470+0000","last_active":"2024-09-16T10:33:31.807086+0000","last_peered":"2024-09-16T10:33:31.807086+0000","last_clean":"2024-09-16T10:33:31.807086+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:33:31.807086+0000","last_undegraded":"2024-09-16T10:33:31.807086+0000","last_fullsized":"2024-09-16T10:33:31.807086+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:16.263399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:16.263399+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"328'27","reported_seq":700,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808694+0000","last_change":"2024-09-16T10:33:06.135162+0000","last_active":"2024-09-16T10:33:31.808694+0000","last_peered":"2024-09-16T10:33:31.808694+0000","last_clean":"2024-09-16T10:33:31.808694+0000","last_became_active":"2024-09-16T10:32:20.414383+0000","last_became_peered":"2024-09-16T10:32:20.414383+0000","last_unstale":"2024-09-16T10:33:31.808694+0000","last_undegraded":"2024-09-16T10:33:31.808694+0000","last_fullsized":"2024-09-16T10:33:31.808694+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:33:06.135084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.135084+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:20.735523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1100,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807036+0000","last_change":"2024-09-16T10:32:20.421467+0000","last_active":"2024-09-16T10:33:31.807036+0000","last_peered":"2024-09-16T10:33:31.807036+0000","last_clean":"2024-09-16T10:33:31.807036+0000","last_became_active":"2024-09-16T10:32:20.417850+0000","last_became_peered":"2024-09-16T10:32:20.417850+0000","last_unstale":"2024-09-16T10:33:31.807036+0000","last_undegraded":"2024-09-16T10:33:31.807036+0000","last_fullsized":"2024-09-16T10:33:31.807036+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.213821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.213821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"383'9","reported_seq":503,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808066+0000","last_change":"2024-09-16T10:30:16.857737+0000","last_active":"2024-09-16T10:33:31.808066+0000","last_peered":"2024-09-16T10:33:31.808066+0000","last_clean":"2024-09-16T10:33:31.808066+0000","last_became_active":"2024-09-16T10:26:28.551639+0000","last_became_peered":"2024-09-16T10:26:28.551639+0000","last_unstale":"2024-09-16T10:33:31.808066+0000","last_undegraded":"2024-09-16T10:33:31.808066+0000","last_fullsized":"2024-09-16T10:33:31.808066+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'9","last_scrub_stamp":"2024-09-16T10:30:16.857676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:16.857676+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1071,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807537+0000","last_change":"2024-09-16T10:32:20.407177+0000","last_active":"2024-09-16T10:33:31.807537+0000","last_peered":"2024-09-16T10:33:31.807537+0000","last_clean":"2024-09-16T10:33:31.807537+0000","last_became_active":"2024-09-16T10:32:20.406800+0000","last_became_peered":"2024-09-16T10:32:20.406800+0000","last_unstale":"2024-09-16T10:33:31.807537+0000","last_undegraded":"2024-09-16T10:33:31.807537+0000","last_fullsized":"2024-09-16T10:33:31.807537+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:14.925284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:14.925284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":496,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261652+0000","last_change":"2024-09-16T10:30:18.655233+0000","last_active":"2024-09-16T10:33:31.261652+0000","last_peered":"2024-09-16T10:33:31.261652+0000","last_clean":"2024-09-16T10:33:31.261652+0000","last_became_active":"2024-09-16T10:27:18.552596+0000","last_became_peered":"2024-09-16T10:27:18.552596+0000","last_unstale":"2024-09-16T10:33:31.261652+0000","last_undegraded":"2024-09-16T10:33:31.261652+0000","last_fullsized":"2024-09-16T10:33:31.261652+0000","mapping_epoch":401,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:18.655172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:18.655172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:24.722180+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"328'30","reported_seq":457,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260781+0000","last_change":"2024-09-16T10:33:04.704385+0000","last_active":"2024-09-16T10:33:31.260781+0000","last_peered":"2024-09-16T10:33:31.260781+0000","last_clean":"2024-09-16T10:33:31.260781+0000","last_became_active":"2024-09-16T10:26:28.557116+0000","last_became_peered":"2024-09-16T10:26:28.557116+0000","last_unstale":"2024-09-16T10:33:31.260781+0000","last_undegraded":"2024-09-16T10:33:31.260781+0000","last_fullsized":"2024-09-16T10:33:31.260781+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'30","last_scrub_stamp":"2024-09-16T10:33:04.704308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:04.704308+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:14.562749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1057,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808989+0000","last_change":"2024-09-16T10:31:14.146422+0000","last_active":"2024-09-16T10:33:31.808989+0000","last_peered":"2024-09-16T10:33:31.808989+0000","last_clean":"2024-09-16T10:33:31.808989+0000","last_became_active":"2024-09-16T10:31:14.146107+0000","last_became_peered":"2024-09-16T10:31:14.146107+0000","last_unstale":"2024-09-16T10:33:31.808989+0000","last_undegraded":"2024-09-16T10:33:31.808989+0000","last_fullsized":"2024-09-16T10:33:31.808989+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:55.254993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.254993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":824,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260373+0000","last_change":"2024-09-16T10:32:06.770331+0000","last_active":"2024-09-16T10:33:31.260373+0000","last_peered":"2024-09-16T10:33:31.260373+0000","last_clean":"2024-09-16T10:33:31.260373+0000","last_became_active":"2024-09-16T10:31:14.164506+0000","last_became_peered":"2024-09-16T10:31:14.164506+0000","last_unstale":"2024-09-16T10:33:31.260373+0000","last_undegraded":"2024-09-16T10:33:31.260373+0000","last_fullsized":"2024-09-16T10:33:31.260373+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:06.770275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.770275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:34.696037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":871,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808464+0000","last_change":"2024-09-16T10:33:23.226943+0000","last_active":"2024-09-16T10:33:31.808464+0000","last_peered":"2024-09-16T10:33:31.808464+0000","last_clean":"2024-09-16T10:33:31.808464+0000","last_became_active":"2024-09-16T10:26:28.550749+0000","last_became_peered":"2024-09-16T10:26:28.550749+0000","last_unstale":"2024-09-16T10:33:31.808464+0000","last_undegraded":"2024-09-16T10:33:31.808464+0000","last_fullsized":"2024-09-16T10:33:31.808464+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:40.298138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:40.298138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:52.274606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":993,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808534+0000","last_change":"2024-09-16T10:31:17.119206+0000","last_active":"2024-09-16T10:33:31.808534+0000","last_peered":"2024-09-16T10:33:31.808534+0000","last_clean":"2024-09-16T10:33:31.808534+0000","last_became_active":"2024-09-16T10:31:17.118842+0000","last_became_peered":"2024-09-16T10:31:17.118842+0000","last_unstale":"2024-09-16T10:33:31.808534+0000","last_undegraded":"2024-09-16T10:33:31.808534+0000","last_fullsized":"2024-09-16T10:33:31.808534+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.198923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.198923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"328'28","reported_seq":1013,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807451+0000","last_change":"2024-09-16T10:30:12.893400+0000","last_active":"2024-09-16T10:33:31.807451+0000","last_peered":"2024-09-16T10:33:31.807451+0000","last_clean":"2024-09-16T10:33:31.807451+0000","last_became_active":"2024-09-16T10:26:28.551995+0000","last_became_peered":"2024-09-16T10:26:28.551995+0000","last_unstale":"2024-09-16T10:33:31.807451+0000","last_undegraded":"2024-09-16T10:33:31.807451+0000","last_fullsized":"2024-09-16T10:33:31.807451+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'28","last_scrub_stamp":"2024-09-16T10:30:12.893340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.893340+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:28.385119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":1072,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134795+0000","last_change":"2024-09-16T10:32:20.425958+0000","last_active":"2024-09-16T10:33:31.134795+0000","last_peered":"2024-09-16T10:33:31.134795+0000","last_clean":"2024-09-16T10:33:31.134795+0000","last_became_active":"2024-09-16T10:32:20.393386+0000","last_became_peered":"2024-09-16T10:32:20.393386+0000","last_unstale":"2024-09-16T10:33:31.134795+0000","last_undegraded":"2024-09-16T10:33:31.134795+0000","last_fullsized":"2024-09-16T10:33:31.134795+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:27:44.624873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:44.624873+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:46.894131+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"536'9","reported_seq":1329,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806734+0000","last_change":"2024-09-16T10:33:15.204469+0000","last_active":"2024-09-16T10:33:31.806734+0000","last_peered":"2024-09-16T10:33:31.806734+0000","last_clean":"2024-09-16T10:33:31.806734+0000","last_became_active":"2024-09-16T10:32:20.418792+0000","last_became_peered":"2024-09-16T10:32:20.418792+0000","last_unstale":"2024-09-16T10:33:31.806734+0000","last_undegraded":"2024-09-16T10:33:31.806734+0000","last_fullsized":"2024-09-16T10:33:31.806734+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'9","last_scrub_stamp":"2024-09-16T10:33:15.201123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:15.201123+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:28.015856+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":941,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261000+0000","last_change":"2024-09-16T10:32:20.413255+0000","last_active":"2024-09-16T10:33:31.261000+0000","last_peered":"2024-09-16T10:33:31.261000+0000","last_clean":"2024-09-16T10:33:31.261000+0000","last_became_active":"2024-09-16T10:32:20.412898+0000","last_became_peered":"2024-09-16T10:32:20.412898+0000","last_unstale":"2024-09-16T10:33:31.261000+0000","last_undegraded":"2024-09-16T10:33:31.261000+0000","last_fullsized":"2024-09-16T10:33:31.261000+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:12.688413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.688413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:40.735663+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":1051,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261475+0000","last_change":"2024-09-16T10:32:16.334773+0000","last_active":"2024-09-16T10:33:31.261475+0000","last_peered":"2024-09-16T10:33:31.261475+0000","last_clean":"2024-09-16T10:33:31.261475+0000","last_became_active":"2024-09-16T10:32:16.334470+0000","last_became_peered":"2024-09-16T10:32:16.334470+0000","last_unstale":"2024-09-16T10:33:31.261475+0000","last_undegraded":"2024-09-16T10:33:31.261475+0000","last_fullsized":"2024-09-16T10:33:31.261475+0000","mapping_epoch":532,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:46.663450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:46.663450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"328'20","reported_seq":699,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807738+0000","last_change":"2024-09-16T10:32:20.446784+0000","last_active":"2024-09-16T10:33:31.807738+0000","last_peered":"2024-09-16T10:33:31.807738+0000","last_clean":"2024-09-16T10:33:31.807738+0000","last_became_active":"2024-09-16T10:32:20.395496+0000","last_became_peered":"2024-09-16T10:32:20.395496+0000","last_unstale":"2024-09-16T10:33:31.807738+0000","last_undegraded":"2024-09-16T10:33:31.807738+0000","last_fullsized":"2024-09-16T10:33:31.807738+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:25.663895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.663895+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:50.360324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1091,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808275+0000","last_change":"2024-09-16T10:33:02.347961+0000","last_active":"2024-09-16T10:33:31.808275+0000","last_peered":"2024-09-16T10:33:31.808275+0000","last_clean":"2024-09-16T10:33:31.808275+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:33:31.808275+0000","last_undegraded":"2024-09-16T10:33:31.808275+0000","last_fullsized":"2024-09-16T10:33:31.808275+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:02.347917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:02.347917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:09.667103+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1300,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135176+0000","last_change":"2024-09-16T10:30:08.183366+0000","last_active":"2024-09-16T10:33:31.135176+0000","last_peered":"2024-09-16T10:33:31.135176+0000","last_clean":"2024-09-16T10:33:31.135176+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:33:31.135176+0000","last_undegraded":"2024-09-16T10:33:31.135176+0000","last_fullsized":"2024-09-16T10:33:31.135176+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:08.182949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:08.182949+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1078,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134462+0000","last_change":"2024-09-16T10:30:12.258492+0000","last_active":"2024-09-16T10:33:31.134462+0000","last_peered":"2024-09-16T10:33:31.134462+0000","last_clean":"2024-09-16T10:33:31.134462+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:33:31.134462+0000","last_undegraded":"2024-09-16T10:33:31.134462+0000","last_fullsized":"2024-09-16T10:33:31.134462+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:12.258426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:12.258426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":1028,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261342+0000","last_change":"2024-09-16T10:32:55.757529+0000","last_active":"2024-09-16T10:33:31.261342+0000","last_peered":"2024-09-16T10:33:31.261342+0000","last_clean":"2024-09-16T10:33:31.261342+0000","last_became_active":"2024-09-16T10:22:56.426133+0000","last_became_peered":"2024-09-16T10:22:56.426133+0000","last_unstale":"2024-09-16T10:33:31.261342+0000","last_undegraded":"2024-09-16T10:33:31.261342+0000","last_fullsized":"2024-09-16T10:33:31.261342+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:55.757468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:55.757468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:58.838895+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"328'29","reported_seq":1184,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134936+0000","last_change":"2024-09-16T10:33:12.305837+0000","last_active":"2024-09-16T10:33:31.134936+0000","last_peered":"2024-09-16T10:33:31.134936+0000","last_clean":"2024-09-16T10:33:31.134936+0000","last_became_active":"2024-09-16T10:32:20.413038+0000","last_became_peered":"2024-09-16T10:32:20.413038+0000","last_unstale":"2024-09-16T10:33:31.134936+0000","last_undegraded":"2024-09-16T10:33:31.134936+0000","last_fullsized":"2024-09-16T10:33:31.134936+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'29","last_scrub_stamp":"2024-09-16T10:33:12.305749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:12.305749+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:35.711805+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":559,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807298+0000","last_change":"2024-09-16T10:32:20.426707+0000","last_active":"2024-09-16T10:33:31.807298+0000","last_peered":"2024-09-16T10:33:31.807298+0000","last_clean":"2024-09-16T10:33:31.807298+0000","last_became_active":"2024-09-16T10:32:20.426427+0000","last_became_peered":"2024-09-16T10:32:20.426427+0000","last_unstale":"2024-09-16T10:33:31.807298+0000","last_undegraded":"2024-09-16T10:33:31.807298+0000","last_fullsized":"2024-09-16T10:33:31.807298+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:26.686221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:26.686221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.303693+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1076,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260657+0000","last_change":"2024-09-16T10:30:50.670023+0000","last_active":"2024-09-16T10:33:31.260657+0000","last_peered":"2024-09-16T10:33:31.260657+0000","last_clean":"2024-09-16T10:33:31.260657+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:33:31.260657+0000","last_undegraded":"2024-09-16T10:33:31.260657+0000","last_fullsized":"2024-09-16T10:33:31.260657+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.669938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.669938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":546,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260491+0000","last_change":"2024-09-16T10:32:20.405226+0000","last_active":"2024-09-16T10:33:31.260491+0000","last_peered":"2024-09-16T10:33:31.260491+0000","last_clean":"2024-09-16T10:33:31.260491+0000","last_became_active":"2024-09-16T10:32:20.404614+0000","last_became_peered":"2024-09-16T10:32:20.404614+0000","last_unstale":"2024-09-16T10:33:31.260491+0000","last_undegraded":"2024-09-16T10:33:31.260491+0000","last_fullsized":"2024-09-16T10:33:31.260491+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:41.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:41.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:42.743403+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":966,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807030+0000","last_change":"2024-09-16T10:30:54.802360+0000","last_active":"2024-09-16T10:33:31.807030+0000","last_peered":"2024-09-16T10:33:31.807030+0000","last_clean":"2024-09-16T10:33:31.807030+0000","last_became_active":"2024-09-16T10:22:56.420514+0000","last_became_peered":"2024-09-16T10:22:56.420514+0000","last_unstale":"2024-09-16T10:33:31.807030+0000","last_undegraded":"2024-09-16T10:33:31.807030+0000","last_fullsized":"2024-09-16T10:33:31.807030+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:54.802277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.802277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"537'43","reported_seq":940,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807505+0000","last_change":"2024-09-16T10:32:20.529917+0000","last_active":"2024-09-16T10:33:31.807505+0000","last_peered":"2024-09-16T10:33:31.807505+0000","last_clean":"2024-09-16T10:33:31.807505+0000","last_became_active":"2024-09-16T10:32:20.435824+0000","last_became_peered":"2024-09-16T10:32:20.435824+0000","last_unstale":"2024-09-16T10:33:31.807505+0000","last_undegraded":"2024-09-16T10:33:31.807505+0000","last_fullsized":"2024-09-16T10:33:31.807505+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'42","last_scrub_stamp":"2024-09-16T10:32:12.197914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.197914+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":384,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134716+0000","last_change":"2024-09-16T10:27:54.628471+0000","last_active":"2024-09-16T10:33:31.134716+0000","last_peered":"2024-09-16T10:33:31.134716+0000","last_clean":"2024-09-16T10:33:31.134716+0000","last_became_active":"2024-09-16T10:26:28.553639+0000","last_became_peered":"2024-09-16T10:26:28.553639+0000","last_unstale":"2024-09-16T10:33:31.134716+0000","last_undegraded":"2024-09-16T10:33:31.134716+0000","last_fullsized":"2024-09-16T10:33:31.134716+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:54.628394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:54.628394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:49.909210+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":1098,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807959+0000","last_change":"2024-09-16T10:32:20.428905+0000","last_active":"2024-09-16T10:33:31.807959+0000","last_peered":"2024-09-16T10:33:31.807959+0000","last_clean":"2024-09-16T10:33:31.807959+0000","last_became_active":"2024-09-16T10:32:20.396067+0000","last_became_peered":"2024-09-16T10:32:20.396067+0000","last_unstale":"2024-09-16T10:33:31.807959+0000","last_undegraded":"2024-09-16T10:33:31.807959+0000","last_fullsized":"2024-09-16T10:33:31.807959+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:30:11.895746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:11.895746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1059,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134523+0000","last_change":"2024-09-16T10:32:20.395692+0000","last_active":"2024-09-16T10:33:31.134523+0000","last_peered":"2024-09-16T10:33:31.134523+0000","last_clean":"2024-09-16T10:33:31.134523+0000","last_became_active":"2024-09-16T10:32:20.395374+0000","last_became_peered":"2024-09-16T10:32:20.395374+0000","last_unstale":"2024-09-16T10:33:31.134523+0000","last_undegraded":"2024-09-16T10:33:31.134523+0000","last_fullsized":"2024-09-16T10:33:31.134523+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.379980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.379980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1089,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.809132+0000","last_change":"2024-09-16T10:31:14.143712+0000","last_active":"2024-09-16T10:33:31.809132+0000","last_peered":"2024-09-16T10:33:31.809132+0000","last_clean":"2024-09-16T10:33:31.809132+0000","last_became_active":"2024-09-16T10:31:14.143349+0000","last_became_peered":"2024-09-16T10:31:14.143349+0000","last_unstale":"2024-09-16T10:33:31.809132+0000","last_undegraded":"2024-09-16T10:33:31.809132+0000","last_fullsized":"2024-09-16T10:33:31.809132+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:04.877913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:04.877913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":559,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135863+0000","last_change":"2024-09-16T10:31:17.108542+0000","last_active":"2024-09-16T10:33:31.135863+0000","last_peered":"2024-09-16T10:33:31.135863+0000","last_clean":"2024-09-16T10:33:31.135863+0000","last_became_active":"2024-09-16T10:31:17.108022+0000","last_became_peered":"2024-09-16T10:31:17.108022+0000","last_unstale":"2024-09-16T10:33:31.135863+0000","last_undegraded":"2024-09-16T10:33:31.135863+0000","last_fullsized":"2024-09-16T10:33:31.135863+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:49.434225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.434225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"327'17","reported_seq":668,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808300+0000","last_change":"2024-09-16T10:32:20.453683+0000","last_active":"2024-09-16T10:33:31.808300+0000","last_peered":"2024-09-16T10:33:31.808300+0000","last_clean":"2024-09-16T10:33:31.808300+0000","last_became_active":"2024-09-16T10:32:20.435036+0000","last_became_peered":"2024-09-16T10:32:20.435036+0000","last_unstale":"2024-09-16T10:33:31.808300+0000","last_undegraded":"2024-09-16T10:33:31.808300+0000","last_fullsized":"2024-09-16T10:33:31.808300+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'17","last_scrub_stamp":"2024-09-16T10:30:05.864159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.864159+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":551,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806471+0000","last_change":"2024-09-16T10:32:20.419274+0000","last_active":"2024-09-16T10:33:31.806471+0000","last_peered":"2024-09-16T10:33:31.806471+0000","last_clean":"2024-09-16T10:33:31.806471+0000","last_became_active":"2024-09-16T10:32:20.418545+0000","last_became_peered":"2024-09-16T10:32:20.418545+0000","last_unstale":"2024-09-16T10:33:31.806471+0000","last_undegraded":"2024-09-16T10:33:31.806471+0000","last_fullsized":"2024-09-16T10:33:31.806471+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:20.690959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:29:20.690959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:43.359607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":606,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807009+0000","last_change":"2024-09-16T10:32:11.066183+0000","last_active":"2024-09-16T10:33:31.807009+0000","last_peered":"2024-09-16T10:33:31.807009+0000","last_clean":"2024-09-16T10:33:31.807009+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:33:31.807009+0000","last_undegraded":"2024-09-16T10:33:31.807009+0000","last_fullsized":"2024-09-16T10:33:31.807009+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:32:11.066076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.066076+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":620,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807499+0000","last_change":"2024-09-16T10:31:22.862857+0000","last_active":"2024-09-16T10:33:31.807499+0000","last_peered":"2024-09-16T10:33:31.807499+0000","last_clean":"2024-09-16T10:33:31.807499+0000","last_became_active":"2024-09-16T10:31:14.142132+0000","last_became_peered":"2024-09-16T10:31:14.142132+0000","last_unstale":"2024-09-16T10:33:31.807499+0000","last_undegraded":"2024-09-16T10:33:31.807499+0000","last_fullsized":"2024-09-16T10:33:31.807499+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:22.862677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:22.862677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:28.712843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":474,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261349+0000","last_change":"2024-09-16T10:31:14.164501+0000","last_active":"2024-09-16T10:33:31.261349+0000","last_peered":"2024-09-16T10:33:31.261349+0000","last_clean":"2024-09-16T10:33:31.261349+0000","last_became_active":"2024-09-16T10:31:14.164133+0000","last_became_peered":"2024-09-16T10:31:14.164133+0000","last_unstale":"2024-09-16T10:33:31.261349+0000","last_undegraded":"2024-09-16T10:33:31.261349+0000","last_fullsized":"2024-09-16T10:33:31.261349+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.706111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.706111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":168,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260570+0000","last_change":"2024-09-16T10:32:53.774953+0000","last_active":"2024-09-16T10:33:31.260570+0000","last_peered":"2024-09-16T10:33:31.260570+0000","last_clean":"2024-09-16T10:33:31.260570+0000","last_became_active":"2024-09-16T10:32:23.420463+0000","last_became_peered":"2024-09-16T10:32:23.420463+0000","last_unstale":"2024-09-16T10:33:31.260570+0000","last_undegraded":"2024-09-16T10:33:31.260570+0000","last_fullsized":"2024-09-16T10:33:31.260570+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:53.774903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:53.774903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.401981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"328'24","reported_seq":1187,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134683+0000","last_change":"2024-09-16T10:32:20.455220+0000","last_active":"2024-09-16T10:33:31.134683+0000","last_peered":"2024-09-16T10:33:31.134683+0000","last_clean":"2024-09-16T10:33:31.134683+0000","last_became_active":"2024-09-16T10:32:20.413651+0000","last_became_peered":"2024-09-16T10:32:20.413651+0000","last_unstale":"2024-09-16T10:33:31.134683+0000","last_undegraded":"2024-09-16T10:33:31.134683+0000","last_fullsized":"2024-09-16T10:33:31.134683+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'24","last_scrub_stamp":"2024-09-16T10:30:20.268438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.268438+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1092,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135278+0000","last_change":"2024-09-16T10:30:15.229737+0000","last_active":"2024-09-16T10:33:31.135278+0000","last_peered":"2024-09-16T10:33:31.135278+0000","last_clean":"2024-09-16T10:33:31.135278+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:33:31.135278+0000","last_undegraded":"2024-09-16T10:33:31.135278+0000","last_fullsized":"2024-09-16T10:33:31.135278+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.229669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.229669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":191,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:33.783217+0000","last_change":"2024-09-16T10:32:20.443363+0000","last_active":"2024-09-16T10:33:33.783217+0000","last_peered":"2024-09-16T10:33:33.783217+0000","last_clean":"2024-09-16T10:33:33.783217+0000","last_became_active":"2024-09-16T10:32:20.396791+0000","last_became_peered":"2024-09-16T10:32:20.396791+0000","last_unstale":"2024-09-16T10:33:33.783217+0000","last_undegraded":"2024-09-16T10:33:33.783217+0000","last_fullsized":"2024-09-16T10:33:33.783217+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:30:39.226399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.226399+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1092,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808199+0000","last_change":"2024-09-16T10:33:20.360912+0000","last_active":"2024-09-16T10:33:31.808199+0000","last_peered":"2024-09-16T10:33:31.808199+0000","last_clean":"2024-09-16T10:33:31.808199+0000","last_became_active":"2024-09-16T10:32:20.395732+0000","last_became_peered":"2024-09-16T10:32:20.395732+0000","last_unstale":"2024-09-16T10:33:31.808199+0000","last_undegraded":"2024-09-16T10:33:31.808199+0000","last_fullsized":"2024-09-16T10:33:31.808199+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:20.360862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:20.360862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:30.710303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":70,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808034+0000","last_change":"2024-09-16T10:32:20.435415+0000","last_active":"2024-09-16T10:33:31.808034+0000","last_peered":"2024-09-16T10:33:31.808034+0000","last_clean":"2024-09-16T10:33:31.808034+0000","last_became_active":"2024-09-16T10:32:20.434853+0000","last_became_peered":"2024-09-16T10:32:20.434853+0000","last_unstale":"2024-09-16T10:33:31.808034+0000","last_undegraded":"2024-09-16T10:33:31.808034+0000","last_fullsized":"2024-09-16T10:33:31.808034+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:19.374407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:27:19.374407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:47.361715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"508'31","reported_seq":1374,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261488+0000","last_change":"2024-09-16T10:31:14.178826+0000","last_active":"2024-09-16T10:33:31.261488+0000","last_peered":"2024-09-16T10:33:31.261488+0000","last_clean":"2024-09-16T10:33:31.261488+0000","last_became_active":"2024-09-16T10:31:14.137441+0000","last_became_peered":"2024-09-16T10:31:14.137441+0000","last_unstale":"2024-09-16T10:33:31.261488+0000","last_undegraded":"2024-09-16T10:33:31.261488+0000","last_fullsized":"2024-09-16T10:33:31.261488+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'30","last_scrub_stamp":"2024-09-16T10:30:54.737043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.737043+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1040,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808710+0000","last_change":"2024-09-16T10:32:07.379538+0000","last_active":"2024-09-16T10:33:31.808710+0000","last_peered":"2024-09-16T10:33:31.808710+0000","last_clean":"2024-09-16T10:33:31.808710+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:33:31.808710+0000","last_undegraded":"2024-09-16T10:33:31.808710+0000","last_fullsized":"2024-09-16T10:33:31.808710+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:07.379336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:07.379336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":576,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134716+0000","last_change":"2024-09-16T10:32:23.435277+0000","last_active":"2024-09-16T10:33:31.134716+0000","last_peered":"2024-09-16T10:33:31.134716+0000","last_clean":"2024-09-16T10:33:31.134716+0000","last_became_active":"2024-09-16T10:32:23.434945+0000","last_became_peered":"2024-09-16T10:32:23.434945+0000","last_unstale":"2024-09-16T10:33:31.134716+0000","last_undegraded":"2024-09-16T10:33:31.134716+0000","last_fullsized":"2024-09-16T10:33:31.134716+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:50.410561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.410561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":945,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807047+0000","last_change":"2024-09-16T10:31:14.146544+0000","last_active":"2024-09-16T10:33:31.807047+0000","last_peered":"2024-09-16T10:33:31.807047+0000","last_clean":"2024-09-16T10:33:31.807047+0000","last_became_active":"2024-09-16T10:31:14.145333+0000","last_became_peered":"2024-09-16T10:31:14.145333+0000","last_unstale":"2024-09-16T10:33:31.807047+0000","last_undegraded":"2024-09-16T10:33:31.807047+0000","last_fullsized":"2024-09-16T10:33:31.807047+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.613780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.613780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:23.684845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1065,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808380+0000","last_change":"2024-09-16T10:30:43.705706+0000","last_active":"2024-09-16T10:33:31.808380+0000","last_peered":"2024-09-16T10:33:31.808380+0000","last_clean":"2024-09-16T10:33:31.808380+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:33:31.808380+0000","last_undegraded":"2024-09-16T10:33:31.808380+0000","last_fullsized":"2024-09-16T10:33:31.808380+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:43.705640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:43.705640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":504,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135377+0000","last_change":"2024-09-16T10:32:53.999521+0000","last_active":"2024-09-16T10:33:31.135377+0000","last_peered":"2024-09-16T10:33:31.135377+0000","last_clean":"2024-09-16T10:33:31.135377+0000","last_became_active":"2024-09-16T10:32:23.432634+0000","last_became_peered":"2024-09-16T10:32:23.432634+0000","last_unstale":"2024-09-16T10:33:31.135377+0000","last_undegraded":"2024-09-16T10:33:31.135377+0000","last_fullsized":"2024-09-16T10:33:31.135377+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:53.999450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:53.999450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.072558+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"328'19","reported_seq":1024,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261565+0000","last_change":"2024-09-16T10:32:20.480209+0000","last_active":"2024-09-16T10:33:31.261565+0000","last_peered":"2024-09-16T10:33:31.261565+0000","last_clean":"2024-09-16T10:33:31.261565+0000","last_became_active":"2024-09-16T10:32:20.395805+0000","last_became_peered":"2024-09-16T10:32:20.395805+0000","last_unstale":"2024-09-16T10:33:31.261565+0000","last_undegraded":"2024-09-16T10:33:31.261565+0000","last_fullsized":"2024-09-16T10:33:31.261565+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:32:08.692093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:08.692093+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:37.858412+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":896,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806617+0000","last_change":"2024-09-16T10:32:20.409986+0000","last_active":"2024-09-16T10:33:31.806617+0000","last_peered":"2024-09-16T10:33:31.806617+0000","last_clean":"2024-09-16T10:33:31.806617+0000","last_became_active":"2024-09-16T10:32:20.406762+0000","last_became_peered":"2024-09-16T10:32:20.406762+0000","last_unstale":"2024-09-16T10:33:31.806617+0000","last_undegraded":"2024-09-16T10:33:31.806617+0000","last_fullsized":"2024-09-16T10:33:31.806617+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:30.867930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:30.867930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1076,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808396+0000","last_change":"2024-09-16T10:33:07.435064+0000","last_active":"2024-09-16T10:33:31.808396+0000","last_peered":"2024-09-16T10:33:31.808396+0000","last_clean":"2024-09-16T10:33:31.808396+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:33:31.808396+0000","last_undegraded":"2024-09-16T10:33:31.808396+0000","last_fullsized":"2024-09-16T10:33:31.808396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:07.434963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:07.434963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:11.222708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1003,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808480+0000","last_change":"2024-09-16T10:29:25.898838+0000","last_active":"2024-09-16T10:33:31.808480+0000","last_peered":"2024-09-16T10:33:31.808480+0000","last_clean":"2024-09-16T10:33:31.808480+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:33:31.808480+0000","last_undegraded":"2024-09-16T10:33:31.808480+0000","last_fullsized":"2024-09-16T10:33:31.808480+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:29:25.898781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:29:25.898781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:29.336913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":937,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807036+0000","last_change":"2024-09-16T10:33:17.369390+0000","last_active":"2024-09-16T10:33:31.807036+0000","last_peered":"2024-09-16T10:33:31.807036+0000","last_clean":"2024-09-16T10:33:31.807036+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:33:31.807036+0000","last_undegraded":"2024-09-16T10:33:31.807036+0000","last_fullsized":"2024-09-16T10:33:31.807036+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:17.369317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:17.369317+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:29.192042+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"328'31","reported_seq":1094,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807278+0000","last_change":"2024-09-16T10:32:20.440999+0000","last_active":"2024-09-16T10:33:31.807278+0000","last_peered":"2024-09-16T10:33:31.807278+0000","last_clean":"2024-09-16T10:33:31.807278+0000","last_became_active":"2024-09-16T10:32:20.402914+0000","last_became_peered":"2024-09-16T10:32:20.402914+0000","last_unstale":"2024-09-16T10:33:31.807278+0000","last_undegraded":"2024-09-16T10:33:31.807278+0000","last_fullsized":"2024-09-16T10:33:31.807278+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'31","last_scrub_stamp":"2024-09-16T10:31:18.771268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:31:18.771268+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:31.659491+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1083,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261126+0000","last_change":"2024-09-16T10:32:54.752441+0000","last_active":"2024-09-16T10:33:31.261126+0000","last_peered":"2024-09-16T10:33:31.261126+0000","last_clean":"2024-09-16T10:33:31.261126+0000","last_became_active":"2024-09-16T10:32:20.406154+0000","last_became_peered":"2024-09-16T10:32:20.406154+0000","last_unstale":"2024-09-16T10:33:31.261126+0000","last_undegraded":"2024-09-16T10:33:31.261126+0000","last_fullsized":"2024-09-16T10:33:31.261126+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:54.752376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:54.752376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:13.545772+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":77,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134335+0000","last_change":"2024-09-16T10:33:06.130249+0000","last_active":"2024-09-16T10:33:31.134335+0000","last_peered":"2024-09-16T10:33:31.134335+0000","last_clean":"2024-09-16T10:33:31.134335+0000","last_became_active":"2024-09-16T10:32:20.395860+0000","last_became_peered":"2024-09-16T10:32:20.395860+0000","last_unstale":"2024-09-16T10:33:31.134335+0000","last_undegraded":"2024-09-16T10:33:31.134335+0000","last_fullsized":"2024-09-16T10:33:31.134335+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:06.130189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.130189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:17.390775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":610,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807779+0000","last_change":"2024-09-16T10:33:16.677107+0000","last_active":"2024-09-16T10:33:31.807779+0000","last_peered":"2024-09-16T10:33:31.807779+0000","last_clean":"2024-09-16T10:33:31.807779+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:33:31.807779+0000","last_undegraded":"2024-09-16T10:33:31.807779+0000","last_fullsized":"2024-09-16T10:33:31.807779+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:16.677041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:16.677041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:39.426050+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":748,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807897+0000","last_change":"2024-09-16T10:31:14.142529+0000","last_active":"2024-09-16T10:33:31.807897+0000","last_peered":"2024-09-16T10:33:31.807897+0000","last_clean":"2024-09-16T10:33:31.807897+0000","last_became_active":"2024-09-16T10:31:14.140156+0000","last_became_peered":"2024-09-16T10:31:14.140156+0000","last_unstale":"2024-09-16T10:33:31.807897+0000","last_undegraded":"2024-09-16T10:33:31.807897+0000","last_fullsized":"2024-09-16T10:33:31.807897+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:07.951845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.951845+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"328'20","reported_seq":977,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808311+0000","last_change":"2024-09-16T10:29:23.889177+0000","last_active":"2024-09-16T10:33:31.808311+0000","last_peered":"2024-09-16T10:33:31.808311+0000","last_clean":"2024-09-16T10:33:31.808311+0000","last_became_active":"2024-09-16T10:26:28.552528+0000","last_became_peered":"2024-09-16T10:26:28.552528+0000","last_unstale":"2024-09-16T10:33:31.808311+0000","last_undegraded":"2024-09-16T10:33:31.808311+0000","last_fullsized":"2024-09-16T10:33:31.808311+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:29:23.889117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:23.889117+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1058,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808305+0000","last_change":"2024-09-16T10:32:09.398843+0000","last_active":"2024-09-16T10:33:31.808305+0000","last_peered":"2024-09-16T10:33:31.808305+0000","last_clean":"2024-09-16T10:33:31.808305+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:33:31.808305+0000","last_undegraded":"2024-09-16T10:33:31.808305+0000","last_fullsized":"2024-09-16T10:33:31.808305+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:09.398706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.398706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"508'13","reported_seq":769,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807442+0000","last_change":"2024-09-16T10:31:27.828520+0000","last_active":"2024-09-16T10:33:31.807442+0000","last_peered":"2024-09-16T10:33:31.807442+0000","last_clean":"2024-09-16T10:33:31.807442+0000","last_became_active":"2024-09-16T10:31:14.138957+0000","last_became_peered":"2024-09-16T10:31:14.138957+0000","last_unstale":"2024-09-16T10:33:31.807442+0000","last_undegraded":"2024-09-16T10:33:31.807442+0000","last_fullsized":"2024-09-16T10:33:31.807442+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'13","last_scrub_stamp":"2024-09-16T10:31:27.828439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:31:27.828439+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":555,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807305+0000","last_change":"2024-09-16T10:33:12.679629+0000","last_active":"2024-09-16T10:33:31.807305+0000","last_peered":"2024-09-16T10:33:31.807305+0000","last_clean":"2024-09-16T10:33:31.807305+0000","last_became_active":"2024-09-16T10:32:20.392061+0000","last_became_peered":"2024-09-16T10:32:20.392061+0000","last_unstale":"2024-09-16T10:33:31.807305+0000","last_undegraded":"2024-09-16T10:33:31.807305+0000","last_fullsized":"2024-09-16T10:33:31.807305+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:12.679545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:12.679545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:31.979630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"73'79","reported_seq":341,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.809047+0000","last_change":"2024-09-16T10:32:17.363539+0000","last_active":"2024-09-16T10:33:31.809047+0000","last_peered":"2024-09-16T10:33:31.809047+0000","last_clean":"2024-09-16T10:33:31.809047+0000","last_became_active":"2024-09-16T10:31:17.119269+0000","last_became_peered":"2024-09-16T10:31:17.119269+0000","last_unstale":"2024-09-16T10:33:31.809047+0000","last_undegraded":"2024-09-16T10:33:31.809047+0000","last_fullsized":"2024-09-16T10:33:31.809047+0000","mapping_epoch":510,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:17.363464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:17.363464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.071616+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"328'15","reported_seq":1105,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135035+0000","last_change":"2024-09-16T10:32:20.468825+0000","last_active":"2024-09-16T10:33:31.135035+0000","last_peered":"2024-09-16T10:33:31.135035+0000","last_clean":"2024-09-16T10:33:31.135035+0000","last_became_active":"2024-09-16T10:32:20.417250+0000","last_became_peered":"2024-09-16T10:32:20.417250+0000","last_unstale":"2024-09-16T10:33:31.135035+0000","last_undegraded":"2024-09-16T10:33:31.135035+0000","last_fullsized":"2024-09-16T10:33:31.135035+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:30:19.209335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:19.209335+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:28.361688+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1084,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808072+0000","last_change":"2024-09-16T10:32:06.250968+0000","last_active":"2024-09-16T10:33:31.808072+0000","last_peered":"2024-09-16T10:33:31.808072+0000","last_clean":"2024-09-16T10:33:31.808072+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:33:31.808072+0000","last_undegraded":"2024-09-16T10:33:31.808072+0000","last_fullsized":"2024-09-16T10:33:31.808072+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:06.250899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.250899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":602,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807179+0000","last_change":"2024-09-16T10:32:12.023979+0000","last_active":"2024-09-16T10:33:31.807179+0000","last_peered":"2024-09-16T10:33:31.807179+0000","last_clean":"2024-09-16T10:33:31.807179+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:33:31.807179+0000","last_undegraded":"2024-09-16T10:33:31.807179+0000","last_fullsized":"2024-09-16T10:33:31.807179+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:12.023740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.023740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:31.931461+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":1115,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135007+0000","last_change":"2024-09-16T10:31:14.161646+0000","last_active":"2024-09-16T10:33:31.135007+0000","last_peered":"2024-09-16T10:33:31.135007+0000","last_clean":"2024-09-16T10:33:31.135007+0000","last_became_active":"2024-09-16T10:31:14.152023+0000","last_became_peered":"2024-09-16T10:31:14.152023+0000","last_unstale":"2024-09-16T10:33:31.135007+0000","last_undegraded":"2024-09-16T10:33:31.135007+0000","last_fullsized":"2024-09-16T10:33:31.135007+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:30:05.174161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:05.174161+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"73'79","reported_seq":1032,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261707+0000","last_change":"2024-09-16T10:30:42.550612+0000","last_active":"2024-09-16T10:33:31.261707+0000","last_peered":"2024-09-16T10:33:31.261707+0000","last_clean":"2024-09-16T10:33:31.261707+0000","last_became_active":"2024-09-16T10:23:31.471414+0000","last_became_peered":"2024-09-16T10:23:31.471414+0000","last_unstale":"2024-09-16T10:33:31.261707+0000","last_undegraded":"2024-09-16T10:33:31.261707+0000","last_fullsized":"2024-09-16T10:33:31.261707+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:30:42.550567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.550567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"328'35","reported_seq":1192,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260716+0000","last_change":"2024-09-16T10:32:20.470230+0000","last_active":"2024-09-16T10:33:31.260716+0000","last_peered":"2024-09-16T10:33:31.260716+0000","last_clean":"2024-09-16T10:33:31.260716+0000","last_became_active":"2024-09-16T10:32:20.412893+0000","last_became_peered":"2024-09-16T10:32:20.412893+0000","last_unstale":"2024-09-16T10:33:31.260716+0000","last_undegraded":"2024-09-16T10:33:31.260716+0000","last_fullsized":"2024-09-16T10:33:31.260716+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:25:35.488518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:25:35.488518+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:41.740678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":472,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807919+0000","last_change":"2024-09-16T10:31:14.148058+0000","last_active":"2024-09-16T10:33:31.807919+0000","last_peered":"2024-09-16T10:33:31.807919+0000","last_clean":"2024-09-16T10:33:31.807919+0000","last_became_active":"2024-09-16T10:31:14.147606+0000","last_became_peered":"2024-09-16T10:31:14.147606+0000","last_unstale":"2024-09-16T10:33:31.807919+0000","last_undegraded":"2024-09-16T10:33:31.807919+0000","last_fullsized":"2024-09-16T10:33:31.807919+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:48.745214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:48.745214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":602,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807091+0000","last_change":"2024-09-16T10:32:18.059677+0000","last_active":"2024-09-16T10:33:31.807091+0000","last_peered":"2024-09-16T10:33:31.807091+0000","last_clean":"2024-09-16T10:33:31.807091+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:33:31.807091+0000","last_undegraded":"2024-09-16T10:33:31.807091+0000","last_fullsized":"2024-09-16T10:33:31.807091+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:18.059602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:18.059602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:35.489290+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":948,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806703+0000","last_change":"2024-09-16T10:31:14.148903+0000","last_active":"2024-09-16T10:33:31.806703+0000","last_peered":"2024-09-16T10:33:31.806703+0000","last_clean":"2024-09-16T10:33:31.806703+0000","last_became_active":"2024-09-16T10:31:14.148572+0000","last_became_peered":"2024-09-16T10:31:14.148572+0000","last_unstale":"2024-09-16T10:33:31.806703+0000","last_undegraded":"2024-09-16T10:33:31.806703+0000","last_fullsized":"2024-09-16T10:33:31.806703+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:49.626720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:49.626720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:27.710427+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":983,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808152+0000","last_change":"2024-09-16T10:32:06.364289+0000","last_active":"2024-09-16T10:33:31.808152+0000","last_peered":"2024-09-16T10:33:31.808152+0000","last_clean":"2024-09-16T10:33:31.808152+0000","last_became_active":"2024-09-16T10:24:00.759030+0000","last_became_peered":"2024-09-16T10:24:00.759030+0000","last_unstale":"2024-09-16T10:33:31.808152+0000","last_undegraded":"2024-09-16T10:33:31.808152+0000","last_fullsized":"2024-09-16T10:33:31.808152+0000","mapping_epoch":333,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:06.364207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:06.364207+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"327'18","reported_seq":682,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807723+0000","last_change":"2024-09-16T10:32:20.466527+0000","last_active":"2024-09-16T10:33:31.807723+0000","last_peered":"2024-09-16T10:33:31.807723+0000","last_clean":"2024-09-16T10:33:31.807723+0000","last_became_active":"2024-09-16T10:32:20.431532+0000","last_became_peered":"2024-09-16T10:32:20.431532+0000","last_unstale":"2024-09-16T10:33:31.807723+0000","last_undegraded":"2024-09-16T10:33:31.807723+0000","last_fullsized":"2024-09-16T10:33:31.807723+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'18","last_scrub_stamp":"2024-09-16T10:30:20.961894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.961894+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":72,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134514+0000","last_change":"2024-09-16T10:32:20.390993+0000","last_active":"2024-09-16T10:33:31.134514+0000","last_peered":"2024-09-16T10:33:31.134514+0000","last_clean":"2024-09-16T10:33:31.134514+0000","last_became_active":"2024-09-16T10:32:20.390515+0000","last_became_peered":"2024-09-16T10:32:20.390515+0000","last_unstale":"2024-09-16T10:33:31.134514+0000","last_undegraded":"2024-09-16T10:33:31.134514+0000","last_fullsized":"2024-09-16T10:33:31.134514+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:27:13.285110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:27:13.285110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.911596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":617,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806794+0000","last_change":"2024-09-16T10:33:13.716266+0000","last_active":"2024-09-16T10:33:31.806794+0000","last_peered":"2024-09-16T10:33:31.806794+0000","last_clean":"2024-09-16T10:33:31.806794+0000","last_became_active":"2024-09-16T10:32:20.418945+0000","last_became_peered":"2024-09-16T10:32:20.418945+0000","last_unstale":"2024-09-16T10:33:31.806794+0000","last_undegraded":"2024-09-16T10:33:31.806794+0000","last_fullsized":"2024-09-16T10:33:31.806794+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:13.715922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:13.715922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:16.525289+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1087,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808000+0000","last_change":"2024-09-16T10:32:20.433825+0000","last_active":"2024-09-16T10:33:31.808000+0000","last_peered":"2024-09-16T10:33:31.808000+0000","last_clean":"2024-09-16T10:33:31.808000+0000","last_became_active":"2024-09-16T10:32:20.423737+0000","last_became_peered":"2024-09-16T10:32:20.423737+0000","last_unstale":"2024-09-16T10:33:31.808000+0000","last_undegraded":"2024-09-16T10:33:31.808000+0000","last_fullsized":"2024-09-16T10:33:31.808000+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:53.230039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.230039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":760,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807271+0000","last_change":"2024-09-16T10:32:23.425365+0000","last_active":"2024-09-16T10:33:31.807271+0000","last_peered":"2024-09-16T10:33:31.807271+0000","last_clean":"2024-09-16T10:33:31.807271+0000","last_became_active":"2024-09-16T10:32:23.424848+0000","last_became_peered":"2024-09-16T10:32:23.424848+0000","last_unstale":"2024-09-16T10:33:31.807271+0000","last_undegraded":"2024-09-16T10:33:31.807271+0000","last_fullsized":"2024-09-16T10:33:31.807271+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:29.763371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:29.763371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:46.291175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"328'27","reported_seq":1134,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807216+0000","last_change":"2024-09-16T10:31:14.162215+0000","last_active":"2024-09-16T10:33:31.807216+0000","last_peered":"2024-09-16T10:33:31.807216+0000","last_clean":"2024-09-16T10:33:31.807216+0000","last_became_active":"2024-09-16T10:31:14.136662+0000","last_became_peered":"2024-09-16T10:31:14.136662+0000","last_unstale":"2024-09-16T10:33:31.807216+0000","last_undegraded":"2024-09-16T10:33:31.807216+0000","last_fullsized":"2024-09-16T10:33:31.807216+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'27","last_scrub_stamp":"2024-09-16T10:30:55.825936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:55.825936+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":431,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134895+0000","last_change":"2024-09-16T10:32:20.422676+0000","last_active":"2024-09-16T10:33:31.134895+0000","last_peered":"2024-09-16T10:33:31.134895+0000","last_clean":"2024-09-16T10:33:31.134895+0000","last_became_active":"2024-09-16T10:32:20.391995+0000","last_became_peered":"2024-09-16T10:32:20.391995+0000","last_unstale":"2024-09-16T10:33:31.134895+0000","last_undegraded":"2024-09-16T10:33:31.134895+0000","last_fullsized":"2024-09-16T10:33:31.134895+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:32:09.730242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.730242+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"536'13","reported_seq":618,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260580+0000","last_change":"2024-09-16T10:32:20.449596+0000","last_active":"2024-09-16T10:33:31.260580+0000","last_peered":"2024-09-16T10:33:31.260580+0000","last_clean":"2024-09-16T10:33:31.260580+0000","last_became_active":"2024-09-16T10:32:20.398138+0000","last_became_peered":"2024-09-16T10:32:20.398138+0000","last_unstale":"2024-09-16T10:33:31.260580+0000","last_undegraded":"2024-09-16T10:33:31.260580+0000","last_fullsized":"2024-09-16T10:33:31.260580+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'11","last_scrub_stamp":"2024-09-16T10:30:53.694883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:53.694883+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1073,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807251+0000","last_change":"2024-09-16T10:30:07.792293+0000","last_active":"2024-09-16T10:33:31.807251+0000","last_peered":"2024-09-16T10:33:31.807251+0000","last_clean":"2024-09-16T10:33:31.807251+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:33:31.807251+0000","last_undegraded":"2024-09-16T10:33:31.807251+0000","last_fullsized":"2024-09-16T10:33:31.807251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:07.792060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.792060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"73'79","reported_seq":665,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807606+0000","last_change":"2024-09-16T10:32:23.427047+0000","last_active":"2024-09-16T10:33:31.807606+0000","last_peered":"2024-09-16T10:33:31.807606+0000","last_clean":"2024-09-16T10:33:31.807606+0000","last_became_active":"2024-09-16T10:32:23.426646+0000","last_became_peered":"2024-09-16T10:32:23.426646+0000","last_unstale":"2024-09-16T10:33:31.807606+0000","last_undegraded":"2024-09-16T10:33:31.807606+0000","last_fullsized":"2024-09-16T10:33:31.807606+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:09.269989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.269989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1059,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260243+0000","last_change":"2024-09-16T10:33:07.769547+0000","last_active":"2024-09-16T10:33:31.260243+0000","last_peered":"2024-09-16T10:33:31.260243+0000","last_clean":"2024-09-16T10:33:31.260243+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:33:31.260243+0000","last_undegraded":"2024-09-16T10:33:31.260243+0000","last_fullsized":"2024-09-16T10:33:31.260243+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:07.769371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:07.769371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:15.987044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"328'12","reported_seq":1121,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808833+0000","last_change":"2024-09-16T10:32:20.455088+0000","last_active":"2024-09-16T10:33:31.808833+0000","last_peered":"2024-09-16T10:33:31.808833+0000","last_clean":"2024-09-16T10:33:31.808833+0000","last_became_active":"2024-09-16T10:32:20.425287+0000","last_became_peered":"2024-09-16T10:32:20.425287+0000","last_unstale":"2024-09-16T10:33:31.808833+0000","last_undegraded":"2024-09-16T10:33:31.808833+0000","last_fullsized":"2024-09-16T10:33:31.808833+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'12","last_scrub_stamp":"2024-09-16T10:30:50.226333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.226333+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1059,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808097+0000","last_change":"2024-09-16T10:32:20.474305+0000","last_active":"2024-09-16T10:33:31.808097+0000","last_peered":"2024-09-16T10:33:31.808097+0000","last_clean":"2024-09-16T10:33:31.808097+0000","last_became_active":"2024-09-16T10:32:20.473984+0000","last_became_peered":"2024-09-16T10:32:20.473984+0000","last_unstale":"2024-09-16T10:33:31.808097+0000","last_undegraded":"2024-09-16T10:33:31.808097+0000","last_fullsized":"2024-09-16T10:33:31.808097+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:26:46.060999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:26:46.060999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:25.381252+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":748,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261283+0000","last_change":"2024-09-16T10:31:14.137702+0000","last_active":"2024-09-16T10:33:31.261283+0000","last_peered":"2024-09-16T10:33:31.261283+0000","last_clean":"2024-09-16T10:33:31.261283+0000","last_became_active":"2024-09-16T10:31:14.136703+0000","last_became_peered":"2024-09-16T10:31:14.136703+0000","last_unstale":"2024-09-16T10:33:31.261283+0000","last_undegraded":"2024-09-16T10:33:31.261283+0000","last_fullsized":"2024-09-16T10:33:31.261283+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:39.865342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:39.865342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:28.714743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"73'79","reported_seq":529,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808136+0000","last_change":"2024-09-16T10:32:55.165175+0000","last_active":"2024-09-16T10:33:31.808136+0000","last_peered":"2024-09-16T10:33:31.808136+0000","last_clean":"2024-09-16T10:33:31.808136+0000","last_became_active":"2024-09-16T10:32:23.433435+0000","last_became_peered":"2024-09-16T10:32:23.433435+0000","last_unstale":"2024-09-16T10:33:31.808136+0000","last_undegraded":"2024-09-16T10:33:31.808136+0000","last_fullsized":"2024-09-16T10:33:31.808136+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:32:55.165009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:32:55.165009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:12.152556+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"327'15","reported_seq":1116,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134770+0000","last_change":"2024-09-16T10:31:14.165639+0000","last_active":"2024-09-16T10:33:31.134770+0000","last_peered":"2024-09-16T10:33:31.134770+0000","last_clean":"2024-09-16T10:33:31.134770+0000","last_became_active":"2024-09-16T10:31:14.151744+0000","last_became_peered":"2024-09-16T10:31:14.151744+0000","last_unstale":"2024-09-16T10:33:31.134770+0000","last_undegraded":"2024-09-16T10:33:31.134770+0000","last_fullsized":"2024-09-16T10:33:31.134770+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:30:07.157828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:30:07.157828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:25.317922+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":412,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261265+0000","last_change":"2024-09-16T10:31:14.137225+0000","last_active":"2024-09-16T10:33:31.261265+0000","last_peered":"2024-09-16T10:33:31.261265+0000","last_clean":"2024-09-16T10:33:31.261265+0000","last_became_active":"2024-09-16T10:31:14.132504+0000","last_became_peered":"2024-09-16T10:31:14.132504+0000","last_unstale":"2024-09-16T10:33:31.261265+0000","last_undegraded":"2024-09-16T10:33:31.261265+0000","last_fullsized":"2024-09-16T10:33:31.261265+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:44.598219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:44.598219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1090,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808623+0000","last_change":"2024-09-16T10:32:20.425604+0000","last_active":"2024-09-16T10:33:31.808623+0000","last_peered":"2024-09-16T10:33:31.808623+0000","last_clean":"2024-09-16T10:33:31.808623+0000","last_became_active":"2024-09-16T10:32:20.424781+0000","last_became_peered":"2024-09-16T10:32:20.424781+0000","last_unstale":"2024-09-16T10:33:31.808623+0000","last_undegraded":"2024-09-16T10:33:31.808623+0000","last_fullsized":"2024-09-16T10:33:31.808623+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:20.058297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:20.058297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":619,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807572+0000","last_change":"2024-09-16T10:31:38.879647+0000","last_active":"2024-09-16T10:33:31.807572+0000","last_peered":"2024-09-16T10:33:31.807572+0000","last_clean":"2024-09-16T10:33:31.807572+0000","last_became_active":"2024-09-16T10:31:14.140004+0000","last_became_peered":"2024-09-16T10:31:14.140004+0000","last_unstale":"2024-09-16T10:33:31.807572+0000","last_undegraded":"2024-09-16T10:33:31.807572+0000","last_fullsized":"2024-09-16T10:33:31.807572+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:38.879579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:38.879579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"73'79","reported_seq":671,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806957+0000","last_change":"2024-09-16T10:32:46.160022+0000","last_active":"2024-09-16T10:33:31.806957+0000","last_peered":"2024-09-16T10:33:31.806957+0000","last_clean":"2024-09-16T10:33:31.806957+0000","last_became_active":"2024-09-16T10:32:46.159664+0000","last_became_peered":"2024-09-16T10:32:46.159664+0000","last_unstale":"2024-09-16T10:33:31.806957+0000","last_undegraded":"2024-09-16T10:33:31.806957+0000","last_fullsized":"2024-09-16T10:33:31.806957+0000","mapping_epoch":546,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:29:34.955153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:29:34.955153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:51.390508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"328'15","reported_seq":1109,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808045+0000","last_change":"2024-09-16T10:33:06.436155+0000","last_active":"2024-09-16T10:33:31.808045+0000","last_peered":"2024-09-16T10:33:31.808045+0000","last_clean":"2024-09-16T10:33:31.808045+0000","last_became_active":"2024-09-16T10:31:14.149019+0000","last_became_peered":"2024-09-16T10:31:14.149019+0000","last_unstale":"2024-09-16T10:33:31.808045+0000","last_undegraded":"2024-09-16T10:33:31.808045+0000","last_fullsized":"2024-09-16T10:33:31.808045+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'15","last_scrub_stamp":"2024-09-16T10:33:06.436076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:06.436076+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:19.257155+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1100,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135961+0000","last_change":"2024-09-16T10:33:18.297728+0000","last_active":"2024-09-16T10:33:31.135961+0000","last_peered":"2024-09-16T10:33:31.135961+0000","last_clean":"2024-09-16T10:33:31.135961+0000","last_became_active":"2024-09-16T10:32:20.416989+0000","last_became_peered":"2024-09-16T10:32:20.416989+0000","last_unstale":"2024-09-16T10:33:31.135961+0000","last_undegraded":"2024-09-16T10:33:31.135961+0000","last_fullsized":"2024-09-16T10:33:31.135961+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:18.297660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:18.297660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:35.219614+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"536'14","reported_seq":1379,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808517+0000","last_change":"2024-09-16T10:32:20.493322+0000","last_active":"2024-09-16T10:33:31.808517+0000","last_peered":"2024-09-16T10:33:31.808517+0000","last_clean":"2024-09-16T10:33:31.808517+0000","last_became_active":"2024-09-16T10:32:20.489447+0000","last_became_peered":"2024-09-16T10:32:20.489447+0000","last_unstale":"2024-09-16T10:33:31.808517+0000","last_undegraded":"2024-09-16T10:33:31.808517+0000","last_fullsized":"2024-09-16T10:33:31.808517+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'12","last_scrub_stamp":"2024-09-16T10:30:42.170419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:42.170419+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":866,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807653+0000","last_change":"2024-09-16T10:32:16.061021+0000","last_active":"2024-09-16T10:33:31.807653+0000","last_peered":"2024-09-16T10:33:31.807653+0000","last_clean":"2024-09-16T10:33:31.807653+0000","last_became_active":"2024-09-16T10:26:28.551438+0000","last_became_peered":"2024-09-16T10:26:28.551438+0000","last_unstale":"2024-09-16T10:33:31.807653+0000","last_undegraded":"2024-09-16T10:33:31.807653+0000","last_fullsized":"2024-09-16T10:33:31.807653+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:16.060965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:16.060965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"73'79","reported_seq":637,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134568+0000","last_change":"2024-09-16T10:26:12.558633+0000","last_active":"2024-09-16T10:33:31.134568+0000","last_peered":"2024-09-16T10:33:31.134568+0000","last_clean":"2024-09-16T10:33:31.134568+0000","last_became_active":"2024-09-16T10:23:31.482639+0000","last_became_peered":"2024-09-16T10:23:31.482639+0000","last_unstale":"2024-09-16T10:33:31.134568+0000","last_undegraded":"2024-09-16T10:33:31.134568+0000","last_fullsized":"2024-09-16T10:33:31.134568+0000","mapping_epoch":322,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:26:12.558542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:26:12.558542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:44.949384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"328'20","reported_seq":1157,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808255+0000","last_change":"2024-09-16T10:33:05.157157+0000","last_active":"2024-09-16T10:33:31.808255+0000","last_peered":"2024-09-16T10:33:31.808255+0000","last_clean":"2024-09-16T10:33:31.808255+0000","last_became_active":"2024-09-16T10:32:20.488822+0000","last_became_peered":"2024-09-16T10:32:20.488822+0000","last_unstale":"2024-09-16T10:33:31.808255+0000","last_undegraded":"2024-09-16T10:33:31.808255+0000","last_fullsized":"2024-09-16T10:33:31.808255+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:05.157106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.157106+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:06.420060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":837,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807021+0000","last_change":"2024-09-16T10:31:34.935251+0000","last_active":"2024-09-16T10:33:31.807021+0000","last_peered":"2024-09-16T10:33:31.807021+0000","last_clean":"2024-09-16T10:33:31.807021+0000","last_became_active":"2024-09-16T10:31:14.132063+0000","last_became_peered":"2024-09-16T10:31:14.132063+0000","last_unstale":"2024-09-16T10:33:31.807021+0000","last_undegraded":"2024-09-16T10:33:31.807021+0000","last_fullsized":"2024-09-16T10:33:31.807021+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:34.935185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:31:34.935185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"536'9","reported_seq":1327,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807905+0000","last_change":"2024-09-16T10:32:20.449280+0000","last_active":"2024-09-16T10:33:31.807905+0000","last_peered":"2024-09-16T10:33:31.807905+0000","last_clean":"2024-09-16T10:33:31.807905+0000","last_became_active":"2024-09-16T10:32:20.436067+0000","last_became_peered":"2024-09-16T10:32:20.436067+0000","last_unstale":"2024-09-16T10:33:31.807905+0000","last_undegraded":"2024-09-16T10:33:31.807905+0000","last_fullsized":"2024-09-16T10:33:31.807905+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:30:56.800991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:56.800991+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1073,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134411+0000","last_change":"2024-09-16T10:33:05.136554+0000","last_active":"2024-09-16T10:33:31.134411+0000","last_peered":"2024-09-16T10:33:31.134411+0000","last_clean":"2024-09-16T10:33:31.134411+0000","last_became_active":"2024-09-16T10:31:14.150453+0000","last_became_peered":"2024-09-16T10:31:14.150453+0000","last_unstale":"2024-09-16T10:33:31.134411+0000","last_undegraded":"2024-09-16T10:33:31.134411+0000","last_fullsized":"2024-09-16T10:33:31.134411+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:05.136512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.136512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:09.897407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"328'19","reported_seq":1024,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261168+0000","last_change":"2024-09-16T10:32:20.399950+0000","last_active":"2024-09-16T10:33:31.261168+0000","last_peered":"2024-09-16T10:33:31.261168+0000","last_clean":"2024-09-16T10:33:31.261168+0000","last_became_active":"2024-09-16T10:32:20.390257+0000","last_became_peered":"2024-09-16T10:32:20.390257+0000","last_unstale":"2024-09-16T10:33:31.261168+0000","last_undegraded":"2024-09-16T10:33:31.261168+0000","last_fullsized":"2024-09-16T10:33:31.261168+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:32:09.683973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:09.683973+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":484,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261360+0000","last_change":"2024-09-16T10:32:20.406554+0000","last_active":"2024-09-16T10:33:31.261360+0000","last_peered":"2024-09-16T10:33:31.261360+0000","last_clean":"2024-09-16T10:33:31.261360+0000","last_became_active":"2024-09-16T10:32:20.405826+0000","last_became_peered":"2024-09-16T10:32:20.405826+0000","last_unstale":"2024-09-16T10:33:31.261360+0000","last_undegraded":"2024-09-16T10:33:31.261360+0000","last_fullsized":"2024-09-16T10:33:31.261360+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:50.216239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:50.216239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":69,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807607+0000","last_change":"2024-09-16T10:32:20.434897+0000","last_active":"2024-09-16T10:33:31.807607+0000","last_peered":"2024-09-16T10:33:31.807607+0000","last_clean":"2024-09-16T10:33:31.807607+0000","last_became_active":"2024-09-16T10:32:20.434526+0000","last_became_peered":"2024-09-16T10:32:20.434526+0000","last_unstale":"2024-09-16T10:33:31.807607+0000","last_undegraded":"2024-09-16T10:33:31.807607+0000","last_fullsized":"2024-09-16T10:33:31.807607+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:11.168381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:11.168381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1054,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807998+0000","last_change":"2024-09-16T10:32:10.448088+0000","last_active":"2024-09-16T10:33:31.807998+0000","last_peered":"2024-09-16T10:33:31.807998+0000","last_clean":"2024-09-16T10:33:31.807998+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:33:31.807998+0000","last_undegraded":"2024-09-16T10:33:31.807998+0000","last_fullsized":"2024-09-16T10:33:31.807998+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:10.447999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:32:10.447999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"327'15","reported_seq":708,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261008+0000","last_change":"2024-09-16T10:32:20.464025+0000","last_active":"2024-09-16T10:33:31.261008+0000","last_peered":"2024-09-16T10:33:31.261008+0000","last_clean":"2024-09-16T10:33:31.261008+0000","last_became_active":"2024-09-16T10:32:20.413318+0000","last_became_peered":"2024-09-16T10:32:20.413318+0000","last_unstale":"2024-09-16T10:33:31.261008+0000","last_undegraded":"2024-09-16T10:33:31.261008+0000","last_fullsized":"2024-09-16T10:33:31.261008+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:29:19.697962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.697962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:43.705811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":471,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.809014+0000","last_change":"2024-09-16T10:31:14.144963+0000","last_active":"2024-09-16T10:33:31.809014+0000","last_peered":"2024-09-16T10:33:31.809014+0000","last_clean":"2024-09-16T10:33:31.809014+0000","last_became_active":"2024-09-16T10:31:14.144634+0000","last_became_peered":"2024-09-16T10:31:14.144634+0000","last_unstale":"2024-09-16T10:33:31.809014+0000","last_undegraded":"2024-09-16T10:33:31.809014+0000","last_fullsized":"2024-09-16T10:33:31.809014+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:03.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:03.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1022,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808100+0000","last_change":"2024-09-16T10:32:20.415681+0000","last_active":"2024-09-16T10:33:31.808100+0000","last_peered":"2024-09-16T10:33:31.808100+0000","last_clean":"2024-09-16T10:33:31.808100+0000","last_became_active":"2024-09-16T10:32:20.415323+0000","last_became_peered":"2024-09-16T10:32:20.415323+0000","last_unstale":"2024-09-16T10:33:31.808100+0000","last_undegraded":"2024-09-16T10:33:31.808100+0000","last_fullsized":"2024-09-16T10:33:31.808100+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:25:40.451599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:25:40.451599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:45.144732+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1046,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134102+0000","last_change":"2024-09-16T10:32:20.402165+0000","last_active":"2024-09-16T10:33:31.134102+0000","last_peered":"2024-09-16T10:33:31.134102+0000","last_clean":"2024-09-16T10:33:31.134102+0000","last_became_active":"2024-09-16T10:32:20.396229+0000","last_became_peered":"2024-09-16T10:32:20.396229+0000","last_unstale":"2024-09-16T10:33:31.134102+0000","last_undegraded":"2024-09-16T10:33:31.134102+0000","last_fullsized":"2024-09-16T10:33:31.134102+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:52.389262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:52.389262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"73'79","reported_seq":1019,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806886+0000","last_change":"2024-09-16T10:32:23.440126+0000","last_active":"2024-09-16T10:33:31.806886+0000","last_peered":"2024-09-16T10:33:31.806886+0000","last_clean":"2024-09-16T10:33:31.806886+0000","last_became_active":"2024-09-16T10:32:23.439740+0000","last_became_peered":"2024-09-16T10:32:23.439740+0000","last_unstale":"2024-09-16T10:33:31.806886+0000","last_undegraded":"2024-09-16T10:33:31.806886+0000","last_fullsized":"2024-09-16T10:33:31.806886+0000","mapping_epoch":537,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:31:31.897681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:31:31.897681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"328'19","reported_seq":1099,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807877+0000","last_change":"2024-09-16T10:31:14.175984+0000","last_active":"2024-09-16T10:33:31.807877+0000","last_peered":"2024-09-16T10:33:31.807877+0000","last_clean":"2024-09-16T10:33:31.807877+0000","last_became_active":"2024-09-16T10:31:14.154997+0000","last_became_peered":"2024-09-16T10:31:14.154997+0000","last_unstale":"2024-09-16T10:33:31.807877+0000","last_undegraded":"2024-09-16T10:33:31.807877+0000","last_fullsized":"2024-09-16T10:33:31.807877+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:29:19.861030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:29:19.861030+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:25.298353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1101,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808265+0000","last_change":"2024-09-16T10:32:20.474361+0000","last_active":"2024-09-16T10:33:31.808265+0000","last_peered":"2024-09-16T10:33:31.808265+0000","last_clean":"2024-09-16T10:33:31.808265+0000","last_became_active":"2024-09-16T10:32:20.474067+0000","last_became_peered":"2024-09-16T10:32:20.474067+0000","last_unstale":"2024-09-16T10:33:31.808265+0000","last_undegraded":"2024-09-16T10:33:31.808265+0000","last_fullsized":"2024-09-16T10:33:31.808265+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:07.250964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:07.250964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":761,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260679+0000","last_change":"2024-09-16T10:32:13.704695+0000","last_active":"2024-09-16T10:33:31.260679+0000","last_peered":"2024-09-16T10:33:31.260679+0000","last_clean":"2024-09-16T10:33:31.260679+0000","last_became_active":"2024-09-16T10:31:14.132459+0000","last_became_peered":"2024-09-16T10:31:14.132459+0000","last_unstale":"2024-09-16T10:33:31.260679+0000","last_undegraded":"2024-09-16T10:33:31.260679+0000","last_fullsized":"2024-09-16T10:33:31.260679+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:13.704652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:32:13.704652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:36.717155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1070,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.808253+0000","last_change":"2024-09-16T10:30:45.211710+0000","last_active":"2024-09-16T10:33:31.808253+0000","last_peered":"2024-09-16T10:33:31.808253+0000","last_clean":"2024-09-16T10:33:31.808253+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:33:31.808253+0000","last_undegraded":"2024-09-16T10:33:31.808253+0000","last_fullsized":"2024-09-16T10:33:31.808253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:45.211638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:45.211638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"327'8","reported_seq":1134,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806788+0000","last_change":"2024-09-16T10:32:20.445935+0000","last_active":"2024-09-16T10:33:31.806788+0000","last_peered":"2024-09-16T10:33:31.806788+0000","last_clean":"2024-09-16T10:33:31.806788+0000","last_became_active":"2024-09-16T10:32:20.396707+0000","last_became_peered":"2024-09-16T10:32:20.396707+0000","last_unstale":"2024-09-16T10:33:31.806788+0000","last_undegraded":"2024-09-16T10:33:31.806788+0000","last_fullsized":"2024-09-16T10:33:31.806788+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:32:12.365003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.365003+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:33:36.671931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":1065,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.135094+0000","last_change":"2024-09-16T10:30:21.275187+0000","last_active":"2024-09-16T10:33:31.135094+0000","last_peered":"2024-09-16T10:33:31.135094+0000","last_clean":"2024-09-16T10:33:31.135094+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:33:31.135094+0000","last_undegraded":"2024-09-16T10:33:31.135094+0000","last_fullsized":"2024-09-16T10:33:31.135094+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:30:21.275103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:30:21.275103+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1085,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.260732+0000","last_change":"2024-09-16T10:31:14.139140+0000","last_active":"2024-09-16T10:33:31.260732+0000","last_peered":"2024-09-16T10:33:31.260732+0000","last_clean":"2024-09-16T10:33:31.260732+0000","last_became_active":"2024-09-16T10:31:14.138772+0000","last_became_peered":"2024-09-16T10:31:14.138772+0000","last_unstale":"2024-09-16T10:33:31.260732+0000","last_undegraded":"2024-09-16T10:33:31.260732+0000","last_fullsized":"2024-09-16T10:33:31.260732+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:51.649610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:51.649610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":703,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807020+0000","last_change":"2024-09-16T10:31:40.861638+0000","last_active":"2024-09-16T10:33:31.807020+0000","last_peered":"2024-09-16T10:33:31.807020+0000","last_clean":"2024-09-16T10:33:31.807020+0000","last_became_active":"2024-09-16T10:31:14.152053+0000","last_became_peered":"2024-09-16T10:31:14.152053+0000","last_unstale":"2024-09-16T10:33:31.807020+0000","last_undegraded":"2024-09-16T10:33:31.807020+0000","last_fullsized":"2024-09-16T10:33:31.807020+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:31:40.861570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:31:40.861570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":70,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.807727+0000","last_change":"2024-09-16T10:32:20.401586+0000","last_active":"2024-09-16T10:33:31.807727+0000","last_peered":"2024-09-16T10:33:31.807727+0000","last_clean":"2024-09-16T10:33:31.807727+0000","last_became_active":"2024-09-16T10:32:20.401249+0000","last_became_peered":"2024-09-16T10:32:20.401249+0000","last_unstale":"2024-09-16T10:33:31.807727+0000","last_undegraded":"2024-09-16T10:33:31.807727+0000","last_fullsized":"2024-09-16T10:33:31.807727+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:15.832904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:30:15.832904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:27.404759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1081,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.806950+0000","last_change":"2024-09-16T10:32:20.428692+0000","last_active":"2024-09-16T10:33:31.806950+0000","last_peered":"2024-09-16T10:33:31.806950+0000","last_clean":"2024-09-16T10:33:31.806950+0000","last_became_active":"2024-09-16T10:32:20.428421+0000","last_became_peered":"2024-09-16T10:32:20.428421+0000","last_unstale":"2024-09-16T10:33:31.806950+0000","last_undegraded":"2024-09-16T10:33:31.806950+0000","last_fullsized":"2024-09-16T10:33:31.806950+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:32:15.364448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:32:15.364448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"328'19","reported_seq":1138,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.261751+0000","last_change":"2024-09-16T10:33:00.718946+0000","last_active":"2024-09-16T10:33:31.261751+0000","last_peered":"2024-09-16T10:33:31.261751+0000","last_clean":"2024-09-16T10:33:31.261751+0000","last_became_active":"2024-09-16T10:32:20.395308+0000","last_became_peered":"2024-09-16T10:32:20.395308+0000","last_unstale":"2024-09-16T10:33:31.261751+0000","last_undegraded":"2024-09-16T10:33:31.261751+0000","last_fullsized":"2024-09-16T10:33:31.261751+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'19","last_scrub_stamp":"2024-09-16T10:33:00.718864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:00.718864+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:18.336822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1099,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-16T10:33:31.134631+0000","last_change":"2024-09-16T10:32:20.414016+0000","last_active":"2024-09-16T10:33:31.134631+0000","last_peered":"2024-09-16T10:33:31.134631+0000","last_clean":"2024-09-16T10:33:31.134631+0000","last_became_active":"2024-09-16T10:32:20.413184+0000","last_became_peered":"2024-09-16T10:32:20.413184+0000","last_unstale":"2024-09-16T10:33:31.134631+0000","last_undegraded":"2024-09-16T10:33:31.134631+0000","last_fullsized":"2024-09-16T10:33:31.134631+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:30:54.399132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:30:54.399132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1110,"num_read_kb":867,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":6068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":5551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":1894080577598,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528567,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834884,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92910268,"statfs":{"total":95995035648,"available":95140114432,"internally_reserved":0,"allocated":1290240,"data_stored":828172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":853604806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790205,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776504,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92968648,"statfs":{"total":95995035648,"available":95199895552,"internally_reserved":0,"allocated":1212416,"data_stored":763779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986434,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681848,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063304,"statfs":{"total":95995035648,"available":95296823296,"internally_reserved":0,"allocated":950272,"data_stored":500143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215367,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917080,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828072,"statfs":{"total":95995035648,"available":95055945728,"internally_reserved":0,"allocated":1572864,"data_stored":1092705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":937490888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411597,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622380,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":93122772,"statfs":{"total":95995035648,"available":95357718528,"internally_reserved":0,"allocated":1200128,"data_stored":762391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575120,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738646,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934840,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92810312,"statfs":{"total":95995035648,"available":95037759488,"internally_reserved":0,"allocated":1282048,"data_stored":828645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:37.213 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-09-16T10:33:37.214 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-16T10:33:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:36 smithi165 ceph-mon[32696]: pgmap v1077: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:33:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:36 smithi165 ceph-mon[32696]: from='client.46745 v1:172.21.15.63:0/1241333132' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:33:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:36 smithi165 ceph-mon[32696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T10:33:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:36 smithi165 ceph-mon[32696]: from='client.46751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T10:33:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:36 smithi165 ceph-mon[32696]: osdmap e564: 8 total, 8 up, 6 in 2024-09-16T10:33:37.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:36 smithi165 ceph-mon[32696]: osdmap e565: 8 total, 8 up, 6 in 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: "size": 20, 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 600, 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: "ops": [ 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.14514.0:3543 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 93976561469696 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:13.112937+0000", 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 144.60657236500001, 2024-09-16T10:33:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.062285218, 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "started", 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.14514", 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 3543 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.724 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112937+0000", 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112937+0000", 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112939+0000", 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2680000000000001e-06 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112946+0000", 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0430000000000002e-06 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112963+0000", 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.734e-05 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.112972+0000", 2024-09-16T10:33:37.726 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.7770000000000008e-06 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.113039+0000", 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7142000000000002e-05 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op must wait for map", 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.113046+0000", 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3069999999999999e-06 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.727 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.145459+0000", 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.032413260999999999 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for peered", 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:13.145468+0000", 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8959999999999992e-06 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.151255+0000", 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0057870840000001 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.728 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for degraded object", 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.151570+0000", 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031488799999999998 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.158957+0000", 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0073872319999999997 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.159066+0000", 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000108924 2024-09-16T10:33:37.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:14.175222+0000", 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.016156055999999999 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_repop(client.26491.0:127346 11.1 e509/476 11:852ef2a9:::benchmark_data_smithi063_176907_object3070:head v 509'7982, mlcod=507'7980)", 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:15.235619+0000", 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 142.48389023999999, 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.28887845699999998, 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235619+0000", 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235619+0000", 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235620+0000", 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1659999999999999e-06 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235627+0000", 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.4790000000000001e-06 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235635+0000", 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.901e-06 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235642+0000", 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2870000000000001e-06 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235857+0000", 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000215639 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.235876+0000", 2024-09-16T10:33:37.733 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.9069999999999999e-05 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.524434+0000", 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.28855762800000001 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:15.524497+0000", 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3287000000000006e-05 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:151673 11.b 11:d5f0724c:::benchmark_data_smithi063_176907_object27397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:39.540122+0000", 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 118.17938721, 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.29731222099999999, 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 151673 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540122+0000", 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540122+0000", 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.736 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540124+0000", 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.2359999999999999e-06 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540130+0000", 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8510000000000004e-06 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540141+0000", 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.093e-05 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540148+0000", 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4390000000000003e-06 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.738 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540833+0000", 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00068436799999999998 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.540953+0000", 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120685 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.541071+0000", 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000117136 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.543941+0000", 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028703090000000001 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.547190+0000", 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0032495639999999999 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.837327+0000", 2024-09-16T10:33:37.740 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.29013633300000002 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.837393+0000", 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6080000000000004e-05 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.837434+0000", 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.1289999999999999e-05 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.741 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:151682 11.b 11:d3c875f7:::benchmark_data_smithi063_176907_object27406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:39.544624+0000", 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 118.174884702, 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.301245704, 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.742 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 151682 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544624+0000", 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544624+0000", 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544626+0000", 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.8810000000000001e-06 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544631+0000", 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0760000000000002e-06 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.744 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544642+0000", 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0413e-05 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544649+0000", 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.216e-06 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544752+0000", 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000102822 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.745 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544889+0000", 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013679999999999999 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1", 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.544995+0000", 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000106654 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.547247+0000", 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.002251435 2024-09-16T10:33:37.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.547566+0000", 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00031900400000000001 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.845719+0000", 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.298153059 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.747 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.845764+0000", 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.4996000000000001e-05 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:39.845870+0000", 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010634800000000001 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.748 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:164823 11.9 11:9c734485:::benchmark_data_smithi063_176907_object40547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e519)", 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:31:52.893527+0000", 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 104.825981919, 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.286719524, 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 164823 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.749 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893527+0000", 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893527+0000", 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893529+0000", 2024-09-16T10:33:37.750 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.308e-06 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893534+0000", 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5079999999999996e-06 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893542+0000", 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2400000000000007e-06 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.751 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893549+0000", 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9890000000000003e-06 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893610+0000", 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0234e-05 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893715+0000", 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00010527999999999999 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.752 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.893826+0000", 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011135499999999999 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.897595+0000", 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0037689350000000002 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:52.903457+0000", 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0058624089999999998 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:53.180165+0000", 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27670773599999998 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:53.180211+0000", 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5881000000000003e-05 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:31:53.180247+0000", 2024-09-16T10:33:37.754 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5648999999999997e-05 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194237 11.b 11:da4e725c:::benchmark_data_smithi063_176907_object69961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.128474+0000", 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.591035275999999, 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.38263244200000002, 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.755 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194237 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.128474+0000", 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.128474+0000", 2024-09-16T10:33:37.756 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.128476+0000", 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.0499999999999999e-06 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.128482+0000", 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1279999999999996e-06 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.757 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.128493+0000", 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1148000000000001e-05 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.128501+0000", 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4399999999999999e-06 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134485+0000", 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0059848200000000001 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.758 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134610+0000", 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012410399999999999 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,4", 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134734+0000", 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012414899999999999 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140733+0000", 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0059992470000000001 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.143750+0000", 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0030174379999999999 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148541+0000", 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0047903809999999998 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.510936+0000", 2024-09-16T10:33:37.760 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36239536500000002 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.510996+0000", 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.0254000000000003e-05 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.511106+0000", 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000109918 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194245 11.b 11:d9a9194d:::benchmark_data_smithi063_176907_object69969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.133889+0000", 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.585620250000005, 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.38075503599999999, 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.762 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194245 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.133889+0000", 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.133889+0000", 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.763 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.133889+0000", 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8699999999999995e-07 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.133893+0000", 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1089999999999998e-06 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.133898+0000", 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.6149999999999996e-06 2024-09-16T10:33:37.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.133903+0000", 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.169e-06 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136779+0000", 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0028752059999999999 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.765 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136904+0000", 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012508800000000001 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,4", 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.137031+0000", 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012714900000000001 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.143798+0000", 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0067667209999999998 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.766 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145593+0000", 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017951810000000001 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148598+0000", 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.003004891 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514481+0000", 2024-09-16T10:33:37.767 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36588312200000001 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514530+0000", 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9419000000000002e-05 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514644+0000", 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113779 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194247 11.b 11:d7c6a0ff:::benchmark_data_smithi063_176907_object69971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.134434+0000", 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.585074758999994, 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.38050467999999998, 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194247 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134434+0000", 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134434+0000", 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134436+0000", 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1209999999999999e-06 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134440+0000", 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2350000000000001e-06 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.771 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134450+0000", 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0285999999999999e-05 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.134457+0000", 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1899999999999998e-06 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140854+0000", 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.006396751 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140974+0000", 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000120059 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,4", 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.141100+0000", 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012585299999999999 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148558+0000", 2024-09-16T10:33:37.773 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0074585199999999997 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148726+0000", 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00016767999999999999 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148785+0000", 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8631000000000002e-05 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514761+0000", 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36597608199999998 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514826+0000", 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.5705999999999997e-05 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514939+0000", 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011256600000000001 2024-09-16T10:33:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194248 11.0 11:02c5ef5c:::benchmark_data_smithi063_176907_object69972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.136496+0000", 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.583013335999993, 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.37319681500000002, 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194248 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136496+0000", 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136496+0000", 2024-09-16T10:33:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136497+0000", 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.296e-06 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136503+0000", 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1789999999999996e-06 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136514+0000", 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0360000000000001e-05 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.136521+0000", 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6199999999999999e-06 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.141524+0000", 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0050029549999999999 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.141636+0000", 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111294 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,6", 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.141815+0000", 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000179428 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.147467+0000", 2024-09-16T10:33:37.780 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0056518769999999996 2024-09-16T10:33:37.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.781 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.781 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.147594+0000", 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000127329 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.147882+0000", 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00028809000000000001 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.782 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.509520+0000", 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36163811899999998 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.509580+0000", 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9175999999999997e-05 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.509693+0000", 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011309199999999999 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.783 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194253 11.9 11:9ab29d2c:::benchmark_data_smithi063_176907_object69977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.140184+0000", 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.579324765999999, 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.374182654, 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.784 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194253 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140184+0000", 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.785 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140184+0000", 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140186+0000", 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6559999999999999e-06 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140191+0000", 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3360000000000002e-06 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140202+0000", 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0708e-05 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140210+0000", 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8460000000000008e-06 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140771+0000", 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00056151200000000001 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.140893+0000", 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012129999999999999 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.141055+0000", 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000162526 2024-09-16T10:33:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146333+0000", 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0052773070000000002 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148093+0000", 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001760129 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514265+0000", 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36617180599999999 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514327+0000", 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.2175999999999996e-05 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.514367+0000", 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.0352000000000003e-05 2024-09-16T10:33:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194255 11.b 11:d3a5c157:::benchmark_data_smithi063_176907_object69979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.145644+0000", 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.573864997000001, 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36961219699999998, 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194255 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145644+0000", 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145644+0000", 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145646+0000", 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.891e-06 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145652+0000", 2024-09-16T10:33:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.9909999999999999e-06 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145663+0000", 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0906e-05 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145670+0000", 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.0280000000000003e-06 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145749+0000", 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.9290999999999997e-05 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145976+0000", 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00022711199999999999 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,4", 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146101+0000", 2024-09-16T10:33:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012502300000000001 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148564+0000", 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0024629650000000001 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148837+0000", 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027306699999999999 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.152989+0000", 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041513770000000004 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515090+0000", 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36210105599999998 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515145+0000", 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4864000000000002e-05 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515256+0000", 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111626 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194256 11.b 11:d4ce1229:::benchmark_data_smithi063_176907_object69980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.145696+0000", 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.573813361000006, 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36995960999999999, 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194256 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145696+0000", 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145696+0000", 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145696+0000", 2024-09-16T10:33:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.8400000000000004e-07 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145699+0000", 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.0340000000000001e-06 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145709+0000", 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 9.0359999999999995e-06 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.145714+0000", 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5570000000000003e-06 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148870+0000", 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0031559840000000001 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.148984+0000", 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000113811 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,4", 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.149109+0000", 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00012476699999999999 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.153056+0000", 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0039478300000000003 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.153108+0000", 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1724999999999997e-05 2024-09-16T10:33:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.153979+0000", 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00087084399999999996 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515476+0000", 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36149698200000002 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515543+0000", 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7158999999999999e-05 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515655+0000", 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112197 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194257 11.b 11:daa733f0:::benchmark_data_smithi063_176907_object69981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.146837+0000", 2024-09-16T10:33:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.572672139999995, 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36907925800000002, 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194257 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146837+0000", 2024-09-16T10:33:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146837+0000", 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146839+0000", 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.6199999999999999e-06 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146844+0000", 2024-09-16T10:33:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.9830000000000004e-06 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146856+0000", 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1919999999999999e-05 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.146863+0000", 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.1960000000000002e-06 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.808 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.151112+0000", 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0042490469999999997 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.151225+0000", 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011312299999999999 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,4", 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.151367+0000", 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000142266 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.153145+0000", 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0017779320000000001 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.153441+0000", 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000295533 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.154110+0000", 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00066975900000000002 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515714+0000", 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.36160399500000001 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515789+0000", 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.4720000000000003e-05 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515916+0000", 2024-09-16T10:33:37.811 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000127164 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:194267 11.b 11:de6f9489:::benchmark_data_smithi063_176907_object69991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:32:24.157737+0000", 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 73.561771716999999, 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.35839808000000001, 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.812 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 194267 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157737+0000", 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157737+0000", 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157739+0000", 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.5799999999999999e-06 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157744+0000", 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.938e-06 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157755+0000", 2024-09-16T10:33:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.1541e-05 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157763+0000", 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.5519999999999999e-06 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157835+0000", 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.2173000000000002e-05 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.815 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.157972+0000", 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000136954 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,1,4", 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.158123+0000", 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00015108499999999999 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.160020+0000", 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018963859999999999 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.816 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.160044+0000", 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 2.4794e-05 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.160338+0000", 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00029311199999999999 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.515968+0000", 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.35563034799999999 2024-09-16T10:33:37.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.516019+0000", 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1566000000000001e-05 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:32:24.516136+0000", 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000116051 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:246025 11.9 11:9406944b:::benchmark_data_smithi063_176907_object121749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:17.473980+0000", 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 20.245529620999999, 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.29001559500000001, 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 246025 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.473980+0000", 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.473980+0000", 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.473980+0000", 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.6700000000000003e-07 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.473984+0000", 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.5480000000000002e-06 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.473990+0000", 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3330000000000002e-06 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.473996+0000", 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.8080000000000001e-06 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.489388+0000", 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.015392375999999999 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.489737+0000", 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00034826600000000002 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.489867+0000", 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00013079000000000001 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.491987+0000", 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0021195290000000002 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.493946+0000", 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0019588539999999999 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763862+0000", 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.269916561 2024-09-16T10:33:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763932+0000", 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9343000000000003e-05 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763995+0000", 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3419999999999999e-05 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:246035 11.b 11:d5c24661:::benchmark_data_smithi063_176907_object121759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:17.481167+0000", 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 20.238341896000001, 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27623598199999999, 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 246035 2024-09-16T10:33:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.481167+0000", 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.481167+0000", 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.481169+0000", 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.621e-06 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.481174+0000", 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.3519999999999997e-06 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.481182+0000", 2024-09-16T10:33:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.8290000000000008e-06 2024-09-16T10:33:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.481187+0000", 2024-09-16T10:33:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.1830000000000002e-06 2024-09-16T10:33:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482184+0000", 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00099633100000000004 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482293+0000", 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000109026 2024-09-16T10:33:37.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482628+0000", 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00033535900000000003 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.487199+0000", 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0045709610000000001 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.833 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.487475+0000", 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027580600000000002 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.757303+0000", 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.269828387 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.757360+0000", 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7374000000000003e-05 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.757403+0000", 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.2753000000000002e-05 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:246037 11.0 11:081e49bd:::benchmark_data_smithi063_176907_object121761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:17.482450+0000", 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 20.237059244000001, 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.27680940100000001, 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.835 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 246037 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482450+0000", 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482450+0000", 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482451+0000", 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.4020000000000001e-06 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482457+0000", 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.2800000000000003e-06 2024-09-16T10:33:37.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482467+0000", 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0557000000000001e-05 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.482474+0000", 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.9779999999999999e-06 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.838 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.484752+0000", 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0022782850000000001 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.484865+0000", 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000112775 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.484967+0000", 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000101959 2024-09-16T10:33:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.489116+0000", 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0041490900000000002 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.490756+0000", 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001640273 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.757821+0000", 2024-09-16T10:33:37.840 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26706465000000001 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.759205+0000", 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0013840530000000001 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.759259+0000", 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.4098999999999998e-05 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:246055 11.9 11:9186a41e:::benchmark_data_smithi063_176907_object121779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:17.495492+0000", 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 20.224016809999998, 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26881865199999999, 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.842 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 246055 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495492+0000", 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495492+0000", 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495523+0000", 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.1129e-05 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495529+0000", 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.5790000000000003e-06 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.844 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495539+0000", 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.0159e-05 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495552+0000", 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 1.2911999999999999e-05 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495599+0000", 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.6835000000000001e-05 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495683+0000", 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.4493000000000006e-05 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495772+0000", 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8313999999999995e-05 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.497645+0000", 2024-09-16T10:33:37.846 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001872905 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.497966+0000", 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00032118899999999999 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.764221+0000", 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26625469699999998 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.847 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.764278+0000", 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.7896000000000003e-05 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.764311+0000", 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.2543999999999999e-05 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:246056 11.b 11:dc3bcad5:::benchmark_data_smithi063_176907_object121780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:17.495584+0000", 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 20.223924775, 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.267472343, 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 246056 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495584+0000", 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495584+0000", 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495585+0000", 2024-09-16T10:33:37.850 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.8199999999999998e-07 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495589+0000", 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.9469999999999996e-06 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495597+0000", 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 8.2300000000000008e-06 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.851 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495604+0000", 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.6449999999999999e-06 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495651+0000", 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.7077999999999999e-05 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495763+0000", 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000111897 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.852 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495882+0000", 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000119166 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.497490+0000", 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.001607399 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.497820+0000", 2024-09-16T10:33:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00033040000000000001 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.762937+0000", 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26511719900000003 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763011+0000", 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 7.3565000000000005e-05 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763057+0000", 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5935e-05 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "description": "osd_op(client.26491.0:246057 11.0 11:07a7f892:::benchmark_data_smithi063_176907_object121781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "initiated_at": "2024-09-16T10:33:17.495617+0000", 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 20.223892016000001, 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26757458699999997, 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "continuous": false, 2024-09-16T10:33:37.855 INFO:teuthology.orchestra.run.smithi063.stdout: "type_data": { 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "client_info": { 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "client": "client.26491", 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "tid": 246057 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "events": [ 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "initiated", 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495617+0000", 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.856 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "header_read", 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495617+0000", 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "throttled", 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495618+0000", 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.7100000000000001e-07 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "all_read", 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495621+0000", 2024-09-16T10:33:37.857 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.264e-06 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "dispatched", 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495627+0000", 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.1680000000000001e-06 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "queued_for_pg", 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495634+0000", 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 6.3960000000000003e-06 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.858 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "reached_pg", 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495684+0000", 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 5.0720000000000002e-05 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "started", 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495788+0000", 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.000103491 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "waiting for subops from 0,6", 2024-09-16T10:33:37.859 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.495900+0000", 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00011209000000000001 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "op_commit", 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.497789+0000", 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.0018886949999999999 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.498064+0000", 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.00027542400000000002 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763109+0000", 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 0.26504503299999999 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "commit_sent", 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763154+0000", 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 4.5025999999999999e-05 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "event": "done", 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "time": "2024-09-16T10:33:17.763192+0000", 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: "duration": 3.7608999999999999e-05 2024-09-16T10:33:37.861 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-09-16T10:33:37.862 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-09-16T10:33:37.862 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-09-16T10:33:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:38.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[29993]: 5.2 scrub starts 2024-09-16T10:33:38.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[29993]: 5.2 scrub ok 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[39470]: 5.2 scrub starts 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[39470]: 5.2 scrub ok 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:33:38.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:37 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:33:38.209 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:37 smithi165 ceph-mon[32696]: 5.2 scrub starts 2024-09-16T10:33:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:37 smithi165 ceph-mon[32696]: 5.2 scrub ok 2024-09-16T10:33:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:37 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:37 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:37 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:33:38.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:37 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:33:38.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T10:33:39.075 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[29993]: pgmap v1080: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 799 B/s rd, 0 op/s 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[29993]: osdmap e566: 8 total, 8 up, 6 in 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[29993]: 3.13 scrub starts 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[29993]: 3.13 scrub ok 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[39470]: pgmap v1080: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 799 B/s rd, 0 op/s 2024-09-16T10:33:39.080 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:33:39.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:33:39.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[39470]: osdmap e566: 8 total, 8 up, 6 in 2024-09-16T10:33:39.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[39470]: 3.13 scrub starts 2024-09-16T10:33:39.081 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:38 smithi063 ceph-mon[39470]: 3.13 scrub ok 2024-09-16T10:33:39.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:38 smithi165 ceph-mon[32696]: pgmap v1080: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 799 B/s rd, 0 op/s 2024-09-16T10:33:39.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:38 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:33:39.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:38 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:33:39.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:38 smithi165 ceph-mon[32696]: osdmap e566: 8 total, 8 up, 6 in 2024-09-16T10:33:39.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:38 smithi165 ceph-mon[32696]: 3.13 scrub starts 2024-09-16T10:33:39.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:38 smithi165 ceph-mon[32696]: 3.13 scrub ok 2024-09-16T10:33:39.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:33:39.944 INFO:teuthology.orchestra.run.smithi063.stderr:nodeep-scrub is unset 2024-09-16T10:33:39.947 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:39.947 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:39.947 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:33:39.947 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: 6.6 scrub starts 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: 6.6 scrub ok 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:33:40.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:39 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/350609184' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:40.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: 6.6 scrub starts 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: 6.6 scrub ok 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/350609184' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: 6.6 scrub starts 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: 6.6 scrub ok 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:40.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:33:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:33:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:33:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:33:40.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:39 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/350609184' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-16T10:33:40.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T10:33:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 1 pg_num to 11 2024-09-16T10:33:41.015 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: pgmap v1082: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: Cluster is now healthy 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/350609184' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: osdmap e567: 8 total, 8 up, 6 in 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/398051715' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: 1.14 scrub starts 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: 1.14 scrub ok 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:40 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:41.201 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: pgmap v1082: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: Cluster is now healthy 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/350609184' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: osdmap e567: 8 total, 8 up, 6 in 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/398051715' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: 1.14 scrub starts 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: 1.14 scrub ok 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:41.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: pgmap v1082: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: Cluster is now healthy 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/350609184' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: osdmap e567: 8 total, 8 up, 6 in 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/398051715' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: 1.14 scrub starts 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: 1.14 scrub ok 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:41.203 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:40 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:41.562 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:41.562 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:41.562 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:33:41.562 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:33:41.562 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:42.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: 5.9 scrub starts 2024-09-16T10:33:42.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: 5.9 scrub ok 2024-09-16T10:33:42.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/398051715' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:33:42.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: osdmap e568: 8 total, 8 up, 6 in 2024-09-16T10:33:42.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: osdmap e569: 8 total, 8 up, 6 in 2024-09-16T10:33:42.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:33:42.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:33:42.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:33:42.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:41 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:33:42.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T10:33:42.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: 5.9 scrub starts 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: 5.9 scrub ok 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/398051715' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: osdmap e568: 8 total, 8 up, 6 in 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: osdmap e569: 8 total, 8 up, 6 in 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:33:42.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: 5.9 scrub starts 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: 5.9 scrub ok 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/398051715' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: osdmap e568: 8 total, 8 up, 6 in 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: osdmap e569: 8 total, 8 up, 6 in 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:33:42.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:41 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-16T10:33:42.546 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:43.059 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:33:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:33:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:33:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.059 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212280 9.e 9:7a91a301:::benchmark_data_smithi063_124717_object96049:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 522.37094401000002, 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30939322400000002, 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212280 2024-09-16T10:33:43.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670146+0000", 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670148+0000", 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.283e-06 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670154+0000", 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8370000000000002e-06 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670167+0000", 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3243e-05 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670176+0000", 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5969999999999992e-06 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670244+0000", 2024-09-16T10:33:43.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8726999999999994e-05 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670254+0000", 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9210000000000004e-06 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.978436+0000", 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30818199899999998 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979539+0000", 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001102617 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212292 9.e 9:780c7a09:::benchmark_data_smithi063_124717_object96061:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 522.37089228900004, 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31685680300000002, 2024-09-16T10:33:43.065 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212292 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670197+0000", 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670198+0000", 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9599999999999999e-07 2024-09-16T10:33:43.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670201+0000", 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1130000000000001e-06 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670207+0000", 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2689999999999996e-06 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670213+0000", 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.164e-06 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670284+0000", 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1136000000000005e-05 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670291+0000", 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9210000000000003e-06 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.979884+0000", 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30959349200000003 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980142+0000", 2024-09-16T10:33:43.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025797 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.980215+0000", 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3054999999999998e-05 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985118+0000", 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049030380000000002 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985153+0000", 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4496999999999999e-05 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986997+0000", 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018447450000000001 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987028+0000", 2024-09-16T10:33:43.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.099e-05 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.987054+0000", 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5717000000000001e-05 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212294 9.e 9:724d3208:::benchmark_data_smithi063_124717_object96063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:33:43.073 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 522.37086051999995, 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.32240128099999998, 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212294 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670229+0000", 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670230+0000", 2024-09-16T10:33:43.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7999999999999995e-07 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670233+0000", 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9919999999999999e-06 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670239+0000", 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3779999999999998e-06 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670245+0000", 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670316+0000", 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1147000000000002e-05 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.670323+0000", 2024-09-16T10:33:43.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7209999999999997e-06 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981661+0000", 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31133887799999999 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981742+0000", 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0136000000000007e-05 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.981814+0000", 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2065999999999997e-05 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985187+0000", 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003373048 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985193+0000", 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7170000000000002e-06 2024-09-16T10:33:43.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992481+0000", 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072876870000000002 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992565+0000", 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3726000000000004e-05 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992630+0000", 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5581999999999997e-05 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212314 9.e 9:73a00d32:::benchmark_data_smithi063_124717_object96083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 522.34581364400003, 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29758563799999999, 2024-09-16T10:33:43.082 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212314 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695276+0000", 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695278+0000", 2024-09-16T10:33:43.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9800000000000001e-06 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695284+0000", 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9349999999999999e-06 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695295+0000", 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0849000000000001e-05 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695303+0000", 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2130000000000008e-06 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695359+0000", 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5859999999999997e-05 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.695366+0000", 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6580000000000002e-06 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982882+0000", 2024-09-16T10:33:43.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.287515939 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.982949+0000", 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6370000000000003e-05 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.983024+0000", 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5723000000000007e-05 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985197+0000", 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002172195 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986274+0000", 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001077733 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992764+0000", 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064893520000000003 2024-09-16T10:33:43.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992820+0000", 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5903000000000003e-05 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992862+0000", 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1928000000000002e-05 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212323 9.e 9:7cb576e6:::benchmark_data_smithi063_124717_object96092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 522.34031951300005, 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29218721399999997, 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212323 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700770+0000", 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700773+0000", 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5299999999999999e-06 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700779+0000", 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1399999999999997e-06 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700801+0000", 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1812999999999999e-05 2024-09-16T10:33:43.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700809+0000", 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1929999999999993e-06 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700877+0000", 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8199000000000002e-05 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:43.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.700889+0000", 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1904e-05 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984122+0000", 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28323306799999998 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984181+0000", 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8847000000000001e-05 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.984241+0000", 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0056000000000002e-05 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986252+0000", 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020111399999999998 2024-09-16T10:33:43.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986461+0000", 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000209273 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992908+0000", 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006446523 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992935+0000", 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7016999999999999e-05 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992957+0000", 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2510999999999998e-05 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.098 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212342 9.e 9:700843e6:::benchmark_data_smithi063_124717_object96111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 522.32077931399999, 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27274648600000001, 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212342 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720310+0000", 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720312+0000", 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0949999999999998e-06 2024-09-16T10:33:43.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720318+0000", 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1059999999999996e-06 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720333+0000", 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4173e-05 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720340+0000", 2024-09-16T10:33:43.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0389999999999999e-06 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720405+0000", 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5079000000000005e-05 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.720415+0000", 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0351e-05 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985207+0000", 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.264792321 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985273+0000", 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5215999999999994e-05 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:43.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.985329+0000", 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6403000000000002e-05 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986392+0000", 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010631939999999999 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.986576+0000", 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018346000000000001 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.992994+0000", 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064186979999999996 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993027+0000", 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2262999999999999e-05 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:00.993057+0000", 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0088000000000001e-05 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:212688 9.e 9:7ab865c0:::benchmark_data_smithi063_124717_object96457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 521.82009297100001, 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12657935300000001, 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.106 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 212688 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220997+0000", 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.220999+0000", 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221004+0000", 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1880000000000001e-06 2024-09-16T10:33:43.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221017+0000", 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2802e-05 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221023+0000", 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8789999999999997e-06 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221090+0000", 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6854999999999996e-05 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221200+0000", 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010959399999999999 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.221304+0000", 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104456 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.223708+0000", 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024038449999999999 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.224936+0000", 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012280119999999999 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347460+0000", 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122523584 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347521+0000", 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1199999999999997e-05 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.347576+0000", 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4972000000000002e-05 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45596.0:213104 9.e 9:709eca86:::benchmark_data_smithi063_124717_object96873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 521.29643973500004, 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12242840100000001, 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.113 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45596", 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2318671687", 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 213104 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:33:43.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744650+0000", 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744652+0000", 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0219999999999999e-06 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744657+0000", 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3020000000000002e-06 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744668+0000", 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.118e-05 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744674+0000", 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6500000000000001e-06 2024-09-16T10:33:43.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744728+0000", 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3705000000000002e-05 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744820+0000", 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2052000000000006e-05 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.744897+0000", 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7200999999999995e-05 2024-09-16T10:33:43.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.746741+0000", 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018441250000000001 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.747036+0000", 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029480100000000003 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.866958+0000", 2024-09-16T10:33:43.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11992201700000001 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867027+0000", 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9314999999999996e-05 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:01.867078+0000", 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1031e-05 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:41313 10.1 10:8e698f2c:::benchmark_data_smithi063_149043_object41312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 468.79716623299998, 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10554345399999999, 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 41313 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.120 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243923+0000", 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243925+0000", 2024-09-16T10:33:43.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0329999999999998e-06 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.243997+0000", 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1767000000000006e-05 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244022+0000", 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4737999999999999e-05 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244030+0000", 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5699999999999993e-06 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244235+0000", 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020468299999999999 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.244353+0000", 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011790699999999999 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.246158+0000", 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018052179999999999 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.251015+0000", 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004856595 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.251463+0000", 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000448369 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349318+0000", 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097854294999999994 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349407+0000", 2024-09-16T10:33:43.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9024999999999996e-05 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:25:54.349467+0000", 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0254000000000003e-05 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73195 10.e 10:780b6aaa:::benchmark_data_smithi063_149043_object73194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:33:43.126 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.606823179, 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10858509, 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73195 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:33:43.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434266+0000", 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434267+0000", 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434306+0000", 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8250000000000001e-05 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434314+0000", 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2949999999999994e-06 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434319+0000", 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5300000000000004e-06 2024-09-16T10:33:43.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434368+0000", 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8321e-05 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434486+0000", 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117924 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:33:43.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.434622+0000", 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013583099999999999 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.437851+0000", 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032291569999999999 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.438873+0000", 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00102191 2024-09-16T10:33:43.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542761+0000", 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10388875 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542818+0000", 2024-09-16T10:33:43.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6863000000000001e-05 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.542852+0000", 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73199 10.e 10:752bb1d5:::benchmark_data_smithi063_149043_object73198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:33:43.133 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.59902942399998, 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112679315, 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73199 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.134 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442060+0000", 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442062+0000", 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.367e-06 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442108+0000", 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6814999999999998e-05 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442117+0000", 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8410000000000006e-06 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442124+0000", 2024-09-16T10:33:43.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6889999999999998e-06 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442179+0000", 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4756000000000002e-05 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442288+0000", 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109372 2024-09-16T10:33:43.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.442378+0000", 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0488000000000005e-05 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.444289+0000", 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001910293 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.444588+0000", 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000299602 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554621+0000", 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.110032625 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554689+0000", 2024-09-16T10:33:43.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8324000000000005e-05 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554739+0000", 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0142999999999997e-05 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73208 10.e 10:776621e8:::benchmark_data_smithi063_149043_object73207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:33:43.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.59390560100002, 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10772002999999999, 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73208 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447184+0000", 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447185+0000", 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2100000000000001e-06 2024-09-16T10:33:43.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447333+0000", 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014735900000000001 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447344+0000", 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0994e-05 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447351+0000", 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1130000000000003e-06 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447410+0000", 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9219999999999999e-05 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447533+0000", 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123325 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.447677+0000", 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143972 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.449865+0000", 2024-09-16T10:33:43.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021879680000000002 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.450597+0000", 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073198000000000002 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554808+0000", 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104210409 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554861+0000", 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3298999999999999e-05 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554904+0000", 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3180999999999999e-05 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:73211 10.e 10:77f82def:::benchmark_data_smithi063_149043_object73210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.58834376300001, 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10288251299999999, 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73211 2024-09-16T10:33:43.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452746+0000", 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452747+0000", 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.643e-06 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452838+0000", 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0611000000000004e-05 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452849+0000", 2024-09-16T10:33:43.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0519e-05 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.452855+0000", 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454022+0000", 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001166582 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454155+0000", 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133013 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.454307+0000", 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015232200000000001 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.459163+0000", 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048564769999999997 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.459272+0000", 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109069 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.554980+0000", 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095707424999999999 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.555077+0000", 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6832999999999999e-05 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:15.555628+0000", 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055162699999999996 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.158 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:80880 10.1 10:8f0a7a66:::benchmark_data_smithi063_149043_object80879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 441.72626605400001, 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109496315, 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 80880 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314824+0000", 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314825+0000", 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.689e-06 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314901+0000", 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5549000000000005e-05 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314912+0000", 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270999999999999e-05 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314917+0000", 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.169e-06 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314973+0000", 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5448999999999997e-05 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315116+0000", 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142986 2024-09-16T10:33:43.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315268+0000", 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015202799999999999 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.319831+0000", 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004563302 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.319869+0000", 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7737000000000001e-05 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424204+0000", 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104335386 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424277+0000", 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2435999999999996e-05 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.424320+0000", 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3312999999999997e-05 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:80882 10.1 10:8d1b387e:::benchmark_data_smithi063_149043_object80881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 441.72613709699999, 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111081265, 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.165 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 80882 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314952+0000", 2024-09-16T10:33:43.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.314954+0000", 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.319e-06 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315013+0000", 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9150000000000001e-05 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315022+0000", 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9309999999999997e-06 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.315027+0000", 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6360000000000002e-06 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317727+0000", 2024-09-16T10:33:43.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027003840000000001 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317832+0000", 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010535999999999999 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.317972+0000", 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014001700000000001 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:43.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.320350+0000", 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023780020000000002 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.320544+0000", 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019368200000000001 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.425942+0000", 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105398164 2024-09-16T10:33:43.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.425992+0000", 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9570000000000001e-05 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:21.426034+0000", 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2049999999999999e-05 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221552 10.c e446/442 10:30346976:::benchmark_data_smithi063_149043_object89115:head v 446'13317, mlcod=446'13315)", 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.613310+0000", 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 306.42777956200001, 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.085401337999999993, 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613310+0000", 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613310+0000", 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613311+0000", 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.006e-06 2024-09-16T10:33:43.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613316+0000", 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0100000000000003e-06 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613323+0000", 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6769999999999998e-06 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.613328+0000", 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4060000000000004e-06 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.689983+0000", 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.076655128000000003 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.690009+0000", 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5837e-05 2024-09-16T10:33:43.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.698621+0000", 2024-09-16T10:33:43.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086123359999999999 2024-09-16T10:33:43.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.698711+0000", 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9938000000000005e-05 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221553 10.12 e446/414 10:4f7e32bd:::benchmark_data_smithi063_149043_object89116:head v 446'12226, mlcod=446'12224)", 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.614067+0000", 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 306.42702212400002, 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.082785922999999997, 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.177 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614067+0000", 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614067+0000", 2024-09-16T10:33:43.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614069+0000", 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.614075+0000", 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2489999999999998e-06 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.689968+0000", 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.075892792000000001 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.689979+0000", 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1164e-05 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.691738+0000", 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017594970000000001 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.691761+0000", 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.283e-05 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.696716+0000", 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049551359999999997 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.696853+0000", 2024-09-16T10:33:43.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137107 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:168344 11.d e522/476 11:b7e3c101:::benchmark_data_smithi063_176907_object44068:head v 522'10226, mlcod=522'10224)", 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:58.045926+0000", 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 104.995163842, 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.125753904, 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.182 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.045926+0000", 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.045926+0000", 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.045927+0000", 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3289999999999999e-06 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.045932+0000", 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.028e-06 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.045940+0000", 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6320000000000008e-06 2024-09-16T10:33:43.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.045945+0000", 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2059999999999998e-06 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.045999+0000", 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4339999999999998e-05 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.046017+0000", 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7689999999999998e-05 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171648+0000", 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12563074699999999 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171680+0000", 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1931999999999999e-05 2024-09-16T10:33:43.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:168342 11.7 e522/476 11:e2652a70:::benchmark_data_smithi063_176907_object44066:head v 522'10459, mlcod=522'10454)", 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:58.047730+0000", 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 104.99335959299999, 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.127721995, 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.047730+0000", 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.047730+0000", 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.047731+0000", 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0759999999999999e-06 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.047736+0000", 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0189999999999997e-06 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.047746+0000", 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5580000000000006e-06 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.047752+0000", 2024-09-16T10:33:43.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5370000000000003e-06 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171611+0000", 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.123858764 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171644+0000", 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2734000000000002e-05 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.175417+0000", 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037738030000000001 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.175452+0000", 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4504e-05 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:168346 11.e e522/508 11:7512fb4a:::benchmark_data_smithi063_176907_object44070:head v 522'10697, mlcod=522'10695)", 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:58.051644+0000", 2024-09-16T10:33:43.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 104.989446089, 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.123631044, 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.051644+0000", 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:43.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.051644+0000", 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.051645+0000", 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.795e-06 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.051653+0000", 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8480000000000001e-06 2024-09-16T10:33:43.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171609+0000", 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119955407 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171622+0000", 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3491e-05 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:43.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171676+0000", 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4216999999999999e-05 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.171702+0000", 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5588999999999998e-05 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.175231+0000", 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035288339999999998 2024-09-16T10:33:43.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:58.175275+0000", 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3863000000000003e-05 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:43.196 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:43.201 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[29993]: pgmap v1086: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[29993]: osdmap e570: 8 total, 8 up, 6 in 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[29993]: 5.3 scrub starts 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[29993]: 5.3 scrub ok 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[29993]: 6.d scrub starts 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[29993]: 6.d scrub ok 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[39470]: pgmap v1086: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[39470]: osdmap e570: 8 total, 8 up, 6 in 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[39470]: 5.3 scrub starts 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[39470]: 5.3 scrub ok 2024-09-16T10:33:43.202 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[39470]: 6.d scrub starts 2024-09-16T10:33:43.203 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:42 smithi063 ceph-mon[39470]: 6.d scrub ok 2024-09-16T10:33:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:42 smithi165 ceph-mon[32696]: pgmap v1086: 152 pgs: 152 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T10:33:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:42 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T10:33:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:42 smithi165 ceph-mon[32696]: osdmap e570: 8 total, 8 up, 6 in 2024-09-16T10:33:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:42 smithi165 ceph-mon[32696]: 5.3 scrub starts 2024-09-16T10:33:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:42 smithi165 ceph-mon[32696]: 5.3 scrub ok 2024-09-16T10:33:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:42 smithi165 ceph-mon[32696]: 6.d scrub starts 2024-09-16T10:33:43.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:42 smithi165 ceph-mon[32696]: 6.d scrub ok 2024-09-16T10:33:44.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T10:33:44.393 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:44.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[39470]: 4.a scrub starts 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[39470]: 4.a scrub ok 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[39470]: osdmap e571: 8 total, 8 up, 6 in 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[39470]: 6.9 scrub starts 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[39470]: 6.9 scrub ok 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[39470]: pgmap v1089: 152 pgs: 1 remapped+peering, 1 unknown, 150 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[29993]: 4.a scrub starts 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[29993]: 4.a scrub ok 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[29993]: osdmap e571: 8 total, 8 up, 6 in 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[29993]: 6.9 scrub starts 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[29993]: 6.9 scrub ok 2024-09-16T10:33:44.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:44 smithi063 ceph-mon[29993]: pgmap v1089: 152 pgs: 1 remapped+peering, 1 unknown, 150 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T10:33:44.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:44 smithi165 ceph-mon[32696]: 4.a scrub starts 2024-09-16T10:33:44.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:44 smithi165 ceph-mon[32696]: 4.a scrub ok 2024-09-16T10:33:44.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:44 smithi165 ceph-mon[32696]: osdmap e571: 8 total, 8 up, 6 in 2024-09-16T10:33:44.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:44 smithi165 ceph-mon[32696]: 6.9 scrub starts 2024-09-16T10:33:44.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:44 smithi165 ceph-mon[32696]: 6.9 scrub ok 2024-09-16T10:33:44.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:44 smithi165 ceph-mon[32696]: pgmap v1089: 152 pgs: 1 remapped+peering, 1 unknown, 150 active+clean; 583 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T10:33:45.136 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:45.137 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:33:45.137 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.9 scrub starts 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.9 scrub ok 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.13 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.12 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.13 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.12 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.10 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.10 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.f starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: osdmap e572: 8 total, 8 up, 6 in 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.11 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.14 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.14 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.11 scrub starts 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 1.11 scrub ok 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 3.1 scrub starts 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 3.1 scrub ok 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 3.0 scrub starts 2024-09-16T10:33:45.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[29993]: 3.0 scrub ok 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.9 scrub starts 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.9 scrub ok 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.13 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.12 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.13 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.12 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.10 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.10 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.f starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: osdmap e572: 8 total, 8 up, 6 in 2024-09-16T10:33:45.454 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.11 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.14 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.14 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.11 scrub starts 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 1.11 scrub ok 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 3.1 scrub starts 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 3.1 scrub ok 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 3.0 scrub starts 2024-09-16T10:33:45.455 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:45 smithi063 ceph-mon[39470]: 3.0 scrub ok 2024-09-16T10:33:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.9 scrub starts 2024-09-16T10:33:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.9 scrub ok 2024-09-16T10:33:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.13 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.12 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.13 starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.12 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.10 starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.10 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.f starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: osdmap e572: 8 total, 8 up, 6 in 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.11 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.14 starting backfill to osd.3 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.14 starting backfill to osd.5 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:45.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.11 scrub starts 2024-09-16T10:33:45.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 1.11 scrub ok 2024-09-16T10:33:45.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 3.1 scrub starts 2024-09-16T10:33:45.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 3.1 scrub ok 2024-09-16T10:33:45.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 3.0 scrub starts 2024-09-16T10:33:45.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:45 smithi165 ceph-mon[32696]: 3.0 scrub ok 2024-09-16T10:33:45.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T10:33:46.128 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:46.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:46 smithi165 ceph-mon[32696]: 3.19 scrub starts 2024-09-16T10:33:46.428 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:46 smithi165 ceph-mon[32696]: 3.19 scrub ok 2024-09-16T10:33:46.428 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:46 smithi165 ceph-mon[32696]: osdmap e573: 8 total, 8 up, 6 in 2024-09-16T10:33:46.428 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:46 smithi165 ceph-mon[32696]: pgmap v1092: 152 pgs: 1 peering, 2 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 609 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 21 KiB/s rd, 18 KiB/s wr, 32 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-16T10:33:46.428 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:46 smithi165 ceph-mon[32696]: 1.d scrub starts 2024-09-16T10:33:46.428 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:46 smithi165 ceph-mon[32696]: 1.d scrub ok 2024-09-16T10:33:46.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[29993]: 3.19 scrub starts 2024-09-16T10:33:46.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[29993]: 3.19 scrub ok 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[29993]: osdmap e573: 8 total, 8 up, 6 in 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[29993]: pgmap v1092: 152 pgs: 1 peering, 2 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 609 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 21 KiB/s rd, 18 KiB/s wr, 32 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[29993]: 1.d scrub starts 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[29993]: 1.d scrub ok 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[39470]: 3.19 scrub starts 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[39470]: 3.19 scrub ok 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[39470]: osdmap e573: 8 total, 8 up, 6 in 2024-09-16T10:33:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[39470]: pgmap v1092: 152 pgs: 1 peering, 2 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 609 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 21 KiB/s rd, 18 KiB/s wr, 32 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-16T10:33:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[39470]: 1.d scrub starts 2024-09-16T10:33:46.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:46 smithi063 ceph-mon[39470]: 1.d scrub ok 2024-09-16T10:33:46.622 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T10:33:46.623 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-16T10:33:46.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-16T10:33:46.650 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:46.650 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:46.650 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:33:46.650 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:33:46.650 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:47.256 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:47 smithi165 ceph-mon[32696]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:33:47.256 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:47 smithi165 ceph-mon[32696]: osdmap e574: 8 total, 8 up, 6 in 2024-09-16T10:33:47.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T10:33:47.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:47 smithi063 ceph-mon[29993]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:33:47.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:47 smithi063 ceph-mon[29993]: osdmap e574: 8 total, 8 up, 6 in 2024-09-16T10:33:47.302 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:33:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:33:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:33:47.302 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:47 smithi063 ceph-mon[39470]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:33:47.303 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:47 smithi063 ceph-mon[39470]: osdmap e574: 8 total, 8 up, 6 in 2024-09-16T10:33:47.534 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:47.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:33:47.591+0000 7f17f27f5640 -1 mon.a@0(leader).osd e574 definitely_dead 0 2024-09-16T10:33:48.050 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:2951 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 93976561478912 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 440.50321507799998, 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0211994769999999, 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2951 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536589+0000", 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536592+0000", 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2979999999999999e-06 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536598+0000", 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536615+0000", 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7496999999999999e-05 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536624+0000", 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9099999999999994e-06 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536869+0000", 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024484799999999998 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.536876+0000", 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1910000000000003e-06 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.547076+0000", 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010199542000000001 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.547092+0000", 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5818e-05 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.552328+0000", 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0052359230000001 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.552503+0000", 2024-09-16T10:33:48.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017550000000000001 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.557789+0000", 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052855139999999998 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98204 10.9 10:9d9b818e:::benchmark_data_smithi063_149043_object98203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:33:48.057 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 431.05821544000003, 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.034086345, 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98204 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981589+0000", 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981591+0000", 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8360000000000001e-06 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981670+0000", 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9151000000000002e-05 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981680+0000", 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8109999999999997e-06 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981687+0000", 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9959999999999996e-06 2024-09-16T10:33:48.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981740+0000", 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3491e-05 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981886+0000", 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014617800000000001 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:33:48.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.982038+0000", 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151927 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.990399+0000", 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083606689999999994 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999930+0000", 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095309130000000002 2024-09-16T10:33:48.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999946+0000", 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6342999999999999e-05 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015600+0000", 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0156542 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015675+0000", 2024-09-16T10:33:48.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.483e-05 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98205 10.9 10:919b6e78:::benchmark_data_smithi063_149043_object98204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 431.05809606600002, 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.046984186, 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:33:48.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98205 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981708+0000", 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981709+0000", 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4499999999999995e-07 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981760+0000", 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0654e-05 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981766+0000", 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6089999999999997e-06 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.981772+0000", 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7620000000000001e-06 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.985245+0000", 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034727030000000002 2024-09-16T10:33:48.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.986079+0000", 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083413300000000003 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.986209+0000", 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012954199999999999 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999975+0000", 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013766733 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:36.999982+0000", 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8759999999999999e-06 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.015907+0000", 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0159249210000001 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016460+0000", 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055315400000000004 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028602+0000", 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012142065 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028692+0000", 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0088999999999998e-05 2024-09-16T10:33:48.070 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98225 10.9 10:9f9b788e:::benchmark_data_smithi063_149043_object98224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 431.01801944200002, 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0071949410000001, 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:48.071 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98225 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021785+0000", 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021788+0000", 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2959999999999999e-06 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021880+0000", 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1638999999999995e-05 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021898+0000", 2024-09-16T10:33:48.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873e-05 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021907+0000", 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1789999999999997e-06 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021961+0000", 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4026000000000001e-05 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.021969+0000", 2024-09-16T10:33:48.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9840000000000001e-06 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016509+0000", 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99453982200000002 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016634+0000", 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012573099999999999 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.016773+0000", 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013850399999999999 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.021953+0000", 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051800379999999997 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028816+0000", 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068627699999999998 2024-09-16T10:33:48.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028873+0000", 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7655999999999998e-05 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.028980+0000", 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010642299999999999 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:98247 10.9 10:95b7428a:::benchmark_data_smithi063_149043_object98246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 431.00315333200001, 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99248770799999997, 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98247 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036651+0000", 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036653+0000", 2024-09-16T10:33:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9139999999999998e-06 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036702+0000", 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9490000000000002e-05 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036716+0000", 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3521e-05 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036723+0000", 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4259999999999997e-06 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036784+0000", 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1013e-05 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:37.036790+0000", 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4739999999999996e-06 2024-09-16T10:33:48.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.018768+0000", 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98197859600000004 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.018953+0000", 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184358 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.019066+0000", 2024-09-16T10:33:48.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011326 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.021961+0000", 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028944869999999998 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029022+0000", 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070617960000000004 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029056+0000", 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4079999999999999e-05 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:38.029139+0000", 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2293000000000006e-05 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:204177 10.23 10:c5a630ae:::benchmark_data_smithi063_149043_object71740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-16T10:33:48.084 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:13.524797+0000", 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 334.51500765600002, 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.922248611, 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 204177 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524797+0000", 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524797+0000", 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524798+0000", 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4839999999999999e-06 2024-09-16T10:33:48.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524803+0000", 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7899999999999999e-06 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524813+0000", 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8570000000000007e-06 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524820+0000", 2024-09-16T10:33:48.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3089999999999998e-06 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524865+0000", 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5302000000000001e-05 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:13.524872+0000", 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.579e-06 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.441098+0000", 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91622587499999997 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.441189+0000", 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1100000000000005e-05 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.441350+0000", 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016148700000000001 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.443761+0000", 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024104009999999999 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.443769+0000", 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8800000000000008e-06 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.446868+0000", 2024-09-16T10:33:48.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030995279999999998 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.446899+0000", 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0568000000000003e-05 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:14.447045+0000", 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146451 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:254094 10.11 10:8ae0866e:::benchmark_data_smithi063_149043_object121657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:07.537563+0000", 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 280.502241029, 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0256367769999999, 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 254094 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.092 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537563+0000", 2024-09-16T10:33:48.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537563+0000", 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537565+0000", 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7910000000000001e-06 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537570+0000", 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.327e-06 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537581+0000", 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0631999999999999e-05 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.537586+0000", 2024-09-16T10:33:48.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.186e-06 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.538277+0000", 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00069087 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:07.538287+0000", 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8849999999999994e-06 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.557618+0000", 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0193307949999999 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.557734+0000", 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011606400000000001 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.558062+0000", 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032821400000000001 2024-09-16T10:33:48.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.562925+0000", 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048629800000000003 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.562990+0000", 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4615000000000004e-05 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.563100+0000", 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011081099999999999 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.563160+0000", 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9432000000000003e-05 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.563200+0000", 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0175000000000001e-05 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127324 11.5 11:a8f03c9e:::benchmark_data_smithi063_176907_object3048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.155742+0000", 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 154.88406269800001, 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0867619980000001, 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127324 2024-09-16T10:33:48.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155742+0000", 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155742+0000", 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155744+0000", 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6139999999999999e-06 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155750+0000", 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2650000000000004e-06 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155762+0000", 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.277e-05 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155771+0000", 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2700000000000004e-06 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155835+0000", 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4236e-05 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155846+0000", 2024-09-16T10:33:48.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0949e-05 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.129500+0000", 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97365438999999998 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.129512+0000", 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1562e-05 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221180+0000", 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0916682090000001 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221302+0000", 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121866 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6,7", 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221688+0000", 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038607600000000001 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231249+0000", 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095608900000000007 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231372+0000", 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122848 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.242302+0000", 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010929960000000001 2024-09-16T10:33:48.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.242341+0000", 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9273000000000002e-05 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.242397+0000", 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6180000000000001e-05 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.242504+0000", 2024-09-16T10:33:48.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010664 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167592 11.5 11:a0b55922:::benchmark_data_smithi063_176907_object43316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.646809+0000", 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.39299486900001, 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0145609769999999, 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167592 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646809+0000", 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646809+0000", 2024-09-16T10:33:48.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646811+0000", 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3e-06 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646814+0000", 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5829999999999998e-06 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646823+0000", 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3559999999999993e-06 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646828+0000", 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2000000000000002e-06 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646870+0000", 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2259000000000003e-05 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.646877+0000", 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7819999999999996e-06 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.655429+0000", 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.008552246 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.655527+0000", 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7813999999999996e-05 2024-09-16T10:33:48.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.655613+0000", 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6114999999999998e-05 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661127+0000", 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055139469999999999 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661155+0000", 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7951000000000001e-05 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661284+0000", 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012932800000000001 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661340+0000", 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5544000000000002e-05 2024-09-16T10:33:48.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661370+0000", 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0552000000000002e-05 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167599 11.5 11:aed60194:::benchmark_data_smithi063_176907_object43323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.654973+0000", 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.384830846, 2024-09-16T10:33:48.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0152166229999999, 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167599 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.654973+0000", 2024-09-16T10:33:48.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.654973+0000", 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.654976+0000", 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019999999999999e-06 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.654981+0000", 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0969999999999996e-06 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.654993+0000", 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2747e-05 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.655001+0000", 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3250000000000002e-06 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.655056+0000", 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5198000000000002e-05 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.655062+0000", 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.241e-06 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.657949+0000", 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0028868369999999 2024-09-16T10:33:48.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.658045+0000", 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5662000000000007e-05 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.658145+0000", 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010042700000000001 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661237+0000", 2024-09-16T10:33:48.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003092053 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661440+0000", 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020238400000000001 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670086+0000", 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086465729999999994 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670144+0000", 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7810999999999999e-05 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670190+0000", 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6165999999999998e-05 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.123 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167621 11.5 11:a795dd8f:::benchmark_data_smithi063_176907_object43345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.673880+0000", 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.365924396, 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99651409099999999, 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167621 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673880+0000", 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673880+0000", 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673882+0000", 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7129999999999999e-06 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673887+0000", 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9089999999999999e-06 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673899+0000", 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2738999999999999e-05 2024-09-16T10:33:48.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673906+0000", 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1110000000000002e-06 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673965+0000", 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8480999999999998e-05 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.673975+0000", 2024-09-16T10:33:48.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0352e-05 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.659862+0000", 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98588630700000002 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.659958+0000", 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6224999999999994e-05 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.660029+0000", 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1001999999999995e-05 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.666154+0000", 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006125559 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.666168+0000", 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4075e-05 2024-09-16T10:33:48.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670305+0000", 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041369620000000001 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670355+0000", 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9704999999999999e-05 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670394+0000", 2024-09-16T10:33:48.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8951e-05 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167624 11.5 11:af5102b9:::benchmark_data_smithi063_176907_object43348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.674913+0000", 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.364891673, 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99568811700000004, 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.131 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167624 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.674913+0000", 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.674913+0000", 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.674914+0000", 2024-09-16T10:33:48.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.454e-06 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.674918+0000", 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3030000000000001e-06 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.674930+0000", 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1089e-05 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.674937+0000", 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.858e-06 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.674992+0000", 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4669999999999997e-05 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.675000+0000", 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0229999999999992e-06 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661493+0000", 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.986493026 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661611+0000", 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117637 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.661728+0000", 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011777000000000001 2024-09-16T10:33:48.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670024+0000", 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008295897 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670244+0000", 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022002299999999999 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670491+0000", 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024683700000000002 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670552+0000", 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0891999999999999e-05 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670601+0000", 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8637999999999997e-05 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.138 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167629 11.5 11:aff6ef00:::benchmark_data_smithi063_176907_object43353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.679568+0000", 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.360236665, 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99120936599999998, 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167629 2024-09-16T10:33:48.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679568+0000", 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679568+0000", 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679570+0000", 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9379999999999999e-06 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679575+0000", 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9359999999999998e-06 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679588+0000", 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3395e-05 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679594+0000", 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.404e-06 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679641+0000", 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6282000000000002e-05 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.679646+0000", 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1630000000000004e-06 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.663930+0000", 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98428404999999997 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.664078+0000", 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000148429 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.664191+0000", 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011322 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670253+0000", 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060618390000000003 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670445+0000", 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019156800000000001 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670675+0000", 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022982699999999999 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670739+0000", 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4395999999999998e-05 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670777+0000", 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7919000000000001e-05 2024-09-16T10:33:48.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167655 11.5 11:a66c4633:::benchmark_data_smithi063_176907_object43379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.702510+0000", 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.33729404899999, 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97085163299999999, 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167655 2024-09-16T10:33:48.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702510+0000", 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702510+0000", 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702512+0000", 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4899999999999999e-06 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702516+0000", 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8920000000000002e-06 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702526+0000", 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0468999999999999e-05 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702532+0000", 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4319999999999998e-06 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702595+0000", 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3713999999999993e-05 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.702605+0000", 2024-09-16T10:33:48.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0132000000000001e-05 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.666195+0000", 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96358928200000005 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.666307+0000", 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111994 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.666410+0000", 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103645 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670259+0000", 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038488009999999998 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673200+0000", 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029406139999999998 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673262+0000", 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1959999999999996e-05 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673316+0000", 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4373999999999998e-05 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673362+0000", 2024-09-16T10:33:48.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5834000000000002e-05 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167661 11.5 11:a8396256:::benchmark_data_smithi063_176907_object43385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.706194+0000", 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.333610119, 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96739863100000001, 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167661 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706194+0000", 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706194+0000", 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706196+0000", 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.99e-06 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706202+0000", 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5419999999999996e-06 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706216+0000", 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4549000000000001e-05 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706223+0000", 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1400000000000002e-06 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706277+0000", 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.393e-05 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.706285+0000", 2024-09-16T10:33:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9990000000000008e-06 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.668213+0000", 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96192758499999997 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.668309+0000", 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6352000000000002e-05 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.668384+0000", 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4369999999999994e-05 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670454+0000", 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020704719999999999 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673414+0000", 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002959546 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673484+0000", 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9821999999999997e-05 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673553+0000", 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9265999999999997e-05 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673593+0000", 2024-09-16T10:33:48.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0068000000000003e-05 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167681 11.5 11:ab938ac5:::benchmark_data_smithi063_176907_object43405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.722021+0000", 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.31778299299999, 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95628283400000003, 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167681 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722021+0000", 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722021+0000", 2024-09-16T10:33:48.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722024+0000", 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4899999999999999e-06 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722030+0000", 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.302e-06 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722045+0000", 2024-09-16T10:33:48.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4949e-05 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722054+0000", 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5140000000000001e-06 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722126+0000", 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2384000000000004e-05 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.722135+0000", 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0699999999999996e-06 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670825+0000", 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94868948600000003 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.670938+0000", 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113866 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.671055+0000", 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011681899999999999 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673428+0000", 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023732800000000002 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678124+0000", 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046951040000000003 2024-09-16T10:33:48.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678198+0000", 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3913999999999997e-05 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678258+0000", 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0004999999999999e-05 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678304+0000", 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6650999999999998e-05 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167721 11.5 11:a362bffe:::benchmark_data_smithi063_176907_object43445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.758179+0000", 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.281625121, 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92208810799999996, 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167721 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758179+0000", 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758179+0000", 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758181+0000", 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758187+0000", 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6590000000000004e-06 2024-09-16T10:33:48.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758198+0000", 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0910999999999999e-05 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758204+0000", 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2439999999999998e-06 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758247+0000", 2024-09-16T10:33:48.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3507000000000002e-05 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758254+0000", 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7490000000000001e-06 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673639+0000", 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91538454899999999 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673722+0000", 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3201000000000005e-05 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.673805+0000", 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3220000000000006e-05 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678319+0000", 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045136100000000004 2024-09-16T10:33:48.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678377+0000", 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8229000000000001e-05 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680213+0000", 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018355839999999999 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680241+0000", 2024-09-16T10:33:48.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8860000000000002e-05 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680267+0000", 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5891999999999999e-05 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167722 11.5 11:ad677cc2:::benchmark_data_smithi063_176907_object43446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.758904+0000", 2024-09-16T10:33:48.174 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.28090005, 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92462509800000003, 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167722 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758904+0000", 2024-09-16T10:33:48.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758904+0000", 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758905+0000", 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4300000000000002e-07 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758908+0000", 2024-09-16T10:33:48.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.151e-06 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758915+0000", 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.511e-06 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758919+0000", 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8210000000000003e-06 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758957+0000", 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7935000000000002e-05 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.758963+0000", 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0889999999999996e-06 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.675902+0000", 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91693932 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.675983+0000", 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0650999999999997e-05 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.676087+0000", 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010471600000000001 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680088+0000", 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040009859999999998 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680148+0000", 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9874999999999998e-05 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683467+0000", 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033190910000000001 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683502+0000", 2024-09-16T10:33:48.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4603000000000001e-05 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683529+0000", 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7506000000000001e-05 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167723 11.5 11:a54919a9:::benchmark_data_smithi063_176907_object43447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.759732+0000", 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.280072159, 2024-09-16T10:33:48.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92394196900000003, 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167723 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759732+0000", 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759732+0000", 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759733+0000", 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.201e-06 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759737+0000", 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0620000000000002e-06 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759745+0000", 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1300000000000003e-06 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759750+0000", 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1250000000000001e-06 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759778+0000", 2024-09-16T10:33:48.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7919000000000002e-05 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.759781+0000", 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4510000000000001e-06 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678418+0000", 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91863659900000005 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678496+0000", 2024-09-16T10:33:48.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8541000000000006e-05 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.678577+0000", 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1105000000000004e-05 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683355+0000", 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00477803 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683416+0000", 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1050999999999995e-05 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683623+0000", 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000206314 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683649+0000", 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5987e-05 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683674+0000", 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5454000000000001e-05 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:167741 11.5 11:a2def33b:::benchmark_data_smithi063_176907_object43465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:55.778634+0000", 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.261169956, 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91551002500000001, 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:48.188 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 167741 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778634+0000", 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778634+0000", 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778637+0000", 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2560000000000001e-06 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778643+0000", 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2770000000000002e-06 2024-09-16T10:33:48.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778658+0000", 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5542000000000001e-05 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778667+0000", 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9570000000000008e-06 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778738+0000", 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0542000000000003e-05 2024-09-16T10:33:48.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:55.778747+0000", 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.789e-06 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680320+0000", 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90157297000000003 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:48.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680397+0000", 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7496000000000005e-05 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.680468+0000", 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0885999999999999e-05 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683574+0000", 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031058040000000002 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.683583+0000", 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9770000000000006e-06 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.693992+0000", 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01040924 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.694091+0000", 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8795999999999994e-05 2024-09-16T10:33:48.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:56.694144+0000", 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3492999999999998e-05 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:48.195 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:48.196 INFO:teuthology.orchestra.run.smithi063.stderr:marked in osd.1. 2024-09-16T10:33:48.301 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:48 smithi165 ceph-mon[32696]: 3.e scrub starts 2024-09-16T10:33:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:48 smithi165 ceph-mon[32696]: 3.e scrub ok 2024-09-16T10:33:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:48 smithi165 ceph-mon[32696]: pgmap v1094: 152 pgs: 1 peering, 2 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 609 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 29 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-09-16T10:33:48.302 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/101221698' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-16T10:33:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[29993]: 3.e scrub starts 2024-09-16T10:33:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[29993]: 3.e scrub ok 2024-09-16T10:33:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[29993]: pgmap v1094: 152 pgs: 1 peering, 2 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 609 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 29 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-09-16T10:33:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/101221698' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-16T10:33:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[39470]: 3.e scrub starts 2024-09-16T10:33:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[39470]: 3.e scrub ok 2024-09-16T10:33:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[39470]: pgmap v1094: 152 pgs: 1 peering, 2 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 609 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 29 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-09-16T10:33:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/101221698' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-16T10:33:48.932 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-16T10:33:48.932 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T10:33:48.932 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-16T10:33:48.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-16T10:33:49.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T10:33:49.192 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[39470]: 6.1 scrub starts 2024-09-16T10:33:49.193 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[39470]: 6.1 scrub ok 2024-09-16T10:33:49.193 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/101221698' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-16T10:33:49.193 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[39470]: osdmap e575: 8 total, 8 up, 7 in 2024-09-16T10:33:49.217 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:49.375 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:49.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[29993]: 6.1 scrub starts 2024-09-16T10:33:49.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[29993]: 6.1 scrub ok 2024-09-16T10:33:49.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/101221698' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-16T10:33:49.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-mon[29993]: osdmap e575: 8 total, 8 up, 7 in 2024-09-16T10:33:49.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:49 smithi165 ceph-mon[32696]: 6.1 scrub starts 2024-09-16T10:33:49.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:49 smithi165 ceph-mon[32696]: 6.1 scrub ok 2024-09-16T10:33:49.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:49 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/101221698' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-16T10:33:49.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:49 smithi165 ceph-mon[32696]: osdmap e575: 8 total, 8 up, 7 in 2024-09-16T10:33:49.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:33:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:33:49.877 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:33:49.877 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:50.179 INFO:teuthology.orchestra.run.smithi063.stderr:marked in osd.7. 2024-09-16T10:33:50.230 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 5.1c scrub starts 2024-09-16T10:33:50.230 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 5.1c scrub ok 2024-09-16T10:33:50.230 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 5.1f scrub starts 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 5.1f scrub ok 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: osdmap e576: 8 total, 8 up, 7 in 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 1.13 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 1.13 continuing backfill to osd.6 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 1.b starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 1.b continuing backfill to osd.4 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 1.b continuing backfill to osd.6 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 5.13 scrub starts 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 5.13 scrub ok 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: 1.13 scrub ok 2024-09-16T10:33:50.231 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: pgmap v1097: 152 pgs: 43 peering, 1 unknown, 6 active+remapped+backfill_toofull, 102 active+clean; 609 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s; 73 KiB/s, 3 objects/s recovering 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3453764985' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3453764985' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:49 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:33:49.964+0000 7f17f27f5640 -1 mon.a@0(leader).osd e576 definitely_dead 0 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 5.1c scrub starts 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 5.1c scrub ok 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 5.1f scrub starts 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 5.1f scrub ok 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: osdmap e576: 8 total, 8 up, 7 in 2024-09-16T10:33:50.232 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 1.13 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:50.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 1.13 continuing backfill to osd.6 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 1.b starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:50.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 1.b continuing backfill to osd.4 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 1.b continuing backfill to osd.6 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:50.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 5.13 scrub starts 2024-09-16T10:33:50.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 5.13 scrub ok 2024-09-16T10:33:50.234 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:33:50.234 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: 1.13 scrub ok 2024-09-16T10:33:50.234 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: pgmap v1097: 152 pgs: 43 peering, 1 unknown, 6 active+remapped+backfill_toofull, 102 active+clean; 609 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s; 73 KiB/s, 3 objects/s recovering 2024-09-16T10:33:50.234 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3453764985' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-16T10:33:50.234 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:50 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3453764985' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-16T10:33:50.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 5.1c scrub starts 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 5.1c scrub ok 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 5.1f scrub starts 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 5.1f scrub ok 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: osdmap e576: 8 total, 8 up, 7 in 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 1.13 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 1.13 continuing backfill to osd.6 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 1.b starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 1.b continuing backfill to osd.4 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 1.b continuing backfill to osd.6 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 5.13 scrub starts 2024-09-16T10:33:50.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 5.13 scrub ok 2024-09-16T10:33:50.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:33:50.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: 1.13 scrub ok 2024-09-16T10:33:50.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: pgmap v1097: 152 pgs: 43 peering, 1 unknown, 6 active+remapped+backfill_toofull, 102 active+clean; 609 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s; 73 KiB/s, 3 objects/s recovering 2024-09-16T10:33:50.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3453764985' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-16T10:33:50.507 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:50 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3453764985' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-16T10:33:50.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T10:33:50.783 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:51.067 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-16T10:33:51.067 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-16T10:33:51.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-16T10:33:51.280 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:33:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:33:51.445 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:51.445 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:51.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[29993]: osdmap e577: 8 total, 8 up, 8 in 2024-09-16T10:33:51.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[29993]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[29993]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[29993]: osdmap e578: 8 total, 8 up, 8 in 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[29993]: 1.f continuing backfill to osd.3 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[29993]: 1.f starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[39470]: osdmap e577: 8 total, 8 up, 8 in 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[39470]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[39470]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[39470]: osdmap e578: 8 total, 8 up, 8 in 2024-09-16T10:33:51.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[39470]: 1.f continuing backfill to osd.3 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:51.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:51 smithi063 ceph-mon[39470]: 1.f starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:51 smithi165 ceph-mon[32696]: osdmap e577: 8 total, 8 up, 8 in 2024-09-16T10:33:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:51 smithi165 ceph-mon[32696]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:51 smithi165 ceph-mon[32696]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-16T10:33:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:51 smithi165 ceph-mon[32696]: osdmap e578: 8 total, 8 up, 8 in 2024-09-16T10:33:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:51 smithi165 ceph-mon[32696]: 1.f continuing backfill to osd.3 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:51 smithi165 ceph-mon[32696]: 1.f starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:51.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T10:33:52.138 INFO:teuthology.orchestra.run.smithi063.stderr:set pool 1 pgp_num to 11 2024-09-16T10:33:52.248 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.14 continuing backfill to osd.3 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.14 continuing backfill to osd.5 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.14 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.12 continuing backfill to osd.4 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.12 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.10 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.13 scrub starts 2024-09-16T10:33:52.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.13 scrub ok 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 4.11 scrub starts 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 4.11 scrub ok 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 5.16 scrub starts 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 5.16 scrub ok 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: pgmap v1100: 152 pgs: 57 peering, 3 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 86 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 149 op/s; 80 KiB/s, 10 objects/s recovering 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/324856281' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/324856281' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: osdmap e579: 8 total, 8 up, 8 in 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[39470]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:33:52.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.14 continuing backfill to osd.3 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.14 continuing backfill to osd.5 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.14 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.12 continuing backfill to osd.4 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.12 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.10 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.13 scrub starts 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.13 scrub ok 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 4.11 scrub starts 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 4.11 scrub ok 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 5.16 scrub starts 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 5.16 scrub ok 2024-09-16T10:33:52.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: pgmap v1100: 152 pgs: 57 peering, 3 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 86 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 149 op/s; 80 KiB/s, 10 objects/s recovering 2024-09-16T10:33:52.455 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/324856281' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-16T10:33:52.455 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/324856281' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-16T10:33:52.455 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: osdmap e579: 8 total, 8 up, 8 in 2024-09-16T10:33:52.455 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:52 smithi063 ceph-mon[29993]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.14 continuing backfill to osd.3 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.14 continuing backfill to osd.5 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.14 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.12 continuing backfill to osd.4 from (73'79,73'79] MIN to 73'79 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.12 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.10 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.509 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.13 scrub starts 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.13 scrub ok 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 4.11 scrub starts 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 4.11 scrub ok 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 5.16 scrub starts 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 5.16 scrub ok 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: pgmap v1100: 152 pgs: 57 peering, 3 remapped+peering, 1 unknown, 5 active+remapped+backfill_toofull, 86 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 149 op/s; 80 KiB/s, 10 objects/s recovering 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/324856281' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/324856281' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: osdmap e579: 8 total, 8 up, 8 in 2024-09-16T10:33:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:52 smithi165 ceph-mon[32696]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88289 10.e 10:777727a3:::benchmark_data_smithi063_149043_object88288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.537589+0000", 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.21929810300003, 2024-09-16T10:33:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0363937000000001, 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88289 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537589+0000", 2024-09-16T10:33:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537589+0000", 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537591+0000", 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.722e-06 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.537831+0000", 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024001099999999999 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540070+0000", 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002238968 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540078+0000", 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5180000000000004e-06 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540979+0000", 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00090050899999999997 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.540990+0000", 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1372000000000001e-05 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.560724+0000", 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0197336770000001 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.562746+0000", 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002021915 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.562908+0000", 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162027 2024-09-16T10:33:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568617+0000", 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057096100000000004 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568662+0000", 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4441999999999999e-05 2024-09-16T10:33:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573892+0000", 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052300309999999996 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573941+0000", 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8789999999999999e-05 2024-09-16T10:33:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573983+0000", 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2107999999999997e-05 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88296 10.e 10:79ee89b5:::benchmark_data_smithi063_149043_object88295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.555357+0000", 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.20153047399998, 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.024583236, 2024-09-16T10:33:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88296 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555357+0000", 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555357+0000", 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555359+0000", 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9989999999999998e-06 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555394+0000", 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5635e-05 2024-09-16T10:33:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555404+0000", 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.302e-06 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555411+0000", 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5839999999999997e-06 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555552+0000", 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014035600000000001 2024-09-16T10:33:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.555561+0000", 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4409999999999999e-06 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.565905+0000", 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0103437479999999 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.566037+0000", 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013203800000000001 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.566191+0000", 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015380799999999999 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568708+0000", 2024-09-16T10:33:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025176560000000001 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579714+0000", 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011005386000000001 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579843+0000", 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012893799999999999 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579894+0000", 2024-09-16T10:33:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1579000000000002e-05 2024-09-16T10:33:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579940+0000", 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5766000000000002e-05 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88311 10.e 10:7eaf0035:::benchmark_data_smithi063_149043_object88310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.574876+0000", 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.18201140799999, 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005137081, 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88311 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574876+0000", 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574876+0000", 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574877+0000", 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9899999999999999e-07 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574916+0000", 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9264000000000002e-05 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574922+0000", 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2940000000000002e-06 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574926+0000", 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8829999999999999e-06 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574980+0000", 2024-09-16T10:33:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3801999999999998e-05 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.574988+0000", 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1100000000000003e-06 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568787+0000", 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99379890400000004 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568896+0000", 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109475 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.568997+0000", 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101127 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.573854+0000", 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048568250000000004 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579746+0000", 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058914609999999997 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579947+0000", 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000201199 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579983+0000", 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5907e-05 2024-09-16T10:33:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580013+0000", 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0031000000000001e-05 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88322 10.e 10:7e8de65a:::benchmark_data_smithi063_149043_object88321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.588082+0000", 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.16880535799999, 2024-09-16T10:33:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99205056999999996, 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88322 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588082+0000", 2024-09-16T10:33:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588082+0000", 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588083+0000", 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4020000000000001e-06 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588139+0000", 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6019e-05 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588148+0000", 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6000000000000007e-06 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588155+0000", 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9120000000000001e-06 2024-09-16T10:33:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588218+0000", 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3280000000000004e-05 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.588226+0000", 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0479999999999999e-06 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.571192+0000", 2024-09-16T10:33:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98296605199999998 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.571327+0000", 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013475499999999999 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.571418+0000", 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0878999999999995e-05 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579676+0000", 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082579100000000003 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579830+0000", 2024-09-16T10:33:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000154139 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580019+0000", 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018949900000000001 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580099+0000", 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9136999999999998e-05 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580132+0000", 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3937999999999999e-05 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:88330 10.e 10:7d45cc0a:::benchmark_data_smithi063_149043_object88329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:26:27.601441+0000", 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.15544590600001, 2024-09-16T10:33:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98658399699999999, 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 88330 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601441+0000", 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601441+0000", 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601442+0000", 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5099999999999998e-07 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601498+0000", 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5744e-05 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601506+0000", 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7000000000000008e-06 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601510+0000", 2024-09-16T10:33:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.267e-06 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601558+0000", 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8062000000000002e-05 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:27.601565+0000", 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1149999999999996e-06 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.574017+0000", 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97245151500000004 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.574149+0000", 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013288800000000001 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.574279+0000", 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012991700000000001 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.579785+0000", 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055055340000000003 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.580138+0000", 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035344700000000001 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.587670+0000", 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075319669999999997 2024-09-16T10:33:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.587984+0000", 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031397599999999999 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:26:28.588025+0000", 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1013999999999997e-05 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:3544 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93976561475456 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.112595+0000", 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.644292697, 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.054160918, 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3544 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112595+0000", 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112595+0000", 2024-09-16T10:33:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112597+0000", 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2249999999999999e-06 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112603+0000", 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2879999999999998e-06 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112616+0000", 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2547e-05 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112624+0000", 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4540000000000007e-06 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112682+0000", 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8171000000000003e-05 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112686+0000", 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2420000000000002e-06 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.126275+0000", 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013588078999999999 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.126285+0000", 2024-09-16T10:33:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0587999999999999e-05 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.138434+0000", 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0121490390000001 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.138696+0000", 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000261506 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.148778+0000", 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010082571 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.148871+0000", 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2923000000000005e-05 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.166755+0000", 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017884285 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:3545 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 93976561468544 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.112644+0000", 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.64424299699999, 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0541178630000001, 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3545 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112644+0000", 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112644+0000", 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112645+0000", 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000001e-07 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112652+0000", 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1779999999999997e-06 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112658+0000", 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0789999999999997e-06 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112663+0000", 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.023e-06 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112709+0000", 2024-09-16T10:33:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5325999999999999e-05 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112714+0000", 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7509999999999997e-06 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.134894+0000", 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022180182999999999 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.134905+0000", 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0069e-05 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.135703+0000", 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000798284 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.137067+0000", 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001364388 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145692+0000", 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086244479999999998 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145759+0000", 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7359000000000003e-05 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.166762+0000", 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021003096999999998 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14514.0:3546 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93976566403584 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.64421156899999, 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.054477669, 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14514", 2024-09-16T10:33:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2920274648", 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3546 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112676+0000", 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7100000000000002e-07 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112679+0000", 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3599999999999999e-06 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112684+0000", 2024-09-16T10:33:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3340000000000001e-06 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112689+0000", 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1660000000000002e-06 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112731+0000", 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2286000000000001e-05 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.112736+0000", 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2039999999999996e-06 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.141374+0000", 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028637040999999998 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.141385+0000", 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0981999999999999e-05 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.137935+0000", 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99655030600000005 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.138328+0000", 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039304800000000001 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145309+0000", 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069808719999999999 2024-09-16T10:33:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.145405+0000", 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6574000000000001e-05 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.167153+0000", 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021748025000000001 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127304 11.4 11:2e776e1e:::benchmark_data_smithi063_176907_object3028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.114285+0000", 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.64260195400001, 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1223989990000001, 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127304 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114285+0000", 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114285+0000", 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114287+0000", 2024-09-16T10:33:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5600000000000001e-06 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114293+0000", 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.755e-06 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114314+0000", 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1268e-05 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114321+0000", 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0600000000000002e-06 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114377+0000", 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6520000000000001e-05 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114516+0000", 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000138687 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-16T10:33:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114847+0000", 2024-09-16T10:33:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000330744 2024-09-16T10:33:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.123432+0000", 2024-09-16T10:33:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0085849419999999999 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.143849+0000", 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020416878999999999 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.143890+0000", 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1791e-05 2024-09-16T10:33:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119563+0000", 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97567231899999995 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119576+0000", 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3191e-05 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218330+0000", 2024-09-16T10:33:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0987538729999999 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218446+0000", 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011647 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236560+0000", 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018113186999999999 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236684+0000", 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012475299999999999 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127307 11.3 11:c802da4f:::benchmark_data_smithi063_176907_object3031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.114339+0000", 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.64254815999999, 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1180333509999998, 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127307 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114339+0000", 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114339+0000", 2024-09-16T10:33:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114340+0000", 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2399999999999998e-07 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114343+0000", 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0819999999999999e-06 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114348+0000", 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114353+0000", 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9219999999999996e-06 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114437+0000", 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4216999999999997e-05 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114561+0000", 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012369400000000001 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.114682+0000", 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120711 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.124095+0000", 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094135409999999992 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.142236+0000", 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018140586 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.142250+0000", 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3715e-05 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.836 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125654+0000", 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98340460100000004 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125667+0000", 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2805e-05 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.220583+0000", 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0949163230000001 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.220891+0000", 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030743399999999998 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.232293+0000", 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011402536 2024-09-16T10:33:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.232372+0000", 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9097999999999994e-05 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127314 11.3 11:c1bf4bc8:::benchmark_data_smithi063_176907_object3038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.117913+0000", 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.638974351, 2024-09-16T10:33:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.118139706, 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127314 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117913+0000", 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117913+0000", 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117914+0000", 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.463e-06 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117920+0000", 2024-09-16T10:33:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2970000000000003e-06 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117930+0000", 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0308e-05 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.117937+0000", 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7819999999999996e-06 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.118037+0000", 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010067 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.118160+0000", 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122414 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,7", 2024-09-16T10:33:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.118277+0000", 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011681 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.124100+0000", 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058237540000000004 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.144098+0000", 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019998096999999999 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.144104+0000", 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6699999999999999e-06 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125697+0000", 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98159304999999997 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125706+0000", 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6750000000000008e-06 2024-09-16T10:33:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221153+0000", 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0954475560000001 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221311+0000", 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015769699999999999 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.235940+0000", 2024-09-16T10:33:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014628611 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236053+0000", 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112852 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127320 11.3 11:c4c194ba:::benchmark_data_smithi063_176907_object3044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.147464+0000", 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.60942313199999, 2024-09-16T10:33:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0989353159999999, 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127320 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147464+0000", 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147464+0000", 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147466+0000", 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.029e-06 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147472+0000", 2024-09-16T10:33:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3630000000000002e-06 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147484+0000", 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2729e-05 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147492+0000", 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.391e-06 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147539+0000", 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7818999999999997e-05 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.147546+0000", 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6340000000000003e-06 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125731+0000", 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97818496499999996 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.125739+0000", 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9489999999999996e-06 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221514+0000", 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.095774936 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.221632+0000", 2024-09-16T10:33:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117859 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5,7", 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.222078+0000", 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000446654 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231188+0000", 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091099849999999993 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.234161+0000", 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002972293 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236228+0000", 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020670900000000002 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.246244+0000", 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010016057 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.246305+0000", 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1068999999999994e-05 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.246399+0000", 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4493999999999993e-05 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127323 11.4 11:22126989:::benchmark_data_smithi063_176907_object3047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.155284+0000", 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.60160352700001, 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0816857770000001, 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127323 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155284+0000", 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155284+0000", 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155285+0000", 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2720000000000001e-06 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155289+0000", 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8990000000000003e-06 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155298+0000", 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5279999999999992e-06 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155304+0000", 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2349999999999999e-06 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155356+0000", 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1968000000000001e-05 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.155365+0000", 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8320000000000001e-06 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119602+0000", 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96423660600000005 2024-09-16T10:33:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119610+0000", 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218745+0000", 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.099135456 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.218881+0000", 2024-09-16T10:33:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013615 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,7", 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.219298+0000", 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000416959 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.235147+0000", 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015848504999999999 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.235187+0000", 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9855999999999999e-05 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236722+0000", 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001535518 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236762+0000", 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9521000000000003e-05 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236852+0000", 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0816999999999995e-05 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.236969+0000", 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117069 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:127341 11.4 11:22f27910:::benchmark_data_smithi063_176907_object3065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-16T10:33:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:13.175921+0000", 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 159.58096632499999, 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0649205770000001, 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 127341 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175921+0000", 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175921+0000", 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175923+0000", 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6109999999999999e-06 2024-09-16T10:33:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175927+0000", 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0679999999999998e-06 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175938+0000", 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1695000000000001e-05 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.175946+0000", 2024-09-16T10:33:52.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5390000000000002e-06 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.176015+0000", 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9643999999999999e-05 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:13.176022+0000", 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1770000000000003e-06 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119642+0000", 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94362000899999998 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:14.119649+0000", 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5129999999999999e-06 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231508+0000", 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1118593029999999 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231642+0000", 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013385099999999999 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,7", 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.231786+0000", 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014369 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.237011+0000", 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052247439999999999 2024-09-16T10:33:52.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.237153+0000", 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142595 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.238291+0000", 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001137569 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.240653+0000", 2024-09-16T10:33:52.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023618630000000001 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.240715+0000", 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1739e-05 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:15.240841+0000", 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012696699999999999 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149389 11.3 11:cc3dd8fd:::benchmark_data_smithi063_176907_object25113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.656263+0000", 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 136.10062412900001, 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98310017500000002, 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149389 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.872 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656263+0000", 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656263+0000", 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656265+0000", 2024-09-16T10:33:52.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.922e-06 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656271+0000", 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0349999999999998e-06 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656286+0000", 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4693e-05 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656348+0000", 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1939000000000004e-05 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656449+0000", 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101623 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.656458+0000", 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7479999999999999e-06 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.634927+0000", 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.978468428 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.635019+0000", 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2315e-05 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.635154+0000", 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013515700000000001 2024-09-16T10:33:52.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639180+0000", 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004026338 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639211+0000", 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0277000000000002e-05 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639278+0000", 2024-09-16T10:33:52.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7537000000000001e-05 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639331+0000", 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2815999999999997e-05 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639363+0000", 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2347e-05 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149392 11.3 11:c4eb902d:::benchmark_data_smithi063_176907_object25116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.659065+0000", 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 136.097822235, 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98378280200000001, 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149392 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.879 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659065+0000", 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659065+0000", 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659067+0000", 2024-09-16T10:33:52.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.573e-06 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659071+0000", 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4970000000000003e-06 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659084+0000", 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2651e-05 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659090+0000", 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9739999999999999e-06 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659140+0000", 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0262000000000001e-05 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659148+0000", 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5939999999999996e-06 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.637503+0000", 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97835563800000003 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.637581+0000", 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7786000000000004e-05 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.637674+0000", 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2897000000000002e-05 2024-09-16T10:33:52.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642747+0000", 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050729399999999997 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642756+0000", 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8429999999999999e-06 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642787+0000", 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1420000000000001e-05 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642820+0000", 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3306000000000003e-05 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642848+0000", 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7421000000000001e-05 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149393 11.3 11:cca5cdfb:::benchmark_data_smithi063_176907_object25117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.659236+0000", 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 136.09765130299999, 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.987797657, 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149393 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.886 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659236+0000", 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659236+0000", 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659237+0000", 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.111e-06 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659241+0000", 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.279e-06 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659249+0000", 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8050000000000007e-06 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659256+0000", 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8709999999999999e-06 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659305+0000", 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9387e-05 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.659312+0000", 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.742e-06 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639407+0000", 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98009459399999999 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639508+0000", 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010131100000000001 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.639582+0000", 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3623999999999998e-05 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642882+0000", 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003300594 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642892+0000", 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5009999999999993e-06 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.646945+0000", 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004053297 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.646993+0000", 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8083e-05 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647034+0000", 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0457999999999998e-05 2024-09-16T10:33:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149430 11.3 11:cf4e2810:::benchmark_data_smithi063_176907_object25154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.692980+0000", 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 136.06390735900001, 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95419773900000004, 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149430 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692980+0000", 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692980+0000", 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692982+0000", 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5809999999999999e-06 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.692988+0000", 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8799999999999996e-06 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693003+0000", 2024-09-16T10:33:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4433999999999999e-05 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693011+0000", 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3559999999999993e-06 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693074+0000", 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2624000000000002e-05 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.693084+0000", 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9559999999999992e-06 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.641063+0000", 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94797921399999996 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.641172+0000", 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108863 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.641270+0000", 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8393000000000005e-05 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642933+0000", 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001662576 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.644750+0000", 2024-09-16T10:33:52.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001817335 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647081+0000", 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023307860000000001 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647136+0000", 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5220999999999999e-05 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647178+0000", 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1520000000000002e-05 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149459 11.3 11:c35b989e:::benchmark_data_smithi063_176907_object25183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.716639+0000", 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 136.040248139, 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93083209499999997, 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.900 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149459 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716639+0000", 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716639+0000", 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716641+0000", 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9719999999999999e-06 2024-09-16T10:33:52.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716647+0000", 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7100000000000004e-06 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716659+0000", 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2567e-05 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716666+0000", 2024-09-16T10:33:52.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9820000000000002e-06 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716722+0000", 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6128999999999997e-05 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.716762+0000", 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9045000000000003e-05 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.642961+0000", 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92619985500000002 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.643021+0000", 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9336000000000002e-05 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.643120+0000", 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9610000000000003e-05 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647238+0000", 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041180230000000002 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647252+0000", 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3749e-05 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647381+0000", 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012919900000000001 2024-09-16T10:33:52.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647433+0000", 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1696000000000001e-05 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647471+0000", 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8222000000000001e-05 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.26491.0:149468 11.3 11:ce0133d3:::benchmark_data_smithi063_176907_object25192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:31:36.721717+0000", 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 136.03517052699999, 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93143064900000005, 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.26491", 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2774945777", 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 149468 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.908 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721717+0000", 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721717+0000", 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721719+0000", 2024-09-16T10:33:52.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0109999999999999e-06 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721724+0000", 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4009999999999996e-06 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721738+0000", 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3915e-05 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:52.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721746+0000", 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0330000000000008e-06 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721819+0000", 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2772000000000001e-05 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:36.721828+0000", 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0389999999999993e-06 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.644810+0000", 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92298208800000003 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.644915+0000", 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010486500000000001 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.645020+0000", 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104895 2024-09-16T10:33:52.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647259+0000", 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022390140000000001 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.647311+0000", 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1975999999999998e-05 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.652409+0000", 2024-09-16T10:33:52.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050983360000000002 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.652460+0000", 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1100999999999997e-05 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:31:37.653147+0000", 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068720299999999995 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:52.915 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:52.915 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:53.183 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:53.462 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: 6.15 scrub starts 2024-09-16T10:33:53.463 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: 6.15 scrub ok 2024-09-16T10:33:53.463 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.463 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.463 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.463 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:33:53.464 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: 1.3 scrub starts 2024-09-16T10:33:53.464 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: 1.3 scrub ok 2024-09-16T10:33:53.464 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:53 smithi165 ceph-mon[32696]: osdmap e580: 8 total, 8 up, 8 in 2024-09-16T10:33:53.468 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: 6.15 scrub starts 2024-09-16T10:33:53.468 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: 6.15 scrub ok 2024-09-16T10:33:53.468 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.468 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.468 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: 1.3 scrub starts 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: 1.3 scrub ok 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[29993]: osdmap e580: 8 total, 8 up, 8 in 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: 6.15 scrub starts 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: 6.15 scrub ok 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 73'79 2024-09-16T10:33:53.469 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:33:53.470 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: 1.3 scrub starts 2024-09-16T10:33:53.470 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: 1.3 scrub ok 2024-09-16T10:33:53.470 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:53 smithi063 ceph-mon[39470]: osdmap e580: 8 total, 8 up, 8 in 2024-09-16T10:33:53.680 INFO:teuthology.orchestra.run.smithi063.stdout:mon.a: {} 2024-09-16T10:33:53.681 INFO:teuthology.orchestra.run.smithi063.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T10:33:53.689 INFO:teuthology.orchestra.run.smithi063.stdout:mon.b: {} 2024-09-16T10:33:53.689 INFO:teuthology.orchestra.run.smithi063.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T10:33:53.699 INFO:teuthology.orchestra.run.smithi063.stdout:mon.c: {} 2024-09-16T10:33:53.699 INFO:teuthology.orchestra.run.smithi063.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T10:33:53.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T10:33:54.031 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:54.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T10:33:54.441 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: 2.2 scrub starts 2024-09-16T10:33:54.441 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: 2.2 scrub ok 2024-09-16T10:33:54.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: 6.2 scrub starts 2024-09-16T10:33:54.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: 6.2 scrub ok 2024-09-16T10:33:54.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: 3.3 scrub starts 2024-09-16T10:33:54.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: 3.3 scrub ok 2024-09-16T10:33:54.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: pgmap v1103: 152 pgs: 38 peering, 3 remapped+peering, 4 active+remapped+backfill_toofull, 107 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 901 B/s, 19 objects/s recovering 2024-09-16T10:33:54.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2190420841' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:33:54.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:54 smithi165 ceph-mon[32696]: osdmap e581: 8 total, 8 up, 8 in 2024-09-16T10:33:54.557 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:54.558 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:54.558 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-16T10:33:54.558 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:54.647 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:54.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: 2.2 scrub starts 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: 2.2 scrub ok 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: 6.2 scrub starts 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: 6.2 scrub ok 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: 3.3 scrub starts 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: 3.3 scrub ok 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: pgmap v1103: 152 pgs: 38 peering, 3 remapped+peering, 4 active+remapped+backfill_toofull, 107 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 901 B/s, 19 objects/s recovering 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2190420841' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[39470]: osdmap e581: 8 total, 8 up, 8 in 2024-09-16T10:33:54.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: 2.2 scrub starts 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: 2.2 scrub ok 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: 6.2 scrub starts 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: 6.2 scrub ok 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: 3.3 scrub starts 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: 3.3 scrub ok 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: pgmap v1103: 152 pgs: 38 peering, 3 remapped+peering, 4 active+remapped+backfill_toofull, 107 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 901 B/s, 19 objects/s recovering 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2190420841' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T10:33:54.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:54 smithi063 ceph-mon[29993]: osdmap e581: 8 total, 8 up, 8 in 2024-09-16T10:33:55.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T10:33:55.449 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: 6.4 scrub starts 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: 6.4 scrub ok 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: 1.5 scrub starts 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: 1.5 scrub ok 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: 6.1b scrub starts 2024-09-16T10:33:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:55 smithi165 ceph-mon[32696]: 6.1b scrub ok 2024-09-16T10:33:55.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: 6.4 scrub starts 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: 6.4 scrub ok 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: 1.5 scrub starts 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: 1.5 scrub ok 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: 6.1b scrub starts 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[29993]: 6.1b scrub ok 2024-09-16T10:33:55.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: 6.4 scrub starts 2024-09-16T10:33:55.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: 6.4 scrub ok 2024-09-16T10:33:55.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:33:55.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:33:55.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: 1.5 scrub starts 2024-09-16T10:33:55.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: 1.5 scrub ok 2024-09-16T10:33:55.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: 6.1b scrub starts 2024-09-16T10:33:55.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:55 smithi063 ceph-mon[39470]: 6.1b scrub ok 2024-09-16T10:33:55.971 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:55.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-16T10:33:55.971 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-16T10:33:55.971 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-16T10:33:55.971 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:56.424 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1886992453' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: from='client.46799 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: 1.7 scrub starts 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: 1.7 scrub ok 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: pgmap v1105: 152 pgs: 1 active+remapped, 7 peering, 4 active+remapped+backfill_toofull, 140 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 1.1 KiB/s, 35 objects/s recovering 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: 5.4 scrub starts 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: 5.4 scrub ok 2024-09-16T10:33:56.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:56 smithi165 ceph-mon[32696]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-16T10:33:56.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T10:33:56.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1886992453' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:56.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: from='client.46799 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:56.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: 1.7 scrub starts 2024-09-16T10:33:56.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: 1.7 scrub ok 2024-09-16T10:33:56.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: pgmap v1105: 152 pgs: 1 active+remapped, 7 peering, 4 active+remapped+backfill_toofull, 140 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 1.1 KiB/s, 35 objects/s recovering 2024-09-16T10:33:56.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: 5.4 scrub starts 2024-09-16T10:33:56.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: 5.4 scrub ok 2024-09-16T10:33:56.704 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[29993]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-16T10:33:56.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1886992453' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:56.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: from='client.46799 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:56.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: 1.7 scrub starts 2024-09-16T10:33:56.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: 1.7 scrub ok 2024-09-16T10:33:56.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: pgmap v1105: 152 pgs: 1 active+remapped, 7 peering, 4 active+remapped+backfill_toofull, 140 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 1.1 KiB/s, 35 objects/s recovering 2024-09-16T10:33:56.705 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: 5.4 scrub starts 2024-09-16T10:33:56.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: 5.4 scrub ok 2024-09-16T10:33:56.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:56 smithi063 ceph-mon[39470]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-16T10:33:56.935 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T10:33:57.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-16T10:33:57.305 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:57.409 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-16T10:33:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-16T10:33:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-16T10:33:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-16T10:33:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.409 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216464 10.11 e443/442 10:8bbe5995:::benchmark_data_smithi063_149043_object84027:head v 443'11994, mlcod=389'6270)", 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.41232971099998, 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180527202, 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:33:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994107+0000", 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994108+0000", 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1179999999999999e-06 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994115+0000", 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7109999999999998e-06 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994124+0000", 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8120000000000002e-06 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.994131+0000", 2024-09-16T10:33:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2929999999999997e-06 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996650+0000", 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002519777 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996675+0000", 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4800999999999999e-05 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174595+0000", 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17792013200000001 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174634+0000", 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8557999999999999e-05 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(10.1 443/442 [PushOp(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head, version: 391'6545, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:82b5de33:::benchmark_data_smithi063_149043_object106368:head@391'6545, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.40968233900003, 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19788486499999999, 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996754+0000", 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996756+0000", 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4810000000000001e-06 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996812+0000", 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6508999999999997e-05 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996831+0000", 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9009e-05 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.996849+0000", 2024-09-16T10:33:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7612999999999999e-05 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193170+0000", 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19632173999999999 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.193181+0000", 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0138e-05 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.194639+0000", 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001458375 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216466 10.9 e443/442 10:96e6e3aa:::benchmark_data_smithi063_149043_object84029:head v 443'12076, mlcod=386'6101)", 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.40924486300003, 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18400087900000001, 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997192+0000", 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997193+0000", 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2759999999999999e-06 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997200+0000", 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8950000000000001e-06 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997209+0000", 2024-09-16T10:33:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2010000000000005e-06 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:30.997226+0000", 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7059e-05 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002158+0000", 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049319699999999999 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.002173+0000", 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4661000000000001e-05 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181097+0000", 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.178924214 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181192+0000", 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5602999999999999e-05 2024-09-16T10:33:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216473 10.9 e443/442 10:9420f438:::benchmark_data_smithi063_149043_object84036:head v 443'12077, mlcod=386'6101)", 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.405614894, 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18039601599999999, 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000822+0000", 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000823+0000", 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.221e-06 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000830+0000", 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1319999999999996e-06 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000840+0000", 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0378e-05 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.000847+0000", 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4710000000000004e-06 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007645+0000", 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067982770000000001 2024-09-16T10:33:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007677+0000", 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2172999999999998e-05 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181195+0000", 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17351757300000001 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.181218+0000", 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2790999999999999e-05 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216481 10.11 e443/442 10:8cd97c52:::benchmark_data_smithi063_149043_object84044:head v 443'11995, mlcod=389'6270)", 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.40045689200002, 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18177754400000001, 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005980+0000", 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005981+0000", 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.618e-06 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.005991+0000", 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4660000000000005e-06 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006004+0000", 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2964e-05 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.006012+0000", 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5939999999999994e-06 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007635+0000", 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001622706 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007678+0000", 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3282000000000003e-05 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187680+0000", 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180001892 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.187757+0000", 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7021999999999995e-05 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216483 10.9 e443/442 10:91f62e6a:::benchmark_data_smithi063_149043_object84046:head v 443'12078, mlcod=386'6101)", 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.39930823499998, 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.185520875, 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007128+0000", 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007129+0000", 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.198e-06 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007136+0000", 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7780000000000002e-06 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007158+0000", 2024-09-16T10:33:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2013000000000001e-05 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.007166+0000", 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999999e-06 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014305+0000", 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071398319999999996 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014326+0000", 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0767999999999998e-05 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192615+0000", 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17828850800000001 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192649+0000", 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4366999999999998e-05 2024-09-16T10:33:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216472 10.8 e443/442 10:167e16b6:::benchmark_data_smithi063_149043_object84035:head v 443'11858, mlcod=389'6215)", 2024-09-16T10:33:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:33:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.39765184800001, 2024-09-16T10:33:57.440 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:33:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:33:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.167553278, 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008785+0000", 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008786+0000", 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.296e-06 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008820+0000", 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4329999999999998e-05 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008830+0000", 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3360000000000001e-06 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.008836+0000", 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3040000000000001e-06 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010938+0000", 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021016780000000001 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.010959+0000", 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1454000000000002e-05 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176290+0000", 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16533061800000001 2024-09-16T10:33:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176338+0000", 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8262e-05 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216489 10.c e443/442 10:36cb7a7c:::benchmark_data_smithi063_149043_object84052:head v 443'13145, mlcod=443'13143)", 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.39378523699997, 2024-09-16T10:33:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16334574800000001, 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012651+0000", 2024-09-16T10:33:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012653+0000", 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.606e-06 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012661+0000", 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0469999999999994e-06 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012673+0000", 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2490000000000001e-05 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012682+0000", 2024-09-16T10:33:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3450000000000006e-06 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012753+0000", 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1565999999999999e-05 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.012773+0000", 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9942999999999999e-05 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175914+0000", 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16314114399999999 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.175997+0000", 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2607000000000004e-05 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216490 10.10 e443/441 10:0ed1dccc:::benchmark_data_smithi063_149043_object84053:head v 443'12109, mlcod=443'12107)", 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.39176330100003, 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16562529300000001, 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014673+0000", 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014675+0000", 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.593e-06 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014687+0000", 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2071e-05 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014697+0000", 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0148e-05 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.014704+0000", 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.613e-06 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174625+0000", 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15992183700000001 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6707e-05 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180235+0000", 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055831880000000002 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180298+0000", 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3136e-05 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216488 10.11 e443/442 10:8a4eba15:::benchmark_data_smithi063_149043_object84051:head v 443'11996, mlcod=389'6270)", 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.39083222599999, 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.177066683, 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015604+0000", 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015606+0000", 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.404e-06 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015617+0000", 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1769e-05 2024-09-16T10:33:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015628+0000", 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0489000000000001e-05 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.015634+0000", 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2960000000000004e-06 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174659+0000", 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15902495699999999 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174681+0000", 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1778999999999999e-05 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192652+0000", 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017971311 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.192671+0000", 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8678e-05 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216491 10.10 e443/441 10:0e46f635:::benchmark_data_smithi063_149043_object84054:head v 443'12110, mlcod=443'12107)", 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.389145875, 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.163036928, 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017291+0000", 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017292+0000", 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.443e-06 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.017300+0000", 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5599999999999996e-06 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174635+0000", 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.157335014 2024-09-16T10:33:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174657+0000", 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1985e-05 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176330+0000", 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016733029999999999 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.176359+0000", 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9564e-05 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180301+0000", 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00394174 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.180328+0000", 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6319e-05 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:216495 10.c e443/442 10:33a8a0ac:::benchmark_data_smithi063_149043_object84058:head v 443'13146, mlcod=443'13143)", 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 326.38785087999997, 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.160016309, 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018586+0000", 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018587+0000", 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3820000000000001e-06 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.018595+0000", 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4759999999999998e-06 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174544+0000", 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15594872200000001 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174566+0000", 2024-09-16T10:33:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1979000000000001e-05 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174626+0000", 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9899999999999999e-05 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.174652+0000", 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.561e-05 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178550+0000", 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038981969999999999 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:31.178602+0000", 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2043000000000003e-05 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221476 10.9 e446/442 10:900b709d:::benchmark_data_smithi063_149043_object89039:head v 446'12225, mlcod=386'6101)", 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:33:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 321.327787683, 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50937916299999997, 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078649+0000", 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078650+0000", 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.091e-06 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078657+0000", 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078665+0000", 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6619999999999994e-06 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078671+0000", 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1700000000000002e-06 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078762+0000", 2024-09-16T10:33:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0346999999999994e-05 2024-09-16T10:33:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.078786+0000", 2024-09-16T10:33:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4314000000000001e-05 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587999+0000", 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.509212514 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.588028+0000", 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9326999999999999e-05 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221483 10.c e446/442 10:32ed840b:::benchmark_data_smithi063_149043_object89046:head v 446'13312, mlcod=445'13310)", 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 321.32705859700002, 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50397308900000004, 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079378+0000", 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079379+0000", 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.395e-06 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079385+0000", 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5639999999999996e-06 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079392+0000", 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.481e-06 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079398+0000", 2024-09-16T10:33:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8950000000000003e-06 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079450+0000", 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1786999999999997e-05 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079469+0000", 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8841000000000001e-05 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583301+0000", 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50383221600000005 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583351+0000", 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9910000000000002e-05 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:221482 10.1f e446/441 10:ffb425b8:::benchmark_data_smithi063_149043_object89045:head v 446'13070, mlcod=445'13068)", 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 321.32685114399999, 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50798407899999998, 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079585+0000", 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079587+0000", 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.187e-06 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079593+0000", 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0929999999999999e-06 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079600+0000", 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5070000000000003e-06 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.079606+0000", 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7509999999999997e-06 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583349+0000", 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50374310499999997 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.583378+0000", 2024-09-16T10:33:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8649e-05 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587530+0000", 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041521559999999997 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:28:36.587569+0000", 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9631000000000001e-05 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:249520 10.1 10:827f4c35:::benchmark_data_smithi063_149043_object117083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 293.94947196499999, 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11004649, 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 249520 2024-09-16T10:33:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456965+0000", 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456966+0000", 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.689e-06 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456972+0000", 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3410000000000002e-06 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456983+0000", 2024-09-16T10:33:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1039999999999999e-05 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.456988+0000", 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6579999999999999e-06 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.457040+0000", 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2166000000000002e-05 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.457143+0000", 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102909 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.458594+0000", 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001450914 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.460804+0000", 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022102369999999999 2024-09-16T10:33:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.461370+0000", 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056559799999999995 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.566893+0000", 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105523061 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.566950+0000", 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6637e-05 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:03.567011+0000", 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1240000000000003e-05 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.16152.0:251245 10.1 10:810fe972:::benchmark_data_smithi063_149043_object118808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-16T10:33:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 292.51612155399999, 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083403509000000001, 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.16152", 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.63:0/2515774729", 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 251245 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890315+0000", 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890317+0000", 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.809e-06 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890322+0000", 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5219999999999998e-06 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890333+0000", 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1069e-05 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890342+0000", 2024-09-16T10:33:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7590000000000003e-06 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890709+0000", 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036659099999999999 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.890839+0000", 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000130147 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,4", 2024-09-16T10:33:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.891118+0000", 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027897099999999998 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.892553+0000", 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014351450000000001 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.892975+0000", 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042191700000000001 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973569+0000", 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080593926999999996 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973658+0000", 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9295000000000005e-05 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:04.973718+0000", 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0356999999999998e-05 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDelete(10.8 e460,442 10:146751e1:::benchmark_data_smithi063_149043_object32211:head,408'10091)", 2024-09-16T10:33:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.29819318199998, 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.111867522, 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:33:57.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108243+0000", 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108244+0000", 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2900000000000002e-07 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.108248+0000", 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.121e-06 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219378+0000", 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11112931500000001 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219407+0000", 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9363e-05 2024-09-16T10:33:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220044+0000", 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063718900000000003 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220053+0000", 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9199999999999993e-06 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220111+0000", 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7685000000000001e-05 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.16152.0:254612 10.22 e460/441 10:460a596d:::benchmark_data_smithi063_149043_object122175:head v 460'12456, mlcod=460'12454)", 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.28707622500002, 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101663158, 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.505 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119360+0000", 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119362+0000", 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4839999999999999e-06 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.119368+0000", 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.348e-06 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219363+0000", 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099994825999999995 2024-09-16T10:33:57.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219389+0000", 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6383000000000001e-05 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219541+0000", 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151476 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.219560+0000", 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9440999999999999e-05 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.220981+0000", 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001421095 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:29:08.221023+0000", 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2104999999999998e-05 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.509 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.26491.0:72894 11.5 e493/476 11:ad5dc03e:::benchmark_data_smithi063_176907_object72893:head v 493'4542, mlcod=493'4539)", 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 213.67432816900001, 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060735732000000001, 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732108+0000", 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732109+0000", 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.119e-06 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732131+0000", 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1999000000000001e-05 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732141+0000", 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4579999999999999e-06 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732147+0000", 2024-09-16T10:33:57.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2759999999999997e-06 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732195+0000", 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8288999999999998e-05 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.732215+0000", 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9961000000000001e-05 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.792747+0000", 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.060531198000000001 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-16T10:30:23.792844+0000", 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7431999999999998e-05 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-16T10:33:57.514 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-16T10:33:57.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[29993]: 5.1a scrub starts 2024-09-16T10:33:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[29993]: 5.1a scrub ok 2024-09-16T10:33:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[29993]: from='client.46799 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:33:57.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[29993]: osdmap e582: 8 total, 8 up, 8 in 2024-09-16T10:33:57.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[39470]: 5.1a scrub starts 2024-09-16T10:33:57.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[39470]: 5.1a scrub ok 2024-09-16T10:33:57.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[39470]: from='client.46799 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:33:57.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:57 smithi063 ceph-mon[39470]: osdmap e582: 8 total, 8 up, 8 in 2024-09-16T10:33:57.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:57 smithi165 ceph-mon[32696]: 5.1a scrub starts 2024-09-16T10:33:57.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:57 smithi165 ceph-mon[32696]: 5.1a scrub ok 2024-09-16T10:33:57.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:57 smithi165 ceph-mon[32696]: from='client.46799 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:33:57.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:57 smithi165 ceph-mon[32696]: osdmap e582: 8 total, 8 up, 8 in 2024-09-16T10:33:58.436 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: 3.14 scrub starts 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: 3.14 scrub ok 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: 5.17 scrub starts 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: 5.17 scrub ok 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: osdmap e583: 8 total, 8 up, 8 in 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: 4.5 scrub starts 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: 4.5 scrub ok 2024-09-16T10:33:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: pgmap v1108: 152 pgs: 1 active+remapped, 2 peering, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 950 B/s, 32 objects/s recovering 2024-09-16T10:33:58.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/543574926' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:58.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:58 smithi165 ceph-mon[32696]: from='client.46805 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:58.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: 3.14 scrub starts 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: 3.14 scrub ok 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: 5.17 scrub starts 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: 5.17 scrub ok 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: osdmap e583: 8 total, 8 up, 8 in 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: 4.5 scrub starts 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: 4.5 scrub ok 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: pgmap v1108: 152 pgs: 1 active+remapped, 2 peering, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 950 B/s, 32 objects/s recovering 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/543574926' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[29993]: from='client.46805 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:58.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: 3.14 scrub starts 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: 3.14 scrub ok 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: 5.17 scrub starts 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: 5.17 scrub ok 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: osdmap e583: 8 total, 8 up, 8 in 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: 4.5 scrub starts 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: 4.5 scrub ok 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: pgmap v1108: 152 pgs: 1 active+remapped, 2 peering, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 950 B/s, 32 objects/s recovering 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/543574926' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:58.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:58 smithi063 ceph-mon[39470]: from='client.46805 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-16T10:33:59.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T10:33:59.401 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:33:59.690 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-16T10:33:59.690 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: from='client.46805 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:33:59.690 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: osdmap e584: 8 total, 8 up, 8 in 2024-09-16T10:33:59.690 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: 3.6 scrub starts 2024-09-16T10:33:59.690 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: 3.6 scrub ok 2024-09-16T10:33:59.690 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: 6.5 scrub starts 2024-09-16T10:33:59.690 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: 6.5 scrub ok 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: 3.4 scrub starts 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[29993]: 3.4 scrub ok 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: from='client.46805 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: osdmap e584: 8 total, 8 up, 8 in 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: 3.6 scrub starts 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: 3.6 scrub ok 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: 6.5 scrub starts 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: 6.5 scrub ok 2024-09-16T10:33:59.691 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: 3.4 scrub starts 2024-09-16T10:33:59.692 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:33:59 smithi063 ceph-mon[39470]: 3.4 scrub ok 2024-09-16T10:33:59.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:33:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-16T10:33:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: from='client.46805 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-16T10:33:59.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: osdmap e584: 8 total, 8 up, 8 in 2024-09-16T10:33:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: 3.6 scrub starts 2024-09-16T10:33:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: 3.6 scrub ok 2024-09-16T10:33:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: 6.5 scrub starts 2024-09-16T10:33:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: 6.5 scrub ok 2024-09-16T10:33:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: 3.4 scrub starts 2024-09-16T10:33:59.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:33:59 smithi165 ceph-mon[32696]: 3.4 scrub ok 2024-09-16T10:34:00.439 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T10:34:00.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[29993]: pgmap v1110: 152 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 100 B/s, 3 objects/s recovering 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3165876979' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[39470]: pgmap v1110: 152 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 100 B/s, 3 objects/s recovering 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:34:00.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:00 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3165876979' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:00 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:00 smithi165 ceph-mon[32696]: pgmap v1110: 152 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 100 B/s, 3 objects/s recovering 2024-09-16T10:34:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:00 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:34:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:00 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-16T10:34:00.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:00 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3165876979' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:01.052 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-16T10:34:01.320 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:01.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: 6.1a scrub starts 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: 6.1a scrub ok 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3165876979' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: osdmap e585: 8 total, 8 up, 8 in 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: 4.2 scrub starts 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: 4.2 scrub ok 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: 6.1a scrub starts 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: 6.1a scrub ok 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-16T10:34:01.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3165876979' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:34:01.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: osdmap e585: 8 total, 8 up, 8 in 2024-09-16T10:34:01.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: 4.2 scrub starts 2024-09-16T10:34:01.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: 4.2 scrub ok 2024-09-16T10:34:01.703 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:01 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: 6.1a scrub starts 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: 6.1a scrub ok 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3165876979' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: osdmap e585: 8 total, 8 up, 8 in 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: 4.2 scrub starts 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: 4.2 scrub ok 2024-09-16T10:34:01.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:01 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:02.458 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T10:34:02.471 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: 6.8 scrub starts 2024-09-16T10:34:02.471 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: 6.8 scrub ok 2024-09-16T10:34:02.471 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: 5.10 scrub starts 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: 5.10 scrub ok 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: osdmap e586: 8 total, 8 up, 8 in 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: pgmap v1113: 152 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2789573073' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: from='client.46817 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: 4.12 scrub starts 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[29993]: 4.12 scrub ok 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: 6.8 scrub starts 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: 6.8 scrub ok 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: 5.10 scrub starts 2024-09-16T10:34:02.472 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: 5.10 scrub ok 2024-09-16T10:34:02.473 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: osdmap e586: 8 total, 8 up, 8 in 2024-09-16T10:34:02.473 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: pgmap v1113: 152 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:02.473 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2789573073' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:02.473 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: from='client.46817 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:02.473 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: 4.12 scrub starts 2024-09-16T10:34:02.473 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:02 smithi063 ceph-mon[39470]: 4.12 scrub ok 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: 6.8 scrub starts 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: 6.8 scrub ok 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: 5.10 scrub starts 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: 5.10 scrub ok 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: osdmap e586: 8 total, 8 up, 8 in 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: pgmap v1113: 152 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 147 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2789573073' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: from='client.46817 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-16T10:34:02.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: 4.12 scrub starts 2024-09-16T10:34:02.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:02 smithi165 ceph-mon[32696]: 4.12 scrub ok 2024-09-16T10:34:03.168 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T10:34:03.430 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:03.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[39470]: 4.1c scrub starts 2024-09-16T10:34:03.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[39470]: 4.1c scrub ok 2024-09-16T10:34:03.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[39470]: 3.1c scrub starts 2024-09-16T10:34:03.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[39470]: 3.1c scrub ok 2024-09-16T10:34:03.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[39470]: from='client.46817 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:34:03.706 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[39470]: osdmap e587: 8 total, 8 up, 8 in 2024-09-16T10:34:03.706 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[29993]: 4.1c scrub starts 2024-09-16T10:34:03.707 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[29993]: 4.1c scrub ok 2024-09-16T10:34:03.707 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[29993]: 3.1c scrub starts 2024-09-16T10:34:03.707 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[29993]: 3.1c scrub ok 2024-09-16T10:34:03.707 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[29993]: from='client.46817 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:34:03.707 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:03 smithi063 ceph-mon[29993]: osdmap e587: 8 total, 8 up, 8 in 2024-09-16T10:34:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:03 smithi165 ceph-mon[32696]: 4.1c scrub starts 2024-09-16T10:34:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:03 smithi165 ceph-mon[32696]: 4.1c scrub ok 2024-09-16T10:34:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:03 smithi165 ceph-mon[32696]: 3.1c scrub starts 2024-09-16T10:34:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:03 smithi165 ceph-mon[32696]: 3.1c scrub ok 2024-09-16T10:34:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:03 smithi165 ceph-mon[32696]: from='client.46817 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T10:34:03.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:03 smithi165 ceph-mon[32696]: osdmap e587: 8 total, 8 up, 8 in 2024-09-16T10:34:04.471 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T10:34:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: 5.6 scrub starts 2024-09-16T10:34:04.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: 5.6 scrub ok 2024-09-16T10:34:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: osdmap e588: 8 total, 8 up, 8 in 2024-09-16T10:34:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: pgmap v1116: 151 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 146 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:34:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:34:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/747680562' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: from='client.46823 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:04.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:04 smithi165 ceph-mon[32696]: 4.19 scrub starts 2024-09-16T10:34:04.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: 5.6 scrub starts 2024-09-16T10:34:04.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: 5.6 scrub ok 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: osdmap e588: 8 total, 8 up, 8 in 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: pgmap v1116: 151 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 146 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/747680562' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: from='client.46823 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[29993]: 4.19 scrub starts 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: 5.6 scrub starts 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: 5.6 scrub ok 2024-09-16T10:34:04.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: osdmap e588: 8 total, 8 up, 8 in 2024-09-16T10:34:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: pgmap v1116: 151 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 146 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:34:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-16T10:34:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/747680562' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: from='client.46823 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:04.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:04 smithi063 ceph-mon[39470]: 4.19 scrub starts 2024-09-16T10:34:05.453 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-16T10:34:05.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[29993]: 4.19 scrub ok 2024-09-16T10:34:05.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[29993]: 5.19 scrub starts 2024-09-16T10:34:05.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[29993]: 5.19 scrub ok 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[29993]: from='client.46823 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[29993]: osdmap e589: 8 total, 8 up, 8 in 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[29993]: osdmap e590: 8 total, 8 up, 8 in 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[39470]: 4.19 scrub ok 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[39470]: 5.19 scrub starts 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[39470]: 5.19 scrub ok 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[39470]: from='client.46823 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[39470]: osdmap e589: 8 total, 8 up, 8 in 2024-09-16T10:34:05.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:05 smithi063 ceph-mon[39470]: osdmap e590: 8 total, 8 up, 8 in 2024-09-16T10:34:05.723 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:05.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:05 smithi165 ceph-mon[32696]: 4.19 scrub ok 2024-09-16T10:34:05.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:05 smithi165 ceph-mon[32696]: 5.19 scrub starts 2024-09-16T10:34:05.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:05 smithi165 ceph-mon[32696]: 5.19 scrub ok 2024-09-16T10:34:05.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:05 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-16T10:34:05.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:05 smithi165 ceph-mon[32696]: from='client.46823 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:34:05.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:05 smithi165 ceph-mon[32696]: osdmap e589: 8 total, 8 up, 8 in 2024-09-16T10:34:05.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:05 smithi165 ceph-mon[32696]: osdmap e590: 8 total, 8 up, 8 in 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 4.d scrub starts 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 4.d scrub ok 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 6.7 scrub starts 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 6.7 scrub ok 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 5.11 scrub starts 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 5.11 scrub ok 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: pgmap v1119: 151 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 146 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: osdmap e591: 8 total, 8 up, 8 in 2024-09-16T10:34:06.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 5.14 scrub starts 2024-09-16T10:34:06.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: 5.14 scrub ok 2024-09-16T10:34:06.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3733992751' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:06.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:06 smithi165 ceph-mon[32696]: from='client.46829 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:06.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 4.d scrub starts 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 4.d scrub ok 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 6.7 scrub starts 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 6.7 scrub ok 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 5.11 scrub starts 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 5.11 scrub ok 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: pgmap v1119: 151 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 146 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: osdmap e591: 8 total, 8 up, 8 in 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 5.14 scrub starts 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: 5.14 scrub ok 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3733992751' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:06.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[29993]: from='client.46829 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 4.d scrub starts 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 4.d scrub ok 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 6.7 scrub starts 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 6.7 scrub ok 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 5.11 scrub starts 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 5.11 scrub ok 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: pgmap v1119: 151 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 146 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: osdmap e591: 8 total, 8 up, 8 in 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 5.14 scrub starts 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: 5.14 scrub ok 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3733992751' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:06.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:06 smithi063 ceph-mon[39470]: from='client.46829 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-16T10:34:07.152 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T10:34:07.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:34:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:34:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:07 smithi165 ceph-mon[32696]: 1.10 scrub starts 2024-09-16T10:34:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:07 smithi165 ceph-mon[32696]: 1.10 scrub ok 2024-09-16T10:34:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:07 smithi165 ceph-mon[32696]: from='client.46829 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:34:07.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:07 smithi165 ceph-mon[32696]: osdmap e592: 8 total, 8 up, 8 in 2024-09-16T10:34:07.840 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[29993]: 1.10 scrub starts 2024-09-16T10:34:07.840 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[29993]: 1.10 scrub ok 2024-09-16T10:34:07.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[29993]: from='client.46829 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:34:07.841 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[29993]: osdmap e592: 8 total, 8 up, 8 in 2024-09-16T10:34:07.841 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[39470]: 1.10 scrub starts 2024-09-16T10:34:07.841 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[39470]: 1.10 scrub ok 2024-09-16T10:34:07.841 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[39470]: from='client.46829 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T10:34:07.841 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:07 smithi063 ceph-mon[39470]: osdmap e592: 8 total, 8 up, 8 in 2024-09-16T10:34:07.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T10:34:08.144 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[29993]: 5.d scrub starts 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[29993]: 5.d scrub ok 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[29993]: pgmap v1122: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[39470]: 5.d scrub starts 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[39470]: 5.d scrub ok 2024-09-16T10:34:08.519 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[39470]: pgmap v1122: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:08.520 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:34:08.520 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:08 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:34:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:08 smithi165 ceph-mon[32696]: 5.d scrub starts 2024-09-16T10:34:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:08 smithi165 ceph-mon[32696]: 5.d scrub ok 2024-09-16T10:34:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:08 smithi165 ceph-mon[32696]: pgmap v1122: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:08 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:34:08.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:08 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:34:09.498 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T10:34:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:09 smithi165 ceph-mon[32696]: osdmap e593: 8 total, 8 up, 8 in 2024-09-16T10:34:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:09 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3981551462' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:09 smithi165 ceph-mon[32696]: from='client.46835 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:09 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:09.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:09 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:09.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:34:09 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:34:09.882 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[29993]: osdmap e593: 8 total, 8 up, 8 in 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3981551462' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[29993]: from='client.46835 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[39470]: osdmap e593: 8 total, 8 up, 8 in 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3981551462' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[39470]: from='client.46835 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:09.883 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:09 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:10.294 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-16T10:34:10.560 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:10.682 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: from='client.46835 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: osdmap e594: 8 total, 8 up, 8 in 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: 4.16 scrub starts 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: 4.16 scrub ok 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: pgmap v1125: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: 6.1d scrub starts 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[29993]: 6.1d scrub ok 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:10.683 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: from='client.46835 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:34:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: osdmap e594: 8 total, 8 up, 8 in 2024-09-16T10:34:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: 4.16 scrub starts 2024-09-16T10:34:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: 4.16 scrub ok 2024-09-16T10:34:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: pgmap v1125: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: 6.1d scrub starts 2024-09-16T10:34:10.684 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:10 smithi063 ceph-mon[39470]: 6.1d scrub ok 2024-09-16T10:34:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: from='client.46835 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:34:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: osdmap e594: 8 total, 8 up, 8 in 2024-09-16T10:34:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: 4.16 scrub starts 2024-09-16T10:34:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: 4.16 scrub ok 2024-09-16T10:34:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: pgmap v1125: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:10.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: 6.1d scrub starts 2024-09-16T10:34:10.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:10 smithi165 ceph-mon[32696]: 6.1d scrub ok 2024-09-16T10:34:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: 5.1b scrub starts 2024-09-16T10:34:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: 5.1b scrub ok 2024-09-16T10:34:11.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: 4.e scrub starts 2024-09-16T10:34:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: 4.e scrub ok 2024-09-16T10:34:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: osdmap e595: 8 total, 8 up, 8 in 2024-09-16T10:34:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]} : dispatch 2024-09-16T10:34:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-16T10:34:11.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:34:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]} : dispatch 2024-09-16T10:34:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-16T10:34:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:34:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3708427332' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:11.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: from='client.46841 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:11.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: 4.7 scrub starts 2024-09-16T10:34:11.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:11 smithi165 ceph-mon[32696]: 4.7 scrub ok 2024-09-16T10:34:11.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: 5.1b scrub starts 2024-09-16T10:34:11.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: 5.1b scrub ok 2024-09-16T10:34:11.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: 4.e scrub starts 2024-09-16T10:34:11.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: 4.e scrub ok 2024-09-16T10:34:11.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: osdmap e595: 8 total, 8 up, 8 in 2024-09-16T10:34:11.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]} : dispatch 2024-09-16T10:34:11.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-16T10:34:11.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]} : dispatch 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3708427332' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: from='client.46841 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: 4.7 scrub starts 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[29993]: 4.7 scrub ok 2024-09-16T10:34:11.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: 5.1b scrub starts 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: 5.1b scrub ok 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: 4.e scrub starts 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: 4.e scrub ok 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: osdmap e595: 8 total, 8 up, 8 in 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]} : dispatch 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]} : dispatch 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-16T10:34:11.955 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T10:34:11.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3708427332' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:11.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: from='client.46841 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-16T10:34:11.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: 4.7 scrub starts 2024-09-16T10:34:11.956 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:11 smithi063 ceph-mon[39470]: 4.7 scrub ok 2024-09-16T10:34:12.150 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T10:34:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: 3.d scrub starts 2024-09-16T10:34:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: 3.d scrub ok 2024-09-16T10:34:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: pgmap v1127: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]': finished 2024-09-16T10:34:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-16T10:34:12.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T10:34:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: from='client.46841 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:34:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: osdmap e596: 8 total, 8 up, 8 in 2024-09-16T10:34:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: 4.9 scrub starts 2024-09-16T10:34:12.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:12 smithi165 ceph-mon[32696]: 4.9 scrub ok 2024-09-16T10:34:12.916 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: 3.d scrub starts 2024-09-16T10:34:12.916 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: 3.d scrub ok 2024-09-16T10:34:12.916 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: pgmap v1127: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:12.916 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]': finished 2024-09-16T10:34:12.916 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: from='client.46841 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: osdmap e596: 8 total, 8 up, 8 in 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: 4.9 scrub starts 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[29993]: 4.9 scrub ok 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: 3.d scrub starts 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: 3.d scrub ok 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: pgmap v1127: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 0]}]': finished 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-16T10:34:12.917 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T10:34:12.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: from='client.46841 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T10:34:12.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: osdmap e596: 8 total, 8 up, 8 in 2024-09-16T10:34:12.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: 4.9 scrub starts 2024-09-16T10:34:12.918 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:12 smithi063 ceph-mon[39470]: 4.9 scrub ok 2024-09-16T10:34:12.952 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T10:34:13.230 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[29993]: 6.14 scrub starts 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[29993]: 6.14 scrub ok 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[29993]: osdmap e597: 8 total, 8 up, 8 in 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[29993]: 6.13 scrub starts 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[29993]: 6.13 scrub ok 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[39470]: 6.14 scrub starts 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[39470]: 6.14 scrub ok 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[39470]: osdmap e597: 8 total, 8 up, 8 in 2024-09-16T10:34:13.644 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[39470]: 6.13 scrub starts 2024-09-16T10:34:13.645 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:13 smithi063 ceph-mon[39470]: 6.13 scrub ok 2024-09-16T10:34:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:13 smithi165 ceph-mon[32696]: 6.14 scrub starts 2024-09-16T10:34:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:13 smithi165 ceph-mon[32696]: 6.14 scrub ok 2024-09-16T10:34:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:13 smithi165 ceph-mon[32696]: osdmap e597: 8 total, 8 up, 8 in 2024-09-16T10:34:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:13 smithi165 ceph-mon[32696]: 6.13 scrub starts 2024-09-16T10:34:13.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:13 smithi165 ceph-mon[32696]: 6.13 scrub ok 2024-09-16T10:34:14.519 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T10:34:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:14 smithi165 ceph-mon[32696]: 1.1 scrub starts 2024-09-16T10:34:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:14 smithi165 ceph-mon[32696]: 1.1 scrub ok 2024-09-16T10:34:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:14 smithi165 ceph-mon[32696]: pgmap v1130: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:14 smithi165 ceph-mon[32696]: 3.9 scrub starts 2024-09-16T10:34:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:14 smithi165 ceph-mon[32696]: 3.9 scrub ok 2024-09-16T10:34:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:14 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3807589428' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:14.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:14 smithi165 ceph-mon[32696]: from='client.46847 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:14.810 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[39470]: 1.1 scrub starts 2024-09-16T10:34:14.810 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[39470]: 1.1 scrub ok 2024-09-16T10:34:14.810 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[39470]: pgmap v1130: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:14.810 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[39470]: 3.9 scrub starts 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[39470]: 3.9 scrub ok 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3807589428' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[39470]: from='client.46847 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[29993]: 1.1 scrub starts 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[29993]: 1.1 scrub ok 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[29993]: pgmap v1130: 150 pgs: 1 active+remapped, 4 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[29993]: 3.9 scrub starts 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[29993]: 3.9 scrub ok 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3807589428' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:14.811 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:14 smithi063 ceph-mon[29993]: from='client.46847 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:15.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-16T10:34:15.496 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:15.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[29993]: 1.4 scrub starts 2024-09-16T10:34:15.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[29993]: 1.4 scrub ok 2024-09-16T10:34:15.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[29993]: from='client.46847 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:34:15.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[29993]: osdmap e598: 8 total, 8 up, 8 in 2024-09-16T10:34:15.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[29993]: 5.1d scrub starts 2024-09-16T10:34:15.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[29993]: 5.1d scrub ok 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[29993]: osdmap e599: 8 total, 8 up, 8 in 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[39470]: 1.4 scrub starts 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[39470]: 1.4 scrub ok 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[39470]: from='client.46847 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[39470]: osdmap e598: 8 total, 8 up, 8 in 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[39470]: 5.1d scrub starts 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[39470]: 5.1d scrub ok 2024-09-16T10:34:15.780 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:15 smithi063 ceph-mon[39470]: osdmap e599: 8 total, 8 up, 8 in 2024-09-16T10:34:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:15 smithi165 ceph-mon[32696]: 1.4 scrub starts 2024-09-16T10:34:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:15 smithi165 ceph-mon[32696]: 1.4 scrub ok 2024-09-16T10:34:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:15 smithi165 ceph-mon[32696]: from='client.46847 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:34:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:15 smithi165 ceph-mon[32696]: osdmap e598: 8 total, 8 up, 8 in 2024-09-16T10:34:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:15 smithi165 ceph-mon[32696]: 5.1d scrub starts 2024-09-16T10:34:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:15 smithi165 ceph-mon[32696]: 5.1d scrub ok 2024-09-16T10:34:15.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:15 smithi165 ceph-mon[32696]: osdmap e599: 8 total, 8 up, 8 in 2024-09-16T10:34:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:34:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:34:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: 4.6 scrub starts 2024-09-16T10:34:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: 4.6 scrub ok 2024-09-16T10:34:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: pgmap v1133: 150 pgs: 1 active+remapped, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 109 B/s, 3 objects/s recovering 2024-09-16T10:34:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: 1.e scrub starts 2024-09-16T10:34:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: 1.e scrub ok 2024-09-16T10:34:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: 6.17 scrub starts 2024-09-16T10:34:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: 6.17 scrub ok 2024-09-16T10:34:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2076414793' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:16 smithi165 ceph-mon[32696]: from='client.46853 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:16.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: 4.6 scrub starts 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: 4.6 scrub ok 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: pgmap v1133: 150 pgs: 1 active+remapped, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 109 B/s, 3 objects/s recovering 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: 1.e scrub starts 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: 1.e scrub ok 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: 6.17 scrub starts 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: 6.17 scrub ok 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2076414793' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:16.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[29993]: from='client.46853 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: 4.6 scrub starts 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: 4.6 scrub ok 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: pgmap v1133: 150 pgs: 1 active+remapped, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 109 B/s, 3 objects/s recovering 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: 1.e scrub starts 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: 1.e scrub ok 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: 6.17 scrub starts 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: 6.17 scrub ok 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2076414793' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:16.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:16 smithi063 ceph-mon[39470]: from='client.46853 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-16T10:34:17.153 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T10:34:17.452 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:34:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:34:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:17 smithi165 ceph-mon[32696]: 6.1c scrub starts 2024-09-16T10:34:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:17 smithi165 ceph-mon[32696]: 6.1c scrub ok 2024-09-16T10:34:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:17 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:17 smithi165 ceph-mon[32696]: 3.17 scrub starts 2024-09-16T10:34:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:17 smithi165 ceph-mon[32696]: 3.17 scrub ok 2024-09-16T10:34:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:17 smithi165 ceph-mon[32696]: from='client.46853 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:34:17.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:17 smithi165 ceph-mon[32696]: osdmap e600: 8 total, 8 up, 8 in 2024-09-16T10:34:17.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[29993]: 6.1c scrub starts 2024-09-16T10:34:17.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[29993]: 6.1c scrub ok 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[29993]: 3.17 scrub starts 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[29993]: 3.17 scrub ok 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[29993]: from='client.46853 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[29993]: osdmap e600: 8 total, 8 up, 8 in 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[39470]: 6.1c scrub starts 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[39470]: 6.1c scrub ok 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[39470]: 3.17 scrub starts 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[39470]: 3.17 scrub ok 2024-09-16T10:34:17.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[39470]: from='client.46853 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-16T10:34:17.850 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:17 smithi063 ceph-mon[39470]: osdmap e600: 8 total, 8 up, 8 in 2024-09-16T10:34:17.885 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T10:34:18.164 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:18 smithi165 ceph-mon[32696]: 6.19 scrub starts 2024-09-16T10:34:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:18 smithi165 ceph-mon[32696]: 6.19 scrub ok 2024-09-16T10:34:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:18 smithi165 ceph-mon[32696]: 3.1e scrub starts 2024-09-16T10:34:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:18 smithi165 ceph-mon[32696]: 3.1e scrub ok 2024-09-16T10:34:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:18 smithi165 ceph-mon[32696]: pgmap v1135: 150 pgs: 1 active+remapped, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 99 B/s, 3 objects/s recovering 2024-09-16T10:34:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:18 smithi165 ceph-mon[32696]: osdmap e601: 8 total, 8 up, 8 in 2024-09-16T10:34:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[29993]: 6.19 scrub starts 2024-09-16T10:34:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[29993]: 6.19 scrub ok 2024-09-16T10:34:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[29993]: 3.1e scrub starts 2024-09-16T10:34:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[29993]: 3.1e scrub ok 2024-09-16T10:34:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[29993]: pgmap v1135: 150 pgs: 1 active+remapped, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 99 B/s, 3 objects/s recovering 2024-09-16T10:34:18.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[29993]: osdmap e601: 8 total, 8 up, 8 in 2024-09-16T10:34:18.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[39470]: 6.19 scrub starts 2024-09-16T10:34:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[39470]: 6.19 scrub ok 2024-09-16T10:34:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[39470]: 3.1e scrub starts 2024-09-16T10:34:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[39470]: 3.1e scrub ok 2024-09-16T10:34:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[39470]: pgmap v1135: 150 pgs: 1 active+remapped, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 145 active+clean; 609 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 99 B/s, 3 objects/s recovering 2024-09-16T10:34:18.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:18 smithi063 ceph-mon[39470]: osdmap e601: 8 total, 8 up, 8 in 2024-09-16T10:34:19.148 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T10:34:19.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:34:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:34:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:34:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:34:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:34:19.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4143877843' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:19.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4143877843' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:34:19.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:19 smithi165 ceph-mon[32696]: osdmap e602: 8 total, 8 up, 8 in 2024-09-16T10:34:19.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:34:19.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:34:19.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:34:19.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:34:19.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4143877843' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:19.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4143877843' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:34:19.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[39470]: osdmap e602: 8 total, 8 up, 8 in 2024-09-16T10:34:19.866 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:34:19.866 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:34:19.866 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:34:19.866 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:34:19.866 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4143877843' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:19.866 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4143877843' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:34:19.866 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:19 smithi063 ceph-mon[29993]: osdmap e602: 8 total, 8 up, 8 in 2024-09-16T10:34:19.902 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-16T10:34:20.165 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:20 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:20 smithi165 ceph-mon[32696]: 6.3 scrub starts 2024-09-16T10:34:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:20 smithi165 ceph-mon[32696]: pgmap v1138: 150 pgs: 1 active+remapped, 3 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:20 smithi165 ceph-mon[32696]: 6.3 scrub ok 2024-09-16T10:34:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:20 smithi165 ceph-mon[32696]: 2.0 scrub starts 2024-09-16T10:34:20.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:20 smithi165 ceph-mon[32696]: 2.0 scrub ok 2024-09-16T10:34:20.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[29993]: 6.3 scrub starts 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[29993]: pgmap v1138: 150 pgs: 1 active+remapped, 3 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[29993]: 6.3 scrub ok 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[29993]: 2.0 scrub starts 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[29993]: 2.0 scrub ok 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[39470]: 6.3 scrub starts 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[39470]: pgmap v1138: 150 pgs: 1 active+remapped, 3 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:20.952 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[39470]: 6.3 scrub ok 2024-09-16T10:34:20.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[39470]: 2.0 scrub starts 2024-09-16T10:34:20.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:20 smithi063 ceph-mon[39470]: 2.0 scrub ok 2024-09-16T10:34:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T10:34:21.864 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: 4.1d scrub starts 2024-09-16T10:34:21.864 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: 4.1d scrub ok 2024-09-16T10:34:21.864 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: osdmap e603: 8 total, 8 up, 8 in 2024-09-16T10:34:21.864 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: 4.b scrub starts 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: 4.b scrub ok 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1522484159' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: from='client.46865 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[29993]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: 4.1d scrub starts 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: 4.1d scrub ok 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: osdmap e603: 8 total, 8 up, 8 in 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: 4.b scrub starts 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: 4.b scrub ok 2024-09-16T10:34:21.865 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1522484159' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:21.866 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: from='client.46865 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:21.866 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:21 smithi063 ceph-mon[39470]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: 4.1d scrub starts 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: 4.1d scrub ok 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: osdmap e603: 8 total, 8 up, 8 in 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: 4.b scrub starts 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: 4.b scrub ok 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1522484159' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: from='client.46865 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-16T10:34:22.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:21 smithi165 ceph-mon[32696]: from='client.46865 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-16T10:34:22.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T10:34:22.465 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:22.834 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[29993]: 1.f scrub starts 2024-09-16T10:34:22.834 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[29993]: 1.f scrub ok 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[29993]: osdmap e604: 8 total, 8 up, 8 in 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[29993]: pgmap v1141: 150 pgs: 1 active+clean+remapped, 1 active+remapped, 2 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[39470]: 1.f scrub starts 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[39470]: 1.f scrub ok 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[39470]: osdmap e604: 8 total, 8 up, 8 in 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[39470]: pgmap v1141: 150 pgs: 1 active+clean+remapped, 1 active+remapped, 2 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:22.835 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:22 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:22 smithi165 ceph-mon[32696]: 1.f scrub starts 2024-09-16T10:34:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:22 smithi165 ceph-mon[32696]: 1.f scrub ok 2024-09-16T10:34:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:22 smithi165 ceph-mon[32696]: osdmap e604: 8 total, 8 up, 8 in 2024-09-16T10:34:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:22 smithi165 ceph-mon[32696]: pgmap v1141: 150 pgs: 1 active+clean+remapped, 1 active+remapped, 2 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:23.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:22 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:23.563 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[29993]: 4.c scrub starts 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[29993]: 4.c scrub ok 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[29993]: osdmap e605: 8 total, 8 up, 8 in 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2534698534' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[39470]: 4.c scrub starts 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[39470]: 4.c scrub ok 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[39470]: osdmap e605: 8 total, 8 up, 8 in 2024-09-16T10:34:23.931 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:23 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2534698534' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:23 smithi165 ceph-mon[32696]: 4.c scrub starts 2024-09-16T10:34:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:23 smithi165 ceph-mon[32696]: 4.c scrub ok 2024-09-16T10:34:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:23 smithi165 ceph-mon[32696]: osdmap e605: 8 total, 8 up, 8 in 2024-09-16T10:34:24.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:23 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2534698534' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:24.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-16T10:34:24.618 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: 2.1 scrub starts 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: 2.1 scrub ok 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: 5.15 scrub starts 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: 5.15 scrub ok 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: pgmap v1143: 150 pgs: 1 active+remapped, 2 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/2534698534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: osdmap e606: 8 total, 8 up, 8 in 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: 4.4 scrub starts 2024-09-16T10:34:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: 4.4 scrub ok 2024-09-16T10:34:24.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:24 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:24.892 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: 2.1 scrub starts 2024-09-16T10:34:24.892 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: 2.1 scrub ok 2024-09-16T10:34:24.892 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: 5.15 scrub starts 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: 5.15 scrub ok 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: pgmap v1143: 150 pgs: 1 active+remapped, 2 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/2534698534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: osdmap e606: 8 total, 8 up, 8 in 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: 4.4 scrub starts 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: 4.4 scrub ok 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: 2.1 scrub starts 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: 2.1 scrub ok 2024-09-16T10:34:24.893 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: 5.15 scrub starts 2024-09-16T10:34:24.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: 5.15 scrub ok 2024-09-16T10:34:24.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: pgmap v1143: 150 pgs: 1 active+remapped, 2 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:24.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/2534698534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:34:24.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: osdmap e606: 8 total, 8 up, 8 in 2024-09-16T10:34:24.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: 4.4 scrub starts 2024-09-16T10:34:24.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: 4.4 scrub ok 2024-09-16T10:34:24.894 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:24 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:25.576 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 5.a scrub starts 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 5.a scrub ok 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 4.8 scrub starts 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 4.8 scrub ok 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 5.1 scrub starts 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 5.1 scrub ok 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 5.12 scrub starts 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: 5.12 scrub ok 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4230986015' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:25.939 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: from='client.46877 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 5.a scrub starts 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 5.a scrub ok 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 4.8 scrub starts 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 4.8 scrub ok 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 5.1 scrub starts 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 5.1 scrub ok 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 5.12 scrub starts 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: 5.12 scrub ok 2024-09-16T10:34:25.940 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4230986015' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:25.941 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: from='client.46877 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:25.941 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:34:25.941 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:25 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 5.a scrub starts 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 5.a scrub ok 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 4.8 scrub starts 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 4.8 scrub ok 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 5.1 scrub starts 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 5.1 scrub ok 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 5.12 scrub starts 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: 5.12 scrub ok 2024-09-16T10:34:26.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4230986015' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:26.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: from='client.46877 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-16T10:34:26.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:34:26.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:25 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-16T10:34:26.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T10:34:26.634 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:26.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: 3.16 scrub starts 2024-09-16T10:34:26.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: 3.16 scrub ok 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: pgmap v1145: 150 pgs: 1 active+remapped+backfill_toofull, 149 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: from='client.46877 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: osdmap e607: 8 total, 8 up, 8 in 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: 6.10 scrub starts 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: 6.10 scrub ok 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: 1.b scrub starts 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[29993]: 1.b scrub ok 2024-09-16T10:34:26.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: 3.16 scrub starts 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: 3.16 scrub ok 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: pgmap v1145: 150 pgs: 1 active+remapped+backfill_toofull, 149 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: from='client.46877 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: osdmap e607: 8 total, 8 up, 8 in 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: 6.10 scrub starts 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: 6.10 scrub ok 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: 1.b scrub starts 2024-09-16T10:34:26.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:26 smithi063 ceph-mon[39470]: 1.b scrub ok 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: 3.16 scrub starts 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: 3.16 scrub ok 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: pgmap v1145: 150 pgs: 1 active+remapped+backfill_toofull, 149 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: from='client.46877 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: osdmap e607: 8 total, 8 up, 8 in 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: 6.10 scrub starts 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: 6.10 scrub ok 2024-09-16T10:34:27.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: 1.b scrub starts 2024-09-16T10:34:27.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:26 smithi165 ceph-mon[32696]: 1.b scrub ok 2024-09-16T10:34:27.451 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:34:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:34:27.595 INFO:teuthology.orchestra.run.smithi063.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T10:34:27.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[29993]: 3.c scrub starts 2024-09-16T10:34:27.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[29993]: 3.c scrub ok 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[29993]: osdmap e608: 8 total, 8 up, 8 in 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1343760864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[29993]: from='client.46883 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[39470]: 3.c scrub starts 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[39470]: 3.c scrub ok 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[39470]: osdmap e608: 8 total, 8 up, 8 in 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1343760864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:27.942 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:27 smithi063 ceph-mon[39470]: from='client.46883 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:28.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:27 smithi165 ceph-mon[32696]: 3.c scrub starts 2024-09-16T10:34:28.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:27 smithi165 ceph-mon[32696]: 3.c scrub ok 2024-09-16T10:34:28.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:27 smithi165 ceph-mon[32696]: osdmap e608: 8 total, 8 up, 8 in 2024-09-16T10:34:28.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:27 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1343760864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:28.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:27 smithi165 ceph-mon[32696]: from='client.46883 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:28.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-16T10:34:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: 6.b scrub starts 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: 6.b scrub ok 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: 1.6 scrub starts 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: 1.6 scrub ok 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: pgmap v1148: 150 pgs: 1 active+remapped+backfill_toofull, 149 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: from='client.46883 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: osdmap e609: 8 total, 8 up, 8 in 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: 5.c scrub starts 2024-09-16T10:34:28.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[39470]: 5.c scrub ok 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: 6.b scrub starts 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: 6.b scrub ok 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: 1.6 scrub starts 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: 1.6 scrub ok 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: pgmap v1148: 150 pgs: 1 active+remapped+backfill_toofull, 149 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: from='client.46883 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: osdmap e609: 8 total, 8 up, 8 in 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: 5.c scrub starts 2024-09-16T10:34:28.849 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:28 smithi063 ceph-mon[29993]: 5.c scrub ok 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: 6.b scrub starts 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: 6.b scrub ok 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: 1.6 scrub starts 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: 1.6 scrub ok 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: pgmap v1148: 150 pgs: 1 active+remapped+backfill_toofull, 149 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: from='client.46883 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: osdmap e609: 8 total, 8 up, 8 in 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: 5.c scrub starts 2024-09-16T10:34:29.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:28 smithi165 ceph-mon[32696]: 5.c scrub ok 2024-09-16T10:34:29.602 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:34:29.607 INFO:teuthology.orchestra.run.smithi063.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T10:34:29.951 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: 4.1 scrub starts 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: 4.1 scrub ok 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: 1.a scrub starts 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: 1.a scrub ok 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: osdmap e610: 8 total, 8 up, 8 in 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/432778447' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: from='client.46889 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: 4.f scrub starts 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: 4.f scrub ok 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:34:29.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: 4.1 scrub starts 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: 4.1 scrub ok 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: 1.a scrub starts 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: 1.a scrub ok 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: osdmap e610: 8 total, 8 up, 8 in 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/432778447' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: from='client.46889 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: 4.f scrub starts 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: 4.f scrub ok 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:34:29.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:29 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:34:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: 4.1 scrub starts 2024-09-16T10:34:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: 4.1 scrub ok 2024-09-16T10:34:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: 1.a scrub starts 2024-09-16T10:34:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: 1.a scrub ok 2024-09-16T10:34:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: osdmap e610: 8 total, 8 up, 8 in 2024-09-16T10:34:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/432778447' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:30.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: from='client.46889 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-16T10:34:30.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: 4.f scrub starts 2024-09-16T10:34:30.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: 4.f scrub ok 2024-09-16T10:34:30.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:34:30.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:29 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-16T10:34:30.392 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T10:34:30.392 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T10:34:30.392 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T10:34:30.392 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T10:34:30.393 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-16T10:34:30.393 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T10:34:30.668 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:30.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:30.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[29993]: pgmap v1151: 149 pgs: 1 active+remapped+backfill_toofull, 148 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:30.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[29993]: from='client.46889 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:34:30.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-16T10:34:30.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[29993]: osdmap e611: 8 total, 8 up, 8 in 2024-09-16T10:34:30.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:30.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[39470]: pgmap v1151: 149 pgs: 1 active+remapped+backfill_toofull, 148 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:30.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[39470]: from='client.46889 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:34:30.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-16T10:34:30.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:30 smithi063 ceph-mon[39470]: osdmap e611: 8 total, 8 up, 8 in 2024-09-16T10:34:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:30 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:30 smithi165 ceph-mon[32696]: pgmap v1151: 149 pgs: 1 active+remapped+backfill_toofull, 148 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:30 smithi165 ceph-mon[32696]: from='client.46889 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-16T10:34:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:30 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-16T10:34:31.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:30 smithi165 ceph-mon[32696]: osdmap e611: 8 total, 8 up, 8 in 2024-09-16T10:34:31.353 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:34:31.353 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":613,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","created":"2024-09-16T10:08:21.081928+0000","modified":"2024-09-16T10:34:31.145600+0000","last_up_change":"2024-09-16T10:28:27.035727+0000","last_in_change":"2024-09-16T10:33:50.175690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:11:23.055843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":17,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":17,"last_pg_merge_meta":{"source_pgid":"1.11","ready_epoch":613,"last_epoch_started":612,"last_epoch_clean":612,"source_version":"73'79","target_version":"73'79"},"last_change":"613","last_force_op_resend":"0","last_force_op_resend_prenautilus":"613","last_force_op_resend_preluminous":"611","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.3499999046325684,"score_stable":2.3499999046325684,"optimal_score":1,"raw_score_acting":2.3499999046325684,"raw_score_stable":2.3499999046325684,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-16T10:13:34.734896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-16T10:13:35.550974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-16T10:13:37.166765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-16T10:13:39.104673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-16T10:13:41.175068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c4234e56-a2f2-467f-9616-190a52181d42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":609,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":2031112888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":2031112888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":2031112888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":2031112888}]},"public_addr":"172.21.15.63:6801/2031112888","cluster_addr":"172.21.15.63:6802/2031112888","heartbeat_back_addr":"172.21.15.63:6804/2031112888","heartbeat_front_addr":"172.21.15.63:6803/2031112888","state":["exists","up"]},{"osd":1,"uuid":"6fe9a4c9-c351-4223-8aab-6405152696c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":603,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":3717451957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":3717451957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":3717451957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":3717451957}]},"public_addr":"172.21.15.63:6805/3717451957","cluster_addr":"172.21.15.63:6806/3717451957","heartbeat_back_addr":"172.21.15.63:6808/3717451957","heartbeat_front_addr":"172.21.15.63:6807/3717451957","state":["exists","up"]},{"osd":2,"uuid":"4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":597,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":196353360}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":196353360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6812","nonce":196353360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":196353360}]},"public_addr":"172.21.15.63:6809/196353360","cluster_addr":"172.21.15.63:6810/196353360","heartbeat_back_addr":"172.21.15.63:6812/196353360","heartbeat_front_addr":"172.21.15.63:6811/196353360","state":["exists","up"]},{"osd":3,"uuid":"355833dd-7cc5-4f59-b036-ed8939892ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":596,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6813","nonce":249840957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6814","nonce":249840957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6816","nonce":249840957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6815","nonce":249840957}]},"public_addr":"172.21.15.63:6813/249840957","cluster_addr":"172.21.15.63:6814/249840957","heartbeat_back_addr":"172.21.15.63:6816/249840957","heartbeat_front_addr":"172.21.15.63:6815/249840957","state":["exists","up"]},{"osd":4,"uuid":"da2fbe06-9462-44ec-9ffb-c5563ccee4b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":600,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":2373552307}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":2373552307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":2373552307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":2373552307}]},"public_addr":"172.21.15.165:6800/2373552307","cluster_addr":"172.21.15.165:6801/2373552307","heartbeat_back_addr":"172.21.15.165:6803/2373552307","heartbeat_front_addr":"172.21.15.165:6802/2373552307","state":["exists","up"]},{"osd":5,"uuid":"b866ad81-ae05-4c8e-9e12-822440fcb713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":611,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":783450148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":783450148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":783450148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":783450148}]},"public_addr":"172.21.15.165:6804/783450148","cluster_addr":"172.21.15.165:6805/783450148","heartbeat_back_addr":"172.21.15.165:6807/783450148","heartbeat_front_addr":"172.21.15.165:6806/783450148","state":["exists","up"]},{"osd":6,"uuid":"ce05cb1a-e5c6-4ea9-a2c0-0039c874affa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":604,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":424252741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":424252741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":424252741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":424252741}]},"public_addr":"172.21.15.165:6808/424252741","cluster_addr":"172.21.15.165:6809/424252741","heartbeat_back_addr":"172.21.15.165:6811/424252741","heartbeat_front_addr":"172.21.15.165:6810/424252741","state":["exists","up"]},{"osd":7,"uuid":"1b464a8a-ad32-47ab-82fb-3097077ebc48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":386,"up_from":441,"up_thru":611,"down_at":387,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1921603083}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1921603083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1921603083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1921603083}]},"public_addr":"172.21.15.165:6812/1921603083","cluster_addr":"172.21.15.165:6813/1921603083","heartbeat_back_addr":"172.21.15.165:6815/1921603083","heartbeat_front_addr":"172.21.15.165:6814/1921603083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:56.960376+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:08.182903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:15.265533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:30.959516+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:40.664776+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:18.487352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:37.138994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:18.615649+0000","dead_epoch":386}],"pg_upmap":[{"pgid":"1.5","osds":[6,3,5]},{"pgid":"4.0","osds":[4,6,2]}],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":4,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":0}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"5.1","mappings":[{"from":4,"to":6}]},{"pgid":"5.1d","mappings":[{"from":0,"to":3}]},{"pgid":"6.0","mappings":[{"from":2,"to":0}]},{"pgid":"6.19","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.c","osds":[3,5,6]}],"primary_temp":[],"blocklist":{"172.21.15.63:0/56975596":"2024-09-17T10:14:06.991973+0000","172.21.15.63:0/455268617":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/3928610681":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/955884223":"2024-09-17T10:14:06.991973+0000","172.21.15.63:0/1387667925":"2024-09-17T10:08:51.022048+0000","172.21.15.63:6800/670090510":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/1037308168":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/2411639026":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/812143427":"2024-09-17T10:14:06.991973+0000","172.21.15.63:6800/274451570":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1121825432":"2024-09-17T10:14:06.991973+0000","172.21.15.63:6800/2945123439":"2024-09-17T10:14:06.991973+0000","172.21.15.63:0/2625337613":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/2993184887":"2024-09-17T10:14:06.991973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: 5.b scrub starts 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: 5.b scrub ok 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: 3.10 scrub starts 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: 3.5 scrub starts 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: 3.10 scrub ok 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: 3.5 scrub ok 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: osdmap e612: 8 total, 8 up, 8 in 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: osdmap e613: 8 total, 8 up, 8 in 2024-09-16T10:34:31.952 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1835948057' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: 5.b scrub starts 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: 5.b scrub ok 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: 3.10 scrub starts 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: 3.5 scrub starts 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: 3.10 scrub ok 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: 3.5 scrub ok 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: osdmap e612: 8 total, 8 up, 8 in 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:31.953 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: osdmap e613: 8 total, 8 up, 8 in 2024-09-16T10:34:31.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1835948057' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:34:31.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:34:31.954 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:31 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: 5.b scrub starts 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: 5.b scrub ok 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: 3.10 scrub starts 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: 3.5 scrub starts 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: 3.10 scrub ok 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: 3.5 scrub ok 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: osdmap e612: 8 total, 8 up, 8 in 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:32.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: osdmap e613: 8 total, 8 up, 8 in 2024-09-16T10:34:32.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1835948057' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:34:32.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:34:32.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:31 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-16T10:34:32.195 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-16T10:34:32.196 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T10:34:32.459 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:32.859 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: 5.f scrub starts 2024-09-16T10:34:32.859 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: 5.f scrub ok 2024-09-16T10:34:32.859 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: 6.c scrub starts 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: 6.c scrub ok 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: pgmap v1155: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%); 0 B/s, 0 objects/s recovering 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: 6.0 scrub starts 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: 6.0 scrub ok 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[29993]: osdmap e614: 8 total, 8 up, 8 in 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: 5.f scrub starts 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: 5.f scrub ok 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: 6.c scrub starts 2024-09-16T10:34:32.860 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: 6.c scrub ok 2024-09-16T10:34:32.861 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: pgmap v1155: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%); 0 B/s, 0 objects/s recovering 2024-09-16T10:34:32.861 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: 6.0 scrub starts 2024-09-16T10:34:32.861 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: 6.0 scrub ok 2024-09-16T10:34:32.861 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-16T10:34:32.861 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:32 smithi063 ceph-mon[39470]: osdmap e614: 8 total, 8 up, 8 in 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: 5.f scrub starts 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: 5.f scrub ok 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: 6.c scrub starts 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: 6.c scrub ok 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: pgmap v1155: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%); 0 B/s, 0 objects/s recovering 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: 6.0 scrub starts 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: 6.0 scrub ok 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-16T10:34:33.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:32 smithi165 ceph-mon[32696]: osdmap e614: 8 total, 8 up, 8 in 2024-09-16T10:34:33.136 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:34:33.137 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":614,"fsid":"7cb51a9a-7413-11ef-bceb-c7b262605968","created":"2024-09-16T10:08:21.081928+0000","modified":"2024-09-16T10:34:32.147561+0000","last_up_change":"2024-09-16T10:28:27.035727+0000","last_in_change":"2024-09-16T10:33:50.175690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:11:23.055843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":17,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"1.11","ready_epoch":613,"last_epoch_started":612,"last_epoch_clean":612,"source_version":"73'79","target_version":"73'79"},"last_change":"614","last_force_op_resend":"0","last_force_op_resend_prenautilus":"614","last_force_op_resend_preluminous":"614","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.3499999046325684,"score_stable":2.3499999046325684,"optimal_score":1,"raw_score_acting":2.3499999046325684,"raw_score_stable":2.3499999046325684,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-16T10:13:34.734896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-16T10:13:35.550974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-16T10:13:37.166765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-16T10:13:39.104673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-16T10:13:41.175068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c4234e56-a2f2-467f-9616-190a52181d42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":609,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":2031112888}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":2031112888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":2031112888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":2031112888}]},"public_addr":"172.21.15.63:6801/2031112888","cluster_addr":"172.21.15.63:6802/2031112888","heartbeat_back_addr":"172.21.15.63:6804/2031112888","heartbeat_front_addr":"172.21.15.63:6803/2031112888","state":["exists","up"]},{"osd":1,"uuid":"6fe9a4c9-c351-4223-8aab-6405152696c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":603,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":3717451957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":3717451957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":3717451957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":3717451957}]},"public_addr":"172.21.15.63:6805/3717451957","cluster_addr":"172.21.15.63:6806/3717451957","heartbeat_back_addr":"172.21.15.63:6808/3717451957","heartbeat_front_addr":"172.21.15.63:6807/3717451957","state":["exists","up"]},{"osd":2,"uuid":"4e3e79da-a17e-4bdf-bc35-fba3a4ebe8d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":597,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":196353360}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":196353360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6812","nonce":196353360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":196353360}]},"public_addr":"172.21.15.63:6809/196353360","cluster_addr":"172.21.15.63:6810/196353360","heartbeat_back_addr":"172.21.15.63:6812/196353360","heartbeat_front_addr":"172.21.15.63:6811/196353360","state":["exists","up"]},{"osd":3,"uuid":"355833dd-7cc5-4f59-b036-ed8939892ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":596,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6813","nonce":249840957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6814","nonce":249840957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6816","nonce":249840957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6815","nonce":249840957}]},"public_addr":"172.21.15.63:6813/249840957","cluster_addr":"172.21.15.63:6814/249840957","heartbeat_back_addr":"172.21.15.63:6816/249840957","heartbeat_front_addr":"172.21.15.63:6815/249840957","state":["exists","up"]},{"osd":4,"uuid":"da2fbe06-9462-44ec-9ffb-c5563ccee4b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":600,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":2373552307}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":2373552307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":2373552307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":2373552307}]},"public_addr":"172.21.15.165:6800/2373552307","cluster_addr":"172.21.15.165:6801/2373552307","heartbeat_back_addr":"172.21.15.165:6803/2373552307","heartbeat_front_addr":"172.21.15.165:6802/2373552307","state":["exists","up"]},{"osd":5,"uuid":"b866ad81-ae05-4c8e-9e12-822440fcb713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":613,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":783450148}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":783450148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":783450148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":783450148}]},"public_addr":"172.21.15.165:6804/783450148","cluster_addr":"172.21.15.165:6805/783450148","heartbeat_back_addr":"172.21.15.165:6807/783450148","heartbeat_front_addr":"172.21.15.165:6806/783450148","state":["exists","up"]},{"osd":6,"uuid":"ce05cb1a-e5c6-4ea9-a2c0-0039c874affa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":604,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":424252741}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":424252741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":424252741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":424252741}]},"public_addr":"172.21.15.165:6808/424252741","cluster_addr":"172.21.15.165:6809/424252741","heartbeat_back_addr":"172.21.15.165:6811/424252741","heartbeat_front_addr":"172.21.15.165:6810/424252741","state":["exists","up"]},{"osd":7,"uuid":"1b464a8a-ad32-47ab-82fb-3097077ebc48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":386,"up_from":441,"up_thru":611,"down_at":387,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1921603083}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1921603083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1921603083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1921603083}]},"public_addr":"172.21.15.165:6812/1921603083","cluster_addr":"172.21.15.165:6813/1921603083","heartbeat_back_addr":"172.21.15.165:6815/1921603083","heartbeat_front_addr":"172.21.15.165:6814/1921603083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:56.960376+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:08.182903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:15.265533+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:30.959516+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:40.664776+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:18.487352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:33:37.138994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:34:18.615649+0000","dead_epoch":386}],"pg_upmap":[{"pgid":"1.5","osds":[6,3,5]},{"pgid":"4.0","osds":[4,6,2]}],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":4,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":0}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"5.1","mappings":[{"from":4,"to":6}]},{"pgid":"5.1d","mappings":[{"from":0,"to":3}]},{"pgid":"6.0","mappings":[{"from":2,"to":0}]},{"pgid":"6.19","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.c","osds":[3,5,6]}],"primary_temp":[],"blocklist":{"172.21.15.63:0/56975596":"2024-09-17T10:14:06.991973+0000","172.21.15.63:0/455268617":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/3928610681":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/955884223":"2024-09-17T10:14:06.991973+0000","172.21.15.63:0/1387667925":"2024-09-17T10:08:51.022048+0000","172.21.15.63:6800/670090510":"2024-09-17T10:08:51.022048+0000","172.21.15.63:0/1037308168":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/2411639026":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/812143427":"2024-09-17T10:14:06.991973+0000","172.21.15.63:6800/274451570":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/1121825432":"2024-09-17T10:14:06.991973+0000","172.21.15.63:6800/2945123439":"2024-09-17T10:14:06.991973+0000","172.21.15.63:0/2625337613":"2024-09-17T10:09:14.602270+0000","172.21.15.63:0/2993184887":"2024-09-17T10:14:06.991973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[29993]: 5.e scrub starts 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[29993]: 5.e scrub ok 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[29993]: 5.1e scrub starts 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[29993]: 5.1e scrub ok 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/312570743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[29993]: osdmap e615: 8 total, 8 up, 8 in 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:34:33.848 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[39470]: 5.e scrub starts 2024-09-16T10:34:33.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[39470]: 5.e scrub ok 2024-09-16T10:34:33.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[39470]: 5.1e scrub starts 2024-09-16T10:34:33.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[39470]: 5.1e scrub ok 2024-09-16T10:34:33.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/312570743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:34:33.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[39470]: osdmap e615: 8 total, 8 up, 8 in 2024-09-16T10:34:33.849 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:33 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:34:33.875 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-16T10:34:33.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-16T10:34:33.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-16T10:34:33.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-16T10:34:33.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-16T10:34:33.877 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-16T10:34:33.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-16T10:34:33.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-16T10:34:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:33 smithi165 ceph-mon[32696]: 5.e scrub starts 2024-09-16T10:34:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:33 smithi165 ceph-mon[32696]: 5.e scrub ok 2024-09-16T10:34:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:33 smithi165 ceph-mon[32696]: 5.1e scrub starts 2024-09-16T10:34:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:33 smithi165 ceph-mon[32696]: 5.1e scrub ok 2024-09-16T10:34:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:33 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/312570743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T10:34:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:33 smithi165 ceph-mon[32696]: osdmap e615: 8 total, 8 up, 8 in 2024-09-16T10:34:34.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:33 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:34:34.414 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.429 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.439 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.489 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.496 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 1.2 scrub starts 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 1.2 scrub ok 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 4.15 scrub starts 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 4.15 scrub ok 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 3.a scrub starts 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 3.a scrub ok 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: pgmap v1158: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%); 0 B/s, 0 objects/s recovering 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 5.8 scrub starts 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: 5.8 scrub ok 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: osdmap e616: 8 total, 8 up, 8 in 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:34:34.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 1.2 scrub starts 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 1.2 scrub ok 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 4.15 scrub starts 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 4.15 scrub ok 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 3.a scrub starts 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 3.a scrub ok 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: pgmap v1158: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%); 0 B/s, 0 objects/s recovering 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 5.8 scrub starts 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: 5.8 scrub ok 2024-09-16T10:34:34.845 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: osdmap e616: 8 total, 8 up, 8 in 2024-09-16T10:34:34.846 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:34:34.846 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:34:34.846 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:34 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 1.2 scrub starts 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 1.2 scrub ok 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 4.15 scrub starts 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 4.15 scrub ok 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 3.a scrub starts 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 3.a scrub ok 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: pgmap v1158: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%); 0 B/s, 0 objects/s recovering 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 5.8 scrub starts 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: 5.8 scrub ok 2024-09-16T10:34:35.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: osdmap e616: 8 total, 8 up, 8 in 2024-09-16T10:34:35.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:34:35.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:34:35.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:34 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:35.820 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[29993]: 3.8 scrub starts 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[29993]: 3.8 scrub ok 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[29993]: 5.5 scrub starts 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[29993]: 5.5 scrub ok 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[29993]: osdmap e617: 8 total, 8 up, 8 in 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[39470]: 3.8 scrub starts 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[39470]: 3.8 scrub ok 2024-09-16T10:34:35.821 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[39470]: 5.5 scrub starts 2024-09-16T10:34:35.822 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[39470]: 5.5 scrub ok 2024-09-16T10:34:35.822 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[39470]: osdmap e617: 8 total, 8 up, 8 in 2024-09-16T10:34:35.822 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:34:35.822 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:35 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:34:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:35 smithi165 ceph-mon[32696]: 3.8 scrub starts 2024-09-16T10:34:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:35 smithi165 ceph-mon[32696]: 3.8 scrub ok 2024-09-16T10:34:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:35 smithi165 ceph-mon[32696]: 5.5 scrub starts 2024-09-16T10:34:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:35 smithi165 ceph-mon[32696]: 5.5 scrub ok 2024-09-16T10:34:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:35 smithi165 ceph-mon[32696]: osdmap e617: 8 total, 8 up, 8 in 2024-09-16T10:34:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:35 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:34:36.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:35 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"} : dispatch 2024-09-16T10:34:36.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 4.1b scrub starts 2024-09-16T10:34:36.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 4.1b scrub ok 2024-09-16T10:34:36.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 1.8 scrub starts 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 1.8 scrub ok 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 4.1f scrub starts 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 4.1f scrub ok 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: pgmap v1161: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 3.1b scrub starts 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: 3.1b scrub ok 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[29993]: osdmap e618: 8 total, 8 up, 8 in 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 4.1b scrub starts 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 4.1b scrub ok 2024-09-16T10:34:36.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 1.8 scrub starts 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 1.8 scrub ok 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 4.1f scrub starts 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 4.1f scrub ok 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: pgmap v1161: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 3.1b scrub starts 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: 3.1b scrub ok 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-16T10:34:36.703 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:36 smithi063 ceph-mon[39470]: osdmap e618: 8 total, 8 up, 8 in 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 4.1b scrub starts 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 4.1b scrub ok 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 1.8 scrub starts 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 1.8 scrub ok 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 4.1f scrub starts 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 4.1f scrub ok 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: pgmap v1161: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 3.1b scrub starts 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: 3.1b scrub ok 2024-09-16T10:34:37.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-16T10:34:37.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:36 smithi165 ceph-mon[32696]: osdmap e618: 8 total, 8 up, 8 in 2024-09-16T10:34:37.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:34:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:34:37.366 INFO:teuthology.orchestra.run.smithi063.stdout:158913790218 2024-09-16T10:34:37.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-16T10:34:37.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[29993]: 6.1e scrub starts 2024-09-16T10:34:37.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[29993]: 6.1e scrub ok 2024-09-16T10:34:37.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[29993]: 4.17 scrub starts 2024-09-16T10:34:37.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[29993]: 4.17 scrub ok 2024-09-16T10:34:37.919 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[29993]: osdmap e619: 8 total, 8 up, 8 in 2024-09-16T10:34:37.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[39470]: 6.1e scrub starts 2024-09-16T10:34:37.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[39470]: 6.1e scrub ok 2024-09-16T10:34:37.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[39470]: 4.17 scrub starts 2024-09-16T10:34:37.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[39470]: 4.17 scrub ok 2024-09-16T10:34:37.920 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:37 smithi063 ceph-mon[39470]: osdmap e619: 8 total, 8 up, 8 in 2024-09-16T10:34:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:37 smithi165 ceph-mon[32696]: 6.1e scrub starts 2024-09-16T10:34:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:37 smithi165 ceph-mon[32696]: 6.1e scrub ok 2024-09-16T10:34:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:37 smithi165 ceph-mon[32696]: 4.17 scrub starts 2024-09-16T10:34:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:37 smithi165 ceph-mon[32696]: 4.17 scrub ok 2024-09-16T10:34:38.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:37 smithi165 ceph-mon[32696]: osdmap e619: 8 total, 8 up, 8 in 2024-09-16T10:34:38.910 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[29993]: 4.13 scrub starts 2024-09-16T10:34:38.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[29993]: 4.13 scrub ok 2024-09-16T10:34:38.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[29993]: pgmap v1164: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:38.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[29993]: osdmap e620: 8 total, 8 up, 8 in 2024-09-16T10:34:38.911 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[39470]: 4.13 scrub starts 2024-09-16T10:34:38.911 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[39470]: 4.13 scrub ok 2024-09-16T10:34:38.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[39470]: pgmap v1164: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:38.912 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:38 smithi063 ceph-mon[39470]: osdmap e620: 8 total, 8 up, 8 in 2024-09-16T10:34:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:38 smithi165 ceph-mon[32696]: 4.13 scrub starts 2024-09-16T10:34:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:38 smithi165 ceph-mon[32696]: 4.13 scrub ok 2024-09-16T10:34:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:38 smithi165 ceph-mon[32696]: pgmap v1164: 147 pgs: 1 active+remapped+backfill_toofull, 146 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:39.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:38 smithi165 ceph-mon[32696]: osdmap e620: 8 total, 8 up, 8 in 2024-09-16T10:34:39.577 INFO:teuthology.orchestra.run.smithi063.stdout:34359738659 2024-09-16T10:34:39.578 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-16T10:34:39.621 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:34:39.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: 3.1a scrub starts 2024-09-16T10:34:39.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: 3.1a scrub ok 2024-09-16T10:34:39.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: 6.18 scrub starts 2024-09-16T10:34:39.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: 6.18 scrub ok 2024-09-16T10:34:39.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: osdmap e621: 8 total, 8 up, 8 in 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: 3.1a scrub starts 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: 3.1a scrub ok 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: 6.18 scrub starts 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: 6.18 scrub ok 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: osdmap e621: 8 total, 8 up, 8 in 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:34:39.877 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:39 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: 3.1a scrub starts 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: 3.1a scrub ok 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: 6.18 scrub starts 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: 6.18 scrub ok 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: osdmap e621: 8 total, 8 up, 8 in 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:34:40.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:39 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"} : dispatch 2024-09-16T10:34:40.865 INFO:teuthology.orchestra.run.smithi063.stdout:77309411611 2024-09-16T10:34:40.865 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: 1.0 scrub starts 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: 1.0 scrub ok 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: 5.7 scrub starts 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: 5.7 scrub ok 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: pgmap v1167: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[29993]: osdmap e622: 8 total, 8 up, 8 in 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: 1.0 scrub starts 2024-09-16T10:34:40.890 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: 1.0 scrub ok 2024-09-16T10:34:40.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:40.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: 5.7 scrub starts 2024-09-16T10:34:40.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: 5.7 scrub ok 2024-09-16T10:34:40.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: pgmap v1167: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:40.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-16T10:34:40.891 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:40 smithi063 ceph-mon[39470]: osdmap e622: 8 total, 8 up, 8 in 2024-09-16T10:34:41.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: 1.0 scrub starts 2024-09-16T10:34:41.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: 1.0 scrub ok 2024-09-16T10:34:41.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:41.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: 5.7 scrub starts 2024-09-16T10:34:41.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: 5.7 scrub ok 2024-09-16T10:34:41.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: pgmap v1167: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:41.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-16T10:34:41.053 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:40 smithi165 ceph-mon[32696]: osdmap e622: 8 total, 8 up, 8 in 2024-09-16T10:34:41.557 INFO:teuthology.orchestra.run.smithi063.stdout:55834575133 2024-09-16T10:34:41.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-16T10:34:41.815 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[29993]: 3.f scrub starts 2024-09-16T10:34:41.815 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[29993]: 3.f scrub ok 2024-09-16T10:34:41.815 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[29993]: 3.7 scrub starts 2024-09-16T10:34:41.816 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[29993]: 3.7 scrub ok 2024-09-16T10:34:41.816 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[29993]: osdmap e623: 8 total, 8 up, 8 in 2024-09-16T10:34:41.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[39470]: 3.f scrub starts 2024-09-16T10:34:41.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[39470]: 3.f scrub ok 2024-09-16T10:34:41.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[39470]: 3.7 scrub starts 2024-09-16T10:34:41.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[39470]: 3.7 scrub ok 2024-09-16T10:34:41.816 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:41 smithi063 ceph-mon[39470]: osdmap e623: 8 total, 8 up, 8 in 2024-09-16T10:34:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:41 smithi165 ceph-mon[32696]: 3.f scrub starts 2024-09-16T10:34:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:41 smithi165 ceph-mon[32696]: 3.f scrub ok 2024-09-16T10:34:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:41 smithi165 ceph-mon[32696]: 3.7 scrub starts 2024-09-16T10:34:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:41 smithi165 ceph-mon[32696]: 3.7 scrub ok 2024-09-16T10:34:42.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:41 smithi165 ceph-mon[32696]: osdmap e623: 8 total, 8 up, 8 in 2024-09-16T10:34:42.217 INFO:teuthology.orchestra.run.smithi063.stdout:133143986448 2024-09-16T10:34:42.217 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-16T10:34:42.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[29993]: 3.1d scrub starts 2024-09-16T10:34:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[29993]: 3.1d scrub ok 2024-09-16T10:34:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[29993]: pgmap v1170: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:42.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[29993]: osdmap e624: 8 total, 8 up, 8 in 2024-09-16T10:34:42.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[39470]: 3.1d scrub starts 2024-09-16T10:34:42.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[39470]: 3.1d scrub ok 2024-09-16T10:34:42.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[39470]: pgmap v1170: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:42.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:42 smithi063 ceph-mon[39470]: osdmap e624: 8 total, 8 up, 8 in 2024-09-16T10:34:42.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:42 smithi165 ceph-mon[32696]: 3.1d scrub starts 2024-09-16T10:34:42.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:42 smithi165 ceph-mon[32696]: 3.1d scrub ok 2024-09-16T10:34:42.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:42 smithi165 ceph-mon[32696]: pgmap v1170: 146 pgs: 1 peering, 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:42.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:42 smithi165 ceph-mon[32696]: osdmap e624: 8 total, 8 up, 8 in 2024-09-16T10:34:43.253 INFO:teuthology.orchestra.run.smithi063.stdout:1894080577613 2024-09-16T10:34:43.253 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-16T10:34:43.781 INFO:teuthology.orchestra.run.smithi063.stdout:103079215380 2024-09-16T10:34:43.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: 6.16 scrub starts 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: 6.16 scrub ok 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: 6.a scrub starts 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: 6.a scrub ok 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: 3.b scrub starts 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: 3.b scrub ok 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:34:43.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: osdmap e625: 8 total, 8 up, 8 in 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: 6.16 scrub starts 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: 6.16 scrub ok 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: 6.a scrub starts 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: 6.a scrub ok 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: 3.b scrub starts 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: 3.b scrub ok 2024-09-16T10:34:43.881 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:34:43.882 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:34:43.882 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: osdmap e625: 8 total, 8 up, 8 in 2024-09-16T10:34:43.882 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.882 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.882 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.882 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:43 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: 6.16 scrub starts 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: 6.16 scrub ok 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: 6.a scrub starts 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: 6.a scrub ok 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: 3.b scrub starts 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: 3.b scrub ok 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:34:43.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:34:43.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: osdmap e625: 8 total, 8 up, 8 in 2024-09-16T10:34:43.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap", "pgid": "1.5"} : dispatch 2024-09-16T10:34:43.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:43 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"} : dispatch 2024-09-16T10:34:44.217 INFO:teuthology.orchestra.run.smithi063.stdout:193273528580 2024-09-16T10:34:44.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-16T10:34:44.386 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:44.386 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:44.386 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:44.387 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:44.393 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:44.421 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:44.424 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:44.828 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[29993]: pgmap v1173: 145 pgs: 1 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:44.828 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.5"}]': finished 2024-09-16T10:34:44.828 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"}]': finished 2024-09-16T10:34:44.828 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[29993]: osdmap e626: 8 total, 8 up, 8 in 2024-09-16T10:34:44.828 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[39470]: pgmap v1173: 145 pgs: 1 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:44.828 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.5"}]': finished 2024-09-16T10:34:44.828 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"}]': finished 2024-09-16T10:34:44.829 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:44 smithi063 ceph-mon[39470]: osdmap e626: 8 total, 8 up, 8 in 2024-09-16T10:34:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:44 smithi165 ceph-mon[32696]: pgmap v1173: 145 pgs: 1 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:44 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.5"}]': finished 2024-09-16T10:34:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:44 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.5"}]': finished 2024-09-16T10:34:45.052 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:44 smithi165 ceph-mon[32696]: osdmap e626: 8 total, 8 up, 8 in 2024-09-16T10:34:45.405 INFO:teuthology.orchestra.run.smithi063.stdout:34359738660 2024-09-16T10:34:45.616 INFO:teuthology.orchestra.run.smithi063.stdout:158913790220 2024-09-16T10:34:46.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: 3.2 scrub starts 2024-09-16T10:34:46.451 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: 3.2 scrub ok 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: osdmap e627: 8 total, 8 up, 8 in 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: 5.18 scrub starts 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: 5.18 scrub ok 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4099632015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: pgmap v1176: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3577579947' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: 3.2 scrub starts 2024-09-16T10:34:46.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: 3.2 scrub ok 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: osdmap e627: 8 total, 8 up, 8 in 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: 5.18 scrub starts 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: 5.18 scrub ok 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4099632015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: pgmap v1176: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:46.453 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:46 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3577579947' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: 3.2 scrub starts 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: 3.2 scrub ok 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: osdmap e627: 8 total, 8 up, 8 in 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: 5.18 scrub starts 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: 5.18 scrub ok 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4099632015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: pgmap v1176: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:46.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3577579947' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T10:34:46.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790218 got 158913790220 for osd.5 2024-09-16T10:34:46.888 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:46.966 INFO:teuthology.orchestra.run.smithi063.stdout:103079215382 2024-09-16T10:34:47.052 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=cleanup t=2024-09-16T10:34:46.752976084Z level=info msg="Completed cleanup jobs" duration=4.885948ms 2024-09-16T10:34:47.052 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=plugins.update.checker t=2024-09-16T10:34:46.899665382Z level=info msg="Update check succeeded" duration=63.595065ms 2024-09-16T10:34:47.052 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:34:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=grafana.update.checker t=2024-09-16T10:34:46.905862668Z level=info msg="Update check succeeded" duration=66.088535ms 2024-09-16T10:34:47.188 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:34:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:34:47.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[29993]: 3.18 scrub starts 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[29993]: 3.18 scrub ok 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[29993]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[29993]: osdmap e628: 8 total, 8 up, 8 in 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[29993]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3396120695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[39470]: 3.18 scrub starts 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[39470]: 3.18 scrub ok 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[39470]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:34:47.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[39470]: osdmap e628: 8 total, 8 up, 8 in 2024-09-16T10:34:47.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[39470]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:34:47.453 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:47 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3396120695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T10:34:47.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:47 smithi165 ceph-mon[32696]: 3.18 scrub starts 2024-09-16T10:34:47.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:47 smithi165 ceph-mon[32696]: 3.18 scrub ok 2024-09-16T10:34:47.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:47 smithi165 ceph-mon[32696]: 1.5 starting backfill to osd.2 from (0'0,0'0] MAX to 73'79 2024-09-16T10:34:47.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:47 smithi165 ceph-mon[32696]: osdmap e628: 8 total, 8 up, 8 in 2024-09-16T10:34:47.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:47 smithi165 ceph-mon[32696]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 73'79 2024-09-16T10:34:47.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:47 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3396120695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T10:34:47.815 INFO:teuthology.orchestra.run.smithi063.stdout:55834575135 2024-09-16T10:34:48.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738659 got 34359738660 for osd.0 2024-09-16T10:34:48.365 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:48.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[29993]: pgmap v1178: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[29993]: 4.3 scrub starts 2024-09-16T10:34:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[29993]: 4.3 scrub ok 2024-09-16T10:34:48.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/3417744895' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T10:34:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[39470]: pgmap v1178: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[39470]: 4.3 scrub starts 2024-09-16T10:34:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[39470]: 4.3 scrub ok 2024-09-16T10:34:48.452 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:48 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/3417744895' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T10:34:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:48 smithi165 ceph-mon[32696]: pgmap v1178: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 934 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-16T10:34:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:48 smithi165 ceph-mon[32696]: 4.3 scrub starts 2024-09-16T10:34:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:48 smithi165 ceph-mon[32696]: 4.3 scrub ok 2024-09-16T10:34:48.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:48 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/3417744895' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T10:34:49.132 INFO:teuthology.orchestra.run.smithi063.stdout:133143986449 2024-09-16T10:34:49.400 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[29993]: 6.f scrub starts 2024-09-16T10:34:49.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[29993]: 6.f scrub ok 2024-09-16T10:34:49.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[29993]: 3.12 scrub starts 2024-09-16T10:34:49.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[29993]: 3.12 scrub ok 2024-09-16T10:34:49.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/387099187' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T10:34:49.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[39470]: 6.f scrub starts 2024-09-16T10:34:49.401 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[39470]: 6.f scrub ok 2024-09-16T10:34:49.402 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[39470]: 3.12 scrub starts 2024-09-16T10:34:49.402 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[39470]: 3.12 scrub ok 2024-09-16T10:34:49.402 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:49 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/387099187' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T10:34:49.427 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215380 got 103079215382 for osd.3 2024-09-16T10:34:49.428 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:49 smithi165 ceph-mon[32696]: 6.f scrub starts 2024-09-16T10:34:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:49 smithi165 ceph-mon[32696]: 6.f scrub ok 2024-09-16T10:34:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:49 smithi165 ceph-mon[32696]: 3.12 scrub starts 2024-09-16T10:34:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:49 smithi165 ceph-mon[32696]: 3.12 scrub ok 2024-09-16T10:34:49.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:49 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/387099187' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T10:34:49.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:34:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:34:50.295 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[29993]: 4.18 scrub starts 2024-09-16T10:34:50.295 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[29993]: 4.18 scrub ok 2024-09-16T10:34:50.295 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[29993]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:50.295 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[29993]: pgmap v1179: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 6.2 KiB/s wr, 1 op/s; 1/597 objects misplaced (0.168%); 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:34:50.296 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[39470]: 4.18 scrub starts 2024-09-16T10:34:50.296 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[39470]: 4.18 scrub ok 2024-09-16T10:34:50.296 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[39470]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:50.296 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:50 smithi063 ceph-mon[39470]: pgmap v1179: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 6.2 KiB/s wr, 1 op/s; 1/597 objects misplaced (0.168%); 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:34:50.341 INFO:teuthology.orchestra.run.smithi063.stdout:1894080577614 2024-09-16T10:34:50.408 INFO:teuthology.orchestra.run.smithi063.stdout:77309411613 2024-09-16T10:34:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:50 smithi165 ceph-mon[32696]: 4.18 scrub starts 2024-09-16T10:34:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:50 smithi165 ceph-mon[32696]: 4.18 scrub ok 2024-09-16T10:34:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:50 smithi165 ceph-mon[32696]: from='client.44468 v1:172.21.15.165:0/3692230971' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T10:34:50.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:50 smithi165 ceph-mon[32696]: pgmap v1179: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 6.2 KiB/s wr, 1 op/s; 1/597 objects misplaced (0.168%); 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:34:51.254 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1894080577613 got 1894080577614 for osd.7 2024-09-16T10:34:51.254 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: 6.11 scrub starts 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: 6.11 scrub ok 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/237435780' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: 6.e scrub starts 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: 6.e scrub ok 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/852259769' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T10:34:51.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:51 smithi165 ceph-mon[32696]: osdmap e629: 8 total, 8 up, 8 in 2024-09-16T10:34:51.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: 6.11 scrub starts 2024-09-16T10:34:51.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: 6.11 scrub ok 2024-09-16T10:34:51.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:51.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/237435780' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T10:34:51.594 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: 6.e scrub starts 2024-09-16T10:34:51.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: 6.e scrub ok 2024-09-16T10:34:51.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/852259769' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T10:34:51.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[29993]: osdmap e629: 8 total, 8 up, 8 in 2024-09-16T10:34:51.595 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: 6.11 scrub starts 2024-09-16T10:34:51.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: 6.11 scrub ok 2024-09-16T10:34:51.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:51.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/237435780' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T10:34:51.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: 6.e scrub starts 2024-09-16T10:34:51.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: 6.e scrub ok 2024-09-16T10:34:51.596 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/852259769' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T10:34:51.597 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:51 smithi063 ceph-mon[39470]: osdmap e629: 8 total, 8 up, 8 in 2024-09-16T10:34:52.015 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575133 got 55834575135 for osd.1 2024-09-16T10:34:52.015 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:52.036 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:52.240 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:52 smithi063 ceph-mon[29993]: pgmap v1181: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 KiB/s rd, 5.8 KiB/s wr, 2 op/s; 1/597 objects misplaced (0.168%); 2.1 KiB/s, 0 objects/s recovering 2024-09-16T10:34:52.240 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:52 smithi063 ceph-mon[39470]: pgmap v1181: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 KiB/s rd, 5.8 KiB/s wr, 2 op/s; 1/597 objects misplaced (0.168%); 2.1 KiB/s, 0 objects/s recovering 2024-09-16T10:34:52.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986448 got 133143986449 for osd.4 2024-09-16T10:34:52.482 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:52.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:52 smithi165 ceph-mon[32696]: pgmap v1181: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 KiB/s rd, 5.8 KiB/s wr, 2 op/s; 1/597 objects misplaced (0.168%); 2.1 KiB/s, 0 objects/s recovering 2024-09-16T10:34:53.014 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411611 got 77309411613 for osd.2 2024-09-16T10:34:53.014 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[29993]: osdmap e630: 8 total, 8 up, 8 in 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[29993]: 3.15 scrub starts 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[29993]: 3.15 scrub ok 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[29993]: 4.0 scrub starts 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[29993]: 4.0 scrub ok 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[39470]: osdmap e630: 8 total, 8 up, 8 in 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[39470]: 3.15 scrub starts 2024-09-16T10:34:53.261 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[39470]: 3.15 scrub ok 2024-09-16T10:34:53.262 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[39470]: 4.0 scrub starts 2024-09-16T10:34:53.262 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:53 smithi063 ceph-mon[39470]: 4.0 scrub ok 2024-09-16T10:34:53.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:53 smithi165 ceph-mon[32696]: osdmap e630: 8 total, 8 up, 8 in 2024-09-16T10:34:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:53 smithi165 ceph-mon[32696]: 3.15 scrub starts 2024-09-16T10:34:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:53 smithi165 ceph-mon[32696]: 3.15 scrub ok 2024-09-16T10:34:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:53 smithi165 ceph-mon[32696]: 4.0 scrub starts 2024-09-16T10:34:53.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:53 smithi165 ceph-mon[32696]: 4.0 scrub ok 2024-09-16T10:34:53.718 INFO:teuthology.orchestra.run.smithi063.stdout:193273528583 2024-09-16T10:34:54.276 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: 3.1f scrub starts 2024-09-16T10:34:54.276 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: 3.1f scrub ok 2024-09-16T10:34:54.276 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: 3.0 scrub starts 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: 3.0 scrub ok 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: osdmap e631: 8 total, 8 up, 8 in 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: 3.11 scrub starts 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: 3.11 scrub ok 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: pgmap v1184: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.2 KiB/s wr, 1 op/s; 1/597 objects misplaced (0.168%); 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/1650003013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: 3.1f scrub starts 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: 3.1f scrub ok 2024-09-16T10:34:54.277 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: 3.0 scrub starts 2024-09-16T10:34:54.278 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: 3.0 scrub ok 2024-09-16T10:34:54.278 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: osdmap e631: 8 total, 8 up, 8 in 2024-09-16T10:34:54.278 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: 3.11 scrub starts 2024-09-16T10:34:54.278 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: 3.11 scrub ok 2024-09-16T10:34:54.278 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: pgmap v1184: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.2 KiB/s wr, 1 op/s; 1/597 objects misplaced (0.168%); 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:34:54.278 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:54 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/1650003013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T10:34:54.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528580 got 193273528583 for osd.6 2024-09-16T10:34:54.305 DEBUG:teuthology.parallel:result is None 2024-09-16T10:34:54.305 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T10:34:54.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: 3.1f scrub starts 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: 3.1f scrub ok 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: 3.0 scrub starts 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: 3.0 scrub ok 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: osdmap e631: 8 total, 8 up, 8 in 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: 3.11 scrub starts 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: 3.11 scrub ok 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: pgmap v1184: 145 pgs: 2 active+remapped+backfill_toofull, 143 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.2 KiB/s wr, 1 op/s; 1/597 objects misplaced (0.168%); 2.2 KiB/s, 0 objects/s recovering 2024-09-16T10:34:54.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:54 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/1650003013' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T10:34:54.571 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:55.232 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[39470]: 1.c scrub starts 2024-09-16T10:34:55.269 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:34:55.271 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:34:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:55 smithi165 ceph-mon[32696]: 1.c scrub starts 2024-09-16T10:34:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:55 smithi165 ceph-mon[32696]: 1.c scrub ok 2024-09-16T10:34:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:55 smithi165 ceph-mon[32696]: 3.13 scrub starts 2024-09-16T10:34:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:55 smithi165 ceph-mon[32696]: 3.13 scrub ok 2024-09-16T10:34:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:55 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:55 smithi165 ceph-mon[32696]: purged_snaps scrub starts 2024-09-16T10:34:55.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:55 smithi165 ceph-mon[32696]: purged_snaps scrub ok 2024-09-16T10:34:55.559 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[29993]: 1.c scrub starts 2024-09-16T10:34:55.560 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[29993]: 1.c scrub ok 2024-09-16T10:34:55.560 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[29993]: 3.13 scrub starts 2024-09-16T10:34:55.560 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[29993]: 3.13 scrub ok 2024-09-16T10:34:55.560 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:55.560 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[29993]: purged_snaps scrub starts 2024-09-16T10:34:55.560 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[29993]: purged_snaps scrub ok 2024-09-16T10:34:55.560 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[39470]: 1.c scrub ok 2024-09-16T10:34:55.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[39470]: 3.13 scrub starts 2024-09-16T10:34:55.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[39470]: 3.13 scrub ok 2024-09-16T10:34:55.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T10:34:55.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[39470]: purged_snaps scrub starts 2024-09-16T10:34:55.561 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:55 smithi063 ceph-mon[39470]: purged_snaps scrub ok 2024-09-16T10:34:55.974 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":1185,"stamp":"2024-09-16T10:34:55.237035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609924,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1649,"num_read_kb":1404,"num_write":1010,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1206112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1322,"ondisk_log_size":1322,"up":435,"acting":435,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":438,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5877788,"kb_used_data":11100,"kb_used_omap":18,"kb_used_meta":5866477,"kb_avail":744083428,"statfs":{"total":767960285184,"available":761941430272,"internally_reserved":0,"allocated":11366400,"data_stored":6681861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19173,"internal_metadata":6007272731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.680500"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1219,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313716+0000","last_change":"2024-09-16T10:33:17.292872+0000","last_active":"2024-09-16T10:34:46.313716+0000","last_peered":"2024-09-16T10:34:46.313716+0000","last_clean":"2024-09-16T10:34:46.313716+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:34:46.313716+0000","last_undegraded":"2024-09-16T10:34:46.313716+0000","last_fullsized":"2024-09-16T10:34:46.313716+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:17.292796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:17.292796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"572'21","reported_seq":1157,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218668+0000","last_change":"2024-09-16T10:34:49.305043+0000","last_active":"2024-09-16T10:34:53.218668+0000","last_peered":"2024-09-16T10:34:53.218668+0000","last_clean":"2024-09-16T10:34:53.218668+0000","last_became_active":"2024-09-16T10:34:18.163340+0000","last_became_peered":"2024-09-16T10:34:18.163340+0000","last_unstale":"2024-09-16T10:34:53.218668+0000","last_undegraded":"2024-09-16T10:34:53.218668+0000","last_fullsized":"2024-09-16T10:34:53.218668+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:49.304961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:49.304961+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:12.506578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":137,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218231+0000","last_change":"2024-09-16T10:34:04.447540+0000","last_active":"2024-09-16T10:34:53.218231+0000","last_peered":"2024-09-16T10:34:53.218231+0000","last_clean":"2024-09-16T10:34:53.218231+0000","last_became_active":"2024-09-16T10:33:51.155115+0000","last_became_peered":"2024-09-16T10:33:51.155115+0000","last_unstale":"2024-09-16T10:34:53.218231+0000","last_undegraded":"2024-09-16T10:34:53.218231+0000","last_fullsized":"2024-09-16T10:34:53.218231+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:04.447488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:04.447488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:18.532117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":768,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218789+0000","last_change":"2024-09-16T10:34:18.164407+0000","last_active":"2024-09-16T10:34:53.218789+0000","last_peered":"2024-09-16T10:34:53.218789+0000","last_clean":"2024-09-16T10:34:53.218789+0000","last_became_active":"2024-09-16T10:34:18.164080+0000","last_became_peered":"2024-09-16T10:34:18.164080+0000","last_unstale":"2024-09-16T10:34:53.218789+0000","last_undegraded":"2024-09-16T10:34:53.218789+0000","last_fullsized":"2024-09-16T10:34:53.218789+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:00.353850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:00.353850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:22.497189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1226,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220898+0000","last_change":"2024-09-16T10:33:55.311542+0000","last_active":"2024-09-16T10:34:53.220898+0000","last_peered":"2024-09-16T10:34:53.220898+0000","last_clean":"2024-09-16T10:34:53.220898+0000","last_became_active":"2024-09-16T10:33:51.155507+0000","last_became_peered":"2024-09-16T10:33:51.155507+0000","last_unstale":"2024-09-16T10:34:53.220898+0000","last_undegraded":"2024-09-16T10:34:53.220898+0000","last_fullsized":"2024-09-16T10:34:53.220898+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:55.311492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:55.311492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.686414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1231,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222649+0000","last_change":"2024-09-16T10:34:17.428552+0000","last_active":"2024-09-16T10:34:53.222649+0000","last_peered":"2024-09-16T10:34:53.222649+0000","last_clean":"2024-09-16T10:34:53.222649+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:34:53.222649+0000","last_undegraded":"2024-09-16T10:34:53.222649+0000","last_fullsized":"2024-09-16T10:34:53.222649+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:17.428493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:17.428493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:46.401772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"572'39","reported_seq":1318,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220113+0000","last_change":"2024-09-16T10:34:04.336597+0000","last_active":"2024-09-16T10:34:53.220113+0000","last_peered":"2024-09-16T10:34:53.220113+0000","last_clean":"2024-09-16T10:34:53.220113+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:34:53.220113+0000","last_undegraded":"2024-09-16T10:34:53.220113+0000","last_fullsized":"2024-09-16T10:34:53.220113+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:04.336533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:04.336533+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:27.000063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":764,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218655+0000","last_change":"2024-09-16T10:34:45.286934+0000","last_active":"2024-09-16T10:34:53.218655+0000","last_peered":"2024-09-16T10:34:53.218655+0000","last_clean":"2024-09-16T10:34:53.218655+0000","last_became_active":"2024-09-16T10:34:18.163264+0000","last_became_peered":"2024-09-16T10:34:18.163264+0000","last_unstale":"2024-09-16T10:34:53.218655+0000","last_undegraded":"2024-09-16T10:34:53.218655+0000","last_fullsized":"2024-09-16T10:34:53.218655+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:45.286890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:45.286890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:47.033188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1240,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.223214+0000","last_change":"2024-09-16T10:34:41.401083+0000","last_active":"2024-09-16T10:34:52.223214+0000","last_peered":"2024-09-16T10:34:52.223214+0000","last_clean":"2024-09-16T10:34:52.223214+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:34:52.223214+0000","last_undegraded":"2024-09-16T10:34:52.223214+0000","last_fullsized":"2024-09-16T10:34:52.223214+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:41.401023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:41.401023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:51.226191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"572'21","reported_seq":1089,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219030+0000","last_change":"2024-09-16T10:34:18.161633+0000","last_active":"2024-09-16T10:34:53.219030+0000","last_peered":"2024-09-16T10:34:53.219030+0000","last_clean":"2024-09-16T10:34:53.219030+0000","last_became_active":"2024-09-16T10:34:18.161281+0000","last_became_peered":"2024-09-16T10:34:18.161281+0000","last_unstale":"2024-09-16T10:34:53.219030+0000","last_undegraded":"2024-09-16T10:34:53.219030+0000","last_fullsized":"2024-09-16T10:34:53.219030+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:33:37.280137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:37.280137+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1233,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222099+0000","last_change":"2024-09-16T10:34:10.336095+0000","last_active":"2024-09-16T10:34:52.222099+0000","last_peered":"2024-09-16T10:34:52.222099+0000","last_clean":"2024-09-16T10:34:52.222099+0000","last_became_active":"2024-09-16T10:33:51.158343+0000","last_became_peered":"2024-09-16T10:33:51.158343+0000","last_unstale":"2024-09-16T10:34:52.222099+0000","last_undegraded":"2024-09-16T10:34:52.222099+0000","last_fullsized":"2024-09-16T10:34:52.222099+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:10.336029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:10.336029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.507579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1238,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312755+0000","last_change":"2024-09-16T10:34:38.765875+0000","last_active":"2024-09-16T10:34:46.312755+0000","last_peered":"2024-09-16T10:34:46.312755+0000","last_clean":"2024-09-16T10:34:46.312755+0000","last_became_active":"2024-09-16T10:33:51.159564+0000","last_became_peered":"2024-09-16T10:33:51.159564+0000","last_unstale":"2024-09-16T10:34:46.312755+0000","last_undegraded":"2024-09-16T10:34:46.312755+0000","last_fullsized":"2024-09-16T10:34:46.312755+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:38.765825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:38.765825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:03.193305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1198,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.224110+0000","last_change":"2024-09-16T10:34:02.378940+0000","last_active":"2024-09-16T10:34:52.224110+0000","last_peered":"2024-09-16T10:34:52.224110+0000","last_clean":"2024-09-16T10:34:52.224110+0000","last_became_active":"2024-09-16T10:33:49.209478+0000","last_became_peered":"2024-09-16T10:33:49.209478+0000","last_unstale":"2024-09-16T10:34:52.224110+0000","last_undegraded":"2024-09-16T10:34:52.224110+0000","last_fullsized":"2024-09-16T10:34:52.224110+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:02.378881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:02.378881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:08.620779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"572'9","reported_seq":1084,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218814+0000","last_change":"2024-09-16T10:34:35.192644+0000","last_active":"2024-09-16T10:34:53.218814+0000","last_peered":"2024-09-16T10:34:53.218814+0000","last_clean":"2024-09-16T10:34:53.218814+0000","last_became_active":"2024-09-16T10:34:18.166777+0000","last_became_peered":"2024-09-16T10:34:18.166777+0000","last_unstale":"2024-09-16T10:34:53.218814+0000","last_undegraded":"2024-09-16T10:34:53.218814+0000","last_fullsized":"2024-09-16T10:34:53.218814+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'9","last_scrub_stamp":"2024-09-16T10:34:35.192587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.192587+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:55.395098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":119,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313190+0000","last_change":"2024-09-16T10:33:56.019688+0000","last_active":"2024-09-16T10:34:46.313190+0000","last_peered":"2024-09-16T10:34:46.313190+0000","last_clean":"2024-09-16T10:34:46.313190+0000","last_became_active":"2024-09-16T10:33:51.158551+0000","last_became_peered":"2024-09-16T10:33:51.158551+0000","last_unstale":"2024-09-16T10:34:46.313190+0000","last_undegraded":"2024-09-16T10:34:46.313190+0000","last_fullsized":"2024-09-16T10:34:46.313190+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:56.019631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:56.019631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:10.047024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1233,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221571+0000","last_change":"2024-09-16T10:34:17.243583+0000","last_active":"2024-09-16T10:34:52.221571+0000","last_peered":"2024-09-16T10:34:52.221571+0000","last_clean":"2024-09-16T10:34:52.221571+0000","last_became_active":"2024-09-16T10:33:49.208831+0000","last_became_peered":"2024-09-16T10:33:49.208831+0000","last_unstale":"2024-09-16T10:34:52.221571+0000","last_undegraded":"2024-09-16T10:34:52.221571+0000","last_fullsized":"2024-09-16T10:34:52.221571+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:17.243521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:17.243521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:21.805421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":753,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219726+0000","last_change":"2024-09-16T10:34:36.192938+0000","last_active":"2024-09-16T10:34:53.219726+0000","last_peered":"2024-09-16T10:34:53.219726+0000","last_clean":"2024-09-16T10:34:53.219726+0000","last_became_active":"2024-09-16T10:34:18.162443+0000","last_became_peered":"2024-09-16T10:34:18.162443+0000","last_unstale":"2024-09-16T10:34:53.219726+0000","last_undegraded":"2024-09-16T10:34:53.219726+0000","last_fullsized":"2024-09-16T10:34:53.219726+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:36.192868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:36.192868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:37.230186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1234,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313205+0000","last_change":"2024-09-16T10:34:35.710276+0000","last_active":"2024-09-16T10:34:46.313205+0000","last_peered":"2024-09-16T10:34:46.313205+0000","last_clean":"2024-09-16T10:34:46.313205+0000","last_became_active":"2024-09-16T10:33:51.164632+0000","last_became_peered":"2024-09-16T10:33:51.164632+0000","last_unstale":"2024-09-16T10:34:46.313205+0000","last_undegraded":"2024-09-16T10:34:46.313205+0000","last_fullsized":"2024-09-16T10:34:46.313205+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:35.710223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.710223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:52.518359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"572'39","reported_seq":844,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.336674+0000","last_change":"2024-09-16T10:34:01.744988+0000","last_active":"2024-09-16T10:34:53.336674+0000","last_peered":"2024-09-16T10:34:53.336674+0000","last_clean":"2024-09-16T10:34:53.336674+0000","last_became_active":"2024-09-16T10:33:49.225899+0000","last_became_peered":"2024-09-16T10:33:49.225899+0000","last_unstale":"2024-09-16T10:34:53.336674+0000","last_undegraded":"2024-09-16T10:34:53.336674+0000","last_fullsized":"2024-09-16T10:34:53.336674+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:01.744914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:01.744914+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:20.032416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219574+0000","last_change":"2024-09-16T10:34:15.333753+0000","last_active":"2024-09-16T10:34:53.219574+0000","last_peered":"2024-09-16T10:34:53.219574+0000","last_clean":"2024-09-16T10:34:53.219574+0000","last_became_active":"2024-09-16T10:33:49.220841+0000","last_became_peered":"2024-09-16T10:33:49.220841+0000","last_unstale":"2024-09-16T10:34:53.219574+0000","last_undegraded":"2024-09-16T10:34:53.219574+0000","last_fullsized":"2024-09-16T10:34:53.219574+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:15.333702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:15.333702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:40.148695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1153,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221657+0000","last_change":"2024-09-16T10:33:08.153114+0000","last_active":"2024-09-16T10:34:53.221657+0000","last_peered":"2024-09-16T10:34:53.221657+0000","last_clean":"2024-09-16T10:34:53.221657+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:34:53.221657+0000","last_undegraded":"2024-09-16T10:34:53.221657+0000","last_fullsized":"2024-09-16T10:34:53.221657+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:08.153044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:08.153044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":778,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219202+0000","last_change":"2024-09-16T10:34:38.229723+0000","last_active":"2024-09-16T10:34:53.219202+0000","last_peered":"2024-09-16T10:34:53.219202+0000","last_clean":"2024-09-16T10:34:53.219202+0000","last_became_active":"2024-09-16T10:34:18.163325+0000","last_became_peered":"2024-09-16T10:34:18.163325+0000","last_unstale":"2024-09-16T10:34:53.219202+0000","last_undegraded":"2024-09-16T10:34:53.219202+0000","last_fullsized":"2024-09-16T10:34:53.219202+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:38.229668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:38.229668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:51.059958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"572'24","reported_seq":1336,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221749+0000","last_change":"2024-09-16T10:34:20.444837+0000","last_active":"2024-09-16T10:34:53.221749+0000","last_peered":"2024-09-16T10:34:53.221749+0000","last_clean":"2024-09-16T10:34:53.221749+0000","last_became_active":"2024-09-16T10:34:13.154733+0000","last_became_peered":"2024-09-16T10:34:13.154733+0000","last_unstale":"2024-09-16T10:34:53.221749+0000","last_undegraded":"2024-09-16T10:34:53.221749+0000","last_fullsized":"2024-09-16T10:34:53.221749+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'24","last_scrub_stamp":"2024-09-16T10:34:20.444765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:20.444765+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:45.950400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":786,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219248+0000","last_change":"2024-09-16T10:34:18.166916+0000","last_active":"2024-09-16T10:34:53.219248+0000","last_peered":"2024-09-16T10:34:53.219248+0000","last_clean":"2024-09-16T10:34:53.219248+0000","last_became_active":"2024-09-16T10:34:18.166206+0000","last_became_peered":"2024-09-16T10:34:18.166206+0000","last_unstale":"2024-09-16T10:34:53.219248+0000","last_undegraded":"2024-09-16T10:34:53.219248+0000","last_fullsized":"2024-09-16T10:34:53.219248+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:48.368758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:48.368758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.320254+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":558,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313445+0000","last_change":"2024-09-16T10:34:30.620887+0000","last_active":"2024-09-16T10:34:46.313445+0000","last_peered":"2024-09-16T10:34:46.313445+0000","last_clean":"2024-09-16T10:34:46.313445+0000","last_became_active":"2024-09-16T10:34:30.620547+0000","last_became_peered":"2024-09-16T10:34:30.620547+0000","last_unstale":"2024-09-16T10:34:46.313445+0000","last_undegraded":"2024-09-16T10:34:46.313445+0000","last_fullsized":"2024-09-16T10:34:46.313445+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:34:16.203535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:16.203535+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:37.486266+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":128,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218388+0000","last_change":"2024-09-16T10:33:49.240616+0000","last_active":"2024-09-16T10:34:53.218388+0000","last_peered":"2024-09-16T10:34:53.218388+0000","last_clean":"2024-09-16T10:34:53.218388+0000","last_became_active":"2024-09-16T10:33:49.220080+0000","last_became_peered":"2024-09-16T10:33:49.220080+0000","last_unstale":"2024-09-16T10:34:53.218388+0000","last_undegraded":"2024-09-16T10:34:53.218388+0000","last_fullsized":"2024-09-16T10:34:53.218388+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:33:44.365752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:44.365752+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.051928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"572'30","reported_seq":1333,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313693+0000","last_change":"2024-09-16T10:33:51.202586+0000","last_active":"2024-09-16T10:34:46.313693+0000","last_peered":"2024-09-16T10:34:46.313693+0000","last_clean":"2024-09-16T10:34:46.313693+0000","last_became_active":"2024-09-16T10:33:51.169227+0000","last_became_peered":"2024-09-16T10:33:51.169227+0000","last_unstale":"2024-09-16T10:34:46.313693+0000","last_undegraded":"2024-09-16T10:34:46.313693+0000","last_fullsized":"2024-09-16T10:34:46.313693+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:14.278843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:14.278843+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"579'12","reported_seq":1505,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.335205+0000","last_change":"2024-09-16T10:33:51.184591+0000","last_active":"2024-09-16T10:34:53.335205+0000","last_peered":"2024-09-16T10:34:53.335205+0000","last_clean":"2024-09-16T10:34:53.335205+0000","last_became_active":"2024-09-16T10:33:51.164458+0000","last_became_peered":"2024-09-16T10:33:51.164458+0000","last_unstale":"2024-09-16T10:34:53.335205+0000","last_undegraded":"2024-09-16T10:34:53.335205+0000","last_fullsized":"2024-09-16T10:34:53.335205+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'11","last_scrub_stamp":"2024-09-16T10:33:48.605415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:48.605415+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.867359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":1079,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221737+0000","last_change":"2024-09-16T10:34:28.375764+0000","last_active":"2024-09-16T10:34:52.221737+0000","last_peered":"2024-09-16T10:34:52.221737+0000","last_clean":"2024-09-16T10:34:52.221737+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:34:52.221737+0000","last_undegraded":"2024-09-16T10:34:52.221737+0000","last_fullsized":"2024-09-16T10:34:52.221737+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:28.375686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:28.375686+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:35.391649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"572'39","reported_seq":146,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221257+0000","last_change":"2024-09-16T10:34:29.319100+0000","last_active":"2024-09-16T10:34:53.221257+0000","last_peered":"2024-09-16T10:34:53.221257+0000","last_clean":"2024-09-16T10:34:53.221257+0000","last_became_active":"2024-09-16T10:33:49.232682+0000","last_became_peered":"2024-09-16T10:33:49.232682+0000","last_unstale":"2024-09-16T10:34:53.221257+0000","last_undegraded":"2024-09-16T10:34:53.221257+0000","last_fullsized":"2024-09-16T10:34:53.221257+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:29.319043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:29.319043+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:59.044672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1255,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220292+0000","last_change":"2024-09-16T10:34:34.390450+0000","last_active":"2024-09-16T10:34:53.220292+0000","last_peered":"2024-09-16T10:34:53.220292+0000","last_clean":"2024-09-16T10:34:53.220292+0000","last_became_active":"2024-09-16T10:33:51.165462+0000","last_became_peered":"2024-09-16T10:33:51.165462+0000","last_unstale":"2024-09-16T10:34:53.220292+0000","last_undegraded":"2024-09-16T10:34:53.220292+0000","last_fullsized":"2024-09-16T10:34:53.220292+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:34.390353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:34.390353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.333117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"601'10","reported_seq":935,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219825+0000","last_change":"2024-09-16T10:34:32.224690+0000","last_active":"2024-09-16T10:34:53.219825+0000","last_peered":"2024-09-16T10:34:53.219825+0000","last_clean":"2024-09-16T10:34:53.219825+0000","last_became_active":"2024-09-16T10:34:18.162934+0000","last_became_peered":"2024-09-16T10:34:18.162934+0000","last_unstale":"2024-09-16T10:34:53.219825+0000","last_undegraded":"2024-09-16T10:34:53.219825+0000","last_fullsized":"2024-09-16T10:34:53.219825+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'10","last_scrub_stamp":"2024-09-16T10:34:32.224631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:32.224631+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:58.452668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1219,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222861+0000","last_change":"2024-09-16T10:33:49.213894+0000","last_active":"2024-09-16T10:34:52.222861+0000","last_peered":"2024-09-16T10:34:52.222861+0000","last_clean":"2024-09-16T10:34:52.222861+0000","last_became_active":"2024-09-16T10:33:49.213427+0000","last_became_peered":"2024-09-16T10:33:49.213427+0000","last_unstale":"2024-09-16T10:34:52.222861+0000","last_undegraded":"2024-09-16T10:34:52.222861+0000","last_fullsized":"2024-09-16T10:34:52.222861+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:42.455925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:42.455925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:11.585766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":160,"reported_epoch":631,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:34:53.221365+0000","last_change":"2024-09-16T10:34:46.315733+0000","last_active":"2024-09-16T10:34:53.221365+0000","last_peered":"2024-09-16T10:34:53.221365+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:34:46.314709+0000","last_became_peered":"2024-09-16T10:34:46.314709+0000","last_unstale":"2024-09-16T10:34:53.221365+0000","last_undegraded":"2024-09-16T10:34:53.221365+0000","last_fullsized":"2024-09-16T10:34:53.221365+0000","mapping_epoch":627,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:54.497762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:54.497762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[3,6,5],"avail_no_missing":["3","2","4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"572'42","reported_seq":570,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218534+0000","last_change":"2024-09-16T10:34:52.355681+0000","last_active":"2024-09-16T10:34:53.218534+0000","last_peered":"2024-09-16T10:34:53.218534+0000","last_clean":"2024-09-16T10:34:53.218534+0000","last_became_active":"2024-09-16T10:34:18.159011+0000","last_became_peered":"2024-09-16T10:34:18.159011+0000","last_unstale":"2024-09-16T10:34:53.218534+0000","last_undegraded":"2024-09-16T10:34:53.218534+0000","last_fullsized":"2024-09-16T10:34:53.218534+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'42","last_scrub_stamp":"2024-09-16T10:34:52.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:52.355594+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:20.728319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1206,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222590+0000","last_change":"2024-09-16T10:34:40.396096+0000","last_active":"2024-09-16T10:34:53.222590+0000","last_peered":"2024-09-16T10:34:53.222590+0000","last_clean":"2024-09-16T10:34:53.222590+0000","last_became_active":"2024-09-16T10:33:51.157669+0000","last_became_peered":"2024-09-16T10:33:51.157669+0000","last_unstale":"2024-09-16T10:34:53.222590+0000","last_undegraded":"2024-09-16T10:34:53.222590+0000","last_fullsized":"2024-09-16T10:34:53.222590+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:40.396048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:40.396048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:40.644426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":974,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.333840+0000","last_change":"2024-09-16T10:34:24.433929+0000","last_active":"2024-09-16T10:34:53.333840+0000","last_peered":"2024-09-16T10:34:53.333840+0000","last_clean":"2024-09-16T10:34:53.333840+0000","last_became_active":"2024-09-16T10:33:51.166245+0000","last_became_peered":"2024-09-16T10:33:51.166245+0000","last_unstale":"2024-09-16T10:34:53.333840+0000","last_undegraded":"2024-09-16T10:34:53.333840+0000","last_fullsized":"2024-09-16T10:34:53.333840+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:24.433865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:24.433865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.653058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":775,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218985+0000","last_change":"2024-09-16T10:34:18.157940+0000","last_active":"2024-09-16T10:34:53.218985+0000","last_peered":"2024-09-16T10:34:53.218985+0000","last_clean":"2024-09-16T10:34:53.218985+0000","last_became_active":"2024-09-16T10:34:18.157585+0000","last_became_peered":"2024-09-16T10:34:18.157585+0000","last_unstale":"2024-09-16T10:34:53.218985+0000","last_undegraded":"2024-09-16T10:34:53.218985+0000","last_fullsized":"2024-09-16T10:34:53.218985+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:53.331351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:53.331351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:09.546382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":1180,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222403+0000","last_change":"2024-09-16T10:34:14.356079+0000","last_active":"2024-09-16T10:34:53.222403+0000","last_peered":"2024-09-16T10:34:53.222403+0000","last_clean":"2024-09-16T10:34:53.222403+0000","last_became_active":"2024-09-16T10:34:03.470999+0000","last_became_peered":"2024-09-16T10:34:03.470999+0000","last_unstale":"2024-09-16T10:34:53.222403+0000","last_undegraded":"2024-09-16T10:34:53.222403+0000","last_fullsized":"2024-09-16T10:34:53.222403+0000","mapping_epoch":580,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":588,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:14.356030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:14.356030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":74,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.832276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"572'42","reported_seq":1129,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218709+0000","last_change":"2024-09-16T10:34:28.282974+0000","last_active":"2024-09-16T10:34:53.218709+0000","last_peered":"2024-09-16T10:34:53.218709+0000","last_clean":"2024-09-16T10:34:53.218709+0000","last_became_active":"2024-09-16T10:34:18.162427+0000","last_became_peered":"2024-09-16T10:34:18.162427+0000","last_unstale":"2024-09-16T10:34:53.218709+0000","last_undegraded":"2024-09-16T10:34:53.218709+0000","last_fullsized":"2024-09-16T10:34:53.218709+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'42","last_scrub_stamp":"2024-09-16T10:34:28.282890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:28.282890+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.624869+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":1219,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313955+0000","last_change":"2024-09-16T10:33:58.447082+0000","last_active":"2024-09-16T10:34:46.313955+0000","last_peered":"2024-09-16T10:34:46.313955+0000","last_clean":"2024-09-16T10:34:46.313955+0000","last_became_active":"2024-09-16T10:33:49.211982+0000","last_became_peered":"2024-09-16T10:33:49.211982+0000","last_unstale":"2024-09-16T10:34:46.313955+0000","last_undegraded":"2024-09-16T10:34:46.313955+0000","last_fullsized":"2024-09-16T10:34:46.313955+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:33:58.447024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:58.447024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:11.900859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"579'10","reported_seq":1490,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219969+0000","last_change":"2024-09-16T10:33:49.280956+0000","last_active":"2024-09-16T10:34:53.219969+0000","last_peered":"2024-09-16T10:34:53.219969+0000","last_clean":"2024-09-16T10:34:53.219969+0000","last_became_active":"2024-09-16T10:33:49.229244+0000","last_became_peered":"2024-09-16T10:33:49.229244+0000","last_unstale":"2024-09-16T10:34:53.219969+0000","last_undegraded":"2024-09-16T10:34:53.219969+0000","last_fullsized":"2024-09-16T10:34:53.219969+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'9","last_scrub_stamp":"2024-09-16T10:33:15.201123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:15.201123+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1017,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312998+0000","last_change":"2024-09-16T10:34:30.621896+0000","last_active":"2024-09-16T10:34:46.312998+0000","last_peered":"2024-09-16T10:34:46.312998+0000","last_clean":"2024-09-16T10:34:46.312998+0000","last_became_active":"2024-09-16T10:34:30.621575+0000","last_became_peered":"2024-09-16T10:34:30.621575+0000","last_unstale":"2024-09-16T10:34:46.312998+0000","last_undegraded":"2024-09-16T10:34:46.312998+0000","last_fullsized":"2024-09-16T10:34:46.312998+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:19.550389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:19.550389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.041848+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":1238,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.335093+0000","last_change":"2024-09-16T10:34:39.212462+0000","last_active":"2024-09-16T10:34:53.335093+0000","last_peered":"2024-09-16T10:34:53.335093+0000","last_clean":"2024-09-16T10:34:53.335093+0000","last_became_active":"2024-09-16T10:34:39.212049+0000","last_became_peered":"2024-09-16T10:34:39.212049+0000","last_unstale":"2024-09-16T10:34:53.335093+0000","last_undegraded":"2024-09-16T10:34:53.335093+0000","last_fullsized":"2024-09-16T10:34:53.335093+0000","mapping_epoch":580,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":621,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:55.533227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:55.533227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:56.206666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"572'30","reported_seq":142,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218707+0000","last_change":"2024-09-16T10:34:00.453400+0000","last_active":"2024-09-16T10:34:53.218707+0000","last_peered":"2024-09-16T10:34:53.218707+0000","last_clean":"2024-09-16T10:34:53.218707+0000","last_became_active":"2024-09-16T10:33:49.208703+0000","last_became_peered":"2024-09-16T10:33:49.208703+0000","last_unstale":"2024-09-16T10:34:53.218707+0000","last_undegraded":"2024-09-16T10:34:53.218707+0000","last_fullsized":"2024-09-16T10:34:53.218707+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'30","last_scrub_stamp":"2024-09-16T10:34:00.453298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:00.453298+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:17.878799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1232,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.223276+0000","last_change":"2024-09-16T10:34:30.439919+0000","last_active":"2024-09-16T10:34:52.223276+0000","last_peered":"2024-09-16T10:34:52.223276+0000","last_clean":"2024-09-16T10:34:52.223276+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:34:52.223276+0000","last_undegraded":"2024-09-16T10:34:52.223276+0000","last_fullsized":"2024-09-16T10:34:52.223276+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:30.439845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:30.439845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:46.644887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1452,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313773+0000","last_change":"2024-09-16T10:33:42.380585+0000","last_active":"2024-09-16T10:34:46.313773+0000","last_peered":"2024-09-16T10:34:46.313773+0000","last_clean":"2024-09-16T10:34:46.313773+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:34:46.313773+0000","last_undegraded":"2024-09-16T10:34:46.313773+0000","last_fullsized":"2024-09-16T10:34:46.313773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:33:42.380504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:42.380504+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:54.620204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1215,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312924+0000","last_change":"2024-09-16T10:34:31.727025+0000","last_active":"2024-09-16T10:34:46.312924+0000","last_peered":"2024-09-16T10:34:46.312924+0000","last_clean":"2024-09-16T10:34:46.312924+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:34:46.312924+0000","last_undegraded":"2024-09-16T10:34:46.312924+0000","last_fullsized":"2024-09-16T10:34:46.312924+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:31.726979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:31.726979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:46.526221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":1199,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334882+0000","last_change":"2024-09-16T10:34:43.178195+0000","last_active":"2024-09-16T10:34:53.334882+0000","last_peered":"2024-09-16T10:34:53.334882+0000","last_clean":"2024-09-16T10:34:53.334882+0000","last_became_active":"2024-09-16T10:34:43.177899+0000","last_became_peered":"2024-09-16T10:34:43.177899+0000","last_unstale":"2024-09-16T10:34:53.334882+0000","last_undegraded":"2024-09-16T10:34:53.334882+0000","last_fullsized":"2024-09-16T10:34:53.334882+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:27.498031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:27.498031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.199776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"572'39","reported_seq":1386,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:47.707078+0000","last_change":"2024-09-16T10:34:47.707078+0000","last_active":"2024-09-16T10:34:47.707078+0000","last_peered":"2024-09-16T10:34:47.707078+0000","last_clean":"2024-09-16T10:34:47.707078+0000","last_became_active":"2024-09-16T10:33:51.164128+0000","last_became_peered":"2024-09-16T10:33:51.164128+0000","last_unstale":"2024-09-16T10:34:47.707078+0000","last_undegraded":"2024-09-16T10:34:47.707078+0000","last_fullsized":"2024-09-16T10:34:47.707078+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:47.706982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:47.706982+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:55.643555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219174+0000","last_change":"2024-09-16T10:33:59.402162+0000","last_active":"2024-09-16T10:34:53.219174+0000","last_peered":"2024-09-16T10:34:53.219174+0000","last_clean":"2024-09-16T10:34:53.219174+0000","last_became_active":"2024-09-16T10:33:49.202668+0000","last_became_peered":"2024-09-16T10:33:49.202668+0000","last_unstale":"2024-09-16T10:34:53.219174+0000","last_undegraded":"2024-09-16T10:34:53.219174+0000","last_fullsized":"2024-09-16T10:34:53.219174+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:59.402107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:59.402107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.409662+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1219,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334009+0000","last_change":"2024-09-16T10:33:36.663621+0000","last_active":"2024-09-16T10:34:53.334009+0000","last_peered":"2024-09-16T10:34:53.334009+0000","last_clean":"2024-09-16T10:34:53.334009+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:34:53.334009+0000","last_undegraded":"2024-09-16T10:34:53.334009+0000","last_fullsized":"2024-09-16T10:34:53.334009+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:36.663561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:36.663561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.772506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218213+0000","last_change":"2024-09-16T10:33:49.199768+0000","last_active":"2024-09-16T10:34:53.218213+0000","last_peered":"2024-09-16T10:34:53.218213+0000","last_clean":"2024-09-16T10:34:53.218213+0000","last_became_active":"2024-09-16T10:33:49.198662+0000","last_became_peered":"2024-09-16T10:33:49.198662+0000","last_unstale":"2024-09-16T10:34:53.218213+0000","last_undegraded":"2024-09-16T10:34:53.218213+0000","last_fullsized":"2024-09-16T10:34:53.218213+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:47.582751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:47.582751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:12.837063+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":1135,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221285+0000","last_change":"2024-09-16T10:34:32.161884+0000","last_active":"2024-09-16T10:34:52.221285+0000","last_peered":"2024-09-16T10:34:52.221285+0000","last_clean":"2024-09-16T10:34:52.221285+0000","last_became_active":"2024-09-16T10:34:32.161436+0000","last_became_peered":"2024-09-16T10:34:32.161436+0000","last_unstale":"2024-09-16T10:34:52.221285+0000","last_undegraded":"2024-09-16T10:34:52.221285+0000","last_fullsized":"2024-09-16T10:34:52.221285+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":614,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:13.269281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:13.269281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:30.798554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"579'56","reported_seq":165,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218994+0000","last_change":"2024-09-16T10:34:24.306070+0000","last_active":"2024-09-16T10:34:53.218994+0000","last_peered":"2024-09-16T10:34:53.218994+0000","last_clean":"2024-09-16T10:34:53.218994+0000","last_became_active":"2024-09-16T10:33:49.222165+0000","last_became_peered":"2024-09-16T10:33:49.222165+0000","last_unstale":"2024-09-16T10:34:53.218994+0000","last_undegraded":"2024-09-16T10:34:53.218994+0000","last_fullsized":"2024-09-16T10:34:53.218994+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'56","last_scrub_stamp":"2024-09-16T10:34:24.305984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:24.305984+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:32.914469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":773,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219238+0000","last_change":"2024-09-16T10:34:18.161353+0000","last_active":"2024-09-16T10:34:53.219238+0000","last_peered":"2024-09-16T10:34:53.219238+0000","last_clean":"2024-09-16T10:34:53.219238+0000","last_became_active":"2024-09-16T10:34:18.161058+0000","last_became_peered":"2024-09-16T10:34:18.161058+0000","last_unstale":"2024-09-16T10:34:53.219238+0000","last_undegraded":"2024-09-16T10:34:53.219238+0000","last_fullsized":"2024-09-16T10:34:53.219238+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:53.495088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:53.495088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:55.014885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":1250,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222279+0000","last_change":"2024-09-16T10:33:53.317359+0000","last_active":"2024-09-16T10:34:52.222279+0000","last_peered":"2024-09-16T10:34:52.222279+0000","last_clean":"2024-09-16T10:34:52.222279+0000","last_became_active":"2024-09-16T10:33:49.213116+0000","last_became_peered":"2024-09-16T10:33:49.213116+0000","last_unstale":"2024-09-16T10:34:52.222279+0000","last_undegraded":"2024-09-16T10:34:52.222279+0000","last_fullsized":"2024-09-16T10:34:52.222279+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:33:53.317300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:33:53.317300+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.362730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1203,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313269+0000","last_change":"2024-09-16T10:34:34.681162+0000","last_active":"2024-09-16T10:34:46.313269+0000","last_peered":"2024-09-16T10:34:46.313269+0000","last_clean":"2024-09-16T10:34:46.313269+0000","last_became_active":"2024-09-16T10:33:49.210355+0000","last_became_peered":"2024-09-16T10:33:49.210355+0000","last_unstale":"2024-09-16T10:34:46.313269+0000","last_undegraded":"2024-09-16T10:34:46.313269+0000","last_fullsized":"2024-09-16T10:34:46.313269+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:34.681117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:34.681117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:50.145347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1237,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222278+0000","last_change":"2024-09-16T10:33:51.156938+0000","last_active":"2024-09-16T10:34:53.222278+0000","last_peered":"2024-09-16T10:34:53.222278+0000","last_clean":"2024-09-16T10:34:53.222278+0000","last_became_active":"2024-09-16T10:33:51.152721+0000","last_became_peered":"2024-09-16T10:33:51.152721+0000","last_unstale":"2024-09-16T10:34:53.222278+0000","last_undegraded":"2024-09-16T10:34:53.222278+0000","last_fullsized":"2024-09-16T10:34:53.222278+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:39.283769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:39.283769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.548316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":655,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313034+0000","last_change":"2024-09-16T10:34:33.194602+0000","last_active":"2024-09-16T10:34:46.313034+0000","last_peered":"2024-09-16T10:34:46.313034+0000","last_clean":"2024-09-16T10:34:46.313034+0000","last_became_active":"2024-09-16T10:34:30.622945+0000","last_became_peered":"2024-09-16T10:34:30.622945+0000","last_unstale":"2024-09-16T10:34:46.313034+0000","last_undegraded":"2024-09-16T10:34:46.313034+0000","last_fullsized":"2024-09-16T10:34:46.313034+0000","mapping_epoch":611,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:33.194525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.194525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:52.999972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"572'21","reported_seq":142,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221110+0000","last_change":"2024-09-16T10:34:11.383837+0000","last_active":"2024-09-16T10:34:53.221110+0000","last_peered":"2024-09-16T10:34:53.221110+0000","last_clean":"2024-09-16T10:34:53.221110+0000","last_became_active":"2024-09-16T10:33:49.208434+0000","last_became_peered":"2024-09-16T10:33:49.208434+0000","last_unstale":"2024-09-16T10:34:53.221110+0000","last_undegraded":"2024-09-16T10:34:53.221110+0000","last_fullsized":"2024-09-16T10:34:53.221110+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:11.383766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:11.383766+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:33.756559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218423+0000","last_change":"2024-09-16T10:34:53.211141+0000","last_active":"2024-09-16T10:34:53.218423+0000","last_peered":"2024-09-16T10:34:53.218423+0000","last_clean":"2024-09-16T10:34:53.218423+0000","last_became_active":"2024-09-16T10:33:49.199335+0000","last_became_peered":"2024-09-16T10:33:49.199335+0000","last_unstale":"2024-09-16T10:34:53.218423+0000","last_undegraded":"2024-09-16T10:34:53.218423+0000","last_fullsized":"2024-09-16T10:34:53.218423+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:53.211046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:53.211046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:00.432857+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":751,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.337066+0000","last_change":"2024-09-16T10:34:22.774392+0000","last_active":"2024-09-16T10:34:53.337066+0000","last_peered":"2024-09-16T10:34:53.337066+0000","last_clean":"2024-09-16T10:34:53.337066+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:34:53.337066+0000","last_undegraded":"2024-09-16T10:34:53.337066+0000","last_fullsized":"2024-09-16T10:34:53.337066+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:34:22.774339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:34:22.774339+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:29.274046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":771,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345596+0000","last_change":"2024-09-16T10:34:02.704295+0000","last_active":"2024-09-16T10:34:53.345596+0000","last_peered":"2024-09-16T10:34:53.345596+0000","last_clean":"2024-09-16T10:34:53.345596+0000","last_became_active":"2024-09-16T10:33:51.156708+0000","last_became_peered":"2024-09-16T10:33:51.156708+0000","last_unstale":"2024-09-16T10:34:53.345596+0000","last_undegraded":"2024-09-16T10:34:53.345596+0000","last_fullsized":"2024-09-16T10:34:53.345596+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:02.704247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:02.704247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.790911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":543,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313453+0000","last_change":"2024-09-16T10:34:30.625497+0000","last_active":"2024-09-16T10:34:46.313453+0000","last_peered":"2024-09-16T10:34:46.313453+0000","last_clean":"2024-09-16T10:34:46.313453+0000","last_became_active":"2024-09-16T10:34:30.621205+0000","last_became_peered":"2024-09-16T10:34:30.621205+0000","last_unstale":"2024-09-16T10:34:46.313453+0000","last_undegraded":"2024-09-16T10:34:46.313453+0000","last_fullsized":"2024-09-16T10:34:46.313453+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:58.563402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:58.563402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:21.828397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":162,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218905+0000","last_change":"2024-09-16T10:34:07.152780+0000","last_active":"2024-09-16T10:34:53.218905+0000","last_peered":"2024-09-16T10:34:53.218905+0000","last_clean":"2024-09-16T10:34:53.218905+0000","last_became_active":"2024-09-16T10:34:07.152171+0000","last_became_peered":"2024-09-16T10:34:07.152171+0000","last_unstale":"2024-09-16T10:34:53.218905+0000","last_undegraded":"2024-09-16T10:34:53.218905+0000","last_fullsized":"2024-09-16T10:34:53.218905+0000","mapping_epoch":578,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:52.304996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:52.304996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:04.380325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"572'36","reported_seq":1362,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313790+0000","last_change":"2024-09-16T10:34:15.538554+0000","last_active":"2024-09-16T10:34:46.313790+0000","last_peered":"2024-09-16T10:34:46.313790+0000","last_clean":"2024-09-16T10:34:46.313790+0000","last_became_active":"2024-09-16T10:33:49.212861+0000","last_became_peered":"2024-09-16T10:33:49.212861+0000","last_unstale":"2024-09-16T10:34:46.313790+0000","last_undegraded":"2024-09-16T10:34:46.313790+0000","last_fullsized":"2024-09-16T10:34:46.313790+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'36","last_scrub_stamp":"2024-09-16T10:34:15.538460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:15.538460+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.810915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1229,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313128+0000","last_change":"2024-09-16T10:33:44.438020+0000","last_active":"2024-09-16T10:34:46.313128+0000","last_peered":"2024-09-16T10:34:46.313128+0000","last_clean":"2024-09-16T10:34:46.313128+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:34:46.313128+0000","last_undegraded":"2024-09-16T10:34:46.313128+0000","last_fullsized":"2024-09-16T10:34:46.313128+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:44.437972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:44.437972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.831239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":83,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:49.656137+0000","last_change":"2024-09-16T10:34:30.625642+0000","last_active":"2024-09-16T10:34:49.656137+0000","last_peered":"2024-09-16T10:34:49.656137+0000","last_clean":"2024-09-16T10:34:49.656137+0000","last_became_active":"2024-09-16T10:34:30.623166+0000","last_became_peered":"2024-09-16T10:34:30.623166+0000","last_unstale":"2024-09-16T10:34:49.656137+0000","last_undegraded":"2024-09-16T10:34:49.656137+0000","last_fullsized":"2024-09-16T10:34:49.656137+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:34:20.296113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:34:20.296113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:24.980292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":114,"num_read_kb":109,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1239,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222401+0000","last_change":"2024-09-16T10:34:39.343719+0000","last_active":"2024-09-16T10:34:52.222401+0000","last_peered":"2024-09-16T10:34:52.222401+0000","last_clean":"2024-09-16T10:34:52.222401+0000","last_became_active":"2024-09-16T10:33:49.210766+0000","last_became_peered":"2024-09-16T10:33:49.210766+0000","last_unstale":"2024-09-16T10:34:52.222401+0000","last_undegraded":"2024-09-16T10:34:52.222401+0000","last_fullsized":"2024-09-16T10:34:52.222401+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:39.343668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:39.343668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:02.971579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219056+0000","last_change":"2024-09-16T10:33:54.309689+0000","last_active":"2024-09-16T10:34:53.219056+0000","last_peered":"2024-09-16T10:34:53.219056+0000","last_clean":"2024-09-16T10:34:53.219056+0000","last_became_active":"2024-09-16T10:33:49.222778+0000","last_became_peered":"2024-09-16T10:33:49.222778+0000","last_unstale":"2024-09-16T10:34:53.219056+0000","last_undegraded":"2024-09-16T10:34:53.219056+0000","last_fullsized":"2024-09-16T10:34:53.219056+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:54.309637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:54.309637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:01.456931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"579'44","reported_seq":1576,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334816+0000","last_change":"2024-09-16T10:33:57.524064+0000","last_active":"2024-09-16T10:34:53.334816+0000","last_peered":"2024-09-16T10:34:53.334816+0000","last_clean":"2024-09-16T10:34:53.334816+0000","last_became_active":"2024-09-16T10:33:51.163308+0000","last_became_peered":"2024-09-16T10:33:51.163308+0000","last_unstale":"2024-09-16T10:34:53.334816+0000","last_undegraded":"2024-09-16T10:34:53.334816+0000","last_fullsized":"2024-09-16T10:34:53.334816+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'44","last_scrub_stamp":"2024-09-16T10:33:57.524000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:57.524000+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:10.448250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1181,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221030+0000","last_change":"2024-09-16T10:34:44.397662+0000","last_active":"2024-09-16T10:34:53.221030+0000","last_peered":"2024-09-16T10:34:53.221030+0000","last_clean":"2024-09-16T10:34:53.221030+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:34:53.221030+0000","last_undegraded":"2024-09-16T10:34:53.221030+0000","last_fullsized":"2024-09-16T10:34:53.221030+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:44.397559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:44.397559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:14.301745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":681,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312930+0000","last_change":"2024-09-16T10:34:39.217482+0000","last_active":"2024-09-16T10:34:46.312930+0000","last_peered":"2024-09-16T10:34:46.312930+0000","last_clean":"2024-09-16T10:34:46.312930+0000","last_became_active":"2024-09-16T10:34:35.174424+0000","last_became_peered":"2024-09-16T10:34:35.174424+0000","last_unstale":"2024-09-16T10:34:46.312930+0000","last_undegraded":"2024-09-16T10:34:46.312930+0000","last_fullsized":"2024-09-16T10:34:46.312930+0000","mapping_epoch":611,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:39.217420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:39.217420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:08.688758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1016,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313326+0000","last_change":"2024-09-16T10:34:30.620052+0000","last_active":"2024-09-16T10:34:46.313326+0000","last_peered":"2024-09-16T10:34:46.313326+0000","last_clean":"2024-09-16T10:34:46.313326+0000","last_became_active":"2024-09-16T10:34:30.619720+0000","last_became_peered":"2024-09-16T10:34:30.619720+0000","last_unstale":"2024-09-16T10:34:46.313326+0000","last_undegraded":"2024-09-16T10:34:46.313326+0000","last_fullsized":"2024-09-16T10:34:46.313326+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:55.616263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:55.616263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:58.137541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1205,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222599+0000","last_change":"2024-09-16T10:34:05.350310+0000","last_active":"2024-09-16T10:34:52.222599+0000","last_peered":"2024-09-16T10:34:52.222599+0000","last_clean":"2024-09-16T10:34:52.222599+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:34:52.222599+0000","last_undegraded":"2024-09-16T10:34:52.222599+0000","last_fullsized":"2024-09-16T10:34:52.222599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:05.350255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:05.350255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:19.398783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":685,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218754+0000","last_change":"2024-09-16T10:34:26.341536+0000","last_active":"2024-09-16T10:34:53.218754+0000","last_peered":"2024-09-16T10:34:53.218754+0000","last_clean":"2024-09-16T10:34:53.218754+0000","last_became_active":"2024-09-16T10:34:21.550673+0000","last_became_peered":"2024-09-16T10:34:21.550673+0000","last_unstale":"2024-09-16T10:34:53.218754+0000","last_undegraded":"2024-09-16T10:34:53.218754+0000","last_fullsized":"2024-09-16T10:34:53.218754+0000","mapping_epoch":603,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:26.341460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:26.341460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:40.126925+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"572'27","reported_seq":1123,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219868+0000","last_change":"2024-09-16T10:34:18.164468+0000","last_active":"2024-09-16T10:34:53.219868+0000","last_peered":"2024-09-16T10:34:53.219868+0000","last_clean":"2024-09-16T10:34:53.219868+0000","last_became_active":"2024-09-16T10:34:18.164080+0000","last_became_peered":"2024-09-16T10:34:18.164080+0000","last_unstale":"2024-09-16T10:34:53.219868+0000","last_undegraded":"2024-09-16T10:34:53.219868+0000","last_fullsized":"2024-09-16T10:34:53.219868+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:34:10.518049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:10.518049+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:12.960322+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1003,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219186+0000","last_change":"2024-09-16T10:34:18.160078+0000","last_active":"2024-09-16T10:34:53.219186+0000","last_peered":"2024-09-16T10:34:53.219186+0000","last_clean":"2024-09-16T10:34:53.219186+0000","last_became_active":"2024-09-16T10:34:18.158678+0000","last_became_peered":"2024-09-16T10:34:18.158678+0000","last_unstale":"2024-09-16T10:34:53.219186+0000","last_undegraded":"2024-09-16T10:34:53.219186+0000","last_fullsized":"2024-09-16T10:34:53.219186+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:13.682998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:13.682998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:20.249907+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1217,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222824+0000","last_change":"2024-09-16T10:34:31.395893+0000","last_active":"2024-09-16T10:34:52.222824+0000","last_peered":"2024-09-16T10:34:52.222824+0000","last_clean":"2024-09-16T10:34:52.222824+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:34:52.222824+0000","last_undegraded":"2024-09-16T10:34:52.222824+0000","last_fullsized":"2024-09-16T10:34:52.222824+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:31.395827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:31.395827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:45.172472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1145,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222532+0000","last_change":"2024-09-16T10:34:31.457282+0000","last_active":"2024-09-16T10:34:53.222532+0000","last_peered":"2024-09-16T10:34:53.222532+0000","last_clean":"2024-09-16T10:34:53.222532+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:34:53.222532+0000","last_undegraded":"2024-09-16T10:34:53.222532+0000","last_fullsized":"2024-09-16T10:34:53.222532+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:31.457232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:31.457232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:55.761379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":1094,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.220753+0000","last_change":"2024-09-16T10:34:35.428777+0000","last_active":"2024-09-16T10:34:52.220753+0000","last_peered":"2024-09-16T10:34:52.220753+0000","last_clean":"2024-09-16T10:34:52.220753+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:34:52.220753+0000","last_undegraded":"2024-09-16T10:34:52.220753+0000","last_fullsized":"2024-09-16T10:34:52.220753+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:35.428712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.428712+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.492709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"572'45","reported_seq":1204,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219867+0000","last_change":"2024-09-16T10:34:18.167545+0000","last_active":"2024-09-16T10:34:53.219867+0000","last_peered":"2024-09-16T10:34:53.219867+0000","last_clean":"2024-09-16T10:34:53.219867+0000","last_became_active":"2024-09-16T10:34:18.166916+0000","last_became_peered":"2024-09-16T10:34:18.166916+0000","last_unstale":"2024-09-16T10:34:53.219867+0000","last_undegraded":"2024-09-16T10:34:53.219867+0000","last_fullsized":"2024-09-16T10:34:53.219867+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'45","last_scrub_stamp":"2024-09-16T10:34:04.694713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:04.694713+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:27.009533+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1232,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334810+0000","last_change":"2024-09-16T10:34:33.460067+0000","last_active":"2024-09-16T10:34:53.334810+0000","last_peered":"2024-09-16T10:34:53.334810+0000","last_clean":"2024-09-16T10:34:53.334810+0000","last_became_active":"2024-09-16T10:33:49.209853+0000","last_became_peered":"2024-09-16T10:33:49.209853+0000","last_unstale":"2024-09-16T10:34:53.334810+0000","last_undegraded":"2024-09-16T10:34:53.334810+0000","last_fullsized":"2024-09-16T10:34:53.334810+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:33.460015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.460015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.224965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219113+0000","last_change":"2024-09-16T10:34:28.321982+0000","last_active":"2024-09-16T10:34:53.219113+0000","last_peered":"2024-09-16T10:34:53.219113+0000","last_clean":"2024-09-16T10:34:53.219113+0000","last_became_active":"2024-09-16T10:33:49.203145+0000","last_became_peered":"2024-09-16T10:33:49.203145+0000","last_unstale":"2024-09-16T10:34:53.219113+0000","last_undegraded":"2024-09-16T10:34:53.219113+0000","last_fullsized":"2024-09-16T10:34:53.219113+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:28.321919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:28.321919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:33.189197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":754,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345659+0000","last_change":"2024-09-16T10:34:47.877123+0000","last_active":"2024-09-16T10:34:53.345659+0000","last_peered":"2024-09-16T10:34:53.345659+0000","last_clean":"2024-09-16T10:34:53.345659+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:34:53.345659+0000","last_undegraded":"2024-09-16T10:34:53.345659+0000","last_fullsized":"2024-09-16T10:34:53.345659+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:47.877067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:47.877067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:06.730966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":835,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312845+0000","last_change":"2024-09-16T10:34:30.623332+0000","last_active":"2024-09-16T10:34:46.312845+0000","last_peered":"2024-09-16T10:34:46.312845+0000","last_clean":"2024-09-16T10:34:46.312845+0000","last_became_active":"2024-09-16T10:34:30.622459+0000","last_became_peered":"2024-09-16T10:34:30.622459+0000","last_unstale":"2024-09-16T10:34:46.312845+0000","last_undegraded":"2024-09-16T10:34:46.312845+0000","last_fullsized":"2024-09-16T10:34:46.312845+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:43.348315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:43.348315+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:04.288198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"572'30","reported_seq":1087,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219346+0000","last_change":"2024-09-16T10:34:22.226635+0000","last_active":"2024-09-16T10:34:53.219346+0000","last_peered":"2024-09-16T10:34:53.219346+0000","last_clean":"2024-09-16T10:34:53.219346+0000","last_became_active":"2024-09-16T10:34:18.158287+0000","last_became_peered":"2024-09-16T10:34:18.158287+0000","last_unstale":"2024-09-16T10:34:53.219346+0000","last_undegraded":"2024-09-16T10:34:53.219346+0000","last_fullsized":"2024-09-16T10:34:53.219346+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'30","last_scrub_stamp":"2024-09-16T10:34:22.226551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:22.226551+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:48.338811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1200,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221259+0000","last_change":"2024-09-16T10:34:42.399434+0000","last_active":"2024-09-16T10:34:53.221259+0000","last_peered":"2024-09-16T10:34:53.221259+0000","last_clean":"2024-09-16T10:34:53.221259+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:34:53.221259+0000","last_undegraded":"2024-09-16T10:34:53.221259+0000","last_fullsized":"2024-09-16T10:34:53.221259+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:42.399351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:42.399351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:02.919984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"579'14","reported_seq":940,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345519+0000","last_change":"2024-09-16T10:34:06.770354+0000","last_active":"2024-09-16T10:34:53.345519+0000","last_peered":"2024-09-16T10:34:53.345519+0000","last_clean":"2024-09-16T10:34:53.345519+0000","last_became_active":"2024-09-16T10:33:51.173662+0000","last_became_peered":"2024-09-16T10:33:51.173662+0000","last_unstale":"2024-09-16T10:34:53.345519+0000","last_undegraded":"2024-09-16T10:34:53.345519+0000","last_fullsized":"2024-09-16T10:34:53.345519+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'14","last_scrub_stamp":"2024-09-16T10:34:06.770304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:06.770304+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:15.467291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":782,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219700+0000","last_change":"2024-09-16T10:34:50.303284+0000","last_active":"2024-09-16T10:34:53.219700+0000","last_peered":"2024-09-16T10:34:53.219700+0000","last_clean":"2024-09-16T10:34:53.219700+0000","last_became_active":"2024-09-16T10:34:18.166006+0000","last_became_peered":"2024-09-16T10:34:18.166006+0000","last_unstale":"2024-09-16T10:34:53.219700+0000","last_undegraded":"2024-09-16T10:34:53.219700+0000","last_fullsized":"2024-09-16T10:34:53.219700+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:50.303228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:50.303228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:05.496420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"572'21","reported_seq":1268,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313616+0000","last_change":"2024-09-16T10:34:20.617016+0000","last_active":"2024-09-16T10:34:46.313616+0000","last_peered":"2024-09-16T10:34:46.313616+0000","last_clean":"2024-09-16T10:34:46.313616+0000","last_became_active":"2024-09-16T10:33:49.209734+0000","last_became_peered":"2024-09-16T10:33:49.209734+0000","last_unstale":"2024-09-16T10:34:46.313616+0000","last_undegraded":"2024-09-16T10:34:46.313616+0000","last_fullsized":"2024-09-16T10:34:46.313616+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:20.616963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:20.616963+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:27.454179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1225,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.224268+0000","last_change":"2024-09-16T10:34:26.348535+0000","last_active":"2024-09-16T10:34:52.224268+0000","last_peered":"2024-09-16T10:34:52.224268+0000","last_clean":"2024-09-16T10:34:52.224268+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:34:52.224268+0000","last_undegraded":"2024-09-16T10:34:52.224268+0000","last_fullsized":"2024-09-16T10:34:52.224268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:26.348476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:26.348476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:51.950897+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":746,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.337017+0000","last_change":"2024-09-16T10:34:23.742483+0000","last_active":"2024-09-16T10:34:53.337017+0000","last_peered":"2024-09-16T10:34:53.337017+0000","last_clean":"2024-09-16T10:34:53.337017+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:34:53.337017+0000","last_undegraded":"2024-09-16T10:34:53.337017+0000","last_fullsized":"2024-09-16T10:34:53.337017+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:23.742433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:23.742433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.639504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":1262,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312833+0000","last_change":"2024-09-16T10:33:51.186724+0000","last_active":"2024-09-16T10:34:46.312833+0000","last_peered":"2024-09-16T10:34:46.312833+0000","last_clean":"2024-09-16T10:34:46.312833+0000","last_became_active":"2024-09-16T10:33:51.168083+0000","last_became_peered":"2024-09-16T10:33:51.168083+0000","last_unstale":"2024-09-16T10:34:46.312833+0000","last_undegraded":"2024-09-16T10:34:46.312833+0000","last_fullsized":"2024-09-16T10:34:46.312833+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:33:43.392949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:43.392949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.703506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":78,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"572'51","reported_seq":1439,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334219+0000","last_change":"2024-09-16T10:34:13.164046+0000","last_active":"2024-09-16T10:34:53.334219+0000","last_peered":"2024-09-16T10:34:53.334219+0000","last_clean":"2024-09-16T10:34:53.334219+0000","last_became_active":"2024-09-16T10:34:13.152342+0000","last_became_peered":"2024-09-16T10:34:13.152342+0000","last_unstale":"2024-09-16T10:34:53.334219+0000","last_undegraded":"2024-09-16T10:34:53.334219+0000","last_fullsized":"2024-09-16T10:34:53.334219+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:33:42.596123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:42.596123+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.725971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":541,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313702+0000","last_change":"2024-09-16T10:34:30.620390+0000","last_active":"2024-09-16T10:34:46.313702+0000","last_peered":"2024-09-16T10:34:46.313702+0000","last_clean":"2024-09-16T10:34:46.313702+0000","last_became_active":"2024-09-16T10:34:30.618306+0000","last_became_peered":"2024-09-16T10:34:30.618306+0000","last_unstale":"2024-09-16T10:34:46.313702+0000","last_undegraded":"2024-09-16T10:34:46.313702+0000","last_fullsized":"2024-09-16T10:34:46.313702+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:11.351964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:11.351964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:21.162619+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":746,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.337168+0000","last_change":"2024-09-16T10:34:29.726300+0000","last_active":"2024-09-16T10:34:53.337168+0000","last_peered":"2024-09-16T10:34:53.337168+0000","last_clean":"2024-09-16T10:34:53.337168+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:34:53.337168+0000","last_undegraded":"2024-09-16T10:34:53.337168+0000","last_fullsized":"2024-09-16T10:34:53.337168+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:29.726249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:29.726249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:36.711014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1020,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312975+0000","last_change":"2024-09-16T10:34:30.621673+0000","last_active":"2024-09-16T10:34:46.312975+0000","last_peered":"2024-09-16T10:34:46.312975+0000","last_clean":"2024-09-16T10:34:46.312975+0000","last_became_active":"2024-09-16T10:34:30.621376+0000","last_became_peered":"2024-09-16T10:34:30.621376+0000","last_unstale":"2024-09-16T10:34:46.312975+0000","last_undegraded":"2024-09-16T10:34:46.312975+0000","last_fullsized":"2024-09-16T10:34:46.312975+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:00.709103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:00.709103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.393842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":1172,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.476369+0000","last_change":"2024-09-16T10:34:53.476369+0000","last_active":"2024-09-16T10:34:53.476369+0000","last_peered":"2024-09-16T10:34:53.476369+0000","last_clean":"2024-09-16T10:34:53.476369+0000","last_became_active":"2024-09-16T10:34:53.223094+0000","last_became_peered":"2024-09-16T10:34:53.223094+0000","last_unstale":"2024-09-16T10:34:53.476369+0000","last_undegraded":"2024-09-16T10:34:53.476369+0000","last_fullsized":"2024-09-16T10:34:53.476369+0000","mapping_epoch":630,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:53.476315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:53.476315+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:22.254895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"572'24","reported_seq":778,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218960+0000","last_change":"2024-09-16T10:34:18.164629+0000","last_active":"2024-09-16T10:34:53.218960+0000","last_peered":"2024-09-16T10:34:53.218960+0000","last_clean":"2024-09-16T10:34:53.218960+0000","last_became_active":"2024-09-16T10:34:18.164205+0000","last_became_peered":"2024-09-16T10:34:18.164205+0000","last_unstale":"2024-09-16T10:34:53.218960+0000","last_undegraded":"2024-09-16T10:34:53.218960+0000","last_fullsized":"2024-09-16T10:34:53.218960+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'24","last_scrub_stamp":"2024-09-16T10:34:12.388261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:12.388261+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.126870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":120,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313677+0000","last_change":"2024-09-16T10:34:30.615211+0000","last_active":"2024-09-16T10:34:46.313677+0000","last_peered":"2024-09-16T10:34:46.313677+0000","last_clean":"2024-09-16T10:34:46.313677+0000","last_became_active":"2024-09-16T10:34:30.614725+0000","last_became_peered":"2024-09-16T10:34:30.614725+0000","last_unstale":"2024-09-16T10:34:46.313677+0000","last_undegraded":"2024-09-16T10:34:46.313677+0000","last_fullsized":"2024-09-16T10:34:46.313677+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:47.490203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:47.490203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:10.886368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":768,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.336671+0000","last_change":"2024-09-16T10:34:33.676291+0000","last_active":"2024-09-16T10:34:53.336671+0000","last_peered":"2024-09-16T10:34:53.336671+0000","last_clean":"2024-09-16T10:34:53.336671+0000","last_became_active":"2024-09-16T10:33:49.222955+0000","last_became_peered":"2024-09-16T10:33:49.222955+0000","last_unstale":"2024-09-16T10:34:53.336671+0000","last_undegraded":"2024-09-16T10:34:53.336671+0000","last_fullsized":"2024-09-16T10:34:53.336671+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:33.676243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.676243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.581118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1243,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221078+0000","last_change":"2024-09-16T10:34:27.484125+0000","last_active":"2024-09-16T10:34:53.221078+0000","last_peered":"2024-09-16T10:34:53.221078+0000","last_clean":"2024-09-16T10:34:53.221078+0000","last_became_active":"2024-09-16T10:33:51.164602+0000","last_became_peered":"2024-09-16T10:33:51.164602+0000","last_unstale":"2024-09-16T10:34:53.221078+0000","last_undegraded":"2024-09-16T10:34:53.221078+0000","last_fullsized":"2024-09-16T10:34:53.221078+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:27.484068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:27.484068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:43.091905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":579,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219768+0000","last_change":"2024-09-16T10:34:18.159367+0000","last_active":"2024-09-16T10:34:53.219768+0000","last_peered":"2024-09-16T10:34:53.219768+0000","last_clean":"2024-09-16T10:34:53.219768+0000","last_became_active":"2024-09-16T10:34:18.158914+0000","last_became_peered":"2024-09-16T10:34:18.158914+0000","last_unstale":"2024-09-16T10:34:53.219768+0000","last_undegraded":"2024-09-16T10:34:53.219768+0000","last_fullsized":"2024-09-16T10:34:53.219768+0000","mapping_epoch":600,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:45.610690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:45.610690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:02.580406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"572'39","reported_seq":1315,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221147+0000","last_change":"2024-09-16T10:34:24.289425+0000","last_active":"2024-09-16T10:34:52.221147+0000","last_peered":"2024-09-16T10:34:52.221147+0000","last_clean":"2024-09-16T10:34:52.221147+0000","last_became_active":"2024-09-16T10:33:51.153340+0000","last_became_peered":"2024-09-16T10:33:51.153340+0000","last_unstale":"2024-09-16T10:34:52.221147+0000","last_undegraded":"2024-09-16T10:34:52.221147+0000","last_fullsized":"2024-09-16T10:34:52.221147+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:24.289354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:24.289354+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:24.336014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":521,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313776+0000","last_change":"2024-09-16T10:34:40.241419+0000","last_active":"2024-09-16T10:34:46.313776+0000","last_peered":"2024-09-16T10:34:46.313776+0000","last_clean":"2024-09-16T10:34:46.313776+0000","last_became_active":"2024-09-16T10:34:30.623804+0000","last_became_peered":"2024-09-16T10:34:30.623804+0000","last_unstale":"2024-09-16T10:34:46.313776+0000","last_undegraded":"2024-09-16T10:34:46.313776+0000","last_fullsized":"2024-09-16T10:34:46.313776+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:34:40.241364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:40.241364+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:53.306728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"614'16","reported_seq":714,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313009+0000","last_change":"2024-09-16T10:34:30.623738+0000","last_active":"2024-09-16T10:34:46.313009+0000","last_peered":"2024-09-16T10:34:46.313009+0000","last_clean":"2024-09-16T10:34:46.313009+0000","last_became_active":"2024-09-16T10:34:30.622941+0000","last_became_peered":"2024-09-16T10:34:30.622941+0000","last_unstale":"2024-09-16T10:34:46.313009+0000","last_undegraded":"2024-09-16T10:34:46.313009+0000","last_fullsized":"2024-09-16T10:34:46.313009+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'13","last_scrub_stamp":"2024-09-16T10:33:40.619814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:40.619814+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:54.354092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1221,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221915+0000","last_change":"2024-09-16T10:34:42.377708+0000","last_active":"2024-09-16T10:34:52.221915+0000","last_peered":"2024-09-16T10:34:52.221915+0000","last_clean":"2024-09-16T10:34:52.221915+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:34:52.221915+0000","last_undegraded":"2024-09-16T10:34:52.221915+0000","last_fullsized":"2024-09-16T10:34:52.221915+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:42.377654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:42.377654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:43.145201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1202,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.333936+0000","last_change":"2024-09-16T10:34:30.439376+0000","last_active":"2024-09-16T10:34:53.333936+0000","last_peered":"2024-09-16T10:34:53.333936+0000","last_clean":"2024-09-16T10:34:53.333936+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:34:53.333936+0000","last_undegraded":"2024-09-16T10:34:53.333936+0000","last_fullsized":"2024-09-16T10:34:53.333936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:30.439316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:30.439316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.929662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"572'18","reported_seq":1301,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222439+0000","last_change":"2024-09-16T10:34:36.433388+0000","last_active":"2024-09-16T10:34:53.222439+0000","last_peered":"2024-09-16T10:34:53.222439+0000","last_clean":"2024-09-16T10:34:53.222439+0000","last_became_active":"2024-09-16T10:33:51.164136+0000","last_became_peered":"2024-09-16T10:33:51.164136+0000","last_unstale":"2024-09-16T10:34:53.222439+0000","last_undegraded":"2024-09-16T10:34:53.222439+0000","last_fullsized":"2024-09-16T10:34:53.222439+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'18","last_scrub_stamp":"2024-09-16T10:34:36.433332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:36.433332+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.154198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1206,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222459+0000","last_change":"2024-09-16T10:33:51.357918+0000","last_active":"2024-09-16T10:34:52.222459+0000","last_peered":"2024-09-16T10:34:52.222459+0000","last_clean":"2024-09-16T10:34:52.222459+0000","last_became_active":"2024-09-16T10:33:49.212617+0000","last_became_peered":"2024-09-16T10:33:49.212617+0000","last_unstale":"2024-09-16T10:34:52.222459+0000","last_undegraded":"2024-09-16T10:34:52.222459+0000","last_fullsized":"2024-09-16T10:34:52.222459+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:51.357869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:51.357869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:19.915195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":119,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313703+0000","last_change":"2024-09-16T10:33:52.019767+0000","last_active":"2024-09-16T10:34:46.313703+0000","last_peered":"2024-09-16T10:34:46.313703+0000","last_clean":"2024-09-16T10:34:46.313703+0000","last_became_active":"2024-09-16T10:33:51.160714+0000","last_became_peered":"2024-09-16T10:33:51.160714+0000","last_unstale":"2024-09-16T10:34:46.313703+0000","last_undegraded":"2024-09-16T10:34:46.313703+0000","last_fullsized":"2024-09-16T10:34:46.313703+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:52.019715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:52.019715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.968879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"572'21","reported_seq":1278,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313898+0000","last_change":"2024-09-16T10:34:09.541086+0000","last_active":"2024-09-16T10:34:46.313898+0000","last_peered":"2024-09-16T10:34:46.313898+0000","last_clean":"2024-09-16T10:34:46.313898+0000","last_became_active":"2024-09-16T10:33:51.168349+0000","last_became_peered":"2024-09-16T10:33:51.168349+0000","last_unstale":"2024-09-16T10:34:46.313898+0000","last_undegraded":"2024-09-16T10:34:46.313898+0000","last_fullsized":"2024-09-16T10:34:46.313898+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:09.541018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:09.541018+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.580562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":491,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313133+0000","last_change":"2024-09-16T10:34:30.624438+0000","last_active":"2024-09-16T10:34:46.313133+0000","last_peered":"2024-09-16T10:34:46.313133+0000","last_clean":"2024-09-16T10:34:46.313133+0000","last_became_active":"2024-09-16T10:34:30.624091+0000","last_became_peered":"2024-09-16T10:34:30.624091+0000","last_unstale":"2024-09-16T10:34:46.313133+0000","last_undegraded":"2024-09-16T10:34:46.313133+0000","last_fullsized":"2024-09-16T10:34:46.313133+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:37.696727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:37.696727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:51.401956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1245,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221846+0000","last_change":"2024-09-16T10:33:57.367133+0000","last_active":"2024-09-16T10:34:53.221846+0000","last_peered":"2024-09-16T10:34:53.221846+0000","last_clean":"2024-09-16T10:34:53.221846+0000","last_became_active":"2024-09-16T10:33:51.163799+0000","last_became_peered":"2024-09-16T10:33:51.163799+0000","last_unstale":"2024-09-16T10:34:53.221846+0000","last_undegraded":"2024-09-16T10:34:53.221846+0000","last_fullsized":"2024-09-16T10:34:53.221846+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:57.367076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:57.367076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:58.674690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":771,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345502+0000","last_change":"2024-09-16T10:34:11.700727+0000","last_active":"2024-09-16T10:34:53.345502+0000","last_peered":"2024-09-16T10:34:53.345502+0000","last_clean":"2024-09-16T10:34:53.345502+0000","last_became_active":"2024-09-16T10:33:51.166834+0000","last_became_peered":"2024-09-16T10:33:51.166834+0000","last_unstale":"2024-09-16T10:34:53.345502+0000","last_undegraded":"2024-09-16T10:34:53.345502+0000","last_fullsized":"2024-09-16T10:34:53.345502+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:11.700675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:11.700675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:26.133615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"572'21","reported_seq":1275,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.223157+0000","last_change":"2024-09-16T10:34:33.400650+0000","last_active":"2024-09-16T10:34:52.223157+0000","last_peered":"2024-09-16T10:34:52.223157+0000","last_clean":"2024-09-16T10:34:52.223157+0000","last_became_active":"2024-09-16T10:33:51.162840+0000","last_became_peered":"2024-09-16T10:33:51.162840+0000","last_unstale":"2024-09-16T10:34:52.223157+0000","last_undegraded":"2024-09-16T10:34:52.223157+0000","last_fullsized":"2024-09-16T10:34:52.223157+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:33.400562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.400562+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.930557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1251,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:48.744945+0000","last_change":"2024-09-16T10:34:48.744945+0000","last_active":"2024-09-16T10:34:48.744945+0000","last_peered":"2024-09-16T10:34:48.744945+0000","last_clean":"2024-09-16T10:34:48.744945+0000","last_became_active":"2024-09-16T10:33:51.165860+0000","last_became_peered":"2024-09-16T10:33:51.165860+0000","last_unstale":"2024-09-16T10:34:48.744945+0000","last_undegraded":"2024-09-16T10:34:48.744945+0000","last_fullsized":"2024-09-16T10:34:48.744945+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:48.744896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:48.744896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:53.373279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"579'16","reported_seq":1560,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222255+0000","last_change":"2024-09-16T10:34:06.318061+0000","last_active":"2024-09-16T10:34:53.222255+0000","last_peered":"2024-09-16T10:34:53.222255+0000","last_clean":"2024-09-16T10:34:53.222255+0000","last_became_active":"2024-09-16T10:33:51.173279+0000","last_became_peered":"2024-09-16T10:33:51.173279+0000","last_unstale":"2024-09-16T10:34:53.222255+0000","last_undegraded":"2024-09-16T10:34:53.222255+0000","last_fullsized":"2024-09-16T10:34:53.222255+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'16","last_scrub_stamp":"2024-09-16T10:34:06.318015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:06.318015+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:08.843206+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":774,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219874+0000","last_change":"2024-09-16T10:34:18.167339+0000","last_active":"2024-09-16T10:34:53.219874+0000","last_peered":"2024-09-16T10:34:53.219874+0000","last_clean":"2024-09-16T10:34:53.219874+0000","last_became_active":"2024-09-16T10:34:18.166960+0000","last_became_peered":"2024-09-16T10:34:18.166960+0000","last_unstale":"2024-09-16T10:34:53.219874+0000","last_undegraded":"2024-09-16T10:34:53.219874+0000","last_fullsized":"2024-09-16T10:34:53.219874+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:15.676759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:15.676759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.640581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"572'30","reported_seq":1344,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221708+0000","last_change":"2024-09-16T10:33:51.193951+0000","last_active":"2024-09-16T10:34:53.221708+0000","last_peered":"2024-09-16T10:34:53.221708+0000","last_clean":"2024-09-16T10:34:53.221708+0000","last_became_active":"2024-09-16T10:33:51.169942+0000","last_became_peered":"2024-09-16T10:33:51.169942+0000","last_unstale":"2024-09-16T10:34:53.221708+0000","last_undegraded":"2024-09-16T10:34:53.221708+0000","last_fullsized":"2024-09-16T10:34:53.221708+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:05.157106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.157106+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":111,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313318+0000","last_change":"2024-09-16T10:33:51.149309+0000","last_active":"2024-09-16T10:34:46.313318+0000","last_peered":"2024-09-16T10:34:46.313318+0000","last_clean":"2024-09-16T10:34:46.313318+0000","last_became_active":"2024-09-16T10:33:51.149009+0000","last_became_peered":"2024-09-16T10:33:51.149009+0000","last_unstale":"2024-09-16T10:34:46.313318+0000","last_undegraded":"2024-09-16T10:34:46.313318+0000","last_fullsized":"2024-09-16T10:34:46.313318+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:38.539979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:38.539979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:53.181167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"579'10","reported_seq":1494,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222983+0000","last_change":"2024-09-16T10:34:23.288603+0000","last_active":"2024-09-16T10:34:52.222983+0000","last_peered":"2024-09-16T10:34:52.222983+0000","last_clean":"2024-09-16T10:34:52.222983+0000","last_became_active":"2024-09-16T10:33:49.212698+0000","last_became_peered":"2024-09-16T10:33:49.212698+0000","last_unstale":"2024-09-16T10:34:52.222983+0000","last_undegraded":"2024-09-16T10:34:52.222983+0000","last_fullsized":"2024-09-16T10:34:52.222983+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'10","last_scrub_stamp":"2024-09-16T10:34:23.288531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:23.288531+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:35.715819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1216,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313447+0000","last_change":"2024-09-16T10:34:41.788432+0000","last_active":"2024-09-16T10:34:46.313447+0000","last_peered":"2024-09-16T10:34:46.313447+0000","last_clean":"2024-09-16T10:34:46.313447+0000","last_became_active":"2024-09-16T10:33:51.169708+0000","last_became_peered":"2024-09-16T10:33:51.169708+0000","last_unstale":"2024-09-16T10:34:46.313447+0000","last_undegraded":"2024-09-16T10:34:46.313447+0000","last_fullsized":"2024-09-16T10:34:46.313447+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:41.788378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:41.788378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.506575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"572'27","reported_seq":1131,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218943+0000","last_change":"2024-09-16T10:34:37.231251+0000","last_active":"2024-09-16T10:34:53.218943+0000","last_peered":"2024-09-16T10:34:53.218943+0000","last_clean":"2024-09-16T10:34:53.218943+0000","last_became_active":"2024-09-16T10:34:18.163371+0000","last_became_peered":"2024-09-16T10:34:18.163371+0000","last_unstale":"2024-09-16T10:34:53.218943+0000","last_undegraded":"2024-09-16T10:34:53.218943+0000","last_fullsized":"2024-09-16T10:34:53.218943+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:34:37.231166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:37.231166+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:48.518957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":490,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219039+0000","last_change":"2024-09-16T10:34:18.161743+0000","last_active":"2024-09-16T10:34:53.219039+0000","last_peered":"2024-09-16T10:34:53.219039+0000","last_clean":"2024-09-16T10:34:53.219039+0000","last_became_active":"2024-09-16T10:34:18.161423+0000","last_became_peered":"2024-09-16T10:34:18.161423+0000","last_unstale":"2024-09-16T10:34:53.219039+0000","last_undegraded":"2024-09-16T10:34:53.219039+0000","last_fullsized":"2024-09-16T10:34:53.219039+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:57.000341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:57.000341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:18.974436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218700+0000","last_change":"2024-09-16T10:34:25.305006+0000","last_active":"2024-09-16T10:34:53.218700+0000","last_peered":"2024-09-16T10:34:53.218700+0000","last_clean":"2024-09-16T10:34:53.218700+0000","last_became_active":"2024-09-16T10:33:49.221969+0000","last_became_peered":"2024-09-16T10:33:49.221969+0000","last_unstale":"2024-09-16T10:34:53.218700+0000","last_undegraded":"2024-09-16T10:34:53.218700+0000","last_fullsized":"2024-09-16T10:34:53.218700+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:25.304959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:25.304959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:47.998074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1196,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220766+0000","last_change":"2024-09-16T10:34:49.459076+0000","last_active":"2024-09-16T10:34:53.220766+0000","last_peered":"2024-09-16T10:34:53.220766+0000","last_clean":"2024-09-16T10:34:53.220766+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:34:53.220766+0000","last_undegraded":"2024-09-16T10:34:53.220766+0000","last_fullsized":"2024-09-16T10:34:53.220766+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:49.459030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:49.459030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:56.485523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"572'21","reported_seq":140,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218593+0000","last_change":"2024-09-16T10:34:02.405828+0000","last_active":"2024-09-16T10:34:53.218593+0000","last_peered":"2024-09-16T10:34:53.218593+0000","last_clean":"2024-09-16T10:34:53.218593+0000","last_became_active":"2024-09-16T10:33:49.206963+0000","last_became_peered":"2024-09-16T10:33:49.206963+0000","last_unstale":"2024-09-16T10:34:53.218593+0000","last_undegraded":"2024-09-16T10:34:53.218593+0000","last_fullsized":"2024-09-16T10:34:53.218593+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:02.405764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:02.405764+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:13.723198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":548,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313572+0000","last_change":"2024-09-16T10:34:30.626188+0000","last_active":"2024-09-16T10:34:46.313572+0000","last_peered":"2024-09-16T10:34:46.313572+0000","last_clean":"2024-09-16T10:34:46.313572+0000","last_became_active":"2024-09-16T10:34:30.622358+0000","last_became_peered":"2024-09-16T10:34:30.622358+0000","last_unstale":"2024-09-16T10:34:46.313572+0000","last_undegraded":"2024-09-16T10:34:46.313572+0000","last_fullsized":"2024-09-16T10:34:46.313572+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:41.322772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:41.322772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1171,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221185+0000","last_change":"2024-09-16T10:33:49.330209+0000","last_active":"2024-09-16T10:34:53.221185+0000","last_peered":"2024-09-16T10:34:53.221185+0000","last_clean":"2024-09-16T10:34:53.221185+0000","last_became_active":"2024-09-16T10:33:49.228689+0000","last_became_peered":"2024-09-16T10:33:49.228689+0000","last_unstale":"2024-09-16T10:34:53.221185+0000","last_undegraded":"2024-09-16T10:34:53.221185+0000","last_fullsized":"2024-09-16T10:34:53.221185+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:49.330162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:49.330162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.043495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1190,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.312588+0000","last_change":"2024-09-16T10:34:25.653261+0000","last_active":"2024-09-16T10:34:46.312588+0000","last_peered":"2024-09-16T10:34:46.312588+0000","last_clean":"2024-09-16T10:34:46.312588+0000","last_became_active":"2024-09-16T10:33:49.210276+0000","last_became_peered":"2024-09-16T10:33:49.210276+0000","last_unstale":"2024-09-16T10:34:46.312588+0000","last_undegraded":"2024-09-16T10:34:46.312588+0000","last_fullsized":"2024-09-16T10:34:46.312588+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:25.653221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:25.653221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.988616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"572'27","reported_seq":1272,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220634+0000","last_change":"2024-09-16T10:33:51.297719+0000","last_active":"2024-09-16T10:34:53.220634+0000","last_peered":"2024-09-16T10:34:53.220634+0000","last_clean":"2024-09-16T10:34:53.220634+0000","last_became_active":"2024-09-16T10:33:51.163081+0000","last_became_peered":"2024-09-16T10:33:51.163081+0000","last_unstale":"2024-09-16T10:34:53.220634+0000","last_undegraded":"2024-09-16T10:34:53.220634+0000","last_fullsized":"2024-09-16T10:34:53.220634+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:33:51.297660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:51.297660+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:08.327471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1255,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222566+0000","last_change":"2024-09-16T10:34:25.316729+0000","last_active":"2024-09-16T10:34:52.222566+0000","last_peered":"2024-09-16T10:34:52.222566+0000","last_clean":"2024-09-16T10:34:52.222566+0000","last_became_active":"2024-09-16T10:33:51.152535+0000","last_became_peered":"2024-09-16T10:33:51.152535+0000","last_unstale":"2024-09-16T10:34:52.222566+0000","last_undegraded":"2024-09-16T10:34:52.222566+0000","last_fullsized":"2024-09-16T10:34:52.222566+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:25.316678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:25.316678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.411867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":119,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313048+0000","last_change":"2024-09-16T10:34:01.029861+0000","last_active":"2024-09-16T10:34:46.313048+0000","last_peered":"2024-09-16T10:34:46.313048+0000","last_clean":"2024-09-16T10:34:46.313048+0000","last_became_active":"2024-09-16T10:33:51.160356+0000","last_became_peered":"2024-09-16T10:33:51.160356+0000","last_unstale":"2024-09-16T10:34:46.313048+0000","last_undegraded":"2024-09-16T10:34:46.313048+0000","last_fullsized":"2024-09-16T10:34:46.313048+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:01.029789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:01.029789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:13.318818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1212,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221251+0000","last_change":"2024-09-16T10:34:13.319956+0000","last_active":"2024-09-16T10:34:53.221251+0000","last_peered":"2024-09-16T10:34:53.221251+0000","last_clean":"2024-09-16T10:34:53.221251+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:34:53.221251+0000","last_undegraded":"2024-09-16T10:34:53.221251+0000","last_fullsized":"2024-09-16T10:34:53.221251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:13.319902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:13.319902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:17.261606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"572'12","reported_seq":1287,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220063+0000","last_change":"2024-09-16T10:33:49.285315+0000","last_active":"2024-09-16T10:34:53.220063+0000","last_peered":"2024-09-16T10:34:53.220063+0000","last_clean":"2024-09-16T10:34:53.220063+0000","last_became_active":"2024-09-16T10:33:49.230962+0000","last_became_peered":"2024-09-16T10:33:49.230962+0000","last_unstale":"2024-09-16T10:34:53.220063+0000","last_undegraded":"2024-09-16T10:34:53.220063+0000","last_fullsized":"2024-09-16T10:34:53.220063+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:32:12.365003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.365003+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":1201,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313848+0000","last_change":"2024-09-16T10:34:16.559557+0000","last_active":"2024-09-16T10:34:46.313848+0000","last_peered":"2024-09-16T10:34:46.313848+0000","last_clean":"2024-09-16T10:34:46.313848+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:34:46.313848+0000","last_undegraded":"2024-09-16T10:34:46.313848+0000","last_fullsized":"2024-09-16T10:34:46.313848+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:34:16.559506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:16.559506+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:41.157665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1235,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334014+0000","last_change":"2024-09-16T10:34:05.544685+0000","last_active":"2024-09-16T10:34:53.334014+0000","last_peered":"2024-09-16T10:34:53.334014+0000","last_clean":"2024-09-16T10:34:53.334014+0000","last_became_active":"2024-09-16T10:33:51.165106+0000","last_became_peered":"2024-09-16T10:33:51.165106+0000","last_unstale":"2024-09-16T10:34:53.334014+0000","last_undegraded":"2024-09-16T10:34:53.334014+0000","last_fullsized":"2024-09-16T10:34:53.334014+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:05.544624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:05.544624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:12.872947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":782,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313055+0000","last_change":"2024-09-16T10:34:30.622092+0000","last_active":"2024-09-16T10:34:46.313055+0000","last_peered":"2024-09-16T10:34:46.313055+0000","last_clean":"2024-09-16T10:34:46.313055+0000","last_became_active":"2024-09-16T10:34:30.621716+0000","last_became_peered":"2024-09-16T10:34:30.621716+0000","last_unstale":"2024-09-16T10:34:46.313055+0000","last_undegraded":"2024-09-16T10:34:46.313055+0000","last_fullsized":"2024-09-16T10:34:46.313055+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:39.584681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:39.584681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:53.936722+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219487+0000","last_change":"2024-09-16T10:34:10.345817+0000","last_active":"2024-09-16T10:34:53.219487+0000","last_peered":"2024-09-16T10:34:53.219487+0000","last_clean":"2024-09-16T10:34:53.219487+0000","last_became_active":"2024-09-16T10:33:49.204475+0000","last_became_peered":"2024-09-16T10:33:49.204475+0000","last_unstale":"2024-09-16T10:34:53.219487+0000","last_undegraded":"2024-09-16T10:34:53.219487+0000","last_fullsized":"2024-09-16T10:34:53.219487+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:10.345755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:10.345755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:18.202511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1229,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220203+0000","last_change":"2024-09-16T10:34:45.433933+0000","last_active":"2024-09-16T10:34:53.220203+0000","last_peered":"2024-09-16T10:34:53.220203+0000","last_clean":"2024-09-16T10:34:53.220203+0000","last_became_active":"2024-09-16T10:33:49.228179+0000","last_became_peered":"2024-09-16T10:33:49.228179+0000","last_unstale":"2024-09-16T10:34:53.220203+0000","last_undegraded":"2024-09-16T10:34:53.220203+0000","last_fullsized":"2024-09-16T10:34:53.220203+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:45.433881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:45.433881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.452621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"572'27","reported_seq":1312,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.335208+0000","last_change":"2024-09-16T10:34:35.474513+0000","last_active":"2024-09-16T10:34:53.335208+0000","last_peered":"2024-09-16T10:34:53.335208+0000","last_clean":"2024-09-16T10:34:53.335208+0000","last_became_active":"2024-09-16T10:33:49.209690+0000","last_became_peered":"2024-09-16T10:33:49.209690+0000","last_unstale":"2024-09-16T10:34:53.335208+0000","last_undegraded":"2024-09-16T10:34:53.335208+0000","last_fullsized":"2024-09-16T10:34:53.335208+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:34:35.474452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.474452+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:43.555576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1250,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-16T10:34:46.313604+0000","last_change":"2024-09-16T10:34:32.708946+0000","last_active":"2024-09-16T10:34:46.313604+0000","last_peered":"2024-09-16T10:34:46.313604+0000","last_clean":"2024-09-16T10:34:46.313604+0000","last_became_active":"2024-09-16T10:33:51.163016+0000","last_became_peered":"2024-09-16T10:33:51.163016+0000","last_unstale":"2024-09-16T10:34:46.313604+0000","last_undegraded":"2024-09-16T10:34:46.313604+0000","last_fullsized":"2024-09-16T10:34:46.313604+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:32.708900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:32.708900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:42.253007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1526,"num_read_kb":1283,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":9330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":973,"ondisk_log_size":973,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":109,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":14,"stat_sum":{"num_bytes":603785,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1194153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1454080,"data_stored":1443424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":42,"acting":42,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":441,"seq":1894080577615,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250416,"kb_used_data":1176,"kb_used_omap":7,"kb_used_meta":249208,"kb_avail":93494736,"statfs":{"total":95995035648,"available":95738609664,"internally_reserved":0,"allocated":1204224,"data_stored":640979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8012,"internal_metadata":255189172},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528584,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834832,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92910320,"statfs":{"total":95995035648,"available":95140167680,"internally_reserved":0,"allocated":1232896,"data_stored":660497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":853604806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790222,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781256,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92963896,"statfs":{"total":95995035648,"available":95195029504,"internally_reserved":0,"allocated":1822720,"data_stored":1245507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986451,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686320,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93058832,"statfs":{"total":95995035648,"available":95292243968,"internally_reserved":0,"allocated":1265664,"data_stored":661538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215384,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917360,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92827792,"statfs":{"total":95995035648,"available":95055659008,"internally_reserved":0,"allocated":1855488,"data_stored":1252745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":937490888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411614,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":606036,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139116,"statfs":{"total":95995035648,"available":95374454784,"internally_reserved":0,"allocated":1503232,"data_stored":923456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575137,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862632,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92882520,"statfs":{"total":95995035648,"available":95111700480,"internally_reserved":0,"allocated":1265664,"data_stored":655348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738662,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938936,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806216,"statfs":{"total":95995035648,"available":95033565184,"internally_reserved":0,"allocated":1216512,"data_stored":641791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:55.981 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T10:34:56.246 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:56.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: osdmap e632: 8 total, 8 up, 8 in 2024-09-16T10:34:56.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: from='client.17169 v1:172.21.15.63:0/1632591296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:34:56.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: 4.1a scrub starts 2024-09-16T10:34:56.493 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: 4.1a scrub ok 2024-09-16T10:34:56.493 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: 6.12 scrub starts 2024-09-16T10:34:56.493 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: 6.12 scrub ok 2024-09-16T10:34:56.493 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: pgmap v1186: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-16T10:34:56.493 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[29993]: osdmap e633: 8 total, 8 up, 8 in 2024-09-16T10:34:56.494 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: osdmap e632: 8 total, 8 up, 8 in 2024-09-16T10:34:56.494 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: from='client.17169 v1:172.21.15.63:0/1632591296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:34:56.494 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: 4.1a scrub starts 2024-09-16T10:34:56.494 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: 4.1a scrub ok 2024-09-16T10:34:56.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: 6.12 scrub starts 2024-09-16T10:34:56.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: 6.12 scrub ok 2024-09-16T10:34:56.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: pgmap v1186: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-16T10:34:56.495 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:56 smithi063 ceph-mon[39470]: osdmap e633: 8 total, 8 up, 8 in 2024-09-16T10:34:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: osdmap e632: 8 total, 8 up, 8 in 2024-09-16T10:34:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: from='client.17169 v1:172.21.15.63:0/1632591296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:34:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: 4.1a scrub starts 2024-09-16T10:34:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: 4.1a scrub ok 2024-09-16T10:34:56.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: 6.12 scrub starts 2024-09-16T10:34:56.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: 6.12 scrub ok 2024-09-16T10:34:56.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: pgmap v1186: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-16T10:34:56.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:56 smithi165 ceph-mon[32696]: osdmap e633: 8 total, 8 up, 8 in 2024-09-16T10:34:56.905 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T10:34:56.907 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-16T10:34:57.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:57 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: ::ffff:172.21.15.165 - - [16/Sep/2024:10:34:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T10:34:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:57 smithi165 ceph-mon[32696]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:57.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:57 smithi165 ceph-mon[32696]: from='client.17175 v1:172.21.15.63:0/4171206559' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:34:57.652 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:57 smithi063 ceph-mon[29993]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:57.653 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:57 smithi063 ceph-mon[29993]: from='client.17175 v1:172.21.15.63:0/4171206559' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:34:57.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:57 smithi063 ceph-mon[39470]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-16T10:34:57.653 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:57 smithi063 ceph-mon[39470]: from='client.17175 v1:172.21.15.63:0/4171206559' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T10:34:57.688 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":1187,"stamp":"2024-09-16T10:34:56.155079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":609924,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1654,"num_read_kb":1409,"num_write":1010,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1206112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1322,"ondisk_log_size":1322,"up":435,"acting":435,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":438,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5878176,"kb_used_data":11492,"kb_used_omap":18,"kb_used_meta":5866477,"kb_avail":744083040,"statfs":{"total":767960285184,"available":761941032960,"internally_reserved":0,"allocated":11767808,"data_stored":7049893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19173,"internal_metadata":6007272731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.597854"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1233,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401685+0000","last_change":"2024-09-16T10:34:52.823129+0000","last_active":"2024-09-16T10:34:53.401685+0000","last_peered":"2024-09-16T10:34:53.401685+0000","last_clean":"2024-09-16T10:34:53.401685+0000","last_became_active":"2024-09-16T10:22:02.897117+0000","last_became_peered":"2024-09-16T10:22:02.897117+0000","last_unstale":"2024-09-16T10:34:53.401685+0000","last_undegraded":"2024-09-16T10:34:53.401685+0000","last_fullsized":"2024-09-16T10:34:53.401685+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:52.823073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:52.823073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:19.209899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"572'21","reported_seq":1157,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218668+0000","last_change":"2024-09-16T10:34:49.305043+0000","last_active":"2024-09-16T10:34:53.218668+0000","last_peered":"2024-09-16T10:34:53.218668+0000","last_clean":"2024-09-16T10:34:53.218668+0000","last_became_active":"2024-09-16T10:34:18.163340+0000","last_became_peered":"2024-09-16T10:34:18.163340+0000","last_unstale":"2024-09-16T10:34:53.218668+0000","last_undegraded":"2024-09-16T10:34:53.218668+0000","last_fullsized":"2024-09-16T10:34:53.218668+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:49.304961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:49.304961+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:12.506578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":137,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218231+0000","last_change":"2024-09-16T10:34:04.447540+0000","last_active":"2024-09-16T10:34:53.218231+0000","last_peered":"2024-09-16T10:34:53.218231+0000","last_clean":"2024-09-16T10:34:53.218231+0000","last_became_active":"2024-09-16T10:33:51.155115+0000","last_became_peered":"2024-09-16T10:33:51.155115+0000","last_unstale":"2024-09-16T10:34:53.218231+0000","last_undegraded":"2024-09-16T10:34:53.218231+0000","last_fullsized":"2024-09-16T10:34:53.218231+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:04.447488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:04.447488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:18.532117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":768,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218789+0000","last_change":"2024-09-16T10:34:18.164407+0000","last_active":"2024-09-16T10:34:53.218789+0000","last_peered":"2024-09-16T10:34:53.218789+0000","last_clean":"2024-09-16T10:34:53.218789+0000","last_became_active":"2024-09-16T10:34:18.164080+0000","last_became_peered":"2024-09-16T10:34:18.164080+0000","last_unstale":"2024-09-16T10:34:53.218789+0000","last_undegraded":"2024-09-16T10:34:53.218789+0000","last_fullsized":"2024-09-16T10:34:53.218789+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:00.353850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:00.353850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:22.497189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1226,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220898+0000","last_change":"2024-09-16T10:33:55.311542+0000","last_active":"2024-09-16T10:34:53.220898+0000","last_peered":"2024-09-16T10:34:53.220898+0000","last_clean":"2024-09-16T10:34:53.220898+0000","last_became_active":"2024-09-16T10:33:51.155507+0000","last_became_peered":"2024-09-16T10:33:51.155507+0000","last_unstale":"2024-09-16T10:34:53.220898+0000","last_undegraded":"2024-09-16T10:34:53.220898+0000","last_fullsized":"2024-09-16T10:34:53.220898+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:55.311492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:55.311492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.686414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1231,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222649+0000","last_change":"2024-09-16T10:34:17.428552+0000","last_active":"2024-09-16T10:34:53.222649+0000","last_peered":"2024-09-16T10:34:53.222649+0000","last_clean":"2024-09-16T10:34:53.222649+0000","last_became_active":"2024-09-16T10:22:02.904973+0000","last_became_peered":"2024-09-16T10:22:02.904973+0000","last_unstale":"2024-09-16T10:34:53.222649+0000","last_undegraded":"2024-09-16T10:34:53.222649+0000","last_fullsized":"2024-09-16T10:34:53.222649+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:17.428493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:17.428493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:46.401772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"572'39","reported_seq":1318,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220113+0000","last_change":"2024-09-16T10:34:04.336597+0000","last_active":"2024-09-16T10:34:53.220113+0000","last_peered":"2024-09-16T10:34:53.220113+0000","last_clean":"2024-09-16T10:34:53.220113+0000","last_became_active":"2024-09-16T10:22:02.904347+0000","last_became_peered":"2024-09-16T10:22:02.904347+0000","last_unstale":"2024-09-16T10:34:53.220113+0000","last_undegraded":"2024-09-16T10:34:53.220113+0000","last_fullsized":"2024-09-16T10:34:53.220113+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:04.336533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:04.336533+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:27.000063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":764,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218655+0000","last_change":"2024-09-16T10:34:45.286934+0000","last_active":"2024-09-16T10:34:53.218655+0000","last_peered":"2024-09-16T10:34:53.218655+0000","last_clean":"2024-09-16T10:34:53.218655+0000","last_became_active":"2024-09-16T10:34:18.163264+0000","last_became_peered":"2024-09-16T10:34:18.163264+0000","last_unstale":"2024-09-16T10:34:53.218655+0000","last_undegraded":"2024-09-16T10:34:53.218655+0000","last_fullsized":"2024-09-16T10:34:53.218655+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:45.286890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:45.286890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:47.033188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1240,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.223214+0000","last_change":"2024-09-16T10:34:41.401083+0000","last_active":"2024-09-16T10:34:52.223214+0000","last_peered":"2024-09-16T10:34:52.223214+0000","last_clean":"2024-09-16T10:34:52.223214+0000","last_became_active":"2024-09-16T10:13:37.092467+0000","last_became_peered":"2024-09-16T10:13:37.092467+0000","last_unstale":"2024-09-16T10:34:52.223214+0000","last_undegraded":"2024-09-16T10:34:52.223214+0000","last_fullsized":"2024-09-16T10:34:52.223214+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:41.401023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:41.401023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:51.226191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"572'21","reported_seq":1089,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219030+0000","last_change":"2024-09-16T10:34:18.161633+0000","last_active":"2024-09-16T10:34:53.219030+0000","last_peered":"2024-09-16T10:34:53.219030+0000","last_clean":"2024-09-16T10:34:53.219030+0000","last_became_active":"2024-09-16T10:34:18.161281+0000","last_became_peered":"2024-09-16T10:34:18.161281+0000","last_unstale":"2024-09-16T10:34:53.219030+0000","last_undegraded":"2024-09-16T10:34:53.219030+0000","last_fullsized":"2024-09-16T10:34:53.219030+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-09-16T10:33:37.280137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:37.280137+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1233,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222099+0000","last_change":"2024-09-16T10:34:10.336095+0000","last_active":"2024-09-16T10:34:52.222099+0000","last_peered":"2024-09-16T10:34:52.222099+0000","last_clean":"2024-09-16T10:34:52.222099+0000","last_became_active":"2024-09-16T10:33:51.158343+0000","last_became_peered":"2024-09-16T10:33:51.158343+0000","last_unstale":"2024-09-16T10:34:52.222099+0000","last_undegraded":"2024-09-16T10:34:52.222099+0000","last_fullsized":"2024-09-16T10:34:52.222099+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:10.336029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:10.336029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.507579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1244,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.400910+0000","last_change":"2024-09-16T10:34:38.765875+0000","last_active":"2024-09-16T10:34:53.400910+0000","last_peered":"2024-09-16T10:34:53.400910+0000","last_clean":"2024-09-16T10:34:53.400910+0000","last_became_active":"2024-09-16T10:33:51.159564+0000","last_became_peered":"2024-09-16T10:33:51.159564+0000","last_unstale":"2024-09-16T10:34:53.400910+0000","last_undegraded":"2024-09-16T10:34:53.400910+0000","last_fullsized":"2024-09-16T10:34:53.400910+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:38.765825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:38.765825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:03.193305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1198,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.224110+0000","last_change":"2024-09-16T10:34:02.378940+0000","last_active":"2024-09-16T10:34:52.224110+0000","last_peered":"2024-09-16T10:34:52.224110+0000","last_clean":"2024-09-16T10:34:52.224110+0000","last_became_active":"2024-09-16T10:33:49.209478+0000","last_became_peered":"2024-09-16T10:33:49.209478+0000","last_unstale":"2024-09-16T10:34:52.224110+0000","last_undegraded":"2024-09-16T10:34:52.224110+0000","last_fullsized":"2024-09-16T10:34:52.224110+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:02.378881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:02.378881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:08.620779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"572'9","reported_seq":1084,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218814+0000","last_change":"2024-09-16T10:34:35.192644+0000","last_active":"2024-09-16T10:34:53.218814+0000","last_peered":"2024-09-16T10:34:53.218814+0000","last_clean":"2024-09-16T10:34:53.218814+0000","last_became_active":"2024-09-16T10:34:18.166777+0000","last_became_peered":"2024-09-16T10:34:18.166777+0000","last_unstale":"2024-09-16T10:34:53.218814+0000","last_undegraded":"2024-09-16T10:34:53.218814+0000","last_fullsized":"2024-09-16T10:34:53.218814+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'9","last_scrub_stamp":"2024-09-16T10:34:35.192587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.192587+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:55.395098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":127,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344061+0000","last_change":"2024-09-16T10:33:56.019688+0000","last_active":"2024-09-16T10:34:55.344061+0000","last_peered":"2024-09-16T10:34:55.344061+0000","last_clean":"2024-09-16T10:34:55.344061+0000","last_became_active":"2024-09-16T10:33:51.158551+0000","last_became_peered":"2024-09-16T10:33:51.158551+0000","last_unstale":"2024-09-16T10:34:55.344061+0000","last_undegraded":"2024-09-16T10:34:55.344061+0000","last_fullsized":"2024-09-16T10:34:55.344061+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:56.019631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:56.019631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:10.047024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1233,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221571+0000","last_change":"2024-09-16T10:34:17.243583+0000","last_active":"2024-09-16T10:34:52.221571+0000","last_peered":"2024-09-16T10:34:52.221571+0000","last_clean":"2024-09-16T10:34:52.221571+0000","last_became_active":"2024-09-16T10:33:49.208831+0000","last_became_peered":"2024-09-16T10:33:49.208831+0000","last_unstale":"2024-09-16T10:34:52.221571+0000","last_undegraded":"2024-09-16T10:34:52.221571+0000","last_fullsized":"2024-09-16T10:34:52.221571+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:17.243521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:17.243521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:21.805421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":753,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219726+0000","last_change":"2024-09-16T10:34:36.192938+0000","last_active":"2024-09-16T10:34:53.219726+0000","last_peered":"2024-09-16T10:34:53.219726+0000","last_clean":"2024-09-16T10:34:53.219726+0000","last_became_active":"2024-09-16T10:34:18.162443+0000","last_became_peered":"2024-09-16T10:34:18.162443+0000","last_unstale":"2024-09-16T10:34:53.219726+0000","last_undegraded":"2024-09-16T10:34:53.219726+0000","last_fullsized":"2024-09-16T10:34:53.219726+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:36.192868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:36.192868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:37.230186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1240,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401240+0000","last_change":"2024-09-16T10:34:35.710276+0000","last_active":"2024-09-16T10:34:53.401240+0000","last_peered":"2024-09-16T10:34:53.401240+0000","last_clean":"2024-09-16T10:34:53.401240+0000","last_became_active":"2024-09-16T10:33:51.164632+0000","last_became_peered":"2024-09-16T10:33:51.164632+0000","last_unstale":"2024-09-16T10:34:53.401240+0000","last_undegraded":"2024-09-16T10:34:53.401240+0000","last_fullsized":"2024-09-16T10:34:53.401240+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:35.710223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.710223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:52.518359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"572'39","reported_seq":844,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.336674+0000","last_change":"2024-09-16T10:34:01.744988+0000","last_active":"2024-09-16T10:34:53.336674+0000","last_peered":"2024-09-16T10:34:53.336674+0000","last_clean":"2024-09-16T10:34:53.336674+0000","last_became_active":"2024-09-16T10:33:49.225899+0000","last_became_peered":"2024-09-16T10:33:49.225899+0000","last_unstale":"2024-09-16T10:34:53.336674+0000","last_undegraded":"2024-09-16T10:34:53.336674+0000","last_fullsized":"2024-09-16T10:34:53.336674+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:01.744914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:01.744914+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:20.032416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219574+0000","last_change":"2024-09-16T10:34:15.333753+0000","last_active":"2024-09-16T10:34:53.219574+0000","last_peered":"2024-09-16T10:34:53.219574+0000","last_clean":"2024-09-16T10:34:53.219574+0000","last_became_active":"2024-09-16T10:33:49.220841+0000","last_became_peered":"2024-09-16T10:33:49.220841+0000","last_unstale":"2024-09-16T10:34:53.219574+0000","last_undegraded":"2024-09-16T10:34:53.219574+0000","last_fullsized":"2024-09-16T10:34:53.219574+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:15.333702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:15.333702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:40.148695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1153,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221657+0000","last_change":"2024-09-16T10:33:08.153114+0000","last_active":"2024-09-16T10:34:53.221657+0000","last_peered":"2024-09-16T10:34:53.221657+0000","last_clean":"2024-09-16T10:34:53.221657+0000","last_became_active":"2024-09-16T10:18:33.824123+0000","last_became_peered":"2024-09-16T10:18:33.824123+0000","last_unstale":"2024-09-16T10:34:53.221657+0000","last_undegraded":"2024-09-16T10:34:53.221657+0000","last_fullsized":"2024-09-16T10:34:53.221657+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:08.153044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:08.153044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":778,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219202+0000","last_change":"2024-09-16T10:34:38.229723+0000","last_active":"2024-09-16T10:34:53.219202+0000","last_peered":"2024-09-16T10:34:53.219202+0000","last_clean":"2024-09-16T10:34:53.219202+0000","last_became_active":"2024-09-16T10:34:18.163325+0000","last_became_peered":"2024-09-16T10:34:18.163325+0000","last_unstale":"2024-09-16T10:34:53.219202+0000","last_undegraded":"2024-09-16T10:34:53.219202+0000","last_fullsized":"2024-09-16T10:34:53.219202+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:38.229668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:38.229668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:51.059958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"572'24","reported_seq":1336,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221749+0000","last_change":"2024-09-16T10:34:20.444837+0000","last_active":"2024-09-16T10:34:53.221749+0000","last_peered":"2024-09-16T10:34:53.221749+0000","last_clean":"2024-09-16T10:34:53.221749+0000","last_became_active":"2024-09-16T10:34:13.154733+0000","last_became_peered":"2024-09-16T10:34:13.154733+0000","last_unstale":"2024-09-16T10:34:53.221749+0000","last_undegraded":"2024-09-16T10:34:53.221749+0000","last_fullsized":"2024-09-16T10:34:53.221749+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'24","last_scrub_stamp":"2024-09-16T10:34:20.444765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:20.444765+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:45.950400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":786,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219248+0000","last_change":"2024-09-16T10:34:18.166916+0000","last_active":"2024-09-16T10:34:53.219248+0000","last_peered":"2024-09-16T10:34:53.219248+0000","last_clean":"2024-09-16T10:34:53.219248+0000","last_became_active":"2024-09-16T10:34:18.166206+0000","last_became_peered":"2024-09-16T10:34:18.166206+0000","last_unstale":"2024-09-16T10:34:53.219248+0000","last_undegraded":"2024-09-16T10:34:53.219248+0000","last_fullsized":"2024-09-16T10:34:53.219248+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:48.368758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:48.368758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.320254+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":566,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344443+0000","last_change":"2024-09-16T10:34:30.620887+0000","last_active":"2024-09-16T10:34:55.344443+0000","last_peered":"2024-09-16T10:34:55.344443+0000","last_clean":"2024-09-16T10:34:55.344443+0000","last_became_active":"2024-09-16T10:34:30.620547+0000","last_became_peered":"2024-09-16T10:34:30.620547+0000","last_unstale":"2024-09-16T10:34:55.344443+0000","last_undegraded":"2024-09-16T10:34:55.344443+0000","last_fullsized":"2024-09-16T10:34:55.344443+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:34:16.203535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:16.203535+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:37.486266+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'2","reported_seq":128,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218388+0000","last_change":"2024-09-16T10:33:49.240616+0000","last_active":"2024-09-16T10:34:53.218388+0000","last_peered":"2024-09-16T10:34:53.218388+0000","last_clean":"2024-09-16T10:34:53.218388+0000","last_became_active":"2024-09-16T10:33:49.220080+0000","last_became_peered":"2024-09-16T10:33:49.220080+0000","last_unstale":"2024-09-16T10:34:53.218388+0000","last_undegraded":"2024-09-16T10:34:53.218388+0000","last_fullsized":"2024-09-16T10:34:53.218388+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:33:44.365752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:44.365752+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.051928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"572'30","reported_seq":1339,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401586+0000","last_change":"2024-09-16T10:33:51.202586+0000","last_active":"2024-09-16T10:34:53.401586+0000","last_peered":"2024-09-16T10:34:53.401586+0000","last_clean":"2024-09-16T10:34:53.401586+0000","last_became_active":"2024-09-16T10:33:51.169227+0000","last_became_peered":"2024-09-16T10:33:51.169227+0000","last_unstale":"2024-09-16T10:34:53.401586+0000","last_undegraded":"2024-09-16T10:34:53.401586+0000","last_fullsized":"2024-09-16T10:34:53.401586+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:14.278843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:14.278843+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"579'12","reported_seq":1505,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.335205+0000","last_change":"2024-09-16T10:33:51.184591+0000","last_active":"2024-09-16T10:34:53.335205+0000","last_peered":"2024-09-16T10:34:53.335205+0000","last_clean":"2024-09-16T10:34:53.335205+0000","last_became_active":"2024-09-16T10:33:51.164458+0000","last_became_peered":"2024-09-16T10:33:51.164458+0000","last_unstale":"2024-09-16T10:34:53.335205+0000","last_undegraded":"2024-09-16T10:34:53.335205+0000","last_fullsized":"2024-09-16T10:34:53.335205+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'11","last_scrub_stamp":"2024-09-16T10:33:48.605415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:48.605415+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.867359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":1079,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221737+0000","last_change":"2024-09-16T10:34:28.375764+0000","last_active":"2024-09-16T10:34:52.221737+0000","last_peered":"2024-09-16T10:34:52.221737+0000","last_clean":"2024-09-16T10:34:52.221737+0000","last_became_active":"2024-09-16T10:22:02.897882+0000","last_became_peered":"2024-09-16T10:22:02.897882+0000","last_unstale":"2024-09-16T10:34:52.221737+0000","last_undegraded":"2024-09-16T10:34:52.221737+0000","last_fullsized":"2024-09-16T10:34:52.221737+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:28.375686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:28.375686+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:35.391649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"572'39","reported_seq":146,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221257+0000","last_change":"2024-09-16T10:34:29.319100+0000","last_active":"2024-09-16T10:34:53.221257+0000","last_peered":"2024-09-16T10:34:53.221257+0000","last_clean":"2024-09-16T10:34:53.221257+0000","last_became_active":"2024-09-16T10:33:49.232682+0000","last_became_peered":"2024-09-16T10:33:49.232682+0000","last_unstale":"2024-09-16T10:34:53.221257+0000","last_undegraded":"2024-09-16T10:34:53.221257+0000","last_fullsized":"2024-09-16T10:34:53.221257+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:29.319043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:29.319043+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:59.044672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1255,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220292+0000","last_change":"2024-09-16T10:34:34.390450+0000","last_active":"2024-09-16T10:34:53.220292+0000","last_peered":"2024-09-16T10:34:53.220292+0000","last_clean":"2024-09-16T10:34:53.220292+0000","last_became_active":"2024-09-16T10:33:51.165462+0000","last_became_peered":"2024-09-16T10:33:51.165462+0000","last_unstale":"2024-09-16T10:34:53.220292+0000","last_undegraded":"2024-09-16T10:34:53.220292+0000","last_fullsized":"2024-09-16T10:34:53.220292+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:34.390353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:34.390353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.333117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"601'10","reported_seq":935,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219825+0000","last_change":"2024-09-16T10:34:32.224690+0000","last_active":"2024-09-16T10:34:53.219825+0000","last_peered":"2024-09-16T10:34:53.219825+0000","last_clean":"2024-09-16T10:34:53.219825+0000","last_became_active":"2024-09-16T10:34:18.162934+0000","last_became_peered":"2024-09-16T10:34:18.162934+0000","last_unstale":"2024-09-16T10:34:53.219825+0000","last_undegraded":"2024-09-16T10:34:53.219825+0000","last_fullsized":"2024-09-16T10:34:53.219825+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'10","last_scrub_stamp":"2024-09-16T10:34:32.224631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:32.224631+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:58.452668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1219,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222861+0000","last_change":"2024-09-16T10:33:49.213894+0000","last_active":"2024-09-16T10:34:52.222861+0000","last_peered":"2024-09-16T10:34:52.222861+0000","last_clean":"2024-09-16T10:34:52.222861+0000","last_became_active":"2024-09-16T10:33:49.213427+0000","last_became_peered":"2024-09-16T10:33:49.213427+0000","last_unstale":"2024-09-16T10:34:52.222861+0000","last_undegraded":"2024-09-16T10:34:52.222861+0000","last_fullsized":"2024-09-16T10:34:52.222861+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:42.455925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:42.455925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:11.585766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":160,"reported_epoch":631,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T10:34:53.221365+0000","last_change":"2024-09-16T10:34:46.315733+0000","last_active":"2024-09-16T10:34:53.221365+0000","last_peered":"2024-09-16T10:34:53.221365+0000","last_clean":"2024-09-16T10:14:09.463956+0000","last_became_active":"2024-09-16T10:34:46.314709+0000","last_became_peered":"2024-09-16T10:34:46.314709+0000","last_unstale":"2024-09-16T10:34:53.221365+0000","last_undegraded":"2024-09-16T10:34:53.221365+0000","last_fullsized":"2024-09-16T10:34:53.221365+0000","mapping_epoch":627,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:54.497762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:54.497762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[3,6,5],"avail_no_missing":["3","2","4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"572'42","reported_seq":570,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218534+0000","last_change":"2024-09-16T10:34:52.355681+0000","last_active":"2024-09-16T10:34:53.218534+0000","last_peered":"2024-09-16T10:34:53.218534+0000","last_clean":"2024-09-16T10:34:53.218534+0000","last_became_active":"2024-09-16T10:34:18.159011+0000","last_became_peered":"2024-09-16T10:34:18.159011+0000","last_unstale":"2024-09-16T10:34:53.218534+0000","last_undegraded":"2024-09-16T10:34:53.218534+0000","last_fullsized":"2024-09-16T10:34:53.218534+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'42","last_scrub_stamp":"2024-09-16T10:34:52.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:52.355594+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:20.728319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1206,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222590+0000","last_change":"2024-09-16T10:34:40.396096+0000","last_active":"2024-09-16T10:34:53.222590+0000","last_peered":"2024-09-16T10:34:53.222590+0000","last_clean":"2024-09-16T10:34:53.222590+0000","last_became_active":"2024-09-16T10:33:51.157669+0000","last_became_peered":"2024-09-16T10:33:51.157669+0000","last_unstale":"2024-09-16T10:34:53.222590+0000","last_undegraded":"2024-09-16T10:34:53.222590+0000","last_fullsized":"2024-09-16T10:34:53.222590+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:40.396048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:40.396048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:40.644426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":974,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.333840+0000","last_change":"2024-09-16T10:34:24.433929+0000","last_active":"2024-09-16T10:34:53.333840+0000","last_peered":"2024-09-16T10:34:53.333840+0000","last_clean":"2024-09-16T10:34:53.333840+0000","last_became_active":"2024-09-16T10:33:51.166245+0000","last_became_peered":"2024-09-16T10:33:51.166245+0000","last_unstale":"2024-09-16T10:34:53.333840+0000","last_undegraded":"2024-09-16T10:34:53.333840+0000","last_fullsized":"2024-09-16T10:34:53.333840+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:24.433865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:24.433865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.653058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":775,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218985+0000","last_change":"2024-09-16T10:34:18.157940+0000","last_active":"2024-09-16T10:34:53.218985+0000","last_peered":"2024-09-16T10:34:53.218985+0000","last_clean":"2024-09-16T10:34:53.218985+0000","last_became_active":"2024-09-16T10:34:18.157585+0000","last_became_peered":"2024-09-16T10:34:18.157585+0000","last_unstale":"2024-09-16T10:34:53.218985+0000","last_undegraded":"2024-09-16T10:34:53.218985+0000","last_fullsized":"2024-09-16T10:34:53.218985+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:53.331351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:53.331351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:09.546382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":1180,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222403+0000","last_change":"2024-09-16T10:34:14.356079+0000","last_active":"2024-09-16T10:34:53.222403+0000","last_peered":"2024-09-16T10:34:53.222403+0000","last_clean":"2024-09-16T10:34:53.222403+0000","last_became_active":"2024-09-16T10:34:03.470999+0000","last_became_peered":"2024-09-16T10:34:03.470999+0000","last_unstale":"2024-09-16T10:34:53.222403+0000","last_undegraded":"2024-09-16T10:34:53.222403+0000","last_fullsized":"2024-09-16T10:34:53.222403+0000","mapping_epoch":580,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":588,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:14.356030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:14.356030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":74,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.832276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"572'42","reported_seq":1129,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218709+0000","last_change":"2024-09-16T10:34:28.282974+0000","last_active":"2024-09-16T10:34:53.218709+0000","last_peered":"2024-09-16T10:34:53.218709+0000","last_clean":"2024-09-16T10:34:53.218709+0000","last_became_active":"2024-09-16T10:34:18.162427+0000","last_became_peered":"2024-09-16T10:34:18.162427+0000","last_unstale":"2024-09-16T10:34:53.218709+0000","last_undegraded":"2024-09-16T10:34:53.218709+0000","last_fullsized":"2024-09-16T10:34:53.218709+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'42","last_scrub_stamp":"2024-09-16T10:34:28.282890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:28.282890+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.624869+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":1225,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401776+0000","last_change":"2024-09-16T10:33:58.447082+0000","last_active":"2024-09-16T10:34:53.401776+0000","last_peered":"2024-09-16T10:34:53.401776+0000","last_clean":"2024-09-16T10:34:53.401776+0000","last_became_active":"2024-09-16T10:33:49.211982+0000","last_became_peered":"2024-09-16T10:33:49.211982+0000","last_unstale":"2024-09-16T10:34:53.401776+0000","last_undegraded":"2024-09-16T10:34:53.401776+0000","last_fullsized":"2024-09-16T10:34:53.401776+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:33:58.447024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:58.447024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:11.900859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"579'10","reported_seq":1490,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219969+0000","last_change":"2024-09-16T10:33:49.280956+0000","last_active":"2024-09-16T10:34:53.219969+0000","last_peered":"2024-09-16T10:34:53.219969+0000","last_clean":"2024-09-16T10:34:53.219969+0000","last_became_active":"2024-09-16T10:33:49.229244+0000","last_became_peered":"2024-09-16T10:33:49.229244+0000","last_unstale":"2024-09-16T10:34:53.219969+0000","last_undegraded":"2024-09-16T10:34:53.219969+0000","last_fullsized":"2024-09-16T10:34:53.219969+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'9","last_scrub_stamp":"2024-09-16T10:33:15.201123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:15.201123+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1025,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343930+0000","last_change":"2024-09-16T10:34:30.621896+0000","last_active":"2024-09-16T10:34:55.343930+0000","last_peered":"2024-09-16T10:34:55.343930+0000","last_clean":"2024-09-16T10:34:55.343930+0000","last_became_active":"2024-09-16T10:34:30.621575+0000","last_became_peered":"2024-09-16T10:34:30.621575+0000","last_unstale":"2024-09-16T10:34:55.343930+0000","last_undegraded":"2024-09-16T10:34:55.343930+0000","last_fullsized":"2024-09-16T10:34:55.343930+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:19.550389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:19.550389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.041848+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":1238,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.335093+0000","last_change":"2024-09-16T10:34:39.212462+0000","last_active":"2024-09-16T10:34:53.335093+0000","last_peered":"2024-09-16T10:34:53.335093+0000","last_clean":"2024-09-16T10:34:53.335093+0000","last_became_active":"2024-09-16T10:34:39.212049+0000","last_became_peered":"2024-09-16T10:34:39.212049+0000","last_unstale":"2024-09-16T10:34:53.335093+0000","last_undegraded":"2024-09-16T10:34:53.335093+0000","last_fullsized":"2024-09-16T10:34:53.335093+0000","mapping_epoch":580,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":621,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:55.533227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:55.533227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:56.206666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"572'30","reported_seq":142,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218707+0000","last_change":"2024-09-16T10:34:00.453400+0000","last_active":"2024-09-16T10:34:53.218707+0000","last_peered":"2024-09-16T10:34:53.218707+0000","last_clean":"2024-09-16T10:34:53.218707+0000","last_became_active":"2024-09-16T10:33:49.208703+0000","last_became_peered":"2024-09-16T10:33:49.208703+0000","last_unstale":"2024-09-16T10:34:53.218707+0000","last_undegraded":"2024-09-16T10:34:53.218707+0000","last_fullsized":"2024-09-16T10:34:53.218707+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'30","last_scrub_stamp":"2024-09-16T10:34:00.453298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:00.453298+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:17.878799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1232,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.223276+0000","last_change":"2024-09-16T10:34:30.439919+0000","last_active":"2024-09-16T10:34:52.223276+0000","last_peered":"2024-09-16T10:34:52.223276+0000","last_clean":"2024-09-16T10:34:52.223276+0000","last_became_active":"2024-09-16T10:22:02.897809+0000","last_became_peered":"2024-09-16T10:22:02.897809+0000","last_unstale":"2024-09-16T10:34:52.223276+0000","last_undegraded":"2024-09-16T10:34:52.223276+0000","last_fullsized":"2024-09-16T10:34:52.223276+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:30.439845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:30.439845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:46.644887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1459,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401460+0000","last_change":"2024-09-16T10:33:42.380585+0000","last_active":"2024-09-16T10:34:53.401460+0000","last_peered":"2024-09-16T10:34:53.401460+0000","last_clean":"2024-09-16T10:34:53.401460+0000","last_became_active":"2024-09-16T10:13:41.114836+0000","last_became_peered":"2024-09-16T10:13:41.114836+0000","last_unstale":"2024-09-16T10:34:53.401460+0000","last_undegraded":"2024-09-16T10:34:53.401460+0000","last_fullsized":"2024-09-16T10:34:53.401460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-16T10:33:42.380504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:42.380504+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:54.620204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1221,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401009+0000","last_change":"2024-09-16T10:34:31.727025+0000","last_active":"2024-09-16T10:34:53.401009+0000","last_peered":"2024-09-16T10:34:53.401009+0000","last_clean":"2024-09-16T10:34:53.401009+0000","last_became_active":"2024-09-16T10:22:02.904460+0000","last_became_peered":"2024-09-16T10:22:02.904460+0000","last_unstale":"2024-09-16T10:34:53.401009+0000","last_undegraded":"2024-09-16T10:34:53.401009+0000","last_fullsized":"2024-09-16T10:34:53.401009+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:31.726979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:31.726979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:46.526221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":1199,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334882+0000","last_change":"2024-09-16T10:34:43.178195+0000","last_active":"2024-09-16T10:34:53.334882+0000","last_peered":"2024-09-16T10:34:53.334882+0000","last_clean":"2024-09-16T10:34:53.334882+0000","last_became_active":"2024-09-16T10:34:43.177899+0000","last_became_peered":"2024-09-16T10:34:43.177899+0000","last_unstale":"2024-09-16T10:34:53.334882+0000","last_undegraded":"2024-09-16T10:34:53.334882+0000","last_fullsized":"2024-09-16T10:34:53.334882+0000","mapping_epoch":156,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:27.498031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:27.498031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.199776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"572'39","reported_seq":1392,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401358+0000","last_change":"2024-09-16T10:34:47.707078+0000","last_active":"2024-09-16T10:34:53.401358+0000","last_peered":"2024-09-16T10:34:53.401358+0000","last_clean":"2024-09-16T10:34:53.401358+0000","last_became_active":"2024-09-16T10:33:51.164128+0000","last_became_peered":"2024-09-16T10:33:51.164128+0000","last_unstale":"2024-09-16T10:34:53.401358+0000","last_undegraded":"2024-09-16T10:34:53.401358+0000","last_fullsized":"2024-09-16T10:34:53.401358+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:47.706982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:47.706982+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:55.643555+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219174+0000","last_change":"2024-09-16T10:33:59.402162+0000","last_active":"2024-09-16T10:34:53.219174+0000","last_peered":"2024-09-16T10:34:53.219174+0000","last_clean":"2024-09-16T10:34:53.219174+0000","last_became_active":"2024-09-16T10:33:49.202668+0000","last_became_peered":"2024-09-16T10:33:49.202668+0000","last_unstale":"2024-09-16T10:34:53.219174+0000","last_undegraded":"2024-09-16T10:34:53.219174+0000","last_fullsized":"2024-09-16T10:34:53.219174+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:59.402107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:59.402107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.409662+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1219,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334009+0000","last_change":"2024-09-16T10:33:36.663621+0000","last_active":"2024-09-16T10:34:53.334009+0000","last_peered":"2024-09-16T10:34:53.334009+0000","last_clean":"2024-09-16T10:34:53.334009+0000","last_became_active":"2024-09-16T10:13:41.114431+0000","last_became_peered":"2024-09-16T10:13:41.114431+0000","last_unstale":"2024-09-16T10:34:53.334009+0000","last_undegraded":"2024-09-16T10:34:53.334009+0000","last_fullsized":"2024-09-16T10:34:53.334009+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:36.663561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:36.663561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.772506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218213+0000","last_change":"2024-09-16T10:33:49.199768+0000","last_active":"2024-09-16T10:34:53.218213+0000","last_peered":"2024-09-16T10:34:53.218213+0000","last_clean":"2024-09-16T10:34:53.218213+0000","last_became_active":"2024-09-16T10:33:49.198662+0000","last_became_peered":"2024-09-16T10:33:49.198662+0000","last_unstale":"2024-09-16T10:34:53.218213+0000","last_undegraded":"2024-09-16T10:34:53.218213+0000","last_fullsized":"2024-09-16T10:34:53.218213+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:47.582751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:47.582751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:12.837063+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":1135,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221285+0000","last_change":"2024-09-16T10:34:32.161884+0000","last_active":"2024-09-16T10:34:52.221285+0000","last_peered":"2024-09-16T10:34:52.221285+0000","last_clean":"2024-09-16T10:34:52.221285+0000","last_became_active":"2024-09-16T10:34:32.161436+0000","last_became_peered":"2024-09-16T10:34:32.161436+0000","last_unstale":"2024-09-16T10:34:52.221285+0000","last_undegraded":"2024-09-16T10:34:52.221285+0000","last_fullsized":"2024-09-16T10:34:52.221285+0000","mapping_epoch":284,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":614,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:13.269281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:13.269281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:30.798554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"579'56","reported_seq":165,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218994+0000","last_change":"2024-09-16T10:34:24.306070+0000","last_active":"2024-09-16T10:34:53.218994+0000","last_peered":"2024-09-16T10:34:53.218994+0000","last_clean":"2024-09-16T10:34:53.218994+0000","last_became_active":"2024-09-16T10:33:49.222165+0000","last_became_peered":"2024-09-16T10:33:49.222165+0000","last_unstale":"2024-09-16T10:34:53.218994+0000","last_undegraded":"2024-09-16T10:34:53.218994+0000","last_fullsized":"2024-09-16T10:34:53.218994+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'56","last_scrub_stamp":"2024-09-16T10:34:24.305984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:24.305984+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:32.914469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":773,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219238+0000","last_change":"2024-09-16T10:34:18.161353+0000","last_active":"2024-09-16T10:34:53.219238+0000","last_peered":"2024-09-16T10:34:53.219238+0000","last_clean":"2024-09-16T10:34:53.219238+0000","last_became_active":"2024-09-16T10:34:18.161058+0000","last_became_peered":"2024-09-16T10:34:18.161058+0000","last_unstale":"2024-09-16T10:34:53.219238+0000","last_undegraded":"2024-09-16T10:34:53.219238+0000","last_fullsized":"2024-09-16T10:34:53.219238+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:53.495088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:53.495088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:55.014885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":1250,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222279+0000","last_change":"2024-09-16T10:33:53.317359+0000","last_active":"2024-09-16T10:34:52.222279+0000","last_peered":"2024-09-16T10:34:52.222279+0000","last_clean":"2024-09-16T10:34:52.222279+0000","last_became_active":"2024-09-16T10:33:49.213116+0000","last_became_peered":"2024-09-16T10:33:49.213116+0000","last_unstale":"2024-09-16T10:34:52.222279+0000","last_undegraded":"2024-09-16T10:34:52.222279+0000","last_fullsized":"2024-09-16T10:34:52.222279+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-16T10:33:53.317300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:33:53.317300+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.362730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001358497,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1209,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401422+0000","last_change":"2024-09-16T10:34:34.681162+0000","last_active":"2024-09-16T10:34:53.401422+0000","last_peered":"2024-09-16T10:34:53.401422+0000","last_clean":"2024-09-16T10:34:53.401422+0000","last_became_active":"2024-09-16T10:33:49.210355+0000","last_became_peered":"2024-09-16T10:33:49.210355+0000","last_unstale":"2024-09-16T10:34:53.401422+0000","last_undegraded":"2024-09-16T10:34:53.401422+0000","last_fullsized":"2024-09-16T10:34:53.401422+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:34.681117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:34.681117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:50.145347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1237,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222278+0000","last_change":"2024-09-16T10:33:51.156938+0000","last_active":"2024-09-16T10:34:53.222278+0000","last_peered":"2024-09-16T10:34:53.222278+0000","last_clean":"2024-09-16T10:34:53.222278+0000","last_became_active":"2024-09-16T10:33:51.152721+0000","last_became_peered":"2024-09-16T10:33:51.152721+0000","last_unstale":"2024-09-16T10:34:53.222278+0000","last_undegraded":"2024-09-16T10:34:53.222278+0000","last_fullsized":"2024-09-16T10:34:53.222278+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:39.283769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:39.283769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.548316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":663,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343952+0000","last_change":"2024-09-16T10:34:33.194602+0000","last_active":"2024-09-16T10:34:55.343952+0000","last_peered":"2024-09-16T10:34:55.343952+0000","last_clean":"2024-09-16T10:34:55.343952+0000","last_became_active":"2024-09-16T10:34:30.622945+0000","last_became_peered":"2024-09-16T10:34:30.622945+0000","last_unstale":"2024-09-16T10:34:55.343952+0000","last_undegraded":"2024-09-16T10:34:55.343952+0000","last_fullsized":"2024-09-16T10:34:55.343952+0000","mapping_epoch":611,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:33.194525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.194525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:52.999972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"572'21","reported_seq":142,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221110+0000","last_change":"2024-09-16T10:34:11.383837+0000","last_active":"2024-09-16T10:34:53.221110+0000","last_peered":"2024-09-16T10:34:53.221110+0000","last_clean":"2024-09-16T10:34:53.221110+0000","last_became_active":"2024-09-16T10:33:49.208434+0000","last_became_peered":"2024-09-16T10:33:49.208434+0000","last_unstale":"2024-09-16T10:34:53.221110+0000","last_undegraded":"2024-09-16T10:34:53.221110+0000","last_fullsized":"2024-09-16T10:34:53.221110+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:11.383766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:11.383766+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:33.756559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218423+0000","last_change":"2024-09-16T10:34:53.211141+0000","last_active":"2024-09-16T10:34:53.218423+0000","last_peered":"2024-09-16T10:34:53.218423+0000","last_clean":"2024-09-16T10:34:53.218423+0000","last_became_active":"2024-09-16T10:33:49.199335+0000","last_became_peered":"2024-09-16T10:33:49.199335+0000","last_unstale":"2024-09-16T10:34:53.218423+0000","last_undegraded":"2024-09-16T10:34:53.218423+0000","last_fullsized":"2024-09-16T10:34:53.218423+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:53.211046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:53.211046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:00.432857+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":751,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.337066+0000","last_change":"2024-09-16T10:34:22.774392+0000","last_active":"2024-09-16T10:34:53.337066+0000","last_peered":"2024-09-16T10:34:53.337066+0000","last_clean":"2024-09-16T10:34:53.337066+0000","last_became_active":"2024-09-16T10:22:02.905322+0000","last_became_peered":"2024-09-16T10:22:02.905322+0000","last_unstale":"2024-09-16T10:34:53.337066+0000","last_undegraded":"2024-09-16T10:34:53.337066+0000","last_fullsized":"2024-09-16T10:34:53.337066+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:34:22.774339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:34:22.774339+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:29.274046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":771,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345596+0000","last_change":"2024-09-16T10:34:02.704295+0000","last_active":"2024-09-16T10:34:53.345596+0000","last_peered":"2024-09-16T10:34:53.345596+0000","last_clean":"2024-09-16T10:34:53.345596+0000","last_became_active":"2024-09-16T10:33:51.156708+0000","last_became_peered":"2024-09-16T10:33:51.156708+0000","last_unstale":"2024-09-16T10:34:53.345596+0000","last_undegraded":"2024-09-16T10:34:53.345596+0000","last_fullsized":"2024-09-16T10:34:53.345596+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:02.704247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:02.704247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.790911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":551,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344269+0000","last_change":"2024-09-16T10:34:30.625497+0000","last_active":"2024-09-16T10:34:55.344269+0000","last_peered":"2024-09-16T10:34:55.344269+0000","last_clean":"2024-09-16T10:34:55.344269+0000","last_became_active":"2024-09-16T10:34:30.621205+0000","last_became_peered":"2024-09-16T10:34:30.621205+0000","last_unstale":"2024-09-16T10:34:55.344269+0000","last_undegraded":"2024-09-16T10:34:55.344269+0000","last_fullsized":"2024-09-16T10:34:55.344269+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:58.563402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:58.563402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:21.828397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":162,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218905+0000","last_change":"2024-09-16T10:34:07.152780+0000","last_active":"2024-09-16T10:34:53.218905+0000","last_peered":"2024-09-16T10:34:53.218905+0000","last_clean":"2024-09-16T10:34:53.218905+0000","last_became_active":"2024-09-16T10:34:07.152171+0000","last_became_peered":"2024-09-16T10:34:07.152171+0000","last_unstale":"2024-09-16T10:34:53.218905+0000","last_undegraded":"2024-09-16T10:34:53.218905+0000","last_fullsized":"2024-09-16T10:34:53.218905+0000","mapping_epoch":578,"log_start":"73'79","ondisk_log_start":"73'79","created":154,"last_epoch_clean":592,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:52.304996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:52.304996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:04.380325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"572'36","reported_seq":1368,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401678+0000","last_change":"2024-09-16T10:34:15.538554+0000","last_active":"2024-09-16T10:34:53.401678+0000","last_peered":"2024-09-16T10:34:53.401678+0000","last_clean":"2024-09-16T10:34:53.401678+0000","last_became_active":"2024-09-16T10:33:49.212861+0000","last_became_peered":"2024-09-16T10:33:49.212861+0000","last_unstale":"2024-09-16T10:34:53.401678+0000","last_undegraded":"2024-09-16T10:34:53.401678+0000","last_fullsized":"2024-09-16T10:34:53.401678+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'36","last_scrub_stamp":"2024-09-16T10:34:15.538460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:15.538460+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.810915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1235,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401147+0000","last_change":"2024-09-16T10:33:44.438020+0000","last_active":"2024-09-16T10:34:53.401147+0000","last_peered":"2024-09-16T10:34:53.401147+0000","last_clean":"2024-09-16T10:34:53.401147+0000","last_became_active":"2024-09-16T10:13:37.096085+0000","last_became_peered":"2024-09-16T10:13:37.096085+0000","last_unstale":"2024-09-16T10:34:53.401147+0000","last_undegraded":"2024-09-16T10:34:53.401147+0000","last_fullsized":"2024-09-16T10:34:53.401147+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:44.437972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:44.437972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.831239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":96,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343724+0000","last_change":"2024-09-16T10:34:30.625642+0000","last_active":"2024-09-16T10:34:55.343724+0000","last_peered":"2024-09-16T10:34:55.343724+0000","last_clean":"2024-09-16T10:34:55.343724+0000","last_became_active":"2024-09-16T10:34:30.623166+0000","last_became_peered":"2024-09-16T10:34:30.623166+0000","last_unstale":"2024-09-16T10:34:55.343724+0000","last_undegraded":"2024-09-16T10:34:55.343724+0000","last_fullsized":"2024-09-16T10:34:55.343724+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-16T10:34:20.296113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:35.065568+0000","last_clean_scrub_stamp":"2024-09-16T10:34:20.296113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:24.980292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000607578,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":119,"num_read_kb":114,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1239,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222401+0000","last_change":"2024-09-16T10:34:39.343719+0000","last_active":"2024-09-16T10:34:52.222401+0000","last_peered":"2024-09-16T10:34:52.222401+0000","last_clean":"2024-09-16T10:34:52.222401+0000","last_became_active":"2024-09-16T10:33:49.210766+0000","last_became_peered":"2024-09-16T10:33:49.210766+0000","last_unstale":"2024-09-16T10:34:52.222401+0000","last_undegraded":"2024-09-16T10:34:52.222401+0000","last_fullsized":"2024-09-16T10:34:52.222401+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:39.343668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:39.343668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:02.971579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219056+0000","last_change":"2024-09-16T10:33:54.309689+0000","last_active":"2024-09-16T10:34:53.219056+0000","last_peered":"2024-09-16T10:34:53.219056+0000","last_clean":"2024-09-16T10:34:53.219056+0000","last_became_active":"2024-09-16T10:33:49.222778+0000","last_became_peered":"2024-09-16T10:33:49.222778+0000","last_unstale":"2024-09-16T10:34:53.219056+0000","last_undegraded":"2024-09-16T10:34:53.219056+0000","last_fullsized":"2024-09-16T10:34:53.219056+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:54.309637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:54.309637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:01.456931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"579'44","reported_seq":1576,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334816+0000","last_change":"2024-09-16T10:33:57.524064+0000","last_active":"2024-09-16T10:34:53.334816+0000","last_peered":"2024-09-16T10:34:53.334816+0000","last_clean":"2024-09-16T10:34:53.334816+0000","last_became_active":"2024-09-16T10:33:51.163308+0000","last_became_peered":"2024-09-16T10:33:51.163308+0000","last_unstale":"2024-09-16T10:34:53.334816+0000","last_undegraded":"2024-09-16T10:34:53.334816+0000","last_fullsized":"2024-09-16T10:34:53.334816+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'44","last_scrub_stamp":"2024-09-16T10:33:57.524000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:57.524000+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:10.448250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1181,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221030+0000","last_change":"2024-09-16T10:34:44.397662+0000","last_active":"2024-09-16T10:34:53.221030+0000","last_peered":"2024-09-16T10:34:53.221030+0000","last_clean":"2024-09-16T10:34:53.221030+0000","last_became_active":"2024-09-16T10:18:33.824599+0000","last_became_peered":"2024-09-16T10:18:33.824599+0000","last_unstale":"2024-09-16T10:34:53.221030+0000","last_undegraded":"2024-09-16T10:34:53.221030+0000","last_fullsized":"2024-09-16T10:34:53.221030+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:44.397559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:44.397559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:14.301745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":689,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343688+0000","last_change":"2024-09-16T10:34:39.217482+0000","last_active":"2024-09-16T10:34:55.343688+0000","last_peered":"2024-09-16T10:34:55.343688+0000","last_clean":"2024-09-16T10:34:55.343688+0000","last_became_active":"2024-09-16T10:34:35.174424+0000","last_became_peered":"2024-09-16T10:34:35.174424+0000","last_unstale":"2024-09-16T10:34:55.343688+0000","last_undegraded":"2024-09-16T10:34:55.343688+0000","last_fullsized":"2024-09-16T10:34:55.343688+0000","mapping_epoch":611,"log_start":"73'79","ondisk_log_start":"73'79","created":19,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:39.217420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:39.217420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:08.688758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1024,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344258+0000","last_change":"2024-09-16T10:34:30.620052+0000","last_active":"2024-09-16T10:34:55.344258+0000","last_peered":"2024-09-16T10:34:55.344258+0000","last_clean":"2024-09-16T10:34:55.344258+0000","last_became_active":"2024-09-16T10:34:30.619720+0000","last_became_peered":"2024-09-16T10:34:30.619720+0000","last_unstale":"2024-09-16T10:34:55.344258+0000","last_undegraded":"2024-09-16T10:34:55.344258+0000","last_fullsized":"2024-09-16T10:34:55.344258+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:55.616263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:55.616263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:58.137541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1205,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222599+0000","last_change":"2024-09-16T10:34:05.350310+0000","last_active":"2024-09-16T10:34:52.222599+0000","last_peered":"2024-09-16T10:34:52.222599+0000","last_clean":"2024-09-16T10:34:52.222599+0000","last_became_active":"2024-09-16T10:13:43.121369+0000","last_became_peered":"2024-09-16T10:13:43.121369+0000","last_unstale":"2024-09-16T10:34:52.222599+0000","last_undegraded":"2024-09-16T10:34:52.222599+0000","last_fullsized":"2024-09-16T10:34:52.222599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:05.350255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:05.350255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:19.398783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"73'79","reported_seq":685,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218754+0000","last_change":"2024-09-16T10:34:26.341536+0000","last_active":"2024-09-16T10:34:53.218754+0000","last_peered":"2024-09-16T10:34:53.218754+0000","last_clean":"2024-09-16T10:34:53.218754+0000","last_became_active":"2024-09-16T10:34:21.550673+0000","last_became_peered":"2024-09-16T10:34:21.550673+0000","last_unstale":"2024-09-16T10:34:53.218754+0000","last_undegraded":"2024-09-16T10:34:53.218754+0000","last_fullsized":"2024-09-16T10:34:53.218754+0000","mapping_epoch":603,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:26.341460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:26.341460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:40.126925+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"572'27","reported_seq":1123,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219868+0000","last_change":"2024-09-16T10:34:18.164468+0000","last_active":"2024-09-16T10:34:53.219868+0000","last_peered":"2024-09-16T10:34:53.219868+0000","last_clean":"2024-09-16T10:34:53.219868+0000","last_became_active":"2024-09-16T10:34:18.164080+0000","last_became_peered":"2024-09-16T10:34:18.164080+0000","last_unstale":"2024-09-16T10:34:53.219868+0000","last_undegraded":"2024-09-16T10:34:53.219868+0000","last_fullsized":"2024-09-16T10:34:53.219868+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:34:10.518049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:10.518049+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:12.960322+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1003,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219186+0000","last_change":"2024-09-16T10:34:18.160078+0000","last_active":"2024-09-16T10:34:53.219186+0000","last_peered":"2024-09-16T10:34:53.219186+0000","last_clean":"2024-09-16T10:34:53.219186+0000","last_became_active":"2024-09-16T10:34:18.158678+0000","last_became_peered":"2024-09-16T10:34:18.158678+0000","last_unstale":"2024-09-16T10:34:53.219186+0000","last_undegraded":"2024-09-16T10:34:53.219186+0000","last_fullsized":"2024-09-16T10:34:53.219186+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:13.682998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:13.682998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:20.249907+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1217,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222824+0000","last_change":"2024-09-16T10:34:31.395893+0000","last_active":"2024-09-16T10:34:52.222824+0000","last_peered":"2024-09-16T10:34:52.222824+0000","last_clean":"2024-09-16T10:34:52.222824+0000","last_became_active":"2024-09-16T10:13:41.115267+0000","last_became_peered":"2024-09-16T10:13:41.115267+0000","last_unstale":"2024-09-16T10:34:52.222824+0000","last_undegraded":"2024-09-16T10:34:52.222824+0000","last_fullsized":"2024-09-16T10:34:52.222824+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:31.395827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:31.395827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:45.172472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1145,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222532+0000","last_change":"2024-09-16T10:34:31.457282+0000","last_active":"2024-09-16T10:34:53.222532+0000","last_peered":"2024-09-16T10:34:53.222532+0000","last_clean":"2024-09-16T10:34:53.222532+0000","last_became_active":"2024-09-16T10:18:33.823558+0000","last_became_peered":"2024-09-16T10:18:33.823558+0000","last_unstale":"2024-09-16T10:34:53.222532+0000","last_undegraded":"2024-09-16T10:34:53.222532+0000","last_fullsized":"2024-09-16T10:34:53.222532+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:31.457232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:31.457232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:55.761379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":1094,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.220753+0000","last_change":"2024-09-16T10:34:35.428777+0000","last_active":"2024-09-16T10:34:52.220753+0000","last_peered":"2024-09-16T10:34:52.220753+0000","last_clean":"2024-09-16T10:34:52.220753+0000","last_became_active":"2024-09-16T10:22:02.908270+0000","last_became_peered":"2024-09-16T10:22:02.908270+0000","last_unstale":"2024-09-16T10:34:52.220753+0000","last_undegraded":"2024-09-16T10:34:52.220753+0000","last_fullsized":"2024-09-16T10:34:52.220753+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:35.428712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.428712+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.492709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"572'45","reported_seq":1204,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219867+0000","last_change":"2024-09-16T10:34:18.167545+0000","last_active":"2024-09-16T10:34:53.219867+0000","last_peered":"2024-09-16T10:34:53.219867+0000","last_clean":"2024-09-16T10:34:53.219867+0000","last_became_active":"2024-09-16T10:34:18.166916+0000","last_became_peered":"2024-09-16T10:34:18.166916+0000","last_unstale":"2024-09-16T10:34:53.219867+0000","last_undegraded":"2024-09-16T10:34:53.219867+0000","last_fullsized":"2024-09-16T10:34:53.219867+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'45","last_scrub_stamp":"2024-09-16T10:34:04.694713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:04.694713+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:27.009533+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1232,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334810+0000","last_change":"2024-09-16T10:34:33.460067+0000","last_active":"2024-09-16T10:34:53.334810+0000","last_peered":"2024-09-16T10:34:53.334810+0000","last_clean":"2024-09-16T10:34:53.334810+0000","last_became_active":"2024-09-16T10:33:49.209853+0000","last_became_peered":"2024-09-16T10:33:49.209853+0000","last_unstale":"2024-09-16T10:34:53.334810+0000","last_undegraded":"2024-09-16T10:34:53.334810+0000","last_fullsized":"2024-09-16T10:34:53.334810+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:33.460015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.460015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.224965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219113+0000","last_change":"2024-09-16T10:34:28.321982+0000","last_active":"2024-09-16T10:34:53.219113+0000","last_peered":"2024-09-16T10:34:53.219113+0000","last_clean":"2024-09-16T10:34:53.219113+0000","last_became_active":"2024-09-16T10:33:49.203145+0000","last_became_peered":"2024-09-16T10:33:49.203145+0000","last_unstale":"2024-09-16T10:34:53.219113+0000","last_undegraded":"2024-09-16T10:34:53.219113+0000","last_fullsized":"2024-09-16T10:34:53.219113+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:28.321919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:28.321919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:33.189197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":754,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345659+0000","last_change":"2024-09-16T10:34:47.877123+0000","last_active":"2024-09-16T10:34:53.345659+0000","last_peered":"2024-09-16T10:34:53.345659+0000","last_clean":"2024-09-16T10:34:53.345659+0000","last_became_active":"2024-09-16T10:22:02.909593+0000","last_became_peered":"2024-09-16T10:22:02.909593+0000","last_unstale":"2024-09-16T10:34:53.345659+0000","last_undegraded":"2024-09-16T10:34:53.345659+0000","last_fullsized":"2024-09-16T10:34:53.345659+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:47.877067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:47.877067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:06.730966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":843,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343647+0000","last_change":"2024-09-16T10:34:30.623332+0000","last_active":"2024-09-16T10:34:55.343647+0000","last_peered":"2024-09-16T10:34:55.343647+0000","last_clean":"2024-09-16T10:34:55.343647+0000","last_became_active":"2024-09-16T10:34:30.622459+0000","last_became_peered":"2024-09-16T10:34:30.622459+0000","last_unstale":"2024-09-16T10:34:55.343647+0000","last_undegraded":"2024-09-16T10:34:55.343647+0000","last_fullsized":"2024-09-16T10:34:55.343647+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:43.348315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:43.348315+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:04.288198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"572'30","reported_seq":1087,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219346+0000","last_change":"2024-09-16T10:34:22.226635+0000","last_active":"2024-09-16T10:34:53.219346+0000","last_peered":"2024-09-16T10:34:53.219346+0000","last_clean":"2024-09-16T10:34:53.219346+0000","last_became_active":"2024-09-16T10:34:18.158287+0000","last_became_peered":"2024-09-16T10:34:18.158287+0000","last_unstale":"2024-09-16T10:34:53.219346+0000","last_undegraded":"2024-09-16T10:34:53.219346+0000","last_fullsized":"2024-09-16T10:34:53.219346+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'30","last_scrub_stamp":"2024-09-16T10:34:22.226551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:22.226551+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:48.338811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1200,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221259+0000","last_change":"2024-09-16T10:34:42.399434+0000","last_active":"2024-09-16T10:34:53.221259+0000","last_peered":"2024-09-16T10:34:53.221259+0000","last_clean":"2024-09-16T10:34:53.221259+0000","last_became_active":"2024-09-16T10:13:37.090721+0000","last_became_peered":"2024-09-16T10:13:37.090721+0000","last_unstale":"2024-09-16T10:34:53.221259+0000","last_undegraded":"2024-09-16T10:34:53.221259+0000","last_fullsized":"2024-09-16T10:34:53.221259+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:42.399351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:42.399351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:02.919984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"579'14","reported_seq":940,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345519+0000","last_change":"2024-09-16T10:34:06.770354+0000","last_active":"2024-09-16T10:34:53.345519+0000","last_peered":"2024-09-16T10:34:53.345519+0000","last_clean":"2024-09-16T10:34:53.345519+0000","last_became_active":"2024-09-16T10:33:51.173662+0000","last_became_peered":"2024-09-16T10:33:51.173662+0000","last_unstale":"2024-09-16T10:34:53.345519+0000","last_undegraded":"2024-09-16T10:34:53.345519+0000","last_fullsized":"2024-09-16T10:34:53.345519+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'14","last_scrub_stamp":"2024-09-16T10:34:06.770304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:06.770304+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:15.467291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":782,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219700+0000","last_change":"2024-09-16T10:34:50.303284+0000","last_active":"2024-09-16T10:34:53.219700+0000","last_peered":"2024-09-16T10:34:53.219700+0000","last_clean":"2024-09-16T10:34:53.219700+0000","last_became_active":"2024-09-16T10:34:18.166006+0000","last_became_peered":"2024-09-16T10:34:18.166006+0000","last_unstale":"2024-09-16T10:34:53.219700+0000","last_undegraded":"2024-09-16T10:34:53.219700+0000","last_fullsized":"2024-09-16T10:34:53.219700+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:50.303228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:50.303228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:05.496420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"572'21","reported_seq":1274,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401408+0000","last_change":"2024-09-16T10:34:20.617016+0000","last_active":"2024-09-16T10:34:53.401408+0000","last_peered":"2024-09-16T10:34:53.401408+0000","last_clean":"2024-09-16T10:34:53.401408+0000","last_became_active":"2024-09-16T10:33:49.209734+0000","last_became_peered":"2024-09-16T10:33:49.209734+0000","last_unstale":"2024-09-16T10:34:53.401408+0000","last_undegraded":"2024-09-16T10:34:53.401408+0000","last_fullsized":"2024-09-16T10:34:53.401408+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:20.616963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:20.616963+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:27.454179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1225,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.224268+0000","last_change":"2024-09-16T10:34:26.348535+0000","last_active":"2024-09-16T10:34:52.224268+0000","last_peered":"2024-09-16T10:34:52.224268+0000","last_clean":"2024-09-16T10:34:52.224268+0000","last_became_active":"2024-09-16T10:13:37.089969+0000","last_became_peered":"2024-09-16T10:13:37.089969+0000","last_unstale":"2024-09-16T10:34:52.224268+0000","last_undegraded":"2024-09-16T10:34:52.224268+0000","last_fullsized":"2024-09-16T10:34:52.224268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:26.348476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:26.348476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:51.950897+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":746,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.337017+0000","last_change":"2024-09-16T10:34:23.742483+0000","last_active":"2024-09-16T10:34:53.337017+0000","last_peered":"2024-09-16T10:34:53.337017+0000","last_clean":"2024-09-16T10:34:53.337017+0000","last_became_active":"2024-09-16T10:22:02.896654+0000","last_became_peered":"2024-09-16T10:22:02.896654+0000","last_unstale":"2024-09-16T10:34:53.337017+0000","last_undegraded":"2024-09-16T10:34:53.337017+0000","last_fullsized":"2024-09-16T10:34:53.337017+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:23.742433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:23.742433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.639504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"66'1","reported_seq":1268,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.400938+0000","last_change":"2024-09-16T10:33:51.186724+0000","last_active":"2024-09-16T10:34:53.400938+0000","last_peered":"2024-09-16T10:34:53.400938+0000","last_clean":"2024-09-16T10:34:53.400938+0000","last_became_active":"2024-09-16T10:33:51.168083+0000","last_became_peered":"2024-09-16T10:33:51.168083+0000","last_unstale":"2024-09-16T10:34:53.400938+0000","last_undegraded":"2024-09-16T10:34:53.400938+0000","last_fullsized":"2024-09-16T10:34:53.400938+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-16T10:33:43.392949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:43.392949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.703506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":78,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"572'51","reported_seq":1439,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334219+0000","last_change":"2024-09-16T10:34:13.164046+0000","last_active":"2024-09-16T10:34:53.334219+0000","last_peered":"2024-09-16T10:34:53.334219+0000","last_clean":"2024-09-16T10:34:53.334219+0000","last_became_active":"2024-09-16T10:34:13.152342+0000","last_became_peered":"2024-09-16T10:34:13.152342+0000","last_unstale":"2024-09-16T10:34:53.334219+0000","last_undegraded":"2024-09-16T10:34:53.334219+0000","last_fullsized":"2024-09-16T10:34:53.334219+0000","mapping_epoch":596,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":597,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'35","last_scrub_stamp":"2024-09-16T10:33:42.596123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:42.596123+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.725971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":549,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344543+0000","last_change":"2024-09-16T10:34:30.620390+0000","last_active":"2024-09-16T10:34:55.344543+0000","last_peered":"2024-09-16T10:34:55.344543+0000","last_clean":"2024-09-16T10:34:55.344543+0000","last_became_active":"2024-09-16T10:34:30.618306+0000","last_became_peered":"2024-09-16T10:34:30.618306+0000","last_unstale":"2024-09-16T10:34:55.344543+0000","last_undegraded":"2024-09-16T10:34:55.344543+0000","last_fullsized":"2024-09-16T10:34:55.344543+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:11.351964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:11.351964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:21.162619+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":746,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.337168+0000","last_change":"2024-09-16T10:34:29.726300+0000","last_active":"2024-09-16T10:34:53.337168+0000","last_peered":"2024-09-16T10:34:53.337168+0000","last_clean":"2024-09-16T10:34:53.337168+0000","last_became_active":"2024-09-16T10:22:02.897873+0000","last_became_peered":"2024-09-16T10:22:02.897873+0000","last_unstale":"2024-09-16T10:34:53.337168+0000","last_undegraded":"2024-09-16T10:34:53.337168+0000","last_fullsized":"2024-09-16T10:34:53.337168+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:29.726249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:29.726249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:36.711014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1028,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343777+0000","last_change":"2024-09-16T10:34:30.621673+0000","last_active":"2024-09-16T10:34:55.343777+0000","last_peered":"2024-09-16T10:34:55.343777+0000","last_clean":"2024-09-16T10:34:55.343777+0000","last_became_active":"2024-09-16T10:34:30.621376+0000","last_became_peered":"2024-09-16T10:34:30.621376+0000","last_unstale":"2024-09-16T10:34:55.343777+0000","last_undegraded":"2024-09-16T10:34:55.343777+0000","last_fullsized":"2024-09-16T10:34:55.343777+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:00.709103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:00.709103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:14.393842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"73'79","reported_seq":1172,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.476369+0000","last_change":"2024-09-16T10:34:53.476369+0000","last_active":"2024-09-16T10:34:53.476369+0000","last_peered":"2024-09-16T10:34:53.476369+0000","last_clean":"2024-09-16T10:34:53.476369+0000","last_became_active":"2024-09-16T10:34:53.223094+0000","last_became_peered":"2024-09-16T10:34:53.223094+0000","last_unstale":"2024-09-16T10:34:53.476369+0000","last_undegraded":"2024-09-16T10:34:53.476369+0000","last_fullsized":"2024-09-16T10:34:53.476369+0000","mapping_epoch":630,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":631,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:34:53.476315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:34:53.476315+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":37,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:22.254895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"572'24","reported_seq":778,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218960+0000","last_change":"2024-09-16T10:34:18.164629+0000","last_active":"2024-09-16T10:34:53.218960+0000","last_peered":"2024-09-16T10:34:53.218960+0000","last_clean":"2024-09-16T10:34:53.218960+0000","last_became_active":"2024-09-16T10:34:18.164205+0000","last_became_peered":"2024-09-16T10:34:18.164205+0000","last_unstale":"2024-09-16T10:34:53.218960+0000","last_undegraded":"2024-09-16T10:34:53.218960+0000","last_fullsized":"2024-09-16T10:34:53.218960+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'24","last_scrub_stamp":"2024-09-16T10:34:12.388261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:12.388261+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.126870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":128,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344667+0000","last_change":"2024-09-16T10:34:30.615211+0000","last_active":"2024-09-16T10:34:55.344667+0000","last_peered":"2024-09-16T10:34:55.344667+0000","last_clean":"2024-09-16T10:34:55.344667+0000","last_became_active":"2024-09-16T10:34:30.614725+0000","last_became_peered":"2024-09-16T10:34:30.614725+0000","last_unstale":"2024-09-16T10:34:55.344667+0000","last_undegraded":"2024-09-16T10:34:55.344667+0000","last_fullsized":"2024-09-16T10:34:55.344667+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:47.490203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:47.490203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:10.886368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":768,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.336671+0000","last_change":"2024-09-16T10:34:33.676291+0000","last_active":"2024-09-16T10:34:53.336671+0000","last_peered":"2024-09-16T10:34:53.336671+0000","last_clean":"2024-09-16T10:34:53.336671+0000","last_became_active":"2024-09-16T10:33:49.222955+0000","last_became_peered":"2024-09-16T10:33:49.222955+0000","last_unstale":"2024-09-16T10:34:53.336671+0000","last_undegraded":"2024-09-16T10:34:53.336671+0000","last_fullsized":"2024-09-16T10:34:53.336671+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:33.676243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.676243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.581118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1243,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221078+0000","last_change":"2024-09-16T10:34:27.484125+0000","last_active":"2024-09-16T10:34:53.221078+0000","last_peered":"2024-09-16T10:34:53.221078+0000","last_clean":"2024-09-16T10:34:53.221078+0000","last_became_active":"2024-09-16T10:33:51.164602+0000","last_became_peered":"2024-09-16T10:33:51.164602+0000","last_unstale":"2024-09-16T10:34:53.221078+0000","last_undegraded":"2024-09-16T10:34:53.221078+0000","last_fullsized":"2024-09-16T10:34:53.221078+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:27.484068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:27.484068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:43.091905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"73'79","reported_seq":579,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219768+0000","last_change":"2024-09-16T10:34:18.159367+0000","last_active":"2024-09-16T10:34:53.219768+0000","last_peered":"2024-09-16T10:34:53.219768+0000","last_clean":"2024-09-16T10:34:53.219768+0000","last_became_active":"2024-09-16T10:34:18.158914+0000","last_became_peered":"2024-09-16T10:34:18.158914+0000","last_unstale":"2024-09-16T10:34:53.219768+0000","last_undegraded":"2024-09-16T10:34:53.219768+0000","last_fullsized":"2024-09-16T10:34:53.219768+0000","mapping_epoch":600,"log_start":"73'79","ondisk_log_start":"73'79","created":306,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-16T10:33:45.610690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:11:23.440798+0000","last_clean_scrub_stamp":"2024-09-16T10:33:45.610690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:02.580406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"572'39","reported_seq":1315,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221147+0000","last_change":"2024-09-16T10:34:24.289425+0000","last_active":"2024-09-16T10:34:52.221147+0000","last_peered":"2024-09-16T10:34:52.221147+0000","last_clean":"2024-09-16T10:34:52.221147+0000","last_became_active":"2024-09-16T10:33:51.153340+0000","last_became_peered":"2024-09-16T10:33:51.153340+0000","last_unstale":"2024-09-16T10:34:52.221147+0000","last_undegraded":"2024-09-16T10:34:52.221147+0000","last_fullsized":"2024-09-16T10:34:52.221147+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'39","last_scrub_stamp":"2024-09-16T10:34:24.289354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:24.289354+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:24.336014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":529,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344721+0000","last_change":"2024-09-16T10:34:40.241419+0000","last_active":"2024-09-16T10:34:55.344721+0000","last_peered":"2024-09-16T10:34:55.344721+0000","last_clean":"2024-09-16T10:34:55.344721+0000","last_became_active":"2024-09-16T10:34:30.623804+0000","last_became_peered":"2024-09-16T10:34:30.623804+0000","last_unstale":"2024-09-16T10:34:55.344721+0000","last_undegraded":"2024-09-16T10:34:55.344721+0000","last_fullsized":"2024-09-16T10:34:55.344721+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-16T10:34:40.241364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:40.241364+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:53.306728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"614'16","reported_seq":723,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343688+0000","last_change":"2024-09-16T10:34:30.623738+0000","last_active":"2024-09-16T10:34:55.343688+0000","last_peered":"2024-09-16T10:34:55.343688+0000","last_clean":"2024-09-16T10:34:55.343688+0000","last_became_active":"2024-09-16T10:34:30.622941+0000","last_became_peered":"2024-09-16T10:34:30.622941+0000","last_unstale":"2024-09-16T10:34:55.343688+0000","last_undegraded":"2024-09-16T10:34:55.343688+0000","last_fullsized":"2024-09-16T10:34:55.343688+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'13","last_scrub_stamp":"2024-09-16T10:33:40.619814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:40.619814+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1221,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221915+0000","last_change":"2024-09-16T10:34:42.377708+0000","last_active":"2024-09-16T10:34:52.221915+0000","last_peered":"2024-09-16T10:34:52.221915+0000","last_clean":"2024-09-16T10:34:52.221915+0000","last_became_active":"2024-09-16T10:13:43.283149+0000","last_became_peered":"2024-09-16T10:13:43.283149+0000","last_unstale":"2024-09-16T10:34:52.221915+0000","last_undegraded":"2024-09-16T10:34:52.221915+0000","last_fullsized":"2024-09-16T10:34:52.221915+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:42.377654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:42.377654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:43.145201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1202,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.333936+0000","last_change":"2024-09-16T10:34:30.439376+0000","last_active":"2024-09-16T10:34:53.333936+0000","last_peered":"2024-09-16T10:34:53.333936+0000","last_clean":"2024-09-16T10:34:53.333936+0000","last_became_active":"2024-09-16T10:13:37.089315+0000","last_became_peered":"2024-09-16T10:13:37.089315+0000","last_unstale":"2024-09-16T10:34:53.333936+0000","last_undegraded":"2024-09-16T10:34:53.333936+0000","last_fullsized":"2024-09-16T10:34:53.333936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:30.439316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:30.439316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.929662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"572'18","reported_seq":1301,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222439+0000","last_change":"2024-09-16T10:34:36.433388+0000","last_active":"2024-09-16T10:34:53.222439+0000","last_peered":"2024-09-16T10:34:53.222439+0000","last_clean":"2024-09-16T10:34:53.222439+0000","last_became_active":"2024-09-16T10:33:51.164136+0000","last_became_peered":"2024-09-16T10:33:51.164136+0000","last_unstale":"2024-09-16T10:34:53.222439+0000","last_undegraded":"2024-09-16T10:34:53.222439+0000","last_fullsized":"2024-09-16T10:34:53.222439+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'18","last_scrub_stamp":"2024-09-16T10:34:36.433332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:36.433332+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.154198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1206,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222459+0000","last_change":"2024-09-16T10:33:51.357918+0000","last_active":"2024-09-16T10:34:52.222459+0000","last_peered":"2024-09-16T10:34:52.222459+0000","last_clean":"2024-09-16T10:34:52.222459+0000","last_became_active":"2024-09-16T10:33:49.212617+0000","last_became_peered":"2024-09-16T10:33:49.212617+0000","last_unstale":"2024-09-16T10:34:52.222459+0000","last_undegraded":"2024-09-16T10:34:52.222459+0000","last_fullsized":"2024-09-16T10:34:52.222459+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:51.357869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:51.357869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:19.915195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":127,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344585+0000","last_change":"2024-09-16T10:33:52.019767+0000","last_active":"2024-09-16T10:34:55.344585+0000","last_peered":"2024-09-16T10:34:55.344585+0000","last_clean":"2024-09-16T10:34:55.344585+0000","last_became_active":"2024-09-16T10:33:51.160714+0000","last_became_peered":"2024-09-16T10:33:51.160714+0000","last_unstale":"2024-09-16T10:34:55.344585+0000","last_undegraded":"2024-09-16T10:34:55.344585+0000","last_fullsized":"2024-09-16T10:34:55.344585+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:52.019715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:33:52.019715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:03.968879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"572'21","reported_seq":1284,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401687+0000","last_change":"2024-09-16T10:34:09.541086+0000","last_active":"2024-09-16T10:34:53.401687+0000","last_peered":"2024-09-16T10:34:53.401687+0000","last_clean":"2024-09-16T10:34:53.401687+0000","last_became_active":"2024-09-16T10:33:51.168349+0000","last_became_peered":"2024-09-16T10:33:51.168349+0000","last_unstale":"2024-09-16T10:34:53.401687+0000","last_undegraded":"2024-09-16T10:34:53.401687+0000","last_fullsized":"2024-09-16T10:34:53.401687+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:09.541018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:09.541018+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:16.580562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":507,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343834+0000","last_change":"2024-09-16T10:34:53.336630+0000","last_active":"2024-09-16T10:34:55.343834+0000","last_peered":"2024-09-16T10:34:55.343834+0000","last_clean":"2024-09-16T10:34:55.343834+0000","last_became_active":"2024-09-16T10:34:30.624091+0000","last_became_peered":"2024-09-16T10:34:30.624091+0000","last_unstale":"2024-09-16T10:34:55.343834+0000","last_undegraded":"2024-09-16T10:34:55.343834+0000","last_fullsized":"2024-09-16T10:34:55.343834+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:53.336569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:53.336569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:22.787661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1245,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221846+0000","last_change":"2024-09-16T10:33:57.367133+0000","last_active":"2024-09-16T10:34:53.221846+0000","last_peered":"2024-09-16T10:34:53.221846+0000","last_clean":"2024-09-16T10:34:53.221846+0000","last_became_active":"2024-09-16T10:33:51.163799+0000","last_became_peered":"2024-09-16T10:33:51.163799+0000","last_unstale":"2024-09-16T10:34:53.221846+0000","last_undegraded":"2024-09-16T10:34:53.221846+0000","last_fullsized":"2024-09-16T10:34:53.221846+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:57.367076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:57.367076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:58.674690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":771,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.345502+0000","last_change":"2024-09-16T10:34:11.700727+0000","last_active":"2024-09-16T10:34:53.345502+0000","last_peered":"2024-09-16T10:34:53.345502+0000","last_clean":"2024-09-16T10:34:53.345502+0000","last_became_active":"2024-09-16T10:33:51.166834+0000","last_became_peered":"2024-09-16T10:33:51.166834+0000","last_unstale":"2024-09-16T10:34:53.345502+0000","last_undegraded":"2024-09-16T10:34:53.345502+0000","last_fullsized":"2024-09-16T10:34:53.345502+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:11.700675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:11.700675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:26.133615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"572'21","reported_seq":1275,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.223157+0000","last_change":"2024-09-16T10:34:33.400650+0000","last_active":"2024-09-16T10:34:52.223157+0000","last_peered":"2024-09-16T10:34:52.223157+0000","last_clean":"2024-09-16T10:34:52.223157+0000","last_became_active":"2024-09-16T10:33:51.162840+0000","last_became_peered":"2024-09-16T10:33:51.162840+0000","last_unstale":"2024-09-16T10:34:52.223157+0000","last_undegraded":"2024-09-16T10:34:52.223157+0000","last_fullsized":"2024-09-16T10:34:52.223157+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:33.400562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:33.400562+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.930557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1257,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401304+0000","last_change":"2024-09-16T10:34:48.744945+0000","last_active":"2024-09-16T10:34:53.401304+0000","last_peered":"2024-09-16T10:34:53.401304+0000","last_clean":"2024-09-16T10:34:53.401304+0000","last_became_active":"2024-09-16T10:33:51.165860+0000","last_became_peered":"2024-09-16T10:33:51.165860+0000","last_unstale":"2024-09-16T10:34:53.401304+0000","last_undegraded":"2024-09-16T10:34:53.401304+0000","last_fullsized":"2024-09-16T10:34:53.401304+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:48.744896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:48.744896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:53.373279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"579'16","reported_seq":1560,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.222255+0000","last_change":"2024-09-16T10:34:06.318061+0000","last_active":"2024-09-16T10:34:53.222255+0000","last_peered":"2024-09-16T10:34:53.222255+0000","last_clean":"2024-09-16T10:34:53.222255+0000","last_became_active":"2024-09-16T10:33:51.173279+0000","last_became_peered":"2024-09-16T10:33:51.173279+0000","last_unstale":"2024-09-16T10:34:53.222255+0000","last_undegraded":"2024-09-16T10:34:53.222255+0000","last_fullsized":"2024-09-16T10:34:53.222255+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'16","last_scrub_stamp":"2024-09-16T10:34:06.318015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:06.318015+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:08.843206+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":774,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219874+0000","last_change":"2024-09-16T10:34:18.167339+0000","last_active":"2024-09-16T10:34:53.219874+0000","last_peered":"2024-09-16T10:34:53.219874+0000","last_clean":"2024-09-16T10:34:53.219874+0000","last_became_active":"2024-09-16T10:34:18.166960+0000","last_became_peered":"2024-09-16T10:34:18.166960+0000","last_unstale":"2024-09-16T10:34:53.219874+0000","last_undegraded":"2024-09-16T10:34:53.219874+0000","last_fullsized":"2024-09-16T10:34:53.219874+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:15.676759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:15.676759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:34.640581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"572'30","reported_seq":1344,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221708+0000","last_change":"2024-09-16T10:33:51.193951+0000","last_active":"2024-09-16T10:34:53.221708+0000","last_peered":"2024-09-16T10:34:53.221708+0000","last_clean":"2024-09-16T10:34:53.221708+0000","last_became_active":"2024-09-16T10:33:51.169942+0000","last_became_peered":"2024-09-16T10:33:51.169942+0000","last_unstale":"2024-09-16T10:34:53.221708+0000","last_undegraded":"2024-09-16T10:34:53.221708+0000","last_fullsized":"2024-09-16T10:34:53.221708+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'20","last_scrub_stamp":"2024-09-16T10:33:05.157106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:05.157106+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":127,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344178+0000","last_change":"2024-09-16T10:34:54.331541+0000","last_active":"2024-09-16T10:34:55.344178+0000","last_peered":"2024-09-16T10:34:55.344178+0000","last_clean":"2024-09-16T10:34:55.344178+0000","last_became_active":"2024-09-16T10:33:51.149009+0000","last_became_peered":"2024-09-16T10:33:51.149009+0000","last_unstale":"2024-09-16T10:34:55.344178+0000","last_undegraded":"2024-09-16T10:34:55.344178+0000","last_fullsized":"2024-09-16T10:34:55.344178+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:54.331474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:54.331474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:57.781474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"579'10","reported_seq":1494,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222983+0000","last_change":"2024-09-16T10:34:23.288603+0000","last_active":"2024-09-16T10:34:52.222983+0000","last_peered":"2024-09-16T10:34:52.222983+0000","last_clean":"2024-09-16T10:34:52.222983+0000","last_became_active":"2024-09-16T10:33:49.212698+0000","last_became_peered":"2024-09-16T10:33:49.212698+0000","last_unstale":"2024-09-16T10:34:52.222983+0000","last_undegraded":"2024-09-16T10:34:52.222983+0000","last_fullsized":"2024-09-16T10:34:52.222983+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'10","last_scrub_stamp":"2024-09-16T10:34:23.288531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:23.288531+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:35.715819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1222,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401473+0000","last_change":"2024-09-16T10:34:41.788432+0000","last_active":"2024-09-16T10:34:53.401473+0000","last_peered":"2024-09-16T10:34:53.401473+0000","last_clean":"2024-09-16T10:34:53.401473+0000","last_became_active":"2024-09-16T10:33:51.169708+0000","last_became_peered":"2024-09-16T10:33:51.169708+0000","last_unstale":"2024-09-16T10:34:53.401473+0000","last_undegraded":"2024-09-16T10:34:53.401473+0000","last_fullsized":"2024-09-16T10:34:53.401473+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:41.788378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:41.788378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:49.506575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"572'27","reported_seq":1131,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218943+0000","last_change":"2024-09-16T10:34:37.231251+0000","last_active":"2024-09-16T10:34:53.218943+0000","last_peered":"2024-09-16T10:34:53.218943+0000","last_clean":"2024-09-16T10:34:53.218943+0000","last_became_active":"2024-09-16T10:34:18.163371+0000","last_became_peered":"2024-09-16T10:34:18.163371+0000","last_unstale":"2024-09-16T10:34:53.218943+0000","last_undegraded":"2024-09-16T10:34:53.218943+0000","last_fullsized":"2024-09-16T10:34:53.218943+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:34:37.231166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:37.231166+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:48.518957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":490,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219039+0000","last_change":"2024-09-16T10:34:18.161743+0000","last_active":"2024-09-16T10:34:53.219039+0000","last_peered":"2024-09-16T10:34:53.219039+0000","last_clean":"2024-09-16T10:34:53.219039+0000","last_became_active":"2024-09-16T10:34:18.161423+0000","last_became_peered":"2024-09-16T10:34:18.161423+0000","last_unstale":"2024-09-16T10:34:53.219039+0000","last_undegraded":"2024-09-16T10:34:53.219039+0000","last_fullsized":"2024-09-16T10:34:53.219039+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:57.000341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:33:57.000341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:18.974436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218700+0000","last_change":"2024-09-16T10:34:25.305006+0000","last_active":"2024-09-16T10:34:53.218700+0000","last_peered":"2024-09-16T10:34:53.218700+0000","last_clean":"2024-09-16T10:34:53.218700+0000","last_became_active":"2024-09-16T10:33:49.221969+0000","last_became_peered":"2024-09-16T10:33:49.221969+0000","last_unstale":"2024-09-16T10:34:53.218700+0000","last_undegraded":"2024-09-16T10:34:53.218700+0000","last_fullsized":"2024-09-16T10:34:53.218700+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:25.304959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:25.304959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:47.998074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1196,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220766+0000","last_change":"2024-09-16T10:34:49.459076+0000","last_active":"2024-09-16T10:34:53.220766+0000","last_peered":"2024-09-16T10:34:53.220766+0000","last_clean":"2024-09-16T10:34:53.220766+0000","last_became_active":"2024-09-16T10:13:43.124588+0000","last_became_peered":"2024-09-16T10:13:43.124588+0000","last_unstale":"2024-09-16T10:34:53.220766+0000","last_undegraded":"2024-09-16T10:34:53.220766+0000","last_fullsized":"2024-09-16T10:34:53.220766+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:49.459030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:49.459030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:56.485523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"572'21","reported_seq":140,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.218593+0000","last_change":"2024-09-16T10:34:02.405828+0000","last_active":"2024-09-16T10:34:53.218593+0000","last_peered":"2024-09-16T10:34:53.218593+0000","last_clean":"2024-09-16T10:34:53.218593+0000","last_became_active":"2024-09-16T10:33:49.206963+0000","last_became_peered":"2024-09-16T10:33:49.206963+0000","last_unstale":"2024-09-16T10:34:53.218593+0000","last_undegraded":"2024-09-16T10:34:53.218593+0000","last_fullsized":"2024-09-16T10:34:53.218593+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'21","last_scrub_stamp":"2024-09-16T10:34:02.405764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:02.405764+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:13.723198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":564,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344357+0000","last_change":"2024-09-16T10:34:52.328405+0000","last_active":"2024-09-16T10:34:55.344357+0000","last_peered":"2024-09-16T10:34:55.344357+0000","last_clean":"2024-09-16T10:34:55.344357+0000","last_became_active":"2024-09-16T10:34:30.622358+0000","last_became_peered":"2024-09-16T10:34:30.622358+0000","last_unstale":"2024-09-16T10:34:55.344357+0000","last_undegraded":"2024-09-16T10:34:55.344357+0000","last_fullsized":"2024-09-16T10:34:55.344357+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:52.328342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:52.328342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:52.484193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1171,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221185+0000","last_change":"2024-09-16T10:33:49.330209+0000","last_active":"2024-09-16T10:34:53.221185+0000","last_peered":"2024-09-16T10:34:53.221185+0000","last_clean":"2024-09-16T10:34:53.221185+0000","last_became_active":"2024-09-16T10:33:49.228689+0000","last_became_peered":"2024-09-16T10:33:49.228689+0000","last_unstale":"2024-09-16T10:34:53.221185+0000","last_undegraded":"2024-09-16T10:34:53.221185+0000","last_fullsized":"2024-09-16T10:34:53.221185+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:33:49.330162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:33:49.330162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:05.043495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1196,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.400836+0000","last_change":"2024-09-16T10:34:25.653261+0000","last_active":"2024-09-16T10:34:53.400836+0000","last_peered":"2024-09-16T10:34:53.400836+0000","last_clean":"2024-09-16T10:34:53.400836+0000","last_became_active":"2024-09-16T10:33:49.210276+0000","last_became_peered":"2024-09-16T10:33:49.210276+0000","last_unstale":"2024-09-16T10:34:53.400836+0000","last_undegraded":"2024-09-16T10:34:53.400836+0000","last_fullsized":"2024-09-16T10:34:53.400836+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:25.653221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:25.653221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.988616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"572'27","reported_seq":1272,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220634+0000","last_change":"2024-09-16T10:33:51.297719+0000","last_active":"2024-09-16T10:34:53.220634+0000","last_peered":"2024-09-16T10:34:53.220634+0000","last_clean":"2024-09-16T10:34:53.220634+0000","last_became_active":"2024-09-16T10:33:51.163081+0000","last_became_peered":"2024-09-16T10:33:51.163081+0000","last_unstale":"2024-09-16T10:34:53.220634+0000","last_undegraded":"2024-09-16T10:34:53.220634+0000","last_fullsized":"2024-09-16T10:34:53.220634+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:33:51.297660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:33:51.297660+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:08.327471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1255,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.222566+0000","last_change":"2024-09-16T10:34:25.316729+0000","last_active":"2024-09-16T10:34:52.222566+0000","last_peered":"2024-09-16T10:34:52.222566+0000","last_clean":"2024-09-16T10:34:52.222566+0000","last_became_active":"2024-09-16T10:33:51.152535+0000","last_became_peered":"2024-09-16T10:33:51.152535+0000","last_unstale":"2024-09-16T10:34:52.222566+0000","last_undegraded":"2024-09-16T10:34:52.222566+0000","last_fullsized":"2024-09-16T10:34:52.222566+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:25.316678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:25.316678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:39.411867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":127,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.343881+0000","last_change":"2024-09-16T10:34:01.029861+0000","last_active":"2024-09-16T10:34:55.343881+0000","last_peered":"2024-09-16T10:34:55.343881+0000","last_clean":"2024-09-16T10:34:55.343881+0000","last_became_active":"2024-09-16T10:33:51.160356+0000","last_became_peered":"2024-09-16T10:33:51.160356+0000","last_unstale":"2024-09-16T10:34:55.343881+0000","last_undegraded":"2024-09-16T10:34:55.343881+0000","last_fullsized":"2024-09-16T10:34:55.343881+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:01.029789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:01.029789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:13.318818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1212,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.221251+0000","last_change":"2024-09-16T10:34:13.319956+0000","last_active":"2024-09-16T10:34:53.221251+0000","last_peered":"2024-09-16T10:34:53.221251+0000","last_clean":"2024-09-16T10:34:53.221251+0000","last_became_active":"2024-09-16T10:13:43.286297+0000","last_became_peered":"2024-09-16T10:13:43.286297+0000","last_unstale":"2024-09-16T10:34:53.221251+0000","last_undegraded":"2024-09-16T10:34:53.221251+0000","last_fullsized":"2024-09-16T10:34:53.221251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:13.319902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:13.319902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:17.261606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"572'12","reported_seq":1287,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220063+0000","last_change":"2024-09-16T10:33:49.285315+0000","last_active":"2024-09-16T10:34:53.220063+0000","last_peered":"2024-09-16T10:34:53.220063+0000","last_clean":"2024-09-16T10:34:53.220063+0000","last_became_active":"2024-09-16T10:33:49.230962+0000","last_became_peered":"2024-09-16T10:33:49.230962+0000","last_unstale":"2024-09-16T10:34:53.220063+0000","last_undegraded":"2024-09-16T10:34:53.220063+0000","last_fullsized":"2024-09-16T10:34:53.220063+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'8","last_scrub_stamp":"2024-09-16T10:32:12.365003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:32:12.365003+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":1207,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401692+0000","last_change":"2024-09-16T10:34:16.559557+0000","last_active":"2024-09-16T10:34:53.401692+0000","last_peered":"2024-09-16T10:34:53.401692+0000","last_clean":"2024-09-16T10:34:53.401692+0000","last_became_active":"2024-09-16T10:13:37.093879+0000","last_became_peered":"2024-09-16T10:13:37.093879+0000","last_unstale":"2024-09-16T10:34:53.401692+0000","last_undegraded":"2024-09-16T10:34:53.401692+0000","last_fullsized":"2024-09-16T10:34:53.401692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-16T10:34:16.559506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:16.559506+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:41.157665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1235,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.334014+0000","last_change":"2024-09-16T10:34:05.544685+0000","last_active":"2024-09-16T10:34:53.334014+0000","last_peered":"2024-09-16T10:34:53.334014+0000","last_clean":"2024-09-16T10:34:53.334014+0000","last_became_active":"2024-09-16T10:33:51.165106+0000","last_became_peered":"2024-09-16T10:33:51.165106+0000","last_unstale":"2024-09-16T10:34:53.334014+0000","last_undegraded":"2024-09-16T10:34:53.334014+0000","last_fullsized":"2024-09-16T10:34:53.334014+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:05.544624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:05.544624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:12.872947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":798,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-16T10:34:55.344510+0000","last_change":"2024-09-16T10:34:55.344510+0000","last_active":"2024-09-16T10:34:55.344510+0000","last_peered":"2024-09-16T10:34:55.344510+0000","last_clean":"2024-09-16T10:34:55.344510+0000","last_became_active":"2024-09-16T10:34:30.621716+0000","last_became_peered":"2024-09-16T10:34:30.621716+0000","last_unstale":"2024-09-16T10:34:55.344510+0000","last_undegraded":"2024-09-16T10:34:55.344510+0000","last_fullsized":"2024-09-16T10:34:55.344510+0000","mapping_epoch":611,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:55.344410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:55.344410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:06.988443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":130,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.219487+0000","last_change":"2024-09-16T10:34:10.345817+0000","last_active":"2024-09-16T10:34:53.219487+0000","last_peered":"2024-09-16T10:34:53.219487+0000","last_clean":"2024-09-16T10:34:53.219487+0000","last_became_active":"2024-09-16T10:33:49.204475+0000","last_became_peered":"2024-09-16T10:33:49.204475+0000","last_unstale":"2024-09-16T10:34:53.219487+0000","last_undegraded":"2024-09-16T10:34:53.219487+0000","last_fullsized":"2024-09-16T10:34:53.219487+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:10.345755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:42.098882+0000","last_clean_scrub_stamp":"2024-09-16T10:34:10.345755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:18.202511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1229,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.220203+0000","last_change":"2024-09-16T10:34:45.433933+0000","last_active":"2024-09-16T10:34:53.220203+0000","last_peered":"2024-09-16T10:34:53.220203+0000","last_clean":"2024-09-16T10:34:53.220203+0000","last_became_active":"2024-09-16T10:33:49.228179+0000","last_became_peered":"2024-09-16T10:33:49.228179+0000","last_unstale":"2024-09-16T10:34:53.220203+0000","last_undegraded":"2024-09-16T10:34:53.220203+0000","last_fullsized":"2024-09-16T10:34:53.220203+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:45.433881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:36.070782+0000","last_clean_scrub_stamp":"2024-09-16T10:34:45.433881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:54.452621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"572'27","reported_seq":1312,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.335208+0000","last_change":"2024-09-16T10:34:35.474513+0000","last_active":"2024-09-16T10:34:53.335208+0000","last_peered":"2024-09-16T10:34:53.335208+0000","last_clean":"2024-09-16T10:34:53.335208+0000","last_became_active":"2024-09-16T10:33:49.209690+0000","last_became_peered":"2024-09-16T10:33:49.209690+0000","last_unstale":"2024-09-16T10:34:53.335208+0000","last_undegraded":"2024-09-16T10:34:53.335208+0000","last_fullsized":"2024-09-16T10:34:53.335208+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":576,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'27","last_scrub_stamp":"2024-09-16T10:34:35.474452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:38.076596+0000","last_clean_scrub_stamp":"2024-09-16T10:34:35.474452+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:43.555576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1256,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-09-16T10:34:53.401580+0000","last_change":"2024-09-16T10:34:32.708946+0000","last_active":"2024-09-16T10:34:53.401580+0000","last_peered":"2024-09-16T10:34:53.401580+0000","last_clean":"2024-09-16T10:34:53.401580+0000","last_became_active":"2024-09-16T10:33:51.163016+0000","last_became_peered":"2024-09-16T10:33:51.163016+0000","last_unstale":"2024-09-16T10:34:53.401580+0000","last_undegraded":"2024-09-16T10:34:53.401580+0000","last_fullsized":"2024-09-16T10:34:53.401580+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:34:32.708900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:13:40.081041+0000","last_clean_scrub_stamp":"2024-09-16T10:34:32.708900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:35:42.253007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1526,"num_read_kb":1283,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":9330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},"log_size":973,"ondisk_log_size":973,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":119,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":14,"stat_sum":{"num_bytes":603785,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1194153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":42,"acting":42,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":441,"seq":1894080577616,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250764,"kb_used_data":1532,"kb_used_omap":7,"kb_used_meta":249208,"kb_avail":93494388,"statfs":{"total":95995035648,"available":95738253312,"internally_reserved":0,"allocated":1568768,"data_stored":988835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8012,"internal_metadata":255189172},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528584,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834832,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92910320,"statfs":{"total":95995035648,"available":95140167680,"internally_reserved":0,"allocated":1232896,"data_stored":660497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":853604806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790222,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781256,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92963896,"statfs":{"total":95995035648,"available":95195029504,"internally_reserved":0,"allocated":1822720,"data_stored":1245507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986451,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686320,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93058832,"statfs":{"total":95995035648,"available":95292243968,"internally_reserved":0,"allocated":1265664,"data_stored":661538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":24,"seq":103079215384,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917360,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92827792,"statfs":{"total":95995035648,"available":95055659008,"internally_reserved":0,"allocated":1855488,"data_stored":1252745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":937490888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411614,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":606036,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139116,"statfs":{"total":95995035648,"available":95374454784,"internally_reserved":0,"allocated":1503232,"data_stored":923456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575137,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862632,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92882520,"statfs":{"total":95995035648,"available":95111700480,"internally_reserved":0,"allocated":1265664,"data_stored":655348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738663,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938976,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806176,"statfs":{"total":95995035648,"available":95033524224,"internally_reserved":0,"allocated":1253376,"data_stored":661967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3572,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:57.695 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T10:34:57.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad shell --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T10:34:57.960 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T10:34:58.348 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:58 smithi063 ceph-mon[29993]: 4.10 scrub starts 2024-09-16T10:34:58.348 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:58 smithi063 ceph-mon[29993]: 4.10 scrub ok 2024-09-16T10:34:58.348 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:58 smithi063 ceph-mon[29993]: pgmap v1188: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-16T10:34:58.348 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:58 smithi063 ceph-mon[39470]: 4.10 scrub starts 2024-09-16T10:34:58.348 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:58 smithi063 ceph-mon[39470]: 4.10 scrub ok 2024-09-16T10:34:58.349 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:58 smithi063 ceph-mon[39470]: pgmap v1188: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-16T10:34:58.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:58 smithi165 ceph-mon[32696]: 4.10 scrub starts 2024-09-16T10:34:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:58 smithi165 ceph-mon[32696]: 4.10 scrub ok 2024-09-16T10:34:58.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:58 smithi165 ceph-mon[32696]: pgmap v1188: 145 pgs: 1 active+remapped+backfill_toofull, 144 active+clean; 596 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 958 B/s rd, 0 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-16T10:34:58.935 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:58 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:34:58] ENGINE Bus STOPPING 2024-09-16T10:34:59.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:34:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T10:34:59.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:34:59] ENGINE Bus STOPPED 2024-09-16T10:34:59.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:34:59] ENGINE Bus STARTING 2024-09-16T10:34:59.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:34:59] ENGINE Serving on http://:::9283 2024-09-16T10:34:59.202 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:34:59] ENGINE Bus STARTED 2024-09-16T10:34:59.289 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-16T10:34:59.300 INFO:tasks.cephadm:Teardown begin 2024-09-16T10:34:59.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T10:34:59.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T10:34:59.402 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-16T10:34:59.402 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T10:34:59.431 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T10:34:59.455 INFO:tasks.cephadm:Stopping all daemons... 2024-09-16T10:34:59.456 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-16T10:34:59.456 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a 2024-09-16T10:34:59.518 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='client.? v1:172.21.15.63:0/4161425105' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-16T10:34:59.518 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='client.47015 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-16T10:34:59.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='client.47015 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-16T10:34:59.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:34:59.519 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:34:59.520 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:34:59.520 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:59.520 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[29993]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:34:59.520 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='client.? v1:172.21.15.63:0/4161425105' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-16T10:34:59.521 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='client.47015 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-16T10:34:59.521 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='client.47015 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-16T10:34:59.521 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:34:59.521 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:34:59.521 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:34:59.522 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:59.522 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-mon[39470]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:34:59.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug there is no tcmu-runner data available 2024-09-16T10:34:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='client.? v1:172.21.15.63:0/4161425105' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-16T10:34:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='client.47015 ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-16T10:34:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='client.47015 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-16T10:34:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T10:34:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T10:34:59.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T10:34:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='mgr.14649 ' entity='mgr.y' 2024-09-16T10:34:59.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:34:59 smithi165 ceph-mon[32696]: from='mgr.14649 v1:172.21.15.63:0/1367109663' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T10:34:59.791 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 systemd[1]: Stopping Ceph mon.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:00.118 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:34:59.789+0000 7f17f8000640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:00.118 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a[29989]: 2024-09-16T10:34:59.789+0000 7f17f8000640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-16T10:35:00.118 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:34:59 smithi063 podman[206218]: 2024-09-16 10:34:59.830508062 +0000 UTC m=+0.154029807 container died c4c048569e6bdc80f3a9f58f75d86c89dc5d61095591bacc7a4ae2b89916c382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T10:35:00.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:35:00 smithi063 podman[206218]: 2024-09-16 10:35:00.214159734 +0000 UTC m=+0.537681478 container cleanup c4c048569e6bdc80f3a9f58f75d86c89dc5d61095591bacc7a4ae2b89916c382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-16T10:35:00.451 INFO:journalctl@ceph.mon.a.smithi063.stdout:Sep 16 10:35:00 smithi063 bash[206218]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-a 2024-09-16T10:35:00.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.a.service' 2024-09-16T10:35:00.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:00.638 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-16T10:35:00.639 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-16T10:35:00.639 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.c 2024-09-16T10:35:00.921 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:35:00 smithi063 systemd[1]: Stopping Ceph mon.c for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:01.192 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:00 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:35:00] ENGINE Bus STOPPING 2024-09-16T10:35:01.192 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:35:00 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-c[39466]: 2024-09-16T10:35:00.918+0000 7f5e4c3ae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:01.192 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:35:00 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-c[39466]: 2024-09-16T10:35:00.918+0000 7f5e4c3ae640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-16T10:35:01.192 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:35:01 smithi063 podman[206339]: 2024-09-16 10:35:01.062696265 +0000 UTC m=+0.239666706 container died 4aa46339769bb956d57b6d83a50d6699ca473289b64ac65066da8c0db17e09f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:01.446 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:35:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T10:35:01.447 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:35:01] ENGINE Bus STOPPED 2024-09-16T10:35:01.447 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:35:01] ENGINE Bus STARTING 2024-09-16T10:35:01.447 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:35:01] ENGINE Serving on http://:::9283 2024-09-16T10:35:01.447 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:01 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: [16/Sep/2024:10:35:01] ENGINE Bus STARTED 2024-09-16T10:35:01.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:35:01 smithi063 podman[206339]: 2024-09-16 10:35:01.44473566 +0000 UTC m=+0.621706093 container cleanup 4aa46339769bb956d57b6d83a50d6699ca473289b64ac65066da8c0db17e09f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-16T10:35:01.701 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:35:01 smithi063 bash[206339]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-c 2024-09-16T10:35:01.702 INFO:journalctl@ceph.mon.c.smithi063.stdout:Sep 16 10:35:01 smithi063 podman[206351]: 2024-09-16 10:35:01.654729406 +0000 UTC m=+0.565701971 container remove 4aa46339769bb956d57b6d83a50d6699ca473289b64ac65066da8c0db17e09f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:01.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.c.service' 2024-09-16T10:35:01.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:01.915 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-16T10:35:01.915 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-16T10:35:01.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.b 2024-09-16T10:35:02.246 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:35:01 smithi165 systemd[1]: Stopping Ceph mon.b for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:35:02 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-b[32692]: 2024-09-16T10:35:02.243+0000 7fba7f107640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:35:02 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-b[32692]: 2024-09-16T10:35:02.243+0000 7fba7f107640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-16T10:35:02.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:35:02 smithi165 podman[148123]: 2024-09-16 10:35:02.264775561 +0000 UTC m=+0.142064297 container died 7c7bce6d8fc5e6fab9e6c6cd91112185f3b1b76a5b9e03e10d63f11334113818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-16T10:35:03.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:35:02 smithi165 podman[148123]: 2024-09-16 10:35:02.652370242 +0000 UTC m=+0.529658978 container cleanup 7c7bce6d8fc5e6fab9e6c6cd91112185f3b1b76a5b9e03e10d63f11334113818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T10:35:03.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 16 10:35:02 smithi165 bash[148123]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mon-b 2024-09-16T10:35:03.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mon.b.service' 2024-09-16T10:35:03.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:03.277 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-16T10:35:03.278 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-16T10:35:03.278 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.y 2024-09-16T10:35:03.575 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:03 smithi063 systemd[1]: Stopping Ceph mgr.y for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:03.901 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:03 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:35:03.573+0000 7f717921b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:03.901 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:03 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y[30218]: 2024-09-16T10:35:03.573+0000 7f717921b640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-16T10:35:03.901 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:03 smithi063 podman[206475]: 2024-09-16 10:35:03.605830755 +0000 UTC m=+0.128422543 container died 7e145543eced72c9a35c52fb052a4700300b514fb6991e436df82e66e7c9a5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux ) 2024-09-16T10:35:04.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:04 smithi063 podman[206475]: 2024-09-16 10:35:04.091592261 +0000 UTC m=+0.614184048 container cleanup 7e145543eced72c9a35c52fb052a4700300b514fb6991e436df82e66e7c9a5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:66e33d6af1496291b0602ee0a859e68145f17dad, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-16T10:35:04.201 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Sep 16 10:35:04 smithi063 bash[206475]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-y 2024-09-16T10:35:04.556 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-16T10:35:04.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:04.595 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-16T10:35:04.595 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-16T10:35:04.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.x 2024-09-16T10:35:04.931 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:35:04 smithi165 systemd[1]: Stopping Ceph mgr.x for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:05.225 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:35:04 smithi165 podman[148247]: 2024-09-16 10:35:04.929244812 +0000 UTC m=+0.144478700 container died ff1338ab8ee94424990d3f3ae81b478e759c4a4f762d2c51859fc1ec0f3aa688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6) 2024-09-16T10:35:05.552 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:35:05 smithi165 podman[148247]: 2024-09-16 10:35:05.331503975 +0000 UTC m=+0.546737860 container cleanup ff1338ab8ee94424990d3f3ae81b478e759c4a4f762d2c51859fc1ec0f3aa688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T10:35:05.552 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 16 10:35:05 smithi165 bash[148247]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-mgr-x 2024-09-16T10:35:05.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-16T10:35:05.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:05.722 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-16T10:35:05.722 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-16T10:35:05.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.0 2024-09-16T10:35:06.023 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:05 smithi063 systemd[1]: Stopping Ceph osd.0 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:06.451 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:06 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0[42760]: 2024-09-16T10:35:06.020+0000 7fdb3df31640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:06.451 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:06 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0[42760]: 2024-09-16T10:35:06.020+0000 7fdb3df31640 -1 osd.0 633 *** Got signal Terminated *** 2024-09-16T10:35:06.451 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:06 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0[42760]: 2024-09-16T10:35:06.020+0000 7fdb3df31640 -1 osd.0 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:35:11.800 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:11 smithi063 podman[206597]: 2024-09-16 10:35:11.496438626 +0000 UTC m=+5.576552708 container died 4685d83a4c70f03747a87f1848f1e8cf2573ab8817fb67be2cf3a90dce20d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:12.201 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:11 smithi063 podman[206597]: 2024-09-16 10:35:11.947761736 +0000 UTC m=+6.027875858 container cleanup 4685d83a4c70f03747a87f1848f1e8cf2573ab8817fb67be2cf3a90dce20d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-16T10:35:12.201 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:11 smithi063 bash[206597]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0 2024-09-16T10:35:12.543 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:12 smithi063 podman[206612]: 2024-09-16 10:35:12.228089086 +0000 UTC m=+0.732740450 container remove 4685d83a4c70f03747a87f1848f1e8cf2573ab8817fb67be2cf3a90dce20d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-16T10:35:12.830 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:12 smithi063 podman[206701]: 2024-09-16 10:35:12.542081621 +0000 UTC m=+0.111825363 container create 2b329989fc5d6c569861d805f71d316d4ca35c048a0818cd1b3515095286e171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-16T10:35:12.830 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:12 smithi063 podman[206701]: 2024-09-16 10:35:12.447381791 +0000 UTC m=+0.017125543 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:35:13.202 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:12 smithi063 podman[206701]: 2024-09-16 10:35:12.828280136 +0000 UTC m=+0.398023876 container init 2b329989fc5d6c569861d805f71d316d4ca35c048a0818cd1b3515095286e171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-16T10:35:13.202 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:12 smithi063 podman[206701]: 2024-09-16 10:35:12.831924 +0000 UTC m=+0.401667749 container start 2b329989fc5d6c569861d805f71d316d4ca35c048a0818cd1b3515095286e171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-16T10:35:13.202 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:12 smithi063 podman[206701]: 2024-09-16 10:35:12.91156094 +0000 UTC m=+0.481304694 container attach 2b329989fc5d6c569861d805f71d316d4ca35c048a0818cd1b3515095286e171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-16T10:35:13.491 INFO:journalctl@ceph.osd.0.smithi063.stdout:Sep 16 10:35:13 smithi063 podman[206701]: 2024-09-16 10:35:13.205851054 +0000 UTC m=+0.775594794 container died 2b329989fc5d6c569861d805f71d316d4ca35c048a0818cd1b3515095286e171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:13.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.0.service' 2024-09-16T10:35:14.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:14.011 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-16T10:35:14.011 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-16T10:35:14.011 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.1 2024-09-16T10:35:14.347 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:14 smithi063 systemd[1]: Stopping Ceph osd.1 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:14.701 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:14 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1[48525]: 2024-09-16T10:35:14.343+0000 7fe4e8c30640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:14.701 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:14 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1[48525]: 2024-09-16T10:35:14.343+0000 7fe4e8c30640 -1 osd.1 633 *** Got signal Terminated *** 2024-09-16T10:35:14.701 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:14 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1[48525]: 2024-09-16T10:35:14.343+0000 7fe4e8c30640 -1 osd.1 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:35:20.281 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:20 smithi063 podman[206813]: 2024-09-16 10:35:20.005936119 +0000 UTC m=+5.746546786 container died af23d5391365255a05d837a7213f52dd8d31798935c740b24c912663f65c54ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:20.639 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:20 smithi063 podman[206813]: 2024-09-16 10:35:20.428735876 +0000 UTC m=+6.169346601 container cleanup af23d5391365255a05d837a7213f52dd8d31798935c740b24c912663f65c54ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:20.640 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:20 smithi063 bash[206813]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1 2024-09-16T10:35:20.951 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:20 smithi063 podman[206828]: 2024-09-16 10:35:20.638568926 +0000 UTC m=+0.629972445 container remove af23d5391365255a05d837a7213f52dd8d31798935c740b24c912663f65c54ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6) 2024-09-16T10:35:21.376 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:21 smithi063 podman[206919]: 2024-09-16 10:35:20.958025684 +0000 UTC m=+0.016549807 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:35:21.376 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:21 smithi063 podman[206919]: 2024-09-16 10:35:21.072969978 +0000 UTC m=+0.131494091 container create e9b824448933e6731e7d014b20537ead36d12456263c18f360ca8b294d5f40ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T10:35:21.376 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:21 smithi063 podman[206919]: 2024-09-16 10:35:21.300386185 +0000 UTC m=+0.358910294 container init e9b824448933e6731e7d014b20537ead36d12456263c18f360ca8b294d5f40ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:21.377 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:21 smithi063 podman[206919]: 2024-09-16 10:35:21.303728329 +0000 UTC m=+0.362252439 container start e9b824448933e6731e7d014b20537ead36d12456263c18f360ca8b294d5f40ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-16T10:35:21.377 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:21 smithi063 podman[206919]: 2024-09-16 10:35:21.375140306 +0000 UTC m=+0.433664415 container attach e9b824448933e6731e7d014b20537ead36d12456263c18f360ca8b294d5f40ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-16T10:35:21.701 INFO:journalctl@ceph.osd.1.smithi063.stdout:Sep 16 10:35:21 smithi063 podman[206919]: 2024-09-16 10:35:21.496466602 +0000 UTC m=+0.554990721 container died e9b824448933e6731e7d014b20537ead36d12456263c18f360ca8b294d5f40ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-16T10:35:22.154 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.1.service' 2024-09-16T10:35:22.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:22.200 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-16T10:35:22.200 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-16T10:35:22.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.2 2024-09-16T10:35:22.452 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:22 smithi063 systemd[1]: Stopping Ceph osd.2 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:22.951 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:22 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2[54013]: 2024-09-16T10:35:22.515+0000 7fee76614640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:22.952 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:22 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2[54013]: 2024-09-16T10:35:22.515+0000 7fee76614640 -1 osd.2 633 *** Got signal Terminated *** 2024-09-16T10:35:22.952 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:22 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2[54013]: 2024-09-16T10:35:22.515+0000 7fee76614640 -1 osd.2 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:35:28.410 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:28 smithi063 podman[207021]: 2024-09-16 10:35:28.137695817 +0000 UTC m=+5.740066365 container died daec8da9377ad356b5559477d42a4689332d1dc4ee9184704fc5c3692c9de975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T10:35:28.701 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:28 smithi063 podman[207021]: 2024-09-16 10:35:28.582892767 +0000 UTC m=+6.185263312 container cleanup daec8da9377ad356b5559477d42a4689332d1dc4ee9184704fc5c3692c9de975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-16T10:35:28.702 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:28 smithi063 bash[207021]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2 2024-09-16T10:35:29.199 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:28 smithi063 podman[207036]: 2024-09-16 10:35:28.876449786 +0000 UTC m=+0.741254440 container remove daec8da9377ad356b5559477d42a4689332d1dc4ee9184704fc5c3692c9de975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-16T10:35:29.451 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:29 smithi063 podman[207129]: 2024-09-16 10:35:29.101294762 +0000 UTC m=+0.016716085 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:35:29.451 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:29 smithi063 podman[207129]: 2024-09-16 10:35:29.210638313 +0000 UTC m=+0.126059661 container create 7d09154bbbdc6e4a298893eb2df5a5bc5d30487e715c784e2cd341e527f06327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-16T10:35:29.451 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:29 smithi063 podman[207129]: 2024-09-16 10:35:29.446299897 +0000 UTC m=+0.361721213 container init 7d09154bbbdc6e4a298893eb2df5a5bc5d30487e715c784e2cd341e527f06327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_CLEAN=True) 2024-09-16T10:35:29.451 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:29 smithi063 podman[207129]: 2024-09-16 10:35:29.449722976 +0000 UTC m=+0.365144285 container start 7d09154bbbdc6e4a298893eb2df5a5bc5d30487e715c784e2cd341e527f06327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:29.701 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:29 smithi063 podman[207129]: 2024-09-16 10:35:29.54643193 +0000 UTC m=+0.461853238 container attach 7d09154bbbdc6e4a298893eb2df5a5bc5d30487e715c784e2cd341e527f06327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-16T10:35:29.702 INFO:journalctl@ceph.osd.2.smithi063.stdout:Sep 16 10:35:29 smithi063 podman[207129]: 2024-09-16 10:35:29.644633999 +0000 UTC m=+0.560055306 container died 7d09154bbbdc6e4a298893eb2df5a5bc5d30487e715c784e2cd341e527f06327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-16T10:35:30.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.2.service' 2024-09-16T10:35:30.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:30.528 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-16T10:35:30.528 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-16T10:35:30.528 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.3 2024-09-16T10:35:30.762 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:30 smithi063 systemd[1]: Stopping Ceph osd.3 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:31.201 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:30 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:35:30.853+0000 7fdd4c1fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:31.201 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:30 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:35:30.853+0000 7fdd4c1fe640 -1 osd.3 633 *** Got signal Terminated *** 2024-09-16T10:35:31.202 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:30 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:35:30.853+0000 7fdd4c1fe640 -1 osd.3 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:35:32.951 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:32 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:35:32.520+0000 7fdd4480b640 -1 osd.3 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.714921+0000 front 2024-09-16T10:35:07.715029+0000 (oldest deadline 2024-09-16T10:35:31.815087+0000) 2024-09-16T10:35:33.052 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:32 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:32.560+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:33.951 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:33 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:35:33.512+0000 7fdd4480b640 -1 osd.3 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.714921+0000 front 2024-09-16T10:35:07.715029+0000 (oldest deadline 2024-09-16T10:35:31.815087+0000) 2024-09-16T10:35:34.051 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:33 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:33.603+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:34.501 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:34 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:34.121+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:34.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:34 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:34.499+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:34.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:34 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:34.581+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:34.803 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:34 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:34.502+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:34.951 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:34 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:35:34.549+0000 7fdd4480b640 -1 osd.3 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.714921+0000 front 2024-09-16T10:35:07.715029+0000 (oldest deadline 2024-09-16T10:35:31.815087+0000) 2024-09-16T10:35:35.471 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:35 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:35.171+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:35.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:35 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:35.589+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:35.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:35 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:35.469+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:35.803 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:35 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:35.491+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:35.951 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:35 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3[59408]: 2024-09-16T10:35:35.564+0000 7fdd4480b640 -1 osd.3 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.714921+0000 front 2024-09-16T10:35:07.715029+0000 (oldest deadline 2024-09-16T10:35:31.815087+0000) 2024-09-16T10:35:36.467 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:36 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:36.148+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:36.764 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:36 smithi063 podman[207234]: 2024-09-16 10:35:36.461461733 +0000 UTC m=+5.730453043 container died 811187484ff6d70085b572773448b781a9d0f3e1a3de00175c0afee53bde329a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-16T10:35:36.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:36 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:36.465+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:36.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:36 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:36.591+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:36.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:36 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:36.467+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:37.201 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:36 smithi063 podman[207234]: 2024-09-16 10:35:36.937017758 +0000 UTC m=+6.206009057 container cleanup 811187484ff6d70085b572773448b781a9d0f3e1a3de00175c0afee53bde329a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-16T10:35:37.201 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:36 smithi063 bash[207234]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3 2024-09-16T10:35:37.459 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:37 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:37.123+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:37.599 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:37 smithi063 podman[207262]: 2024-09-16 10:35:37.280611933 +0000 UTC m=+0.816566197 container remove 811187484ff6d70085b572773448b781a9d0f3e1a3de00175c0afee53bde329a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6) 2024-09-16T10:35:37.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:37 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:37.591+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:37.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:37 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:37.457+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:37.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:37 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:37.486+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:37.868 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:37 smithi063 podman[207352]: 2024-09-16 10:35:37.597974945 +0000 UTC m=+0.111114287 container create 126553f3b4e45721b2157fe62a7132caccf6820ddcd35adfa33ab0a75abdd88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-16T10:35:37.868 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:37 smithi063 podman[207352]: 2024-09-16 10:35:37.503445962 +0000 UTC m=+0.016585313 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:35:37.868 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:37 smithi063 podman[207352]: 2024-09-16 10:35:37.792356632 +0000 UTC m=+0.305495986 container init 126553f3b4e45721b2157fe62a7132caccf6820ddcd35adfa33ab0a75abdd88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-16T10:35:37.869 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:37 smithi063 podman[207352]: 2024-09-16 10:35:37.795660632 +0000 UTC m=+0.308799974 container start 126553f3b4e45721b2157fe62a7132caccf6820ddcd35adfa33ab0a75abdd88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-16T10:35:37.869 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:37 smithi063 podman[207352]: 2024-09-16 10:35:37.867206935 +0000 UTC m=+0.380346277 container attach 126553f3b4e45721b2157fe62a7132caccf6820ddcd35adfa33ab0a75abdd88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-09-16T10:35:38.201 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:37 smithi063 podman[207352]: 2024-09-16 10:35:37.993436259 +0000 UTC m=+0.506575609 container died 126553f3b4e45721b2157fe62a7132caccf6820ddcd35adfa33ab0a75abdd88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T10:35:38.439 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:38.151+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:38.546 INFO:journalctl@ceph.osd.3.smithi063.stdout:Sep 16 10:35:38 smithi063 podman[207374]: 2024-09-16 10:35:38.529006378 +0000 UTC m=+0.529449462 container remove 126553f3b4e45721b2157fe62a7132caccf6820ddcd35adfa33ab0a75abdd88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-16T10:35:38.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.3.service' 2024-09-16T10:35:38.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:38.595 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-16T10:35:38.596 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-16T10:35:38.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.4 2024-09-16T10:35:38.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:38.486+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:38.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:38.584+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:38.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:38 smithi165 systemd[1]: Stopping Ceph osd.4 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:38.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:38.437+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:39.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:39.152+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:39.302 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:38.941+0000 7f2b21042640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:39.302 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:38.941+0000 7f2b21042640 -1 osd.4 633 *** Got signal Terminated *** 2024-09-16T10:35:39.302 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:38 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:38.941+0000 7f2b21042640 -1 osd.4 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:35:39.803 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:39.560+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:39.803 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:39.470+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:39.803 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:39 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:39.443+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:40.514 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:40 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:40.394+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:40.515 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:40 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:40.186+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:40.801 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:40 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:40.516+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:40.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:40 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:40.513+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:41.540 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:41 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:41.405+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:41.540 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:41 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:41.405+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:41.540 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:41 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:41.158+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:41.540 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:41 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:41.158+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:41.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:41 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:41.548+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:41.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:41 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:41.538+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:42.422 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:42.143+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:42.422 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:42.143+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:42.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:42.527+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:42.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:42.511+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:42.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:42.420+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:42.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:42.420+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:43.512 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:43.372+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:43.513 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:43.372+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:43.513 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:43.187+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:43.513 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:43.187+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:43.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:43.510+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:08.720857+0000 front 2024-09-16T10:35:08.720655+0000 (oldest deadline 2024-09-16T10:35:31.620584+0000) 2024-09-16T10:35:43.802 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4[36800]: 2024-09-16T10:35:43.510+0000 7f2b1964f640 -1 osd.4 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:13.921597+0000 front 2024-09-16T10:35:13.921482+0000 (oldest deadline 2024-09-16T10:35:43.321965+0000) 2024-09-16T10:35:43.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:43.544+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:43.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:43 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:43.544+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:44.551 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:44 smithi165 podman[148384]: 2024-09-16 10:35:44.380377481 +0000 UTC m=+5.584445308 container died f6a58053597e3b274c3750b81d1571113861928a32443a8e813459c0bbfa8332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-16T10:35:44.552 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:44 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:44.368+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:44.552 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:44 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:44.368+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:44.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:44 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:44.183+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:44.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:44 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:44.183+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:44.808 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:44 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:44.581+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:44.808 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:44 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:44.581+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:45.135 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:44 smithi165 podman[148384]: 2024-09-16 10:35:44.815787916 +0000 UTC m=+6.019855742 container cleanup f6a58053597e3b274c3750b81d1571113861928a32443a8e813459c0bbfa8332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6) 2024-09-16T10:35:45.135 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:44 smithi165 bash[148384]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4 2024-09-16T10:35:45.136 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:45 smithi165 podman[148399]: 2024-09-16 10:35:45.025905889 +0000 UTC m=+0.647470506 container remove f6a58053597e3b274c3750b81d1571113861928a32443a8e813459c0bbfa8332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:45.443 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:45.380+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:45.444 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:45.380+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:45.444 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:45.133+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:45.444 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:45.133+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:45.721 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:45.593+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:45.722 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:45 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:45.593+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:45.722 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:45 smithi165 podman[148497]: 2024-09-16 10:35:45.345646632 +0000 UTC m=+0.017171760 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:35:45.722 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:45 smithi165 podman[148497]: 2024-09-16 10:35:45.468426561 +0000 UTC m=+0.139951681 container create 8cc466e869808c0735792db3baaebba31faba52d2904ccd98917355f446809a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-16T10:35:46.034 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:45 smithi165 podman[148497]: 2024-09-16 10:35:45.720670249 +0000 UTC m=+0.392195368 container init 8cc466e869808c0735792db3baaebba31faba52d2904ccd98917355f446809a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-16T10:35:46.034 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:45 smithi165 podman[148497]: 2024-09-16 10:35:45.724639599 +0000 UTC m=+0.396164723 container start 8cc466e869808c0735792db3baaebba31faba52d2904ccd98917355f446809a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-16T10:35:46.034 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:45 smithi165 podman[148497]: 2024-09-16 10:35:45.82908552 +0000 UTC m=+0.500610657 container attach 8cc466e869808c0735792db3baaebba31faba52d2904ccd98917355f446809a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-16T10:35:46.302 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 16 10:35:46 smithi165 podman[148497]: 2024-09-16 10:35:46.032832669 +0000 UTC m=+0.704357788 container died 8cc466e869808c0735792db3baaebba31faba52d2904ccd98917355f446809a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-16T10:35:46.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:46.158+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:46.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:46.158+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:46.667 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:46.416+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:46.667 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:46.416+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:46.668 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:46.619+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:46.668 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:46 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:46.619+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:46.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.4.service' 2024-09-16T10:35:46.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:46.734 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-16T10:35:46.734 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-16T10:35:46.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.5 2024-09-16T10:35:47.052 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:46 smithi165 systemd[1]: Stopping Ceph osd.5 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:47.373 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:47.127+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:47.373 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:47.127+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:47.374 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:47.077+0000 7f0963718640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:47.374 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:47.077+0000 7f0963718640 -1 osd.5 633 *** Got signal Terminated *** 2024-09-16T10:35:47.374 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:47.077+0000 7f0963718640 -1 osd.5 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:35:47.374 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:47.371+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:47.374 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:47.371+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:48.052 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:47.622+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:48.052 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:47 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:47.622+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:48.416 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:48.415+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:48.417 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:48.415+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:48.417 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:48.165+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:48.417 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:48.165+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:48.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:48.636+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:48.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:48 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:48.636+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:49.460 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:49.155+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:49.461 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:49.155+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:49.461 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:49.155+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:49.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:49.458+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:49.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:49.458+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:49.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:49.678+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:49.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:49 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:49.678+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:50.449 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:50 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:50.131+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:50.449 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:50 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:50.131+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:50.449 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:50 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:50.131+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:50.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:50 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:50.447+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:50.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:50 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:50.447+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:50.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:50 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:50.669+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:50.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:50 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:50.669+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:51.399 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:51.138+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:51.399 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:51.138+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:51.399 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:51.138+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:51.656 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:51.398+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:10.803022+0000 front 2024-09-16T10:35:10.803108+0000 (oldest deadline 2024-09-16T10:35:33.702991+0000) 2024-09-16T10:35:51.656 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:51.398+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:15.404032+0000 front 2024-09-16T10:35:15.403937+0000 (oldest deadline 2024-09-16T10:35:40.703790+0000) 2024-09-16T10:35:51.656 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5[42067]: 2024-09-16T10:35:51.398+0000 7f095fd2d640 -1 osd.5 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:26.406001+0000 front 2024-09-16T10:35:26.405908+0000 (oldest deadline 2024-09-16T10:35:50.505899+0000) 2024-09-16T10:35:52.052 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:51.654+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:52.052 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:51 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:51.654+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:52.527 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:52 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:52.126+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:52.527 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:52 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:52.126+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:52.527 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:52 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:52.126+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:52.802 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:52 smithi165 podman[148600]: 2024-09-16 10:35:52.526421171 +0000 UTC m=+5.568070037 container died 8bfb27825d39640c82dae209aeabbac8cd80c84ab176f43265dcd0306722ef51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-16T10:35:52.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:52 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:52.632+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:52.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:52 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:52.632+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:52.803 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:52 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:52.632+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:53.137 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:52 smithi165 podman[148600]: 2024-09-16 10:35:52.918404346 +0000 UTC m=+5.960053197 container cleanup 8bfb27825d39640c82dae209aeabbac8cd80c84ab176f43265dcd0306722ef51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-16T10:35:53.137 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:52 smithi165 bash[148600]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5 2024-09-16T10:35:53.473 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:53 smithi165 podman[148616]: 2024-09-16 10:35:53.161125587 +0000 UTC m=+0.629022271 container remove 8bfb27825d39640c82dae209aeabbac8cd80c84ab176f43265dcd0306722ef51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-16T10:35:53.474 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:53.135+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:53.474 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:53.135+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:53.474 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:53.135+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:53.732 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:53 smithi165 podman[148705]: 2024-09-16 10:35:53.375453997 +0000 UTC m=+0.016263677 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:35:53.733 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:53 smithi165 podman[148705]: 2024-09-16 10:35:53.495345041 +0000 UTC m=+0.136154716 container create 35119a3644c20a77fe47029f2b9c06f02e3b6b0cf7e39bf8b888c2836054b0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-16T10:35:53.733 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:53.583+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:53.733 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:53.583+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:53.733 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:53 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:53.583+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:54.052 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:53 smithi165 podman[148705]: 2024-09-16 10:35:53.731685829 +0000 UTC m=+0.372495516 container init 35119a3644c20a77fe47029f2b9c06f02e3b6b0cf7e39bf8b888c2836054b0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-16T10:35:54.052 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:53 smithi165 podman[148705]: 2024-09-16 10:35:53.735408292 +0000 UTC m=+0.376217971 container start 35119a3644c20a77fe47029f2b9c06f02e3b6b0cf7e39bf8b888c2836054b0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-16T10:35:54.052 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:53 smithi165 podman[148705]: 2024-09-16 10:35:53.822550661 +0000 UTC m=+0.463360348 container attach 35119a3644c20a77fe47029f2b9c06f02e3b6b0cf7e39bf8b888c2836054b0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-16T10:35:54.053 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 16 10:35:53 smithi165 podman[148705]: 2024-09-16 10:35:53.923450497 +0000 UTC m=+0.564260174 container died 35119a3644c20a77fe47029f2b9c06f02e3b6b0cf7e39bf8b888c2836054b0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:35:54.527 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:54.091+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:54.527 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:54.091+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:54.527 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:54.091+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:54.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.5.service' 2024-09-16T10:35:54.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:35:54.590 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-16T10:35:54.590 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-16T10:35:54.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.6 2024-09-16T10:35:54.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:54.621+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:54.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:54.621+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:54.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:54.621+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:54.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:54 smithi165 systemd[1]: Stopping Ceph osd.6 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:35:55.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:55.047+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:55.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:55.047+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:55.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:55.047+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:55.302 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:54.911+0000 7fecf44a9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:35:55.303 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:54.911+0000 7fecf44a9640 -1 osd.6 633 *** Got signal Terminated *** 2024-09-16T10:35:55.303 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:54 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:54.911+0000 7fecf44a9640 -1 osd.6 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:35:55.911 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:55.599+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:55.911 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:55.599+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:55.911 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:55.599+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:55.911 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:35:55.577Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.63:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.63:8765: connect: connection refused" 2024-09-16T10:35:55.911 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:35:55.581Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.63:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.63:8765: connect: connection refused" 2024-09-16T10:35:55.912 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:35:55.581Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.63:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.63:8765: connect: connection refused" 2024-09-16T10:35:55.912 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:35:55.581Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.63:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.63:8765: connect: connection refused" 2024-09-16T10:35:55.912 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:35:55.581Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.63:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.63:8765: connect: connection refused" 2024-09-16T10:35:55.912 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:35:55.583Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.63:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.63:8765: connect: connection refused" 2024-09-16T10:35:55.912 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:35:55 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:35:55.584Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.63:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.63:8765: connect: connection refused" 2024-09-16T10:35:56.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:56.020+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:56.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:56.020+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:56.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:56.020+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:56.984 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:56.645+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:56.984 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:56.645+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:56.984 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:56.645+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:57.301 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:56.982+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:57.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:56.982+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:57.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:56.982+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:57.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:56 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:56.982+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:35:57.994 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:57.599+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:57.995 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:57.599+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:57.995 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:57.599+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:57.995 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:57.599+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:32.777647+0000 front 2024-09-16T10:35:32.777564+0000 (oldest deadline 2024-09-16T10:35:57.378144+0000) 2024-09-16T10:35:58.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:57.993+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:58.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:57.993+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:58.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:57.993+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:58.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:57 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:57.993+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:35:59.044 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:58.574+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:59.044 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:58.574+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:59.044 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:58.574+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:59.044 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:58 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:58.574+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:32.777647+0000 front 2024-09-16T10:35:32.777564+0000 (oldest deadline 2024-09-16T10:35:57.378144+0000) 2024-09-16T10:35:59.301 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:59.042+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:35:59.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:59.042+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:35:59.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:59.042+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:35:59.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:35:59.042+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:35:59.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:59.527+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.873147+0000 front 2024-09-16T10:35:07.873121+0000 (oldest deadline 2024-09-16T10:35:33.772884+0000) 2024-09-16T10:35:59.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:59.527+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:18.474067+0000 front 2024-09-16T10:35:18.473995+0000 (oldest deadline 2024-09-16T10:35:43.074381+0000) 2024-09-16T10:35:59.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:59.527+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.876088+0000 front 2024-09-16T10:35:25.876102+0000 (oldest deadline 2024-09-16T10:35:51.776036+0000) 2024-09-16T10:35:59.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:35:59 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6[47483]: 2024-09-16T10:35:59.527+0000 7fececab6640 -1 osd.6 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:32.777647+0000 front 2024-09-16T10:35:32.777564+0000 (oldest deadline 2024-09-16T10:35:57.378144+0000) 2024-09-16T10:36:00.430 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:00.080+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:00.430 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:00.080+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:00.431 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:00.080+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:00.431 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:00.080+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:00.431 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:00 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:00.080+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:00.802 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:00 smithi165 podman[148807]: 2024-09-16 10:36:00.429210497 +0000 UTC m=+5.618384793 container died 5a099f12fb1761d92d677bf7967391823608c7c25c818d3694165c50fbafcfa3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-16T10:36:01.205 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:00 smithi165 podman[148807]: 2024-09-16 10:36:00.944376181 +0000 UTC m=+6.133550465 container cleanup 5a099f12fb1761d92d677bf7967391823608c7c25c818d3694165c50fbafcfa3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6, CEPH_POINT_RELEASE=, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T10:36:01.205 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:00 smithi165 bash[148807]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6 2024-09-16T10:36:01.205 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:01.081+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:01.205 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:01.081+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:01.205 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:01.081+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:01.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:01.081+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:01.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:01 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:01.081+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:01.552 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:01 smithi165 podman[148823]: 2024-09-16 10:36:01.204019895 +0000 UTC m=+0.770907853 container remove 5a099f12fb1761d92d677bf7967391823608c7c25c818d3694165c50fbafcfa3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-16T10:36:01.858 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:01 smithi165 podman[148912]: 2024-09-16 10:36:01.466419413 +0000 UTC m=+0.016573481 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:36:01.858 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:01 smithi165 podman[148912]: 2024-09-16 10:36:01.57984314 +0000 UTC m=+0.129997198 container create 052636b8e735264ed732e0ad0ba97938094153538576fd8ac40f5c5742cb0cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-16T10:36:01.858 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:01 smithi165 podman[148912]: 2024-09-16 10:36:01.85723067 +0000 UTC m=+0.407384728 container init 052636b8e735264ed732e0ad0ba97938094153538576fd8ac40f5c5742cb0cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:01 smithi165 podman[148912]: 2024-09-16 10:36:01.860545966 +0000 UTC m=+0.410700038 container start 052636b8e735264ed732e0ad0ba97938094153538576fd8ac40f5c5742cb0cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:01 smithi165 podman[148912]: 2024-09-16 10:36:01.957341806 +0000 UTC m=+0.507495893 container attach 052636b8e735264ed732e0ad0ba97938094153538576fd8ac40f5c5742cb0cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 16 10:36:02 smithi165 podman[148912]: 2024-09-16 10:36:02.055580662 +0000 UTC m=+0.605734721 container died 052636b8e735264ed732e0ad0ba97938094153538576fd8ac40f5c5742cb0cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:02 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:02.043+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:02 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:02.043+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:02 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:02.043+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:02 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:02.043+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:02.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:02 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:02.043+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:02.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.6.service' 2024-09-16T10:36:02.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:36:02.695 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-16T10:36:02.695 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-16T10:36:02.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.7 2024-09-16T10:36:03.023 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:02 smithi165 systemd[1]: Stopping Ceph osd.7 for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.021+0000 7f6495545640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.021+0000 7f6495545640 -1 osd.7 633 *** Got signal Terminated *** 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.021+0000 7f6495545640 -1 osd.7 633 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.071+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.071+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.071+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.071+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:03.302 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:03 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:03.072+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:04.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:04 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:04.068+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:04.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:04 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:04.068+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:04.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:04 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:04.068+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:04.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:04 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:04.068+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:04.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:04 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:04.068+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:05.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:05 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:05.098+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:05.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:05 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:05.098+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:05.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:05 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:05.098+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:05.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:05 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:05.098+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:05.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:05 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:05.098+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:06.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:06.142+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:06.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:06.142+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:06.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:06.142+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:06.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:06.142+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:06.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:06.142+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.165:6802 osd.4 since back 2024-09-16T10:35:39.439411+0000 front 2024-09-16T10:35:39.439458+0000 (oldest deadline 2024-09-16T10:36:05.339582+0000) 2024-09-16T10:36:06.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:06 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:06.143+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:07.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:07 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:07.146+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6803 osd.0 since back 2024-09-16T10:35:07.935596+0000 front 2024-09-16T10:35:07.935721+0000 (oldest deadline 2024-09-16T10:35:33.835397+0000) 2024-09-16T10:36:07.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:07 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:07.146+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6807 osd.1 since back 2024-09-16T10:35:14.336275+0000 front 2024-09-16T10:35:14.336306+0000 (oldest deadline 2024-09-16T10:35:40.236291+0000) 2024-09-16T10:36:07.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:07 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:07.146+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6811 osd.2 since back 2024-09-16T10:35:25.537352+0000 front 2024-09-16T10:35:25.537317+0000 (oldest deadline 2024-09-16T10:35:48.437261+0000) 2024-09-16T10:36:07.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:07 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:07.146+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.63:6815 osd.3 since back 2024-09-16T10:35:35.339288+0000 front 2024-09-16T10:35:35.339358+0000 (oldest deadline 2024-09-16T10:35:59.439177+0000) 2024-09-16T10:36:07.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:07 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:07.146+0000 7f648db52640 -1 osd.7 633 heartbeat_check: no reply from 172.21.15.165:6802 osd.4 since back 2024-09-16T10:35:39.439411+0000 front 2024-09-16T10:35:39.439458+0000 (oldest deadline 2024-09-16T10:36:05.339582+0000) 2024-09-16T10:36:07.552 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:07 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7[123487]: 2024-09-16T10:36:07.146+0000 7f648db52640 -1 osd.7 633 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24481.0:1291 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e633) 2024-09-16T10:36:08.907 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:08 smithi165 podman[149014]: 2024-09-16 10:36:08.617501818 +0000 UTC m=+5.702102648 container died 76d1aad57994b2a8f80541148854e80d92fc8016749ed4a0bc8e4ee7c7af2e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-16T10:36:09.248 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 podman[149014]: 2024-09-16 10:36:09.004222196 +0000 UTC m=+6.088823026 container cleanup 76d1aad57994b2a8f80541148854e80d92fc8016749ed4a0bc8e4ee7c7af2e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T10:36:09.248 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 bash[149014]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7 2024-09-16T10:36:09.551 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 podman[149031]: 2024-09-16 10:36:09.246837259 +0000 UTC m=+0.625984957 container remove 76d1aad57994b2a8f80541148854e80d92fc8016749ed4a0bc8e4ee7c7af2e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-16T10:36:09.817 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 podman[149121]: 2024-09-16 10:36:09.468526292 +0000 UTC m=+0.016416065 image pull 261f2dc7b20f078cba20226e1a82a867f88de1a71ca34322497f8520cc32e62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f 2024-09-16T10:36:09.818 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 podman[149121]: 2024-09-16 10:36:09.589586733 +0000 UTC m=+0.137476496 container create b1f8ef9ef509bf5f686b1d2301e9ef68d85ca481347fe562d53e4fef5a557c82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-16T10:36:09.818 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 podman[149121]: 2024-09-16 10:36:09.816863719 +0000 UTC m=+0.364753483 container init b1f8ef9ef509bf5f686b1d2301e9ef68d85ca481347fe562d53e4fef5a557c82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-16T10:36:10.295 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 podman[149121]: 2024-09-16 10:36:09.820116143 +0000 UTC m=+0.368005911 container start b1f8ef9ef509bf5f686b1d2301e9ef68d85ca481347fe562d53e4fef5a557c82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-16T10:36:10.295 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:09 smithi165 podman[149121]: 2024-09-16 10:36:09.899922929 +0000 UTC m=+0.447812706 container attach b1f8ef9ef509bf5f686b1d2301e9ef68d85ca481347fe562d53e4fef5a557c82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-16T10:36:10.295 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 16 10:36:10 smithi165 podman[149121]: 2024-09-16 10:36:10.007202622 +0000 UTC m=+0.555092400 container died b1f8ef9ef509bf5f686b1d2301e9ef68d85ca481347fe562d53e4fef5a557c82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-16T10:36:10.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@osd.7.service' 2024-09-16T10:36:10.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:36:10.657 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-16T10:36:10.657 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-16T10:36:10.658 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-16T10:36:10.951 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:36:10 smithi063 systemd[1]: Stopping Ceph rgw.foo.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:11.451 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:36:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-rgw-foo-a[63716]: 2024-09-16T10:36:10.971+0000 7f0a80149640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T10:36:11.452 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:36:10 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-rgw-foo-a[63716]: 2024-09-16T10:36:10.971+0000 7f0a839b5880 -1 shutting down 2024-09-16T10:36:21.240 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:36:20 smithi063 bash[207467]: time="2024-09-16T10:36:20Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-16T10:36:21.240 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:36:20 smithi063 podman[207467]: 2024-09-16 10:36:20.990236445 +0000 UTC m=+10.125723608 container died 871b2952b58194a1c5c8842490513bd6c01e57d9012ae866ed3ee2af6e2ee2bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-09-16T10:36:21.831 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:36:21 smithi063 podman[207467]: 2024-09-16 10:36:21.554092135 +0000 UTC m=+10.689579333 container cleanup 871b2952b58194a1c5c8842490513bd6c01e57d9012ae866ed3ee2af6e2ee2bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True) 2024-09-16T10:36:21.831 INFO:journalctl@ceph.rgw.foo.a.smithi063.stdout:Sep 16 10:36:21 smithi063 bash[207467]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-16T10:36:22.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-16T10:36:22.040 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:36:22.040 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-16T10:36:22.040 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-16T10:36:22.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@prometheus.a 2024-09-16T10:36:22.380 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 systemd[1]: Stopping Ceph prometheus.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-16T10:36:22.639 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-16T10:36:22.640 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.379Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-16T10:36:22.640 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.380Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-16T10:36:22.640 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.380Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-16T10:36:22.640 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a[65369]: ts=2024-09-16T10:36:22.380Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-16T10:36:22.640 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 podman[149222]: 2024-09-16 10:36:22.389514277 +0000 UTC m=+0.128793584 container died 6fecfcdf6c17bb87e9ac226e24f82093b36bcc651117fb3eb67de184751a3869 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:23.048 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 podman[149222]: 2024-09-16 10:36:22.787635095 +0000 UTC m=+0.526914399 container cleanup 6fecfcdf6c17bb87e9ac226e24f82093b36bcc651117fb3eb67de184751a3869 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:23.048 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 16 10:36:22 smithi165 bash[149222]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-prometheus-a 2024-09-16T10:36:23.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-16T10:36:23.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T10:36:23.229 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-16T10:36:23.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T10:36:23.443 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:36:25.396 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:25 smithi063 systemd[1]: Stopping Ceph alertmanager.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:25.396 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:25 smithi063 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a[79966]: ts=2024-09-16T10:36:25.343Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-16T10:36:25.695 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:25 smithi063 podman[207955]: 2024-09-16 10:36:25.394880186 +0000 UTC m=+0.151870806 container died c82a0fcfe301e57df27402a073c295e3a01020bd41cbc363cd671d9c941e7716 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:25.951 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:25 smithi063 podman[207955]: 2024-09-16 10:36:25.851900599 +0000 UTC m=+0.608891211 container cleanup c82a0fcfe301e57df27402a073c295e3a01020bd41cbc363cd671d9c941e7716 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:25.951 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:25 smithi063 bash[207955]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a 2024-09-16T10:36:26.451 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:26 smithi063 podman[207969]: 2024-09-16 10:36:26.12022759 +0000 UTC m=+0.718316911 container remove c82a0fcfe301e57df27402a073c295e3a01020bd41cbc363cd671d9c941e7716 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:26.452 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:26 smithi063 podman[207969]: 2024-09-16 10:36:26.187840855 +0000 UTC m=+0.785930133 volume remove 3e05a0e8bf3db90169849de1c5861262611af42f6acbd08e64ea67a63b5fc221 2024-09-16T10:36:26.452 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:26 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-16T10:36:26.452 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:26 smithi063 systemd[1]: Stopped Ceph alertmanager.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:36:26.452 INFO:journalctl@ceph.alertmanager.a.smithi063.stdout:Sep 16 10:36:26 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.725s CPU time. 2024-09-16T10:36:26.810 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:26 smithi063 systemd[1]: Stopping Ceph node-exporter.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:27.201 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:26 smithi063 podman[208080]: 2024-09-16 10:36:26.809597923 +0000 UTC m=+0.114513984 container died e72b9c1b21673229e09f2c498590216431c0c9968da61f9906ed70b3cdee833c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:27.451 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:27 smithi063 podman[208080]: 2024-09-16 10:36:27.3658921 +0000 UTC m=+0.670808160 container cleanup e72b9c1b21673229e09f2c498590216431c0c9968da61f9906ed70b3cdee833c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:27.451 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:27 smithi063 bash[208080]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a 2024-09-16T10:36:27.893 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:27 smithi063 podman[208092]: 2024-09-16 10:36:27.625444873 +0000 UTC m=+0.819586132 container remove e72b9c1b21673229e09f2c498590216431c0c9968da61f9906ed70b3cdee833c (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T10:36:27.893 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:27 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-16T10:36:27.893 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:27 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-16T10:36:27.893 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:27 smithi063 systemd[1]: Stopped Ceph node-exporter.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:36:27.893 INFO:journalctl@ceph.node-exporter.a.smithi063.stdout:Sep 16 10:36:27 smithi063 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 5.152s CPU time. 2024-09-16T10:36:28.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T10:36:28.647 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:36:30.227 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:29 smithi165 systemd[1]: Stopping Ceph iscsi.iscsi.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:30.551 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:30 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a[57218]: debug Shutdown received 2024-09-16T10:36:40.550 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:40 smithi165 bash[149642]: time="2024-09-16T10:36:40Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-16T10:36:40.550 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:40 smithi165 podman[149642]: 2024-09-16 10:36:40.242697504 +0000 UTC m=+10.130153191 container died dfec6f40657bdab50034dc088d10999af77e2d9b09d2ea157e3a0231c6c109f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-16T10:36:40.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:40 smithi165 podman[149642]: 2024-09-16 10:36:40.640298666 +0000 UTC m=+10.527754354 container cleanup dfec6f40657bdab50034dc088d10999af77e2d9b09d2ea157e3a0231c6c109f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-16T10:36:40.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:40 smithi165 bash[149642]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-16T10:36:41.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:40 smithi165 podman[149655]: 2024-09-16 10:36:40.910052415 +0000 UTC m=+0.662789592 container remove dfec6f40657bdab50034dc088d10999af77e2d9b09d2ea157e3a0231c6c109f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:2dbeb26ef01ea72be3b27ac61f923de34ec1a7d8b2fcd428841fc646734c133f, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-athakkar-testing-2024-09-12-1857-66e33d6, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-16T10:36:41.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:40 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-16T10:36:41.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:41 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-16T10:36:41.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:41 smithi165 systemd[1]: Stopped Ceph iscsi.iscsi.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:36:41.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 16 10:36:41 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Consumed 2.657s CPU time. 2024-09-16T10:36:42.662 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:42 smithi165 systemd[1]: Stopping Ceph grafana.a for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:42.662 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=server t=2024-09-16T10:36:42.640199152Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-16T10:36:42.662 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=ticker t=2024-09-16T10:36:42.640528308Z level=info msg=stopped last_tick=2024-09-16T10:36:40Z 2024-09-16T10:36:42.662 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=tracing t=2024-09-16T10:36:42.640572773Z level=info msg="Closing tracing" 2024-09-16T10:36:42.662 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:42 smithi165 ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a[59834]: logger=grafana-apiserver t=2024-09-16T10:36:42.640585677Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-16T10:36:42.662 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:42 smithi165 podman[149918]: 2024-09-16 10:36:42.660665174 +0000 UTC m=+0.128818356 container died ccf521d17e6e50bb3b84daff580f744c6f8045c225c4eca6a9a637ef2f821a47 (image=quay.io/ceph/grafana:10.4.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T10:36:43.283 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:43 smithi165 podman[149918]: 2024-09-16 10:36:43.0395425 +0000 UTC m=+0.507695723 container cleanup ccf521d17e6e50bb3b84daff580f744c6f8045c225c4eca6a9a637ef2f821a47 (image=quay.io/ceph/grafana:10.4.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T10:36:43.283 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:43 smithi165 bash[149918]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a 2024-09-16T10:36:43.551 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:43 smithi165 podman[149929]: 2024-09-16 10:36:43.282069185 +0000 UTC m=+0.622999591 container remove ccf521d17e6e50bb3b84daff580f744c6f8045c225c4eca6a9a637ef2f821a47 (image=quay.io/ceph/grafana:10.4.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T10:36:43.552 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:43 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-16T10:36:43.552 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:43 smithi165 systemd[1]: Stopped Ceph grafana.a for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:36:43.552 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 16 10:36:43 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@grafana.a.service: Consumed 14.475s CPU time. 2024-09-16T10:36:44.052 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:43 smithi165 systemd[1]: Stopping Ceph node-exporter.b for 7cb51a9a-7413-11ef-bceb-c7b262605968... 2024-09-16T10:36:44.052 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:43 smithi165 podman[150045]: 2024-09-16 10:36:43.837286285 +0000 UTC m=+0.104567090 container died 640aeecfab3246c99db2bff4b960e0e7a66f96fb50580dd969f8c9ea6d5b669d (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T10:36:44.470 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:44 smithi165 podman[150045]: 2024-09-16 10:36:44.184875999 +0000 UTC m=+0.452156802 container cleanup 640aeecfab3246c99db2bff4b960e0e7a66f96fb50580dd969f8c9ea6d5b669d (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T10:36:44.470 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:44 smithi165 bash[150045]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b 2024-09-16T10:36:44.756 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:44 smithi165 podman[150058]: 2024-09-16 10:36:44.469244453 +0000 UTC m=+0.631908535 container remove 640aeecfab3246c99db2bff4b960e0e7a66f96fb50580dd969f8c9ea6d5b669d (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-7cb51a9a-7413-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T10:36:44.756 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:44 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-16T10:36:44.756 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:44 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-16T10:36:44.756 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:44 smithi165 systemd[1]: Stopped Ceph node-exporter.b for 7cb51a9a-7413-11ef-bceb-c7b262605968. 2024-09-16T10:36:44.756 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 16 10:36:44 smithi165 systemd[1]: ceph-7cb51a9a-7413-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 4.389s CPU time. 2024-09-16T10:36:45.293 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T10:36:45.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T10:36:45.356 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-16T10:36:45.359 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111/remote/smithi063/crash 2024-09-16T10:36:45.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/crash -- . 2024-09-16T10:36:45.406 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T10:36:45.406 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-09-16T10:36:45.408 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111/remote/smithi165/crash 2024-09-16T10:36:45.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/crash -- . 2024-09-16T10:36:45.465 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T10:36:45.465 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-09-16T10:36:45.467 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-16T10:36:45.467 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-16T10:36:45.562 INFO:tasks.cephadm:Compressing logs... 2024-09-16T10:36:45.562 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T10:36:45.565 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T10:36:45.608 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/tcmu-runner.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:: No such file or directory 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr: 63.5% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.log 2024-09-16T10:36:45.609 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-16T10:36:45.610 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-16T10:36:45.610 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T10:36:45.617 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-09-16T10:36:45.618 INFO:teuthology.orchestra.run.smithi063.stderr: /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-16T10:36:45.618 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-16T10:36:45.618 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T10:36:45.618 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T10:36:45.620 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-16T10:36:45.620 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T10:36:45.620 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-16T10:36:45.620 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-16T10:36:45.622 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.cephadm.log: /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-16T10:36:45.622 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.log 2024-09-16T10:36:45.622 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-16T10:36:45.622 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-16T10:36:45.622 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.log: /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T10:36:45.622 INFO:teuthology.orchestra.run.smithi063.stderr: 58.9% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-16T10:36:45.631 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.9% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T10:36:45.702 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 79.8% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T10:36:45.706 INFO:teuthology.orchestra.run.smithi063.stderr: 95.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T10:36:45.724 INFO:teuthology.orchestra.run.smithi063.stderr: 89.2% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T10:36:45.798 INFO:teuthology.orchestra.run.smithi063.stderr: 91.2% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T10:36:45.808 INFO:teuthology.orchestra.run.smithi165.stderr: 89.9% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T10:36:45.812 INFO:teuthology.orchestra.run.smithi165.stderr: 85.7% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T10:36:45.816 INFO:teuthology.orchestra.run.smithi165.stderr: 91.4% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-16T10:36:45.912 INFO:teuthology.orchestra.run.smithi165.stderr: 91.2% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T10:36:46.058 INFO:teuthology.orchestra.run.smithi063.stderr: 93.9% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T10:36:46.114 INFO:teuthology.orchestra.run.smithi165.stderr: 94.3% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-16T10:36:46.153 INFO:teuthology.orchestra.run.smithi165.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T10:36:46.612 INFO:teuthology.orchestra.run.smithi165.stderr: 90.4% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-16T10:36:47.206 INFO:teuthology.orchestra.run.smithi063.stderr: 89.6% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-16T10:36:52.363 INFO:teuthology.orchestra.run.smithi063.stderr: 90.5% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-16T10:37:05.985 INFO:teuthology.orchestra.run.smithi063.stderr: 90.1% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-16T10:38:26.977 INFO:teuthology.orchestra.run.smithi165.stderr: 93.4% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-16T10:40:08.166 INFO:teuthology.orchestra.run.smithi165.stderr: 94.0% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-16T10:42:02.994 INFO:teuthology.orchestra.run.smithi165.stderr: 94.0% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-16T10:43:14.396 INFO:teuthology.orchestra.run.smithi165.stderr: 92.6% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-16T10:43:14.398 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-16T10:43:14.398 INFO:teuthology.orchestra.run.smithi165.stderr:real 6m28.819s 2024-09-16T10:43:14.398 INFO:teuthology.orchestra.run.smithi165.stderr:user 4m57.658s 2024-09-16T10:43:14.398 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m13.255s 2024-09-16T10:43:45.291 INFO:teuthology.orchestra.run.smithi063.stderr: 93.2% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-16T10:44:28.487 INFO:teuthology.orchestra.run.smithi063.stderr: 93.4% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-16T10:49:46.074 INFO:teuthology.orchestra.run.smithi063.stderr: 93.1% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-16T10:50:48.985 INFO:teuthology.orchestra.run.smithi063.stderr: 92.5% -- replaced with /var/log/ceph/7cb51a9a-7413-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-16T10:50:48.988 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T10:50:48.988 INFO:teuthology.orchestra.run.smithi063.stderr:real 14m3.409s 2024-09-16T10:50:48.988 INFO:teuthology.orchestra.run.smithi063.stderr:user 7m18.112s 2024-09-16T10:50:48.988 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m19.355s 2024-09-16T10:50:48.988 INFO:tasks.cephadm:Archiving logs... 2024-09-16T10:50:48.989 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111/remote/smithi063/log 2024-09-16T10:50:48.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T10:51:51.156 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111/remote/smithi165/log 2024-09-16T10:51:51.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T10:52:36.228 INFO:tasks.cephadm:Removing cluster... 2024-09-16T10:52:36.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 --force 2024-09-16T10:52:36.474 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:52:37.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7cb51a9a-7413-11ef-bceb-c7b262605968 --force 2024-09-16T10:52:37.448 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 7cb51a9a-7413-11ef-bceb-c7b262605968 2024-09-16T10:52:38.098 INFO:tasks.cephadm:Removing cephadm ... 2024-09-16T10:52:38.098 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T10:52:38.118 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T10:52:38.134 INFO:tasks.cephadm:Teardown complete 2024-09-16T10:52:38.134 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T10:52:38.143 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T10:52:38.144 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T10:52:38.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T10:52:38.276 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T10:52:38.285 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T10:52:38.285 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:52:38.288 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:52:38.302 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:52:38.304 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:52:38.330 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:52:38.330 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-09-16T10:52:38.330 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 -382us[ -374us] +/- 37ms 2024-09-16T10:52:38.330 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 +1519us[+1519us] +/- 74ms 2024-09-16T10:52:38.330 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -599us[ -599us] +/- 61ms 2024-09-16T10:52:38.330 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:52:38.344 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:52:38.344 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-09-16T10:52:38.344 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv01.front.sepia.ceph.com 3 6 377 7 -408us[ -403us] +/- 37ms 2024-09-16T10:52:38.345 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 7 +1532us[+1532us] +/- 74ms 2024-09-16T10:52:38.345 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -611us[ -606us] +/- 61ms 2024-09-16T10:52:38.345 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:52:38.346 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T10:52:38.356 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T10:52:38.356 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T10:52:38.387 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T10:52:38.480 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T10:52:38.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:52:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:52:38.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:52:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:52:38.597 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T10:52:38.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T10:52:38.650 INFO:teuthology.task.internal:Duration was 3089.525947 seconds 2024-09-16T10:52:38.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T10:52:38.702 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T10:52:38.703 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T10:52:38.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T10:52:38.758 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:52:38.762 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:52:39.262 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T10:52:39.263 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T10:52:39.263 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T10:52:39.291 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-09-16T10:52:39.292 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T10:52:39.324 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T10:52:39.325 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:52:39.334 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:52:39.391 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T10:52:39.392 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T10:52:39.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T10:52:41.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T10:52:41.739 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T10:52:41.740 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T10:52:41.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T10:52:41.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T10:52:41.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:41.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:41.865 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-09-16T10:52:41.890 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-09-16T10:52:41.926 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:52:41.941 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:52:41.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T10:52:41.992 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T10:52:41.993 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111/remote/smithi063 2024-09-16T10:52:41.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T10:52:42.067 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/avan-2024-09-16_08:23:53-orch:cephadm-wip-athakkar-testing-2024-09-12-1857-distro-default-smithi/7907111/remote/smithi165 2024-09-16T10:52:42.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T10:52:42.110 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T10:52:42.110 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T10:52:42.113 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T10:52:42.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T10:52:42.171 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T10:52:42.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T10:52:42.196 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T10:52:42.196 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T10:52:42.199 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T10:52:42.212 INFO:teuthology.orchestra.run.smithi063.stdout: 266078 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 10:52 /home/ubuntu/cephtest 2024-09-16T10:52:42.216 INFO:teuthology.orchestra.run.smithi165.stdout: 265613 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 10:52 /home/ubuntu/cephtest 2024-09-16T10:52:42.217 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T10:52:42.293 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T10:52:42.400 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3089.5259466171265 flavor: default owner: scheduled_avan@teuthology success: true 2024-09-16T10:52:42.400 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T10:52:42.476 INFO:teuthology.run:pass